Custom serialization of complex process variable

[ajjain]

I wished to extend Activiti's default object serialization scheme to json based serialization. This was intended to handle changes in process variable. When any change is made in any process variable structure, I had to always clean up the Activiti's database other wise it gives me deserialization issues. I wish to handle this by serializing all changes as JSON and an offline tool to adapt any changes made to the proc var class.

package test.custom.type;import org.activiti.engine.ActivitiException;import org.activiti.engine.impl.persistence.entity.VariableInstanceEntity;import org.activiti.engine.impl.variable.ByteArrayType;import org.activiti.engine.impl.variable.ValueFields;import org.activiti.engine.impl.context.Context;import org.codehaus.jackson.map.ObjectMapper;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import test.custom.CustomProcVar;/** * @author ajjain *  * Represents Custom variable type in Activiti. This supports serializing process variables as JSON strings * as byte arrays.  */public class CustomVariableType extends ByteArrayType {   /** The type name. */   protected String typeName;   /** The class. */   protected Class<? extends CustomProcVar> theClass;   /**    * represents the JSON mapper object    */   private static final ObjectMapper jsonMapper = new ObjectMapper();   /**    * Instantiates a new view object variable type.    *    * @param typeName the type name    * @param theClass the the class    */   public CustomVariableType(String typeName, Class<? extends CustomProcVar> theClass) {      logger.trace("constructing view object variable type name {}, class {}", typeName, theClass);      this.typeName = typeName;      this.theClass = theClass;   }   /**    * @return the typeName    */   @Override   public String getTypeName() {      return typeName;   }   @Override   public Object getValue(ValueFields valueFields) {      logger.trace("CustomVariableType.getValue valueFields {}", valueFields);      Object cachedObject = valueFields.getCachedValue();      if (cachedObject != null) {         return cachedObject;      }      byte[] bytes = (byte[]) super.getValue(valueFields);      Object deserializedObject;      try {         deserializedObject = jsonMapper.readValue(bytes, this.getTheClass());         logger.trace("getValue returning object {}", deserializedObject);         valueFields.setCachedValue(deserializedObject);         if (valueFields instanceof VariableInstanceEntity) {            Context.getCommandContext()            .getDbSqlSession()            .addDeserializedObject(bytes, bytes, (VariableInstanceEntity) valueFields);         }         return deserializedObject;      }      catch (Throwable e) {         logger.warn("error in deserializing the output", e);         throw new ActivitiException("Couldn't deserialize object in variable '"+valueFields.getName()+"'", e);      }   }   @Override   public boolean isCachable() {      return true;   }   @Override   public boolean isAbleToStore(Object value) {      logger.trace("isAbleToStore value {}", value, value instanceof CustomProcVar);      return (value instanceof CustomProcVar) && (theClass.isAssignableFrom(value.getClass()));   }   @Override   public void setValue(Object value, ValueFields valueFields) {      byte[] byteArray = serialize(value, valueFields);      valueFields.setCachedValue(value);      if (valueFields.getBytes() == null) {         if (valueFields instanceof VariableInstanceEntity) {            Context.getCommandContext()            .getDbSqlSession()            .addDeserializedObject(byteArray, byteArray, (VariableInstanceEntity)valueFields);         }      }      super.setValue(byteArray, valueFields);      }   /**    * Serialize.    *    * @param value the value    * @param valueFields the value fields    * @return the byte[]    */   public static byte[] serialize(Object value, ValueFields valueFields) {      if (value == null) {         return null;      }      try {         byte[] valueBytes = jsonMapper.writeValueAsBytes(value);         return valueBytes;      } catch (Exception e) {         throw new ActivitiException("Couldn't serialize value '"+value+"' in variable '"+valueFields.getName()+"'", e);      }    }   /**    * @return the theClass    */   public Class<?> getTheClass() {      return theClass;   }   /**    * @param theClass the theClass to set    */   public void setTheClass(Class<? extends CustomProcVar> theClass) {      this.theClass = theClass;   }   /**    * @param typeName the typeName to set    */   public void setTypeName(String typeName) {      this.typeName = typeName;   }   /** The logger. */   private Logger logger = LoggerFactory.getLogger(CustomVariableType.class);   /**    * @return the json mapper    */   public static ObjectMapper getJsonmapper() {      return jsonMapper;   }}‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

After integrating the above code with the application, I found issues with getValue method. On further analysis of issue, I realized it is DeserializedObject class which defaults to serializing objects using default mechanism. Also, this doesn't open any extension points so that the same can be used for any custom serialization mechanism.
I wanted to bring this issue on the community so that this feature can be contributed. I am finding this to be very useful feature which every Activiti user will wish for applications deployed in production.

[frederikherema1]

The DeserializedObject is intended to make sure that Serializable variables are updated, when they are updated (eg. a member is set) without an explicit call to a setVariable(). So if you use the variable value in an execution-listener and call a setter on it, it will be flushed with the new values.

However, if you're using your own mechanism for serializing, you shouldn't use the DeserializedObject. Rather, do a setVariable() call with the updated object when needed instead. Another solution may be to extend the DeserializedObject class, and override the flush() method to use YOUR logic to check if a byte-array has been changed… Pass that custom DeserializedObject to the Context.getCommandContext().getDbSqlSession().addDeserializedObject(…) instead.

[ajjain]

Thanks Frederik for comment. I will give it a try.
Do you see this new VariableType useful? I will be pleased to share this back to the community. With updating requirements in agile world, there are chances that proc variable definition may change. And this feature shall provide developer the next level of control in managing those changes.

[ajjain]

I have given a thought on your opinions Frederik, below are my thoughts:
Option 1. Default serialization, this will n't work for this case, since I want to get rid of default serialization mechanism.
Option 2. Explicitly calling setVariable, this sounds out of track to me. If I understood it right you mean in event listener I have to explicitly call setVariable() right. If I do so, end user has to be careful of my new VariableType and do a special handling inside the BPMN flow. I wish to get this implemented exhactly as how default scheme is implemented.
Option 3. Extending DeserializedObject, this is exactly what I have thought of, but after analyzing code my observation was lacks extension points in DbSqlSession class. DbSqlSession.addDeserializedObject() creates an object of its own and doesn't provide user a mean to add object from outside. Also, deserializedObjects type declared marked protected and with no getters and setters, thereby another restriction. That's why I initially commented that this is not open for extension. I would suggest if we can have another overloaded method like:
<java>
public void addDeserializedObject(DeserializedObject deserializedObject) {
    deserializedObjects.add(deserializedObject);
}
</java>
This will help us in solving this issue and a very useful feature can be contributed to activiti community.

[frederikherema1]

Okay, I think I was too quick when proposing the solution about overriding the DeserializedObject. It makes sense indeed to open up the method you mentioned, to add custom deserialization-difference handling. I'll add this to the current Master, so it's included in 5.15.

[frederikherema1]

See https://github.com/Activiti/Activiti/commit/e127424d26307881b09fa63f0d2d7bafe0629dec

[ajjain]

Frederik,
How about getting JSON based serialization feature made available?
I have the code ready, I am keen to share it to the community. Every one wants to be a contributor, so is me. When the project is like Activiti, who'll want to miss a chance.
Thanks,
Abhishek

[frederikherema1]

If you're implementation is based on a "widely adopted standard" to create JSON, we can consider it. However, we want to keep the engine as free from external dependencies as we want, so if we could make it like an additional "plugin" or something, that whould be great.

Without saying yes or no, can you perhaps share the code you have right now?

[ajjain]

For serialization to and from JSON, I have used Jackson APIs. I fee Activiti uses the same in other modules, so including this feature does not need any other third party integration.

I have uploaded the implemented code in my GISTs:
JsonVariableType.java : https://gist.github.com/ajjain/7803027
JsonizedObject.java : https://gist.github.com/ajjain/7802924
JsonType.java : https://gist.github.com/ajjain/7803074

[frederikherema1]

Well, it actually does create additional depencencies, as the engine itself does not depend on Jackson, other modules do… I'll take a look at this asap.