Extending Activiti with Custom Properties

[rhafner]

We are migrating from a (non-bpmn) proprietary workflow system which support "attributes" on various workflow objects such as UserTask, SequenceFlow, and DataObject. The attributes are classified in to two types: mutable and immutable. Mutable attributes are key/value pairs that can be modified on a workflow object (such as UserTask) of a running workflow process. Immutable attributes are ones that are modeled in the BPMN Template and can not be modified on a running workflow process. In terms of use cases, the immutable attributes are used to support localization of the various workflow object names.

For this support we have implemented the following Activiti extensions:
1) Custom Extension Elements in the BPMN. For example:
<userTask id="usertask1" name="Task A">
        <extensionElements>
            <xyz:attributes>
                <xyz:attribute xyz:id="1" xyz:name="Attr1" xyz:value="1" />
                <xyz:attribute xyz:id="2" xyz:name="Attr2" xyz:value="2" />
            </xyz:attributes>
            <xyz:i18ln>
                <xyz:labeledEntityIdForName xyz:entityID="6c7826fb-effd-45c2-8097-2bb59e7ba8b9" xyz:defaultlocale="en">
                    <xyz:locale xyz:name="en">leifn-1</xyz:locale>
                    <xyz:locale xyz:name="it">itleifn-1</xyz:locale>
                </xyz:labeledEntityIdForName>
                <xyz:labeledEntityIdForDescription xyz:entityID="f8a34a02-5d57-4a7d-90bb-1116f4304b97" xyz:defaultlocale="en">
                    <xyz:locale xyz:name="en">leifd-1</xyz:locale>
                    <xyz:locale xyz:name="it">itleifd-1</xyz:locale>
                </xyz:labeledEntityIdForDescription>
            </xyz:i18ln>
            <xyz:resourceBundleKeyForName>rbkfn-1</xyz:resourceBundleKeyForName>
            <xyz:resourceBundleKeyForDescription>rbkfd-1</xyz:resourceBundleKeyForDescription>
        </extensionElements>
    </userTask>

2) Custom ParseHandlers that parse the attribute extension elements and add them as properties to the respective activiti object in the process definition.

3) Custom database tables to store the attributes. For example:
create table SAS_WFS_EXT_TASK_ATTRIBUTE
(
  ID varchar(64) not null,
  TASK_ID varchar(64) not null,
  ATTRIBUTE_NAME varchar(255) not null,
  ATTRIBUTE_VALUE varchar(255),
  primary key(ID),
  unique(TASK_ID, ATTRIBUTE_NAME)
);

4) Custom Task and Execution Listeners to create and delete the attributes from the extension tables when the appropriate lifecycle event occurs.

All of the above works fine, the remaining question we have is how should the Database CRUD operations for the attributes be implemented? Thus far we have explored the following options:
1) JDBC via Spring's JdbcTemplate
2) Custom MyBatis Mappers as described here: http://www.jorambarrez.be/blog/2014/01/17/execute-custom-sql-in-activiti/
3) Update our Attribute Entity classes to extend org.activiti.engine.impl.db.PersistentObject,  create custom AttributeEntityManager classes that extend org.activiti.engine.impl.persistence.AbstractManager, and create custom commands to set and get the attributes.

Options 1 & 2 prevent us from having referential integrity between the custom attribute table and the associated Activiti table due to behavior differences with org.activiti.engine.impl.db.DbSqlSession. Both JDBC and the Custom MyBatis Mappers flush earlier than the DbSqlSession resulting in a referential integrity violation.

Example:
alter table SAS_WFS_EXT_TASK_ATTRIBUTE add constraint SAS_WFS_EXT_FK_TASK_ID foreign key (TASK_ID) references SAS_WFS_ACT_RU_TASK;

Invoking DbSqlSession.flush() prior to creating the attributes resolves the referential integrity violation but causes behavior differences and other failures in our tests which I have not investigated in detail yet.

Some questions:
1) Can extension code invoke DbSqlSession.flush()? Should it in this case?
2) Is there a way to hook the Custom MyBatis Mapper into flush lifecycle of DbSqlSession?

We've begun exploring Option 3 some more and thus far it has provided the ability to define foreign key constraints and tie in with the DbSqlSession flush lifecycle. But are still running into some problems with referential integrity on some edge cases. For example:

Template:
   Flow: StartEvent -> Service Task -> UserTask -> EndEvent.
   Modeled DataObjects: DataObject1
Use Case:
1) Start Template
2) StartProcessInstanceCmd sets DataObjects and their respective attributes (via a SetDataObjectAttributesComand).
3) Service Task executes and updates the value of DataObject1 (but not its attributes).

The behavior we see in this case is the order of the insertObjects changes from VariableInstanceEntity followed by DataObjectAttributeEntity (our extension) to DataObjectAttributeEntity followed by VariableInstanceEntity after the service task runs resulting in a referential integrity violation.

Additional questions:
1) Are there any concerns with defining foreign key constraints between Activiti tables and custom Extension tables?
2) Is Option 3 the recommended path for implementing extensions that need referential integrity with Activiti tables? Due to the optimizations made in DbSqlSession which eliminates the need to execute statements that insert and delete a PersistentObject with the same id I don’t see another alternative for the reason stated earlier.

[rhafner]

Forgot code tag around bpmn snippet for item 1. Adding again.

<code>
<userTask id="usertask1" name="Task A">
        <extensionElements>
            <xyz:attributes>
                <xyz:attribute xyz:id="1" xyz:name="Attr1" xyz:value="1" />
                <xyz:attribute xyz:id="2" xyz:name="Attr2" xyz:value="2" />
            </xyz:attributes>
            <xyz:i18ln>
                <xyz:labeledEntityIdForName xyz:entityID="6c7826fb-effd-45c2-8097-2bb59e7ba8b9" xyz:defaultlocale="en">
                    <xyz:locale xyz:name="en">leifn-1</xyz:locale>
                    <xyz:locale xyz:name="it">itleifn-1</xyz:locale>
                </xyz:labeledEntityIdForName>
                <xyz:labeledEntityIdForDescription xyz:entityID="f8a34a02-5d57-4a7d-90bb-1116f4304b97" xyz:defaultlocale="en">
                    <xyz:locale xyz:name="en">leifd-1</xyz:locale>
                    <xyz:locale xyz:name="it">itleifd-1</xyz:locale>
                </xyz:labeledEntityIdForDescription>
            </xyz:i18ln>
            <xyz:resourceBundleKeyForName>rbkfn-1</xyz:resourceBundleKeyForName>
            <xyz:resourceBundleKeyForDescription>rbkfd-1</xyz:resourceBundleKeyForDescription>
        </extensionElements>
    </userTask>
</code>

[jbarrez]

Wow - that's a lot of text

Personally, I would go for Spring transaction management and jdbc / hibernate for your custom stuff to let the transactions behave properly.

"1) Can extension code invoke DbSqlSession.flush()? Should it in this case?"

It could. You can get the current DbSqlSession, but I'm not sure if this would have consequences which you cannot see yet now.

"2) Is there a way to hook the Custom MyBatis Mapper into flush lifecycle of DbSqlSession?"

Yes, you can, you can always do Context.getCurrentXXX to get to those things. But im not sure if it's a good idea.

"1) Are there any concerns with defining foreign key constraints between Activiti tables and custom Extension tables?"

It can only go wrong if you don't delete your stuff before activiti stuff is deleted.

"2) Is Option 3 the recommended path for implementing extensions that need referential integrity with Activiti tables? Due to the optimizations made in DbSqlSession which eliminates the need to execute statements that insert and delete a PersistentObject with the same id I don’t see another alternative for the reason stated earlier."

It would for sure fit nicely, but it also does mean you are very dependent on Activiti internal code. Which in theory, will not change that much, but it could happen of course.

Why the need for 'real' foreign keys? Isn't a 'soft' reference enough?

[rhafner]

Thanks for the quick response! I wasn't expecting one so fast since the thread is quite lengthy .

This morning I tracked down the cause of the referential integrity issue I'm seeing with Option 3. For reference here is the behavior:

1) Start Process Instance.
2) Modeled DataObjects and our Extension Attributes are initialized. (put in DbSqlSession.insertedObjects but not flushed).
3) Our Service Task runs and updates the value of one of the Modeled DataObjects but not its attributes. This results in a call to VariableScopeImpl.updateVariables() which performs a VariableInstanceEntity.touch() call on the updated VariableInstanceEntity. The touch call removes the existing VariableInstanceEntity from the insertedObjects list and appends it to the end of the list. This results in the VariableInstanceEntity being after our custom extension (VariableAttributeEntity) entry which results in the constraint violation.

It's unclear to me why the touch call reorders the updated VariableInstanceEntity in the insertedObjects list. Can you explain why the reordering occurs?

For now, I have been able to workaround this issue by calling touch on the attributes after the VariableInstanceEntity has been updated to correct the order.

I don't see any choice but to be very dependent on the Activiti internal code due to the insert/delete optimizations that occur in the DbSqlSession flush method (assuming we choose to use referential integrity).

We are still evaluating whether we want to use referential integrity at the moment. For now we are attempting to use it to ensure our extension attributes are cleaned up when the related activiti object no longer exists since we have run into cases where the attributes were not being cleaned up and our tests were still passing.

[jbarrez]

I think the reorder happens for exactly the problem you state: referential constraints, if a change was made to that object we put it at the end of the list so we are sure any dependent objects are inserted before.