I've implemented an extensible BPMN parser for Activiti

[iravanchi]

As a part of my project, I needed to extend BPMN parser to support my own activities in the process definition.
I imported a few of the source codes of Activiti into my project and implemented the following solution:

BpmnParser supports multiple xmlns definitions, and each element / attribute is delegated to a handler class depending on its xmlns. Handlers can be added in the BpmnParseListener implementations. So, by implementing custom deployers, a developer can easily extend the activities and BPMN notation.

I refactored the current parser to my new design, and it passes the initial tests. I think it's pretty good support for extensibility. It passed the few tests that I did. For example, I could successfully run the following process:

<definitions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL"
          xmlns:activiti="http://activiti.org/bpmn-extensions"
          xmlns:io="http://my/custom/namespace"
             xsi:schemaLocation="http://schema.omg.org/spec/BPMN/2.0 BPMN20.xsd"
             typeLanguage="http://www.w3.org/2001/XMLSchema"
             expressionLanguage="http://www.w3.org/1999/XPath"
             targetNamespace="http://www.activiti.org/bpmn2.0">

   <process id="fileCopyTest" name="File Copy Test">

      <startEvent id="theStart"/>

      <sequenceFlow id="flow1" sourceRef="theStart" targetRef="readFromSource"/>

      <io:readFile id="readFromSource" sourceFile="D:\new.txt" targetVariable="fileContent" />
      
      <sequenceFlow id="flow2" sourceRef="readFromSource" targetRef="writeToTarget" />
      
      <io:writeFile id="writeToTarget" sourceVariable="fileContent" targetFile="D:\new-copied.txt" />

      <sequenceFlow id="flow3" sourceRef="writeToTarget" targetRef="theEnd" />

      <endEvent id="theEnd"/>

   </process>
</definitions>

Now I want to contribute the code to the original project, so that I can give you a hand, and plus if the custom code is included in the JAR files, the maintenance would be easier for me too. I have already created the accounts, and will send the form to you too, but here are a few questions:
1. What's your idea about the whole thing?
2. Do you think that it can be included in the 5.0.rc? or GA? (I think a few other people are waiting for this feature too)
3. I have different coding style setup, and my code will look a bit different. Should I change it, or can you reformat the code to match yours when merging with Trunk?

-Hamed

[farrukh_najmi]

from a first incomplete glance, i would be concerned with 1) i don't think your proposal is valid bpmn.  it's not allowed to define custom activities like that.

+1

Looking at Hamed's BPMN example more carefully I now see that it is based upon invalid BPMN and does not extend BPMN using extensionElement as pointed by another post on this thread.

I think Hamed has the right idea (though details are off) that there is a need to provide a framework in the impl for adding handlers for custom extension elements. The same extension framework should be used internally by actviti dev team for actviti-defined extensions and exposed to activiti developers in user community to implement support for their own custom extensions.

The details of Hamed's proposed solution is not correct since it is not valid BPMN.

Tom, does the dev team already have a framework for handling custom extensions in mind? If not, is there support and encouragement from dev team
for Hamed and I to work out an extensibility proposal that is based on valid BPMN?

On a related note what are the plans for using JAXB or BPMN2 EMF project within the parser?

[tombaeyens]

We're don't think its a good idea to switch our BpmnParser to JAXB.
We can consider improvements to the current parser, but then we need to be sure that the solution covers also the tooling. 

We have an idea in mind that looks like this

<serviceTask activiti:type="someTypeOrAClassName">
  <extensionElements>
    <field name="cfgParam1">stringValue</field>
    <field name="cfgParam2"><integer value="293" /></field>  
In case short type names are used, then the BpmnParser (as part of the ProcessEngineConfiguration) should have a list of full class names that implement ActivityBehavior.  Those classes should indicate their short type name with an annotation.

This way, the modeler and designer tools can dynamically add support for ActivityBehavior's that are plugged in by doing introspection on the ActivityBehavior class.

thats our current line of thinking.

[iravanchi]

Hi,

I agree with not using JAXB. The current parser, in my opinion, is both more flexible and more efficient, than if you'd used JAXB. (except the long list of if .. else if … else if … comparisons in the parser, which can be made more efficient using a lookup map)

I have a few notes, that won't clash with your current idea. I want to know what you think about it:

1. I think if <serviceTask> can have custom attributes, such as activiti:type, it should be able to have other custom attributes too, like "myOwnNs:type", through some extensibility.

2. If you agree with 1, then I think the best design would be the one that handles "activiti:" extensions the same way that it handles other extensions. You can even distinguish between pure-BPMN parsing, and BPMN with activiti extensions parsing.

3. If you provide some features in the engine, it doesn't necessarily mean that you have to support it in the designer. For example, you allow a list of "deployers" in the engine, but the designer only creates BPMN files. This way, a user can extend the engine and use it in his own way, in addition to what Activiti provides already.

4. Being able to re-use any part of the class library is a good feature to have. For example, I might want to use the BpmnDeployer (or Parser) to validate an XML that I get from my user, and to see if the engine supports it already, and gather error messages independently, without deploying the XML into the Activiti tables.

I'd be happy to hear what you think about the above.
Meanwhile, I will take a look at the BPMN specs one more time, to see exactly what it allows for extensions. I'll update my design accordingly, and post a couple of class / sequence diagrams for you to see what I have.

-Hamed

[falko_menge]

As BPMN's XML schema doesn't block Element Substitution, I was just about to argue that the custom Flow Nodes suggested by iravanchi could be made valid with the following XML schema:

<?xml version="1.0" encoding="UTF-8"?>
<schema
    xmlns="http://www.w3.org/2001/XMLSchema"
    targetNamespace="http://my/custom/namespace"
    xmlns:tns="http://my/custom/namespace"
    xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
    elementFormDefault="qualified">

    <!–
        This is a schema-valid BPMN extension, which is invalid with respect
        to the meta model as described in section "8.2.3 Extensibility" of the
        BPMN specification.
    –>

    <import namespace="http://www.omg.org/spec/BPMN/20100524/MODEL" />

    <!– <io:readFile id="readFromSource" sourceFile="D:\new.txt" targetVariable="fileContent" /> –>

    <element name="readFile" type="tns:tReadFile" substitutionGroup="bpmn:flowElement" />

    <complexType name="tReadFile">
        <complexContent>
            <extension base="bpmn:tTask">
                <attribute name="sourceFile" type="string" />
                <attribute name="targetVariable" type="string" />
            </extension>
        </complexContent>
    </complexType>

</schema>
However, after reading section "8.2.3 Extensibility" of the BPMN specification again, I learned that the BPMN Meta Model only allows custom XML elements that are contained in extensionElements elements. This means that the direct children of the process element can only be elements from the official BPMN XML schema.

This still leaves the question of how to deal with the quite frequent request for custom elements open.
In general, I believe that it is good to have a compact set of task categories that gives users hint about what they have to do and what the machines do for them. I think that the seven Task Types of BPMN 2.0 are broad enough to put any custom task into one of those seven categories. So why don't we restrict us to, e.g., Service Tasks with Extension Elements and be sure that our processes stay compatible with the majority of BPMN tools? We could still extend our particular tools to display our extended Service Task, e.g., with another icon or even completely different.

[tombaeyens]

@iravanchi  did you see the ParseListeners already in the code?  with that (undocumented for now) feature you can parse custom attributes

[iravanchi]

@falko
I've planned to read BPMN extensibility again tomorrow, but as you already did and you're fresh, does BPMN allow such <extensionElements> in all of the process children? For example, can I have extensions on flows and user tasks too?
During the previous version of our project (which used jBPM 3) the requirement to add extra custom information to such elements in the XML raised a few times, and I think I'll probably need the same stuff in BPMN too.
For example, I'm sure that I'd be adding additional event attributes for allowing processes to be triggered by my application's custom events.

@tom
Yes, I saw the ParseListeners, and I've actually extended it to have two methods (before and after). I don't remember why I added the "after" method, I'd check my code tomorrow at work and write back.
What I did (in brief) was to create some kind of ElementParseHandler, each of which can declare a set of element names they can handle (plus xmlns).
The parser then creates a Map of the handlers, and delegates parsing of each element to its handler.

-Hamed

[farrukh_najmi]

We're don't think its a good idea to switch our BpmnParser to JAXB.

I agree with not using JAXB. The current parser, in my opinion, is both more flexible and more efficient, than if you'd used JAXB. (except the long list of if .. else if … else if … comparisons in the parser, which can be made more efficient using a lookup map)

I do not wish to beat a dead horse but only interested in learning the rationale behind not wanting to use jAXB.

I realize JAXB is not as efficient as raw XML processing. What other considerations make you both feel that its not a good fit for parsing BPMN2?
Again I only wish to understand your rationale and am not trying to make a case for JAXB. Thanks.

[iravanchi]

What other considerations make you both feel that its not a good fit for parsing BPMN2?

Maybe if I wanted to write the BPMN Parser from scratch, I might have used JAXB. Specially provided that we have an XSD ready for generating JAXB classes. So, I'm not saying that JAXB is not a good fit. Anyway, here's why I felt I'm not comfortable with JAXB in here:
(Note that none of these are concrete reasons, just generally the way I feel about the issue)

1. BpmnParser does two things. One is parsing the XML, and one is generating PVM objects based on what is read from XML. So if we use JAXB, most of the current code will remain, not replaced.

2. Using JAXB means strict conformance of the XML to the object model. In the current implementation, it's rather easier to handle errors in custom ways.

3. It's easier to change / extend the current parser, than if the parser have used JAXB. For example, in JAXB code you need to specify possible sub-classes of a super class user @XmlSeeAlso. So if you want the parser to handle a new element, you'll either have to change the library code, or re-implement the whole JAXB class hierarchy that is bound to BPMN. In the current parser, it's a matter of comparing strings.

All of the above items can be considered a positive point for JAXB too, depending on your point of view.
Even for the performance, since there is many string comparisons and string lookups in the current code, we can't be sure that using JAXB means lower performance.

I think the concrete reason for not changing it would be: there's no (special) gain for the user of Activiti to do so.
From your point of view, as a user of Activiti, what can be better if Activiti uses JAXB or not internally?

[farrukh_najmi]

What other considerations make you both feel that its not a good fit for parsing BPMN2?

Maybe if I wanted to write the BPMN Parser from scratch, I might have used JAXB. Specially provided that we have an XSD ready for generating JAXB classes. So, I'm not saying that JAXB is not a good fit. Anyway, here's why I felt I'm not comfortable with JAXB in here:
(Note that none of these are concrete reasons, just generally the way I feel about the issue)

1. BpmnParser does two things. One is parsing the XML, and one is generating PVM objects based on what is read from XML. So if we use JAXB, most of the current code will remain, not replaced.

2. Using JAXB means strict conformance of the XML to the object model. In the current implementation, it's rather easier to handle errors in custom ways.

3. It's easier to change / extend the current parser, than if the parser have used JAXB. For example, in JAXB code you need to specify possible sub-classes of a super class user @XmlSeeAlso. So if you want the parser to handle a new element, you'll either have to change the library code, or re-implement the whole JAXB class hierarchy that is bound to BPMN. In the current parser, it's a matter of comparing strings.

All of the above items can be considered a positive point for JAXB too, depending on your point of view.
Even for the performance, since there is many string comparisons and string lookups in the current code, we can't be sure that using JAXB means lower performance.

I think the concrete reason for not changing it would be: there's no (special) gain for the user of Activiti to do so.
From your point of view, as a user of Activiti, what can be better if Activiti uses JAXB or not internally?

Thanks for sharing your thoughts Ahmed.

The main benefit of using JAXB that I see is that it will likely encourage a tighter support for BPMN2 syntax in the parser and will be much more robust and easier to maintain in the long term. Perhaps that is not compelling enough benefit in the short term.

BTW, handling of extension elements can be done in JAXB RI via separate compilation of the extension schema and using the generated bindings of extension schema with the core schema:

http://weblogs.java.net/blog/2006/09/05/separate-compilation-jaxb-ri-21

[iravanchi]

Yes, I agree with you about the long-term maintainability, and cleaner code.
Maybe Tom has some more concrete reasons for why he didn't think JAXB is a good choice.

And, BTW, I'm NOT "Ahmed". I'm "Hamed"

-Hamed