Extending BPMN Parser

[iravanchi]

Regarding the discussion about extending the BPMN parser (http://forums.activiti.org/en/viewtopic.php?f=4&t=305)

The BPMN XSD, allows extending all BPMN elements (including flows, tasks, gateways, etc.) with any custom property, provided that the additional attributes are not in the BPMN namespace:

   <xsd:element name="baseElement" type="tBaseElement"/>
   <xsd:complexType name="tBaseElement" abstract="true">
      <xsd:sequence>
         <xsd:element ref="documentation" minOccurs="0" maxOccurs="unbounded"/>
         <xsd:element ref="extensionElements" minOccurs="0" maxOccurs="1" />
      </xsd:sequence>
      <xsd:attribute name="id" type="xsd:ID" use="optional"/>
      <xsd:anyAttribute namespace="##other" processContents="lax"/>
   </xsd:complexType>
‍‍‍‍‍‍‍‍‍‍‍

Also, all BPMN elements can contain a <extensionElements> element, which is free to have any XML content:

   <xsd:element name="extensionElements" type="tExtensionElements" /> 
   <xsd:complexType name="tExtensionElements">
      <xsd:sequence>
         <xsd:any namespace="##other" processContents="lax" minOccurs="0" maxOccurs="unbounded" />
      </xsd:sequence>
   </xsd:complexType>
‍‍‍‍‍‍‍‍

The <extension> element is used to declare extensions required to parse the file.
So, I guess we need to look at the <extension> elements in the XML file, and see if there are extension handlers ready to handle such definitions. If there are <extension> declarations that their "mustUnderstand" is true, and there's no one to handle, the parser should raise an error.

Otherwise, I'm thinking that when parsing each XML element, if there are any extensions specified (either attribute, or in <extensionElements>) we should group them in namespaces (the extensions can be from different namespaces) and then hand them to the extension handlers.

The handler should have the ability to influence PVM object model generation in any way.

I think "activiti:" extensions should also be treated the same way Parser treats other extensions.

What's your idea? Does it seem right?

I have two specific questions too, if anyone has any idea about it:

1. What is the formal meaning of the "definition" attribute in <extension> element? It should be an xsd:QName. Is it just an identifier, or does it have any specific meaning (relation to extension XSD?) that is specified in the BPMN 2.0 spec?

2. In the example in BPMN2.0 spec document, there are two violations of this XSD thing. Here's the sample:

<bpmn:task name="Retrieve Customer Record" id="ID_2">
    <bpmn:dataInput name="Order Input" id="ID_3">
        <bpmn:typeDefinition typeRef="bo:Order" id="ID_4"/>
    </bpmn:dataInput>
    <bpmn:dataOutput name="Customer Record Output" id="ID_5">
        <bpmn:typeDefinition typeRef="bo:CustomerRecord" id="ID_6"/>
    </bpmn:dataOutput>
    <bpmn:inputSet name="Inputs" id="ID_7" dataInputRefs="ID_3"/>
    <bpmn:outputSet name="Outputs" id="ID_8" dataOutputRefs="ID_5"/>
</bpmn:task>
‍‍‍‍‍‍‍‍‍‍‍‍

First violation: extension elements are not contained in an <extensionElements> node.
Second violation: extensions are also using the same "bpmn:" namespace, where XSD specifies namespace="##other".
Am I right? Or there's something I'm missing here?

[iravanchi]

To add to my above proposal, I prepared three variations of a process that is supposed to copy a file from one location to the other location, using some extension called "io".


<definitions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL"
             xmlns:activiti="http://activiti.org/bpmn-extensions"
             xmlns:io="http://mynamespaces.com/bpmn-extensions/io"
             xsi:schemaLocation="http://schema.omg.org/spec/BPMN/2.0 BPMN20.xsd"
             typeLanguage="http://www.w3.org/2001/XMLSchema"
             expressionLanguage="http://www.w3.org/1999/XPath"
             targetNamespace="http://www.activiti.org/bpmn2.0">

    <process id="fileCopySample1" name="File Copy Sample One">
        <startEvent id="theStart"/>
        <sequenceFlow id="flow1" sourceRef="theStart" targetRef="readFromSource"/>
        <serviceTask id="readFromSource"
                     operationRef="io:readFile"
                     io:sourceFile="D:\new.txt"
                     io:targetVariable="fileContent" />
        <sequenceFlow id="flow2" sourceRef="readFromSource" targetRef="waitForSignal" />
        <manualTask id="waitForSignal" name="Wait for the operator to signal"/>
        <sequenceFlow id="flow3" sourceRef="waitForSignal" targetRef="writeToTarget" />
        <serviceTask id="writeToTarget"
                     operationRef="io:writeFile"
                     io:targetFile="D:\new-copy.txt"
                     io:sourceVariable="fileContent" />
        <sequenceFlow id="flow4" sourceRef="writeToTarget" targetRef="theEnd" />
        <endEvent id="theEnd"/>
    </process>

    <process id="fileCopySample2" name="File Copy Sample Two">
        <startEvent id="theStart"/>
        <sequenceFlow id="flow1" sourceRef="theStart" targetRef="readFromSource"/>
        <serviceTask id="readFromSource"
                     io:action="readFile"
                     io:sourceFile="D:\new.txt"
                     io:targetVariable="fileContent" />
        <sequenceFlow id="flow2" sourceRef="readFromSource" targetRef="waitForSignal" />
        <manualTask id="waitForSignal" name="Wait for the operator to signal"/>
        <sequenceFlow id="flow3" sourceRef="waitForSignal" targetRef="writeToTarget" />
        <serviceTask id="writeToTarget"
                     io:action="writeFile"
                     io:targetFile="D:\new-copy.txt"
                     io:sourceVariable="fileContent" />
        <sequenceFlow id="flow4" sourceRef="writeToTarget" targetRef="theEnd" />
        <endEvent id="theEnd"/>
    </process>
   
    <process id="fileCopySample3" name="File Copy Sample Three">
        <startEvent id="theStart"/>
        <sequenceFlow id="flow1" sourceRef="theStart" targetRef="readFromSource"/>
        <serviceTask id="readFromSource">
            <extensionElements>
                <io:action>readFile</io:action>
                <io:sourceFile>D:\new.txt</sourceFile>
                <io:targetVariable>fileContent</io:targetVariable>
            </extensionElements>
        </serviceTask>
        <sequenceFlow id="flow2" sourceRef="readFromSource" targetRef="waitForSignal" />
        <manualTask id="waitForSignal" name="Wait for the operator to signal"/>
        <sequenceFlow id="flow3" sourceRef="waitForSignal" targetRef="writeToTarget" />
        <serviceTask id="writeToTarget">
            <extensionElements>
                <io:action>writeFile</io:action>
                <io:targetFile>D:\new-copy.txt</targetFile>
                <io:sourceVariable>fileContent</io:sourceVariable>
            </extensionElements>
        </serviceTask>
        <sequenceFlow id="flow4" sourceRef="writeToTarget" targetRef="theEnd" />
        <endEvent id="theEnd"/>
    </process>
   
</definitions>

I'm not sure if there are other semantic limitations on BPMN 2.0 that may result in some / all of the above being invalid BPMN. But considering the XSD, all three variations are valid BPMN 2.0. (a <extension> element should be added under <definitions>).

In my opinion, the BPMN parser should be designed in a way that it should be possible for an extension developer to create an extension that supports any of the above cases. (deciding which one is supported by the extension is the extension developer's choice, but the parser should be extensible to handle all the cases).

In the first process, operationRef of the <serviceTask> is used to specify the type of the operation. Note the "io:" prefix. Since operationRef is a QName, "io:" maps to the namespace URI of the "xmlns:io" declaration at the top.

In the second process, all of the values needed for the extension activity is specified using extension attributes, where in the third sample they are all specified in the <extensionElements>.

This is a super-set of what Tom said he has in mind, and I tried to cover everything that BPMN 2.0 formally allows, to the extent I comprehended from the specification.

If you give me your ideas about this, I can refine my design and post back.

-Hamed

[farrukh_najmi]

Hi Hamed,

I tried to grok your example but was having trouble groking the main points. Perhaps you can simply your example by getting rid of extraineous stuff and add some comments to the rest to make it clear what are the different flavors of extensibility you are trying to illustrate by example. Sorry if I am being slow or lazy.

[iravanchi]

I tried to highlight my points in the XML file when I was posting, but in the code block, the color tags don't work.
Anyway,…

The XML file above contains three <process> elements.
You just need to pay attention to the <serviceTask> elements inside them.

First variation:

<serviceTask id="readFromSource"
    operationRef="io:readFile"
    io:sourceFile="D:\new.txt"
    io:targetVariable="fileContent" />

Second variation:

<serviceTask id="readFromSource"
    io:action="readFile"
    io:sourceFile="D:\new.txt"
    io:targetVariable="fileContent" />

Third variation:

<serviceTask id="readFromSource">
            <extensionElements>
                <io:action>readFile</io:action>
                <io:sourceFile>D:\new.txt</sourceFile>
                <io:targetVariable>fileContent</io:targetVariable>
            </extensionElements>
</serviceTask>

I think that all of the above variations should be possible to Parse (using an extension to the parser) for ALL different elements in the process definition, as they are allowed in all elements (XSD considers them valid for the tBaseElement type)

Note that "io" ns prefix is defined in the root (definitions) element:

<definitions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL"
             xmlns:activiti="http://activiti.org/bpmn-extensions"
             xmlns:io="http://mynamespaces.com/bpmn-extensions/io"
             xsi:schemaLocation="http://schema.omg.org/spec/BPMN/2.0 BPMN20.xsd"
             typeLanguage="http://www.w3.org/2001/XMLSchema"
             expressionLanguage="http://www.w3.org/1999/XPath"
             targetNamespace="http://www.activiti.org/bpmn2.0">

And there needs to be an <extension> added under definitions, so that the parser can be told which extensions to allow in the current BPMN file.

-Hamed

[farrukh_najmi]

Hi Hamed,

Thank for helping me understand.

The key points I get are:

• First and second variation: Activiti should allow customized processing of any extension attribute


• Third variation: Activiti should allow customized processing of any extension element

I agree with both requirements above.

The points I dont get are:

• The use of operationRef in first variation. It seems specific to serviceTask which leads to my next point….

I have one concern. Both in your examples and the examples shared by Tom in another email (see below) seem to focus on serviceTask:

We have an idea in mind that looks like this

<serviceTask activiti:type="someTypeOrAClassName">
  <extensionElements>
    <field name="cfgParam1">stringValue</field>
    <field name="cfgParam2"><integer value="293" /></field>

I would like to generalize to the following requirements that allow customized processing of any task using external mapping from task to the handler as follows:

• Activiti should allow customized processing of any type of task and not just serviceTask


• Specialized processing should allow handling extension attributes and elements in the task as you suggested


• There should be some way to define a mapping from a task type (e.g. receiveTask) to a task handler that is defined externally to the BPMN2 file (e.g. in spring configuration)


• There should be some way to define a mapping from a task instance (by its id) to a task handler that is defined externally to the BPMN2 file (e.g. in spring
configuration)

The reason why I am suggesting that the mapping between a task and its handler be external to BPMN is because I am working on an OASIS standard (ebXML RegRep 4) that plans to use BPMN2 and specify how it should be used within a registry / repository to manage the lifecycle of its content. Such a spec would like to define certain task patters and how they should be handled. The spec needs to avoid specifying how the mapping of handlers to task patterns is done and leave it up to implementations. The spec also plans to specify certain canonical BPMN2 processes that should be supported by any compliant implementation. Thus the requirement I see for defining the task to handler mapping external to the BPMN.

What do folks think of these requirements?

[iravanchi]

Thanks for replying. Here's the points in my mind, in reply:

1. I too believe that the customizations should be applicable to ALL elements, as BPMN XSD allows it. (Read my line after the "third variation" above) serviceTask is just an easy way to illustrate it.

2. The operationRef thing: operationRef should have a QName value, which means, a value qualified with a namespace. I think one extension option is to use such QName attributes, and define custom handlers (from outside Activiti core) that can comprehend them, based on the namespace. In my example, the parser can look at the namespace qualifier of the operationRef, and delegate mapping it to a concrete activity to the extension registered with the "io" namespace.
In other words, this is a specific extension point for serviceTask which is not applicable to all other elements. I meant that such extension points should also be considered.

3. I am a believer in Spring based development (generally, IoC/DI and compositional architecture). So, allowing all such extensions and integrations to be easy to do with Spring beans is what I have in mind too. But I couldn't quite understand your idea about your third and forth requirement.

4. To summarize what we have till now, here's what we have as a list of constraints and requirements:

• All customization / extensions should adhere to BPMN 2.0 specification.


• Any extension used in each file should be declared using an <extension> element.


• BPMN Parser should allow any custom attributes on all elements.


• BPMN Parser should allow any custom elements in the <extensionElements> element on all elements.


• BPMN attributes with abstract specification, and QName type, may be extended and handled by an extension.


• Extensions should be able to affect generation of PVM objects based on extention attributes / elements in any way.


• Spring-based composition for the extensibility features should be easily possible.

5. I have a customizability requirement in mind too:

• Extensions / customizations should be able to alter / replace the default PVM objects generation algorithm for BPMN elements.

What I mean, is that just including an extension in the file (or by other means) one should be able to make BPMN elements (without any extension attributes / elements) parsed in a different way. Since this can allow changing the BPMN semantics, I don't believe supporting this as a first-class requirement is a good idea. But I think this should not be impossible. (We can deliberately make it hard to do).

An example use case would be, if I want to have my own language which is very similar to BPMN. It's nice if I can re-use the code from BPMN Parser to write my custom Deployer.

Another example would be if I want to write a simulation program, say for education purposes, or if I want to support test-running of the processes where some tasks and activities are not actually run, but just produce a log so that the designer can review it and debug the process definition. I might want to replace, for example, serviceTask behavior with something that shows an animation or produces a log. Again, re-using the code here is a nice thing if it is possible.

Just want to see what you think about this, as if we are thinking about BPMN Parser, we can consider other requirements (in addition to extensibility) too.

Edit: I just came along this post by chance, which is an example of the "simulation" case I was talking about:
http://forums.activiti.org/en/viewtopic.php?f=4&t=122

[farrukh_najmi]

3. I am a believer in Spring based development (generally, IoC/DI and compositional architecture). So, allowing all such extensions and integrations to be easy to do with Spring beans is what I have in mind too. But I couldn't quite understand your idea about your third and forth requirement.

What I mean is that I prefer not to define and extension element or attribute that instructs what Java handler class to use as in "activiti:type" Tom's example below:

We have an idea in mind that looks like this

<serviceTask activiti:type="someTypeOrAClassName">
  <extensionElements>
    <field name="cfgParam1">stringValue</field>
    <field name="cfgParam2"><integer value="293" /></field>

Instead I am suggesting that there be an external configuration that defines the handler mapping. Here is an example of what this could look like in spring:


    <bean id="BPMNParse"
        class="org.activiti.engine.impl.bpmn.parser.BpmnParse">
        <property name="taskTypeToHandlerMap">
            <map>
                <entry key="serviceTask" value-ref="DefaultServiceTaskHandler"/>
                <entry key="sendTask" value-ref="DefaultSeendTaskHandler"/>
                …
            </map>
        </property>
        <property name="taskInstanceToHandlerMap">
            <map>
                <entry key="serviceTask01" value-ref="CustomServiceTaskHandler1"/>
                <entry key="serviceTask02" value-ref="CustomServiceTaskHandler2"/>
                …
            </map>
        </property>
    </bean>


Above would allow for an external database or registry to also keep the mapping tables if desired and the API would allow changing them dynamically.
But teh default of spring based config is really quite trivial.

4. To summarize what we have till now, here's what we have as a list of constraints and requirements:

• All customization / extensions should adhere to BPMN 2.0 specification.


• Any extension used in each file should be declared using an <extension> element.


• BPMN Parser should allow any custom attributes on all elements.


• BPMN Parser should allow any custom elements in the <extensionElements> element on all elements.


• BPMN attributes with abstract specification, and QName type, may be extended and handled by an extension.


• Extensions should be able to affect generation of PVM objects based on extention attributes / elements in any way.


• Spring-based composition for the extensibility features should be easily possible.

+1. The only thing is that your second bullet should include custom attributes.

5. I have a customizability requirement in mind too:

• Extensions / customizations should be able to alter / replace the default PVM objects generation algorithm for BPMN elements.

What I mean, is that just including an extension in the file (or by other means) one should be able to make BPMN elements (without any extension attributes / elements) parsed in a different way. Since this can allow changing the BPMN semantics, I don't believe supporting this as a first-class requirement is a good idea. But I think this should not be impossible. (We can deliberately make it hard to do).

An example use case would be, if I want to have my own language which is very similar to BPMN. It's nice if I can re-use the code from BPMN Parser to write my custom Deployer.

Another example would be if I want to write a simulation program, say for education purposes, or if I want to support test-running of the processes where some tasks and activities are not actually run, but just produce a log so that the designer can review it and debug the process definition. I might want to replace, for example, serviceTask behavior with something that shows an animation or produces a log. Again, re-using the code here is a nice thing if it is possible.

Just want to see what you think about this, as if we are thinking about BPMN Parser, we can consider other requirements (in addition to extensibility) too.

Edit: I just came along this post by chance, which is an example of the "simulation" case I was talking about:
http://forums.activiti.org/en/viewtopic.php?f=4&t=122

I propose we keep things as minimal and simple as possible initially to and hope to get it right. Then I propose looking at above ideas in a followup effort incrementally at a later point.


It would be good if Tom, Joram and other dev team memebrs could participate in this discussion. I would hate to spend time on this and learn that dev team does not like the proposed direction.

[tombaeyens]

Check out class ParseListener.  You can configure a list of those that in the BpmnDeployer in the ProcessEngineConfiguration.
You can add an implementation that reads your extension elements and custom attributes and store the information in the generic properties map in the PvmActivity.  Then it is available during runtime.

Is that the kind of pluggability your looking for?

[iravanchi]

@tom

That can do it for some of the cases (at least the cases that I have) but not all.

In addition to that:

1. It's not just a matter of storing additional properties for use during execution. Sometimes the parser behavior is expected to change based on the extensions, provided that PVM is very flexible and extensible.

2. What is your idea about <extension> element in BPMN? I mean, what is your plan about what should Parser do with it?

3. I think a better way would be for the Parser to "invoke" extensions when it sees the extension elements / attributes, rather that the extension looking for the extended attributes using ParserListener.

4. If all extensibility we're talking about is to store extension elements and custom attributes in the PvmActivity, then the best would be to change the current parser to store all such elements and attributes with a specific key in the PvmActivity. This means support for reading BPMN extensions, but it doesn't mean an extensible parser.

-Hamed

[mcella]

Hi all,

I'am interested to know if there are dev team news about the BPMN Parser extensibility and if there has been some step forward about extensions development in BPMN Parser.

About custom BPMN elements: does the BPMN 2.0 specification explicitly prohibits use of custom process-child elements ? Taking as reference the example provided by @iravanchi :


<definitions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL"
    xmlns:activiti="http://activiti.org/bpmn-extensions"
    xmlns:io="http://my/custom/namespace"
             xsi:schemaLocation="http://schema.omg.org/spec/BPMN/2.0 BPMN20.xsd"
             typeLanguage="http://www.w3.org/2001/XMLSchema"
             expressionLanguage="http://www.w3.org/1999/XPath"
             targetNamespace="http://www.activiti.org/bpmn2.0">

<process id="fileCopyTest" name="File Copy Test">

  <startEvent id="theStart"/>

  <sequenceFlow id="flow1" sourceRef="theStart" targetRef="readFromSource"/>

  <io:readFile id="readFromSource" sourceFile="D:\new.txt" targetVariable="fileContent" />
 
  <sequenceFlow id="flow2" sourceRef="readFromSource" targetRef="writeToTarget" />
 
  <io:writeFile id="writeToTarget" sourceVariable="fileContent" targetFile="D:\new-copied.txt" />

  <sequenceFlow id="flow3" sourceRef="writeToTarget" targetRef="theEnd" />

  <endEvent id="theEnd"/>

</process>
</definitions>