Hello Ben,
we are currently running Activiti successfully under WebSphere Application Server 8.x. You are right, there are some things to take care of, mainly transaction handling and asynchronous support via managed threads.
While the first is rather easy as Activiti supports Spring and Spring has a generalization of a transaction management which includes the use of WebSphere UnitOfWork Transaction management, simply search for an Activiti setup utilizing the WebSphereUOWTransactionManager and the SpringProcessEngineConfiguration. This shall lead you into the right direction. If this does not help, simply reply here and I will try to give further information. If your project does not allow spring, you are screwed here. We don't use (and don't like) Spring, and use it only because of its adaption for transaction management in our application designs. We rather stick with CDI which works great in combination with Activiti. For that to work, you'd have to combine the setup found in the SpringProcessEngineConfiguration with those found in the CdiProcessEngineConfiguration which is quite easy as there is not much added here.
The second part, asynchronous support is actually more difficult, cause Activiti by default does not use managed threads in a JEE environment, thus you loose the JEE context as soon as your application becomes asynchronous. This includes JNDI lookups, EJB (including transaction management) as well as CDI. The worst part here is, that WebSphere does its own thing for handling 'managed' asynchronous threads. Only the Liberty Profile of WAS supports the JEE 7 ManagedExecutionService. For earlier versions of WebSphere we simply applied the following scenario:
We subclassed the JobExecutor and the AquireJobsRunnable and let these use a programmable JEE timer (via TimerService) instead of a simple thread. The programmable timers timeout()-method works exactly as the AquireJobsRunnable run()-method (without the unnecessary endless while-loop). By default the timer goes off every 5 seconds to check whether there is work to do. It then is rescheduled for the next round (making it possible to even fire right away in case multiple jobs need to be handled) or in time frames shorter than the 5 seconds if work is due in less than these 5 seconds.
Our version of the Job Executor (see the DefaultJobExecutor) is again done asynchronously as proposed by Activiti. To make sure, the whole JEE context will be active, we simply lookup a local EJB via JNDI from our Job Executor and call a method annotated with @Asynchronous within it, passing in the ExecuteJobsRunable which is then executed within the asynchronous method (obviously asynchronous and transactional). This works great even with CDI. One could think of using a WorkManager here too (WebSphere supports commonj.WorkManager API) but the easy API of just annotating a method within a Stateless bean with @Asynchronous made our work so much easier.
One thing you must be aware of is in case you are using JPA entities within your process as process variables and become asynchronous, the JPA entities will ALWAYS become detached, as their attachment is based on the EntityManager from the previous thread. So in case you become asynchronous, the first thing to do is to reattach your entities.
This all would have been much easier, if Activiti supported an easier plugin-interface to change the underlying implementation of the task manager. The drawback of our solution is, that we have a different way of 'AquireJobsRunnable' and we must take care, that this implementation is consistent with Activiti (there was actually a slight modification between 5.15 and 5.16-SNAPSHOT).
Just one note to Camunda. They've taken a completely different approach of implementing container managed asynchronous support via JCA and a MessageDriven Bean. This approach is far more complex and does not work with Activiti, as Camunda needed to change the Job Execution Framework quite a lot to fit this in. Additionally, Camunda offers WebSphere support as a enterprise supported platform.
Regards,
Heiko