Alternative approach that I can think of is using an additional TransactionSynchronisationAdapter (if you're using Spring, that is). After all changes have been rolled back (transaction-rollback) to both activiti and your service, you can use the adapter to force a new transaction (REQUIRES_NEW) and try running the compensation-behaviour manually. Offcourse, this won't be using the "compensation" that is built in into BPMN, as the process has been rolled back and the process-state is not suited for a compensation since it doesn't know anything went wrong (take a look at the org.activiti.engine.impl.jobexecutor.FailedJobListener which we use to decrement job retries after rollback in job-execution).
In the current implement ion, you should either rely on the transaction being rolled back completely (so process-state remains as it was before the API-call was made) or either use compensation for scenario's where the transaction has not been rolled back by the failure…
I understand that this is a valid use-case for compensation working with @Transactional rollback, although really hard to implement as I mentioned before due to the fact that the process-changes are done in the rolled-back transaction. So I don't see such behavior being added any time soon to the engine, I'm afraid.