Cycle (and Engine?) Persistence need refactoring

[bernd_ruecker]

Hey guys.

We checked in the first DB persistence for Cycle in the meantime. Currently it just saved the Cycle Configuration to the database as a first step. We wanted to leverage which was already there in the DbSqlSessionFactory (e.g. create-drop mechanism). But one goal was, that cycle persistence keeps independent of the engine in a way, that we want to have the possibility to run Cycle only with the Cycle database but without any tables for the engine. But we still want to use the Bootstrapping and configuration mechanism of the Engine/PVM.

But that seems to be harder than I thought. Currently we extended the DbSqlSessionFactory with a CycleDbSqlSessionFactory and overwrote some methods to use the correct mapping and sql files. But that feels a bit hacky.

And we use the ProcessEngines.getDefaultProcessEngine()).getProcessEngineConfiguration() to create that class. This internally already starts the Engine DB Session, so we need the tables for the engine again 😉

I think it would be good if somebody who wrote the wiring and persistence can have a look on that and maybe refactor it (after the code freeze in our branch or after the release of course). Maybe Tom?

Thanks a lot and cheers
Bernd

[tombaeyens]

I think engine and cycle should always use the same DbSqlSessionFactory.  If something needs to be changed to that, bring it up.

I think the most difficult part will be in the setup scripts.  If we really want to support that people select the individual components (with their respective DB tables) from activiti, then the first thing we need to tackle is the schema creation.  But this will be a very big challenge.  Especially going forward.  Think about how it complicates the schema upgrade for instance.

First thing we need to do is analyse if this is really so important.  And let's distinct between capabilities that the engine has for developers (=power users) like yourself, and the features that we expose to our users and document in the userguide.

Can you summarize the requirements that you targetted when you came to this design?  That makes it easier for me to review and discuss.

[bernd_ruecker]

Hi Tom.

The most important requirement is, that we can use Cycle without the tables for the Engine. We have a pilot customer currently for Cycle, but like big companies are, creating databases must be done via some DBA's. Any they want to know what certain tables are for, and if you tell them "hey we only need that one tables, the rest are empty but we need them created" they will tell you to get a proper software 😉 At least I think that will happen.

Despite from that, if you make different modules, they should be different. Kind of clean speration I think.

I think as well we should use the same Factory, look at the Cycle Factory and you will get an idea of which hard coded things make problems in cycle for that. Feel free to refactor that, so we can get rid of it!

If we really want to support that people select the individual components (with their respective DB tables) from activiti, then the first thing we need to tackle is the schema creation. But this will be a very big challenge. Especially going forward. Think about how it complicates the schema upgrade for instance.

Where is the difference? I have either 10 Engine + 2 Cycle Tables to create/update or 12 Activiti tables. Same effort or not?

What may be a bit tricky are tables commonly used, like the properties table at the moment. This table we maybe need in Cycle standalone as well?

Cheers
Bernd

[tombaeyens]

1) can you pinpoint the reason why you couldn't use the DbSqlSessionFactory as-is, even if you only created the cycle tables?

2) think twice or even verify the expected DBA's comments.  how much time are you willing to invest to fix this.  how will this impact QA, CI and the setup?  as an indication, I think i spend more time in QA and CI these days then in feature development.  you could also consider these 2 simple fixes for the DBA problem: a) don't tell that many of those tables are empty in the first place or b) stick some dummy data in there to make em happy 🙂

[bernd_ruecker]

For 2) Actually, if you have a good DBA they ask a lot of questions what data, how much in which intervals will go into it. That makes it possible to create indexes, tablespaces and whatever correctly…

For me the goal is, I want to tell everybody, that Cycle is as well usable without the engine. The jar dependency is not problem, the tables IMHO are. Maybe we can ask Christian for an oppinion, as he is the guy currently always adjusting the DB scripts 🙂

For 1) Look in the Cycle class, there you should already see it. From my head:
- The mybatis mapping is referenced hard coded (and we need a different mapping file9
- Same for the create-update-drop scripts, hard coded files

[tombaeyens]

For 2) Actually, if you have a good DBA they ask a lot of questions what data, how much in which intervals will go into it. That makes it possible to create indexes, tablespaces and whatever correctly…

can you documented that feedback on the forum so that we can shareit with companies like next level integration?

For me the goal is, I want to tell everybody, that Cycle is as well usable without the engine. The jar dependency is not problem, the tables IMHO are. Maybe we can ask Christian for an oppinion, as he is the guy currently always adjusting the DB scripts 🙂

you have to ensure that we're in sync on the architecture.  and the the componentization is straightened out.  if you just hack it in, then over time you will get frustrated that our impl changes lead to more maintenance for you on your integration.  we need clear documented features and a clear interface that is guarded by ci.  interface is to be taken really broad here.   interface means all aspects that users are confronted with.  creating tables is something that users have to do so it is part of the interface.  think beyond, what line needs to be changed and try to think about all aspects of the interface is and how we can build QA+CI on that.  while working out the QA and CI, that's when things have to get concrete and that will force us to have the discussions to get and keep in sync.

- The mybatis mapping is referenced hard coded (and we need a different mapping file9

i don't see this yet.  can you elaborate? 

- Same for the create-update-drop scripts, hard coded files

on first sight this seems doable.  though i'll have to think about how this will be exposed in the setup scripts towards the users

[cmuelder]

From my point of view:

1. Bernd is right! DBAs are almost always like he described. So i would not start a discussion with em nor would i follow your advice, tom!

2. I also agree with bernd, that cycle should be running without the engine database tables. I haven´t looked into cycle code yet. Is it designed like that?

3. But, the effort for qa and ci are already very high and i don´t see a chance to do that now.

4. The point of time is also not very helpful.

It might at least be a good idea, if someone could come up with a plan. What has to be changed! How much time has to be spend to update qa,ci and setup. Right now, i can also only guess. But i might be mistaken.

Best regards,
Christian

[bernd_ruecker]

Hey guys.

We discussed that topic today again. So the summary is: Somehow a clear separation would make sense, but seems pretty hard to get and increases QA overhead. And nobody really has time for this now.

So we could live with a solution, where you cannot run Cycle without the Engine tables, at least for the GA. And we could still think about separating it later on.

But that means, that we refer to cycle mappings from within the engine and the engine needs the cycle always as well. So we put that together. Okay?

Who wants to do that? We have an issue for that, but actually not with the highest priority, so we will not come to that the next month I guess 😕

Cheers
bernd

[cstettler]

But that means, that we refer to cycle mappings from within the engine and the engine needs the cycle always as well. So we put that together. Okay?

Does that mean that one cannot deploy the activiti engine without deploying activiti cycle as well? We definitely have the use case where we have to integrate the activiti engine into an existing technology stack / application, but without deploying (even on jar level) the activiti cycle component. IMHO this needs to be possible for sure.

@Bernd: can you elaborate at bit more on this dependency? Thank you!

Regards,
Christian

[bernd_ruecker]

Hi Christian.

Could you maybe have a look in the code base in the CycleDbSqlSessionFactory and the corresponding mapping and sql files. I think you will see "the problem". I leave for vacation tomorrow, so I will actually have no time to get into that in the next two weeks, maybe Kristin can assist you.

Cheers
Bernd