topic Re: Cycle (and Engine?) Persistence need refactoring in Alfresco Archive

Cycle (and Engine?) Persistence need refactoring

bernd_ruecker — Fri, 24 Sep 2010 07:39:28 GMT

Hey guys.We checked in the first DB persistence for Cycle in the meantime. Currently it just saved the Cycle Configuration to the database as a first step. We wanted to leverage which was already there in the DbSqlSessionFactory (e.g. create-drop mechanism). But one goal was, that cycle persistence

Re: Cycle (and Engine?) Persistence need refactoring

tombaeyens — Mon, 27 Sep 2010 06:58:41 GMT

I think engine and cycle should always use the same DbSqlSessionFactory. If something needs to be changed to that, bring it up.

I think the most difficult part will be in the setup scripts. If we really want to support that people select the individual components (with their respective DB tables) from activiti, then the first thing we need to tackle is the schema creation. But this will be a very big challenge. Especially going forward. Think about how it complicates the schema upgrade for instance.

First thing we need to do is analyse if this is really so important. And let's distinct between capabilities that the engine has for developers (=power users) like yourself, and the features that we expose to our users and document in the userguide.

Can you summarize the requirements that you targetted when you came to this design? That makes it easier for me to review and discuss.

Re: Cycle (and Engine?) Persistence need refactoring

bernd_ruecker — Mon, 27 Sep 2010 07:22:14 GMT

Hi Tom.

The most important requirement is, that we can use Cycle without the tables for the Engine. We have a pilot customer currently for Cycle, but like big companies are, creating databases must be done via some DBA's. Any they want to know what certain tables are for, and if you tell them "hey we only need that one tables, the rest are empty but we need them created" they will tell you to get a proper software 😉 At least I think that will happen.

Despite from that, if you make different modules, they should be different. Kind of clean speration I think.

I think as well we should use the same Factory, look at the Cycle Factory and you will get an idea of which hard coded things make problems in cycle for that. Feel free to refactor that, so we can get rid of it!

If we really want to support that people select the individual components (with their respective DB tables) from activiti, then the first thing we need to tackle is the schema creation. But this will be a very big challenge. Especially going forward. Think about how it complicates the schema upgrade for instance.

Where is the difference? I have either 10 Engine + 2 Cycle Tables to create/update or 12 Activiti tables. Same effort or not?

What may be a bit tricky are tables commonly used, like the properties table at the moment. This table we maybe need in Cycle standalone as well?

Cheers
Bernd

Re: Cycle (and Engine?) Persistence need refactoring

tombaeyens — Mon, 27 Sep 2010 07:28:52 GMT

1) can you pinpoint the reason why you couldn't use the DbSqlSessionFactory as-is, even if you only created the cycle tables?

2) think twice or even verify the expected DBA's comments. how much time are you willing to invest to fix this. how will this impact QA, CI and the setup? as an indication, I think i spend more time in QA and CI these days then in feature development. you could also consider these 2 simple fixes for the DBA problem: a) don't tell that many of those tables are empty in the first place or b) stick some dummy data in there to make em happy 🙂

Re: Cycle (and Engine?) Persistence need refactoring

bernd_ruecker — Mon, 27 Sep 2010 07:45:34 GMT

For 2) Actually, if you have a good DBA they ask a lot of questions what data, how much in which intervals will go into it. That makes it possible to create indexes, tablespaces and whatever correctly…

For me the goal is, I want to tell everybody, that Cycle is as well usable without the engine. The jar dependency is not problem, the tables IMHO are. Maybe we can ask Christian for an oppinion, as he is the guy currently always adjusting the DB scripts 🙂

For 1) Look in the Cycle class, there you should already see it. From my head:
- The mybatis mapping is referenced hard coded (and we need a different mapping file9
- Same for the create-update-drop scripts, hard coded files

Re: Cycle (and Engine?) Persistence need refactoring

tombaeyens — Mon, 27 Sep 2010 07:56:46 GMT

For 2) Actually, if you have a good DBA they ask a lot of questions what data, how much in which intervals will go into it. That makes it possible to create indexes, tablespaces and whatever correctly…

can you documented that feedback on the forum so that we can shareit with companies like next level integration?

For me the goal is, I want to tell everybody, that Cycle is as well usable without the engine. The jar dependency is not problem, the tables IMHO are. Maybe we can ask Christian for an oppinion, as he is the guy currently always adjusting the DB scripts 🙂

you have to ensure that we're in sync on the architecture. and the the componentization is straightened out. if you just hack it in, then over time you will get frustrated that our impl changes lead to more maintenance for you on your integration. we need clear documented features and a clear interface that is guarded by ci. interface is to be taken really broad here. interface means all aspects that users are confronted with. creating tables is something that users have to do so it is part of the interface. think beyond, what line needs to be changed and try to think about all aspects of the interface is and how we can build QA+CI on that. while working out the QA and CI, that's when things have to get concrete and that will force us to have the discussions to get and keep in sync.

- The mybatis mapping is referenced hard coded (and we need a different mapping file9

i don't see this yet. can you elaborate?

- Same for the create-update-drop scripts, hard coded files

on first sight this seems doable. though i'll have to think about how this will be exposed in the setup scripts towards the users

Re: Cycle (and Engine?) Persistence need refactoring

cmuelder — Mon, 27 Sep 2010 19:17:15 GMT

From my point of view:

1. Bernd is right! DBAs are almost always like he described. So i would not start a discussion with em nor would i follow your advice, tom!

2. I also agree with bernd, that cycle should be running without the engine database tables. I haven´t looked into cycle code yet. Is it designed like that?

3. But, the effort for qa and ci are already very high and i don´t see a chance to do that now.

4. The point of time is also not very helpful.

It might at least be a good idea, if someone could come up with a plan. What has to be changed! How much time has to be spend to update qa,ci and setup. Right now, i can also only guess. But i might be mistaken.

Best regards,
Christian

Re: Cycle (and Engine?) Persistence need refactoring

bernd_ruecker — Wed, 29 Sep 2010 14:50:41 GMT

Hey guys.

We discussed that topic today again. So the summary is: Somehow a clear separation would make sense, but seems pretty hard to get and increases QA overhead. And nobody really has time for this now.

So we could live with a solution, where you cannot run Cycle without the Engine tables, at least for the GA. And we could still think about separating it later on.

But that means, that we refer to cycle mappings from within the engine and the engine needs the cycle always as well. So we put that together. Okay?

Who wants to do that? We have an issue for that, but actually not with the highest priority, so we will not come to that the next month I guess 😕

Cheers
bernd

Re: Cycle (and Engine?) Persistence need refactoring

cstettler — Thu, 30 Sep 2010 07:08:44 GMT

But that means, that we refer to cycle mappings from within the engine and the engine needs the cycle always as well. So we put that together. Okay?

Does that mean that one cannot deploy the activiti engine without deploying activiti cycle as well? We definitely have the use case where we have to integrate the activiti engine into an existing technology stack / application, but without deploying (even on jar level) the activiti cycle component. IMHO this needs to be possible for sure.

@Bernd: can you elaborate at bit more on this dependency? Thank you!

Regards,
Christian

Re: Cycle (and Engine?) Persistence need refactoring

bernd_ruecker — Thu, 30 Sep 2010 07:17:11 GMT

Hi Christian.

Could you maybe have a look in the code base in the CycleDbSqlSessionFactory and the corresponding mapping and sql files. I think you will see "the problem". I leave for vacation tomorrow, so I will actually have no time to get into that in the next two weeks, maybe Kristin can assist you.

Cheers
Bernd

Re: Cycle (and Engine?) Persistence need refactoring

bernd_ruecker — Mon, 01 Nov 2010 11:44:15 GMT

Hey!

This issue gets more urgent, since the persistence code in the engine already has changed and I just have quickly adopted our code to it. But we currently already have no support for the "DBSCHEMASTRATEGY_CREATE_IF_NECESSARY" strategy and the whole implementation is a bit hacky.

@Tom: Could you maybe have a look into it? It is basically the class "AbstractCycleDaoMyBatisImpl" and you will see the problem we had there…

That would be really nice!

Cheers
Bernd

Re: Cycle (and Engine?) Persistence need refactoring

tombaeyens — Tue, 02 Nov 2010 20:37:10 GMT

In the meantime, I think it is a good idea if the Cycle tables are separate from the Engine tables so that Cycle can run independant of the engine. Does that answer the most important part of this topic? Are there other outstanding questions?

Last time I looked, you had a lot of duplicated code with respect to the DbSqlSessionFactory/DbSqlSession. Is that still the case? If yes, do you think we could allow better reuse here? I think it should be possible that cycle can work independent (no engine tables) but still leverage the base persistence infrastructure from the engine. WDYT?

Re: Cycle (and Engine?) Persistence need refactoring

bernd_ruecker — Wed, 03 Nov 2010 12:43:37 GMT

Hey Tom.

Hmm, that's exactly what I wanted the whole time 🙂 As I already wrote, I want to get rid of the duplicated code as well, but that needs some refactoring in the SessionFactory & Session code in the engine. And this would be good, if you can have a look at it (especially AbstractCycleDaoMyBatisImpl). Would that be possible for you?

Re: Cycle (and Engine?) Persistence need refactoring

bernd_ruecker — Wed, 03 Nov 2010 12:45:57 GMT

Short addition: The InternalActivitiTestCase offers a method assertAndEnsureCleanDb(). This currently only works hard coded with Engine tables. So this would be nice as well to extend/improve to be usable in both components. If you have an idea to this as well? Unfortunately I will not really have much time till the GA to look into this stuff myself 😞