topic Re: Maven2 Build for Alfresco in Alfresco Archive

Maven2 Build for Alfresco

rdanner — Sat, 21 Jul 2007 19:07:26 GMT

Hello everyone… we (the royal 'we' as in the Alfresco Community) need your input on a very important subject. BUILD.Alfresco has been using a custom build harness for the last two years and in that time there seems to be a growing support for the use of Maven2. I'd like to open the floor here for y

Re: Maven2 Build for Alfresco

rdanner — Sat, 21 Jul 2007 19:27:35 GMT

We use maven 2 here for everything. I love maven Ã¢â‚¬â€œ I hate maven – but Ã¢â‚¬â€œ itÃ¢â‚¬â„¢s the best we have and it makes sense for open source projects. Once you know maven – you know it. Everyone I know who has switched resisted it at first because Maven 1.x stank but they are all glad they made the shift to Maven2. This is another area where we can seek de facto based standards and reduce the barrier to entry for Alfresco.

That said… my gripes with maven are (Maven has some flaws that have nothing to do with Maven per se.):

           First is the issue of Sun Jars which are not allowed to be housed in the Maven repository or any other repository because of some silliness on the part of Sun. Second are all the other jars that are not in the maven repository because vendors (cough-fresco-cough <GRIN>) have not made them available.

           The second issue is that the central maven repository is a superfund site where people have managed to place the same libraries at 30 different groups and names Ã¢â‚¬â€œ this is mavens fault because they donÃ¢â‚¬â„¢t seem to manage the issue.

           The third issue is that Eclipse plug-ins for maven are Terrible (yes Ã¢â‚¬â€œ with a capital Ã¢â‚¬ËœTÃ¢â‚¬â„¢) but they are just as good as any support for home cooked ant builds so I am not sure it matters Ã¢â‚¬â€œ we just donÃ¢â‚¬â„¢t gain much on the eclipse front. Netbeans however seems to have nice tools (and a profiler) so in some ways we are gaining there for people who use netbeans.

           The last issue I can think of is that working offline with maven has sometimes bitten me because I can't reach a dependency – this is not much different that just not having the Jar on hand except that with Maven you don't ship with the jars the way the Alfresco source is currently shipped – good for the download, it's small if you just want to sniff around the source but it does mean you must be on-line for your first build.

Other than that – my concerns are with logistics:

           We need a strong plan with guidelines for defining groups, artifact ids, dependencies and other practices.

           We need a plan for the overall source configuration – to be honest Alfresco is pretty well suited for a port: we already have things broken down in to sub projects.

           I think we need a proof of concept that demonstates
                * nightly builds will continue to work
                * junit tests will continue to run
                * the sun will continue to rise and so fourth

           The actual porting of the code to a Maven2 build is simple and we could do it very quickly with the help of a few – that isnt the hard part. The hard part would be actually making the switch.

           Alfresco's engineering team has to be on board and has to participate in moving and the community has to be on board and know whats happening and when its going to happen and then have support after it has happened.

           That is the tough part – making the change where it touches the people

           If we do decide to make the move (aka there is strong support and the POC works well) we would need to move with the quickness (kung-fu sword noises here) and make this happen. We can't sit on the fence with two builds, it is too far down the stack and way too important to mess around. It is also essentially a fork until it is one build again … so once we decide to go (should we decide to go) we need to execute the act quickly.

Re: Maven2 Build for Alfresco

jcox — Sun, 22 Jul 2007 00:27:33 GMT

Good topic Russ!

Before we get too deeply into the fine points of Ant versus Maven II,
I think it's worth stepping back and defining our needs. This way, we'll
be in the much better shape to identify the best way to proceed
(or at least to weight tradeoffs with a checklist in hand).

Here's what I'd like to see regardless of our choice of build framework:

Remove inter-project dependency issues for good

Remove all .classpath issues in Eclipse for good

Make the source tree reflect the java package organization

Default: one jar per java package; build others as-needed.

All build output can be configured to go outside the source tree

Make Javadoc build better docs & overall indices w/o special tweaking

Improve build speed

Allow builds to occur directly from read-only source archive

This could be done by reorganizing the code within Ant so that all the java
code is in a single project. Essentially, I think this is just the next logical
step along a path begun when we got rid of recursive ant calls in our
build; we're letting javac do its job in sorting out dependencies rather
than hard-coding them ourselves (in 2 places). I dont' know how easy
or hard that is to do in Maven II compared to Ant.
Does somebody care to jump in here?

There are always funny corner-cases when it comes to building
non-trivial programs so I thought it might be nice to think about
it collectively. The one real technical issue I can see up-front with
this is the following:

        
         Package  org.alfreco.foobar  has classes a, b, c, and d.
         Classes a & b are in project ab, and go into ab.jar, while
         classes c & d are in project cd, and go into cd.jar.  Due 
         to some security/classloader/bundling reason, ab.jar and
         cd.jar must reside in different locations.   
‍‍‍‍‍‍‍

This could be resolved in three different ways:

Punt. If you need a jar in more than one place, the fact that it has a bit of extra

Make targets that create more selective jars by picking & choosing which classes they want.

Reorganizing the code so that packages don't need to be split across different jars.

Note that [1], [2] and [3] are not mutually exclusive options.
We could get something working in a hackish way fairly quickly
via [1] and [2], and clean things up as we go via [3].

There are probably a bunch of other issues I'm not considering at all, so
if you can think of anything big offhand, it would be nice to hear about.
My guess is that the pros strongly outweigh the cons, but that could
certainly change if someone thinks of a show-stopper.

In the end, I'd like us to focus on what problem we're solving first,
rather than the specific technology we're using. The inter-project
dependency issues we've got now have become a bit painful,
a finer-grained set of jars would give those who accept "hot patches"
more peace of mind, the mapping of packages to paths has
become annoying in places, and I'm sick of javadoc indexes not
being quite right.   Those are the pain points I can see immediately,
but perhaps there are others too.   If so, what are they?

In what ways would Maven II help or hurt relative to what could be
done by reorganizing the 22 ant projects into 1 (as it probably should
have been originally)? Ultimately, if Maven II helps with respect to
these specific "pain point" issues, and does not create other problems,
I'm very open to understanding how and why this is so.   Otherwise,
I'm inclined to stick with the known pros/cons of Ant, but use it in a
better way. When the dust settles a bit on 2.1, we're definitely going to
do some homework on Maven II.

I'm not speaking for all of engineering here, just myself.
From what I can tell, nobody in engineering has a very strong
opinion on this yet because we just haven't had time to discuss
it or analyze it in detail.   It's a conversation that's well worth having.

Thanks for kicking off the discussion!

       Cheers,
       -Jon

Re: Maven2 Build for Alfresco

rdanner — Sun, 22 Jul 2007 01:12:52 GMT

Good topic Russ!

Before we get too deeply into the fine points of Ant versus Maven II,
I think it's worth stepping back and defining our needs. This way, we'll
be in the much better shape to identify the best way to proceed
(or at least to weight tradeoffs with a checklist in hand).

…

You make some really good points. I think a bunch of us were looking at making a one for one switch just to head in the direction of a somewhat industry or at least open source standard for the sake of lowering the barrier to entry for folks who have never built Alfresco. Frankly speaking, I wasn't even aware of two or three of the issues you surfaced.

There are some features that come with Maven II which are very good. One that I am particularly fond of is the way it performs tests. It goes through a lot to make sure that classloaders are 'sterile' and that the test environment has only what is needed (declared) for the test.

Other features include plugins for eclipse, javadoc, clean, and other such capabilities.

I want to go back and look at your post more carefully. Maybe there are some points I can address. I have a couple of answers concerning maven and also some comments which don't have anything to do with maven that I'll take a swing at.

Like you said: it's probably a good idea to make sure we have a good list of the concerns we want to address before we go too far down the technology path but I'll throw out some maven trivia to address your points and maybe you can respond to my question concerning API(s) below.

RE: Remove inter-project dependency issues for good

This doesn't seem to be a build harness question per se, but rather a packaging question. I'm trying to think through the issue in the moment here and one thing that pops out to me is that there are some basic dependencies like core that you will have between projects. That is, other projects will need core and possibly other low level libraries in order to build. I think you are addressing something different and I am just not quite sure what.

Something that I brought up out in CA was pulling the API's out in to separate jars. I think there are several obvious pros and only one constant con and one situational that I can think of.

Pros:

Cons:

Remove all .classpath issues in Eclipse for good

What do you mean by this? Maven II will generate your eclipse machinery but perhaps the issue you are speaking of will still be presnent. Typically the eclipse machinery is not kept in SVN, its generated by maven by typing mvn eclispe:eclipse or by using the mvn plugin to import the project.

Make the source tree reflect the java package organization
This is independent of the build harness correct? We only need make sure that we comply with this requirement, and that the build tool doesn't stop us from keeping compliant.

RE: Default: one jar per java package; build others as-needed.
RE: All build output can be configured to go outside the source tree

Maven builds in a target folder. I never looked to see if it can be configured to be put outside the source tree – by default it is in the source tree but at the top level (not in line with the java source files)

RE: Make Javadoc build better docs & overall indices w/o special tweaking

What is throwing them off currently?

RE: Improve build speed

I think granularity and organization come to play here. You need the capability to easily build only what you need to build. I am not sure Maven would be "faster" for its own sake but we could probably make some optimizations in the organization which would impact build time.
We should make sure that whatever we do, build time doesnÃ¢â‚¬â„¢t get worse.

RE: Allow builds to occur directly from read-only source archive

I don't see any issues with this.

Good perspective Jon. Does anyone else have concerns about the current build harness that we aught to investigate?

Re: Maven2 Build for Alfresco

rdanner — Sun, 22 Jul 2007 01:36:46 GMT

Good topic Russ!

Here's what I'd like to see regardless of our choice of build framework:
Remove inter-project dependency issues for good

Remove all .classpath issues in Eclipse for good

Make the source tree reflect the java package organization

Default: one jar per java package; build others as-needed.

All build output can be configured to go outside the source tree

Make Javadoc build better docs & overall indices w/o special tweaking

Improve build speed

Allow builds to occur directly from read-only source archive
…

I'll put together a wiki page on requirements so we have something we can use as a measuring stick when considering our next steps and so that we have something to build a test plan from.

Re: Maven2 Build for Alfresco

jcox — Sun, 22 Jul 2007 03:50:16 GMT

The inter-project dependencies are a needless pain in the neck
for developers.   It ends up forcing certain things to live in
places because of when they're built, rather than what would
be the ideal logical organization. In at few cases, it forces
us to do lazy-init within spring…etc.   Cruft like this only gets worse
over time, so I'd like us to fix now while it's still relatively easy.
This is a developer's perspective, not a user's… but that matters too!

I'd like to learn a lot more about Maven II. If it can meet all our
requirements, and most folks prefer it, then that's great.   Ant is ok,
but I don't love it.   That said, I think we could be using Ant a lot
better than we are currently.

This doesn't seem to be a build harness question per se, but rather a packaging question.

It's not just about the packaging. Structuring things so that one big
javac can see everything at once means perfect build dependencies
with 0 effort forever. Otherwise we're right back to:

class X needs to be built before Y, but it needs class P
which must be put in location Q but then that generates a circular
dependency that Ant can handle but Eclipse can't… unless you
sprinkle lazy inits here and there in various spring config files…
and then muck around with .classpath files used by the continuous
builds done by IDEs such as Eclipse… and… and….

Ugh. The picture I'm painting isn't that bad yet, but it's always there,
gnawing, gnawing, gnawing….

Regarding your concern about many jar files, you can actually
compose jars, so there can actually be *fewer* of them in the
final product regardless of the intermediate jars you choose to
create (or not).   If we go the many-smaller-jars route in the
final product, then the way of handling compatability is via
a major/minor numbering scheme.   Tried and true. The existence
proof: UNIX libs   (kudos to Sun for getting it right).

Check out One-JAR:
http://www.ibm.com/developerworks/library/j-onejar/index.html?ca=drs-j4904
As far as I'm concerned, everything is on the table.

> Make the source tree reflect the java package organization
This is independent of the build harness correct?

It depends on the underlying capabilities of that harness.
For example, the way we're using Ant currently, source code
that belongs to the same package cannot live in the same directory,
due to build dependency issues. That's not a fundamental problem
with Ant, but rather with our current use of Ant.   It's my understanding
that Maven II lets you do things at a higher level than Ant, but sometimes
that can mean loss of lower-level control.

For example, some tools force you to build things a certain way,
and if your goals and their model are badly matched, the tool gets
in the way).    I'm not saying that's the case with Maven II,
but I don't know for sure that it's not either… yet.

Being able to build from a read-only source is great.
It adds a lot of flexibility to the system in terms of
permissions, media types, etc.   It's also just cleaner
(e.g.: we'll be able to get rid of a lot of svn:ignore tags).

As for build speed, one javac is faster than n, and only ever builds
exactly what's out of date.   When you have n projects, you've got
to be a lot more careful, and even then people tend to throw in a
"clean" operation because they get scared.   As soon as you do that,
you're rebuilding way more than you really need to; if you fail to
do that when things are slightly wrong in the dependencies
(which is easy to do), then you waste time in a different way…
which is no fun either.

Essentially, as soon as you break things up into multiple projects,
you take ownership of hand-crafting dependencies rather than just
letting the compiler do its job.    It's bad for exactly the same reasons
recursive make is bad.

How these compiled classes are *packaged* however, is a totally
orthogonal issue, so long as the build framework supports multiple
artifacts per project.   In Maven I, that was an issue, but I don't
know what the deal is with Maven II.   Some stuff needs to go
into server/lib while other stuff needs to be in common/lib,
so this actually is a hard requirement.   I don't want to be forced
into a different project / javac invocation just because the build
framework thinks it knows better and refuses to generate multiple
artifacts in one project.   If that's the case with Maven II, it's a
non-starter in my view.

Again, this isn't something we've had a chance to discuss a lot
internally, so I'm not summarizing a collectively held viewpoint,
nor am I even expressing misgivings regarding Maven II.
For all I know right now, I'd love it.   That said, this is at least
part of the checklist I'll use when I get around to forming my
own opinion on the topic.   I'm sure there will be other stuff
too, such as learning curve, interop with tools, how others
in the community & engineering feel about it all, and so forth.

Well, you asked for my opinion, and all you got was my meta-opinion!
I hope to do better than that soon.

Cheers,
-Jon

Re: Maven2 Build for Alfresco

rdanner — Sun, 22 Jul 2007 04:07:02 GMT

I've started a wiki page concerning issues with the current build and other requirements we would like to consider moving forward.

Please feel free to add your own requirements, thoughts etc. I only ask that you try and organize your thoughts along with eveyone elses. The page doesn't have much structure yet but I think we will see some very soon.

http://wiki.alfresco.com/wiki/Build_Harness_Requirements

Re: Maven2 Build for Alfresco

rdanner — Sun, 22 Jul 2007 05:06:16 GMT

The inter-project dependencies are a needless pain in the neck
for developers. It ends up forcing certain things to live in
places because of when they're built, rather than what would
be the ideal logical organization. In at few cases, it forces
us to do lazy-init within spring…etc. Cruft like this only gets worse
over time, so I'd like us to fix now while it's still relatively easy.
This is a developer's perspective, not a user's… but that matters too!

The comments on the build dependencies are really interesting for me and I am going to give them a lot of thought.

I've always liked the idea of keeping things in separate projects and using maven to include what needs to be included. I tend to think of it as a virtue that each piece only gets compiled with what it needs and what it needs is either declared or missing (in the POM file) in which case [need not declared] an explosion will occur. The pom file is a sort of documentation on the cohesion of the code. I also like the fact that you can enforce certain layering and strict use of APIs across libraries (check for cohesion with implementation) by breaking things in to sub projects and declaring dependencies. I haven't fretted much that I am taking some of the dependency management off the shoulders of javac but then, I've never worried too much about build time – which can become an issue if you have to rebuild and you freak and run a clean – and some of the symptoms you mention (like clean) are actualities.

Really good stuff Jon. I have a lot to think about here because what you are saying is very much different then what I am currently doing. We have a lot of small projects linked together by dummy pom files and Maven is responsible for determining the build order. I always like having to come from a completely different direction and examine how we are doing things. One of the reasons I though Maven would be a natural fit for Alfresco was the fact that Alfresco is currently split up in a similar fashion.

So if I get what you are saying: you would like packages to be atomic so that they can be individually built and deployed as a patch, but that the whole project can also be built in one javac.

ItÃ¢â‚¬â„¢s interesting because really never thought of Alfresco as a single project. I think of the core and repository as very different from the web client for example and IÃ¢â‚¬â„¢ve always thought those lines would deepen with time. Now, again that is a bit of deployment versus source tree talk, and they are not the same thing so again, I may have to go rethink. I am stuck in that mind set because I do break my libraries up as separate projects and I build my applications as yet separate projects. In most cases the applications have only spring configuration, web configuration and a few domain classes and maven pulls in all the dependencies.

The fact that the Alfresco approach was similar (not the same but not too far off) re-enforced my thought process around that kind of organization. IÃ¢â‚¬â„¢ve been aware of at least some of the basic CONS for some time but I have always felt the PROS outweighed them. IÃ¢â‚¬â„¢ll be interested to hear what others throw in on this subject but I really appreciate the thoughts you have put here especially because they are very different from my own on the subject.

Re: Maven2 Build for Alfresco

jcox — Sun, 22 Jul 2007 06:48:38 GMT

I've always liked the idea of keeping things in separate projects
and using maven to include what needs to be included. I tend to
think of it as a virtue that each piece only gets compiled with what
it needs and what it needs

I think taking over any of these dependencies just invites brittleness,
crufty hand-written configs, needless compilations, and/or bad builds.
As for enforcing proper layering, public/protected/package APIs
already handle this; a separate package is the wrong mechanism for
the job. In short, I want my *compiler* worrying about compilation,
not me.

Regarding code cohesion, I think that's what my packaging, distribution
system, SVN branches, and automatically enforced jar naming
conventions should handle, not my compiler.

So if I get what you are saying: you would like packages to be atomic
so that they can be individually built and deployed as a patch, but that
the whole project can also be built in one javac.

Ideally, I'd like the package hierarchy to be mirrored exactly in the
directory structure, then have one javac compile all my .java files
into class files. This means that anything that's out of date gets
rebuilt, and anything that's not isn't. No build artifact (e.g.: class file)
should be forced back into the source tree.    Then, depending on
how I want things packaged up, jar files get created.   There are many
ways this last bit could be done.   We could have a set of explicit
package-to-jar rules, and/or do jars-of-jars as we see fit via One-JAR
(again keeping these the heck away from the source tree). One nice
thing about One-JAR is that we could have neater-looking installs
yet at the same time, clear partitions within the One-JAR if it came
time to do patches in the field.

Different projects seem useful when you're integrating work across
different organization units, such as when you've got 3rd party plugins.
The benefit there is that if the 3rd party hands you something that
won't compile, the core system can forge ahead. However, within an
organization, something that breaks the build can be rolled back
because you've got SVN history right there.   Therefore, if it's
in the same SVN branch, crippling javac's ability to work out the
dependencies for you by busting things up into different projects
represents a substantial maintenance cost for no gain.

Hence, a single project with more flexibility on the backend
when it comes to packaging (jar/war creation & naming)
seems better to me than N projects hand-cobbled together
with hardcoded glue.

Sleepy now…

- Jon

Re: Maven2 Build for Alfresco

stk137 — Wed, 15 Aug 2007 21:57:36 GMT

One thing I'd like to gain out Alfresco moving to Maven would be a better SDK. There could be some custom web client architype to get started and unlike the current SDK this should be a proper JEE-Web project in my IDE that's aware of all the JSF-Facets, tags, and so on. As it is now, I open something like SDK CustomJSP and Eclipse (jee-europa) doesn't know it's a web project and there's bunch of errors for all the unknown tags.

The other benefit to replacing the current eclipse based SDK, with Maven
and architypes, is that it'd work in Eclipse (and derivatives like Red Hat Developer Studio), Netbeans, and IntelliJ as a JEE-Web-JSF project.
(Last I checked importing an Eclipse Web project in Netbeans didn't work, only basic Java projects)

I also I thought I'd mention this:
http://ancientprogramming.blogspot.com/2007/08/badly-packaged-project-will-break.html
upshot: create one distro rather than two like Spring and choose your group name carefully and don't change it.

Re: Maven2 Build for Alfresco

franz_see — Sun, 16 Sep 2007 07:16:13 GMT

Good day,

I'd just like to share my experiences with Maven2.

Maven2 offers convention over configuration. Meaning most of the defaults in Maven2 are what the community behind it considers to be the "best practices".

Remove all .classpath issues in Eclipse for good

This is actually more of a source code management issue rather than a build issue ( not unless your build depends on the .classpath ). In terms of Maven2, it does not depend upon the .classpath files. Furthermore, it can generate the .classpath files of your maven project based from your pom.xml ( which is the 'description' of your maven project ).

Make the source tree reflect the java package organization

Part of convention-over-configuration: This is the default of Maven2

( To give you a better of how Maven2 does its compilation, in the background, what Maven2 actually does is calling your javac command and supplying the parameters for it )

Default: one jar per java package; build others as-needed.

The default of Maven2 outputs one jar ( or binary ) per Maven project. If you have several java packages in one Maven Project, then all of those would be assembled into one binary.

So if you want one jar per java package, you'd have to create one maven project per java package ( though you can use the assembly plugin to create several binaries in one maven project, for this use case, i prefer one maven project per java package ).

Btw, what is the rationality for this? Although it's nice to componentizes your project, Im not a fan of over-componentizing of projects. IMHO, it makes things harder to read. Personally, I prefer if you break down things into smaller components only when needed ( i.e. when it becomes too big ). At least if you have to "hot patch" several java packages, users will only see it as "hot patch"'ing one component.

All build output can be configured to go outside the source tree

Part of the convention-over-configuration, by default, Maven2 will output the build in a directory separate from your source.

( Furthermore, you can install the built artifacts in your local repository or deploy them in some remote repository, so that other projects depending on them will not have to rebuild them ).

Make Javadoc build better docs & overall indices w/o special tweaking

Maven2 can generate javadocs for individual projects ( w/ proper indices ), but there was a bug before that it was not aggregating the javadocs of several maven projects. I'm not sure if this is fixed right now. Haven't tried it again.

I.m.p.r.o.v.e build speed

Maven2 does not actually specialize in build speed, but IMHO, it's not significantly slower either ( assuming all the required binaries are already in your local repository ), not unless your project is gigantic!

As for building the 'root' maven project - it can be slow. But you don't usually do that anyway. Furthermore, if you want to build the whole thing to make sure nothing breaks, you can use a Continuous Integration for that ( i.e. Continuum ).

Allow builds to occur directly from read-only source archive

Part of the convention-over-configuration, Maven2 does not overwrite your source by default ( or at least most of the plugins i've encountered anyway ).

If you have to modify the source files during the build process, what Maven2 does is that it first reads your source files, does the necessary changes, then outputs the modified sources in the build output ( and automatically updates Maven2 to look at those modified sources instead ).

This could be done by reorganizing the code within Ant so that all the java
code is in a single project. Essentially, I think this is just the next logical
step along a path begun when we got rid of recursive ant calls in our
build; we're letting javac do its job in sorting out dependencies rather
than hard-coding them ourselves (in 2 places). I dont' know how easy
or hard that is to do in Maven II compared to Ant.
Does somebody care to jump in here?

By default, Maven2 just javac's the java files in the current maven project.

But if your problem is recursive calls to build the dependencies, Maven2 has a solution for that.

Maven2 does not need to call javac on its dependencies. It will just look for its dependencies' binaries ( from the remote repositories & your local repository ), and include those in its classpath.

Note however that Maven2 cannot resolve cyclic dependencies ( 'A' depends on 'B', 'B' depends on 'A' ). But then again, if you have a cyclic dependency, then maybe there's something wrong with the componentization.

There are always funny corner-cases when it comes to building
non-trivial programs so I thought it might be nice to think about
it collectively. The one real technical issue I can see up-front with
this is the following:

        
         Package  org.alfreco.foobar  has classes a, b, c, and d.
         Classes a & b are in project ab, and go into ab.jar, while
         classes c & d are in project cd, and go into cd.jar.  Due 
         to some security/classloader/bundling reason, ab.jar and
         cd.jar must reside in different locations.   
‍‍‍‍‍‍‍

In Maven2, you can simply create two maven projects - one for ab, and another of cd.

And if there's cyclic dependency between the two, then you can probably take the 'cyclic parts' and create a foobar-commons project.

There are probably a bunch of other issues I'm not considering at all, so
if you can think of anything big offhand, it would be nice to hear about.
My guess is that the pros strongly outweigh the cons, but that could
certainly change if someone thinks of a show-stopper.

I am not sure if I've addressed all your concerns here.

May I ask for clarifications from your listed concerns ( i.e the problem being addressed by those solutions ).

Thanks,
Franz

Re: Maven2 Build for Alfresco

franz_see — Sun, 16 Sep 2007 07:58:48 GMT

The inter-project dependencies are a needless pain in the neck
for developers. It ends up forcing certain things to live in
places because of when they're built, rather than what would
be the ideal logical organization. In at few cases, it forces
us to do lazy-init within spring…etc. Cruft like this only gets worse
over time, so I'd like us to fix now while it's still relatively easy.
This is a developer's perspective, not a user's… but that matters too!

In Maven2, it does require the dependencies to live in certain places - remote repositories and local repository.

Pardon, I'm not familiar with your lazy-init examples

It's not just about the packaging. Structuring things so that one big
javac can see everything at once means perfect build dependencies
with 0 effort forever.

True, but for our price. If I would to load your source in my IDE, I would be overwhelmed by the number of files that I have to browse, as opposed to looking at the available components, openning that in my IDE, and searching in that component only.

Furthermore, if somebody would to use a part of your code, they would have to use the whole alfresco binary ( when they only need a part of it ).

Otherwise we're right back to:

class X needs to be built before Y, but it needs class P
which must be put in location Q but then that generates a circular
dependency that Ant can handle but Eclipse can't… unless you
sprinkle lazy inits here and there in various spring config files…
and then muck around with .classpath files used by the continuous
builds done by IDEs such as Eclipse… and… and….

In Maven2, you don't have to be rebuild everything. Once you've built it and installed in your local repository ( deployed in some remote repository ), Maven2 will just pick it up without rebuilding it. So you can build X once, then when you're working on Y, you can just rebuild Y ( even if X is cleaned ).

Also, locations of the binary is ( usually ) not a problem in Maven2, because by default, all binaries should be available in your local repository or in the remote repositories.

Furthermore, if you would need a binary to be in a specific location, the usual practice is to depend on that binary, and extract it ( from my experience, binaries that need to be in a specific location are usally extracted ) in your current build output directory.

And as for cyclic dependency among the components, Maven2 usually tries to avoid that.

Ugh. The picture I'm painting isn't that bad yet, but it's always there,
gnawing, gnawing, gnawing….

Regarding your concern about many jar files, you can actually
compose jars, so there can actually be *fewer* of them in the
final product regardless of the intermediate jars you choose to
create (or not).   If we go the many-smaller-jars route in the
final product, then the way of handling compatability is via
a major/minor numbering scheme.   Tried and true. The existence
proof: UNIX libs   (kudos to Sun for getting it right).

This can also be done in Maven2 ( i.e assembly plugin, uber jar )

Check out One-JAR:
http://www.ibm.com/developerworks/library/j-onejar/index.html?ca=drs-j4904
As far as I'm concerned, everything is on the table.

> Make the source tree reflect the java package organization
This is independent of the build harness correct?
It depends on the underlying capabilities of that harness.
For example, the way we're using Ant currently, source code
that belongs to the same package cannot live in the same directory,
due to build dependency issues. That's not a fundamental problem
with Ant, but rather with our current use of Ant. It's my understanding
that Maven II lets you do things at a higher level than Ant, but sometimes
that can mean loss of lower-level control.

True, it does let you do things at a 'higher-level' for some 'lower-level control'. But the cost is usually not a problem since those 'higher level' stuffs are "best practices".

Moreoever, you can use ant tasks in Maven2 if you're stuck

For example, some tools force you to build things a certain way,
and if your goals and their model are badly matched, the tool gets
in the way). I'm not saying that's the case with Maven II,
but I don't know for sure that it's not either… yet.

I'm not sure as well what problems alfresco may face in migrating to Maven2.

Being able to build from a read-only source is great.
It adds a lot of flexibility to the system in terms of
permissions, media types, etc. It's also just cleaner
(e.g.: we'll be able to get rid of a lot of svn:ignore tags).

Maven2 can handle that ( as I've explained in my previous post ).

As for build speed, one javac is faster than n, and only ever builds
exactly what's out of date. When you have n projects, you've got
to be a lot more careful, and even then people tend to throw in a
"clean" operation because they get scared. As soon as you do that,
you're rebuilding way more than you really need to; if you fail to
do that when things are slightly wrong in the dependencies
(which is easy to do), then you waste time in a different way…
which is no fun either.

Maven2 only builds the current maven project it is working on ( and not its dependencies ). As to checking whether the changes you've done in one component would break the whole thing, you can use a Continuous Integration ( i.e Continuum ) to run every now and then and build everything. And if you're components are lously copuled from each other, you would reduce the risk of breaking the whole thing when you change one part.

Essentially, as soon as you break things up into multiple projects,
you take ownership of hand-crafting dependencies rather than just
letting the compiler do its job. It's bad for exactly the same reasons
recursive make is bad.

That's usually not a problem ( I don't think I've encountered any serious compilation issues due to breaking things down into smaller projects).

Furthermore, it's easier to debug several small things instead of one big thing ( debugging - this I usually do, and because the group of sources that I look at is smaller, I've immediately isolated the problem to a smaller segment ).

How these compiled classes are *packaged* however, is a totally
orthogonal issue, so long as the build framework supports multiple
artifacts per project.   In Maven I, that was an issue, but I don't
know what the deal is with Maven II.   Some stuff needs to go
into server/lib while other stuff needs to be in common/lib,
so this actually is a hard requirement.   I don't want to be forced
into a different project / javac invocation just because the build
framework thinks it knows better and refuses to generate multiple
artifacts in one project.   If that's the case with Maven II, it's a
non-starter in my view.

By default, Maven2 outputs one package per project. As for configuring the setup of binaries into specific location, that's usually handled by the specific plugin ( maven-war-plugin for wars, maven-ear-plugin for ear's, etc). And if all else fails, you can use ant tasks to fix Maven2's mess

Again, this isn't something we've had a chance to discuss a lot
internally, so I'm not summarizing a collectively held viewpoint,
nor am I even expressing misgivings regarding Maven II.
For all I know right now, I'd love it. That said, this is at least
part of the checklist I'll use when I get around to forming my
own opinion on the topic. I'm sure there will be other stuff
too, such as learning curve, interop with tools, how others
in the community & engineering feel about it all, and so forth.

Well, you asked for my opinion, and all you got was my meta-opinion!
I hope to do better than that soon.

Cheers,
-Jon

I am not sure as well whether Maven2 would be best for your use case since I am not familiar with your build.

Cheers,
Franz

Re: Maven2 Build for Alfresco

mindthegab — Mon, 07 Apr 2008 18:37:17 GMT

Hi all, a bit delayed but wanted to leave my 2 cents on this *very interesting* topic.

I believe that independently from the technology that is used, which hardly depends upon the context you plan to adopt and deploy Alfresco in,
using Maven2 can be far more useful in structured enterprise environments where some of the mentioned requirements (strict dependency management and versioning or releasing policies / ownerships) which Ant is certainly the low key entry point for a test drive / evaluation or for very simple customizations for which maven would be overkill.

ATM, as no real news have happened in one year on the Alfresco side on the mavenization of the Alfresco sources,
we proceeded (in Sourcesense, http://www.sourcesense.com, alfresco partner) in using maven to build and customize Alfresco at our clients, typically big and potentially intricate enterprise environments.

To be clear: we use maven ATM to build alfresco extensions (in the future also AMP should be supported) and that does not mandatorily mean that Alfresco itself should build with maven, even if how it works now (basically a maven-webapp-archetype with a war dependency on the alfresco war) is not completely neat as it flattens the whole transitive dependency resolving magic of m2.

The process of improvement of this lifecycle management (more than just build) tool has been later on released on the alfresco forge (http://forge.alfresco.com/projects/m2alfresco/ and http://repository.sourcesense.com/maven2-sites/maven-alfresco-extension-archetype-1.0.0/) and provides a number of interesting features (like jetty embedded run, clean property filtering management, documentation, jboss/tomcat local/remote deploy, maven release support etc.).
For supporting this release, we made alfresco community artifacts (jar and war) available on Sourcesense repositories in order to have a zero conf startup for even junior developers (and to lower the maven learning curve). See http://repository.sourcesense.com/maven2/alfresco.

For enterprise customization instead, we happily use internal repositories (Sourcesense or clients') with the same approach.

About my 2 cents, it would be much better if Alfresco itself releases its artifacts on m2 repos (thus complete the m10n - mavenization - of source tree) for a number of reasons

- standardization of the naming/versioning
- easy startup
- Alfresco declares itself a ECM systems, as we know 'E' stands for "enterprise", and as we all (sadly) know "enterprise" stands for processes, release policies, bureaucracy, ownerships, lifecycle, development vs. maintenance. A m2 structured approach will ease to me integration in highly complex environments (and with many enterprise applications) in which is not just a matter of "taking and xml file, dropping it in the shared folder, and restart the server" , because it's *simply* not possible
- Having an archetype based approach can really gear community development and lower Alfresco "real" learning curve, as it's pretty much easier to have a list of properties to edit more than go-search-crawl-find on the wiki which is the Spring snippet to be modified. This is true for the extension projects (already in the forge) as well as AMP (let's say module build).
- Alfresco grows faster and faster, to me it's the moment (especially with the 3.0 release) to provide a real neat basis for the even greater growth alfresco expects from the great improvements they are doing: Apache maven is basis for most open source projects, and even tough and complex frameworks (e.g. Apache Cocoon) have completed (not without hassles) the m10n process, towards a more integrated and documented open source vision. The bigger it grows the more difficult will be to do this change and to me it may really make it an outstanding example of "enterprise opensource"
- I don't have to maintain a public m2 repository almost just for Alfresco

Ok, now you may think I'm a maven evangelist trying to find useful applications of that technology at all costs

On the other hand I believe that (especially being a partner) we should not impose technologies, while adapt the way we work to the open source (typically high) standards of development, and this is a characteristic that sometimes I see missing in Alfresco extension development: in this sense the archetype we've released builds *also* with Apache Ant (see http://forge.alfresco.com/forum/forum.php?forum_id=512), exploiting the tidy and standard maven project hierarchy and sharing the same property filtering policies.
Of course all the nice m2 features (like jetty run, jboss and tomcat deploy, release management, etc.) are not supported by this build, but what you just need is to checkout the jar, unzip it, and run ant to have a properly packaged, environment aware alfresco WAR build.

This was more a POC for the feasibility of this double approach, so I'm not planning to maintain the ant part while focusing on the interesting growth perspectives of the m2 part, but it can be useful to provide the double point of access I was mentioning before (enterprise vs. community).

So wrapping up:
I would be (and so my company Sourcesense) to see Alfresco move (and to help/support in this transition) to maven for building its source code, document and release it. But I understand it's a complex and risky process (but risks can be lowered supporting a bottom line Ant based build which provides continuity with current developments).

So ATM I'd love to have some feedback on the possibility of building extension (and AMP in a while) using archetypes / prototype based approach (even before an Alfresco official move to maven) and hopefully some contributions to the project. I think that it's vitally important to have a (still agile but) sort of a process when developing and releasing and application.
That's especially true/valuable for a partner but I think that will especially gear the professional open source community interest in Alfresco.

Hope this helps,
Gab

PS:
For some additional references on the maven topic you may be interested in (sorry for the spamming :

Library conventions - http://forums.alfresco.com/viewtopic.php?f=10&t=1017&p=39172#p39172
Maven repositories - http://forums.alfresco.com/viewtopic.php?f=12&t=10210&p=39175#p39175
Running outside a container - http://forums.alfresco.com/viewtopic.php?f=4&t=9317&p=39176#p39176
Maven4alfresco - http://forums.alfresco.com/viewtopic.php?f=32&t=9268&p=39179#p39179

Re: Maven2 Build for Alfresco

mindthegab — Wed, 06 Jun 2012 08:24:49 GMT

On step forward here;: Following yesterday's announcement of the official Alfresco Artifact Repository (http://blogs.alfresco.com/wp/its-official-alfresco-artifacts-repository/), wehave Alfresco Community and Enterprise artifacts are hosted at http://artifacts.alfresco.com .

Please check:

https://wiki.alfresco.com/wiki/Alfresco_Artifacts_Repository
http://mindthegab.com/2012/06/05/introducing-the-alfresco-artifacts-repository-yes-with-alfresco-enterprise/

On the building with Maven topic follow:

https://issues.alfresco.com/jira/browse/ALF-14390