Maven2 Build for Alfresco

[rdanner]

Hello everyone… we (the royal 'we' as in the Alfresco Community) need your input on a very important subject. BUILD.

Alfresco has been using a custom build harness for the last two years and in that time there seems to be a growing support for the use of Maven2.  I'd like to open the floor here for your input on Alfresco moving to a MAven2 based build.  

Please at least cast a ballot – as I mentioned, this is very important and it makes sense that if you have support or concerns which have not been voiced or require re-enforcement that you speak up.

For the sake of discussion I think there are several things we need to talk about.

First and foremost are the pros and cons of moving to maven. 
Second would be concerning the logistics of the move if it were to happen. 

Just to be clear there is no official movement from Alfresco to change the build process – this is just an exploratory question. There seems to be a lot of interest and several people are willing to help make the transition if it makes sense but there is no official plan from alfresco to switch. We won't know the facts unless you vote and unless you speak up. 

[size=200]Please make your thoughts known![/size]

[franz_see]

Good day,

I'd just like to share my experiences with Maven2.

Maven2 offers convention over configuration. Meaning most of the defaults in Maven2 are what the community behind it considers to be the "best practices".

Remove all .classpath issues in Eclipse for good

This is actually more of a source code management issue rather than a build issue ( not unless your build depends on the .classpath ). In terms of Maven2, it does not depend upon the .classpath files. Furthermore, it can generate the .classpath files of your maven project based from your pom.xml ( which is the 'description' of your maven project ).

Make the source tree reflect the java package organization

Part of convention-over-configuration: This is the default of Maven2

( To give you a better of how Maven2 does its compilation, in the background, what Maven2 actually does is calling your javac command and supplying the parameters for it )

Default: one jar per java package; build others as-needed.

The default of Maven2 outputs one jar ( or binary ) per Maven project. If you have several java packages in one Maven Project, then all of those would be assembled into one binary.

So if you want one jar per java package, you'd have to create one maven project per java package ( though you can use the assembly plugin to create several binaries in one maven project, for this use case, i prefer one maven project per java package ).

Btw, what is the rationality for this? Although it's nice to componentizes your project, Im not a fan of over-componentizing of projects. IMHO, it makes things harder to read. Personally, I prefer if you break down things into smaller components only when needed ( i.e. when it becomes too big ). At least if you have to "hot patch" several java packages, users will only see it as "hot patch"'ing one component. 

All build output can be configured to go outside the source tree

Part of the convention-over-configuration, by default, Maven2 will output the build in a directory separate from your source.

( Furthermore, you can install the built artifacts in your local repository or deploy them in some remote repository, so that other projects depending on them will not have to rebuild them ).

Make Javadoc build better docs & overall indices w/o special tweaking

Maven2 can generate javadocs for individual projects ( w/ proper indices ), but there was a bug before that it was not aggregating the javadocs of several maven projects. I'm not sure if this is fixed right now. Haven't tried it again.

I.m.p.r.o.v.e build speed

Maven2 does not actually specialize in build speed, but IMHO, it's not significantly slower either ( assuming all the required binaries are already in your local repository ), not unless your project is gigantic!

As for building the 'root' maven project - it can be slow. But you don't usually do that anyway. Furthermore, if you want to build the whole thing to make sure nothing breaks, you can use a Continuous Integration for that ( i.e. Continuum ).

Allow builds to occur directly from read-only source archive

Part of the convention-over-configuration, Maven2 does not overwrite your source by default ( or at least most of the plugins i've encountered anyway ).

If you have to modify the source files during the build process, what Maven2 does is that it first reads your source files, does the necessary changes, then outputs the modified sources in the build output ( and automatically updates Maven2 to look at those modified sources instead ).

This could be done by reorganizing the code within Ant so that all the java
code is in a single project. Essentially, I think this is just the next logical
step along a path  begun when we got rid of recursive ant calls in our
build;   we're  letting javac do its job in sorting out dependencies rather
than  hard-coding them ourselves (in 2 places).   I dont' know how easy
or hard that is to do in Maven II compared to Ant.
Does somebody care to jump in here?

By default, Maven2 just javac's the java files in the current maven project.

But if your problem is recursive calls to build the dependencies, Maven2 has a solution for that.

Maven2 does not need to call javac on its dependencies. It will just look for its dependencies' binaries ( from the remote repositories & your local repository ), and include those in its classpath.

Note however that Maven2 cannot resolve cyclic dependencies ( 'A' depends on 'B', 'B' depends on 'A' ). But then again, if you have a cyclic dependency, then maybe there's something wrong with the componentization.

There are always funny corner-cases when it comes to building
non-trivial programs so I thought it might be nice to think about
it collectively. The one real technical issue I can see up-front with
this is the following:

        
         Package  org.alfreco.foobar  has classes a, b, c, and d.
         Classes a & b are in project ab, and go into ab.jar, while
         classes c & d are in project cd, and go into cd.jar.  Due 
         to some security/classloader/bundling reason, ab.jar and
         cd.jar must reside in different locations.   
‍‍‍‍‍‍‍

In Maven2, you can simply create two maven projects - one for ab, and another of cd.

And if there's cyclic dependency between the two, then you can probably take the 'cyclic parts' and create a foobar-commons project.

There are probably a bunch of other issues I'm not considering at all, so
if you can think of anything big offhand, it would be nice to hear about.
My guess is that the pros strongly outweigh the cons, but that could
certainly change if someone thinks of a  show-stopper.

I am not sure if I've addressed all your concerns here.

May I ask for clarifications from your listed concerns ( i.e the problem being addressed by those solutions ).

Thanks,
Franz

[franz_see]

The inter-project dependencies are a needless pain in the neck
for developers.   It ends up forcing certain things to live in
places because of when they're built, rather than what would
be the ideal logical organization.  In at few cases, it forces
us to do lazy-init within spring…etc.   Cruft like this only gets worse
over time, so I'd like us to fix now while it's still relatively easy.
This is a developer's perspective, not a user's… but that matters too!

In Maven2, it does require the dependencies to live in certain places - remote repositories and local repository.

Pardon, I'm not familiar with your lazy-init examples

It's not just about the packaging. Structuring things so that one big
javac can see everything at once means perfect build dependencies
with 0 effort forever.

True, but for our price. If I would to load your source in my IDE, I would be overwhelmed by the number of files that I have to browse, as opposed to looking at the available components, openning that in my IDE, and searching in that component only.

Furthermore, if somebody would to use a part of your code, they would have to use the whole alfresco binary ( when they only need a part of it ).

Otherwise we're right back to:

class X needs to be built before Y, but it needs class P
which must be put in location Q but then that generates a circular
dependency that Ant can handle but Eclipse can't… unless you
sprinkle lazy inits here and there in various spring config files…
and then muck around with .classpath files used by the continuous
builds done by IDEs such as Eclipse… and… and….

In Maven2, you don't have to be rebuild everything. Once you've built it and installed in your local repository ( deployed in some remote repository ), Maven2 will just pick it up without rebuilding it. So you can build X once, then when you're working on Y, you can just rebuild Y ( even if X is cleaned ).

Also, locations of the binary is ( usually ) not a problem in Maven2, because by default, all binaries should be available in your local repository or in the remote repositories.

Furthermore, if you would need a binary to be in a specific location, the usual practice is to depend on that binary, and extract it ( from my experience, binaries that need to be in a specific location are usally extracted ) in your current build output directory.

And as for cyclic dependency among the components, Maven2 usually tries to avoid that.

Ugh. The picture I'm painting isn't that bad yet, but it's always there,
gnawing, gnawing, gnawing….

Regarding your concern about many jar files,  you can actually
compose jars, so there can actually be *fewer* of them in the
final product regardless of the intermediate jars you choose to
create (or not).   If we go the many-smaller-jars route in the
final product, then the way of handling compatability is via
a major/minor numbering scheme.   Tried and true.  The existence
proof:  UNIX libs   (kudos to Sun for getting it right).

This can also be done in Maven2 ( i.e assembly plugin, uber jar )

Check out One-JAR:
http://www.ibm.com/developerworks/library/j-onejar/index.html?ca=drs-j4904
As far as I'm concerned, everything is on the table.

> Make the source tree reflect the java package organization
This is independent of the build harness correct?

It depends on the underlying capabilities of that harness.
For example,  the way we're using Ant currently, source code
that belongs to the same package cannot live in the same directory,
due to build dependency issues.  That's not a fundamental problem
with Ant, but rather with our current use of Ant.   It's my understanding
that Maven II lets you do things at a higher level than Ant, but sometimes
that can mean loss of lower-level control.

True, it does let you do things at a 'higher-level' for some 'lower-level control'. But the cost is usually not a problem since those 'higher level' stuffs are "best practices".

Moreoever, you can use ant tasks in Maven2 if you're stuck

For example, some tools force you to build things a certain way,
and if your goals and their model are badly matched, the tool gets
in the way).    I'm not saying that's the case with Maven II,
but I don't know for sure that it's not either… yet.

I'm not sure as well what problems alfresco may face in migrating to Maven2.

Being able to build from a read-only source is great.
It adds a lot of flexibility to the system in terms of
permissions, media types, etc.   It's also just cleaner
(e.g.: we'll be able to get rid of a lot of  svn:ignore tags).

Maven2 can handle that ( as I've explained in my previous post ).

As for build speed, one javac is faster than n, and only ever builds
exactly what's out of date.   When you have n projects, you've got
to be a lot more careful, and even then people tend to throw in a
"clean" operation because they get scared.   As soon as you do that,
you're rebuilding way more than you really need to;  if you fail to
do that when things are slightly wrong in the dependencies
(which is easy to do), then you waste time in a different way…
which is no fun either.

Maven2 only builds the current maven project it is working on ( and not its dependencies ). As to checking whether the changes you've done in one component would break the whole thing, you can use a Continuous Integration ( i.e Continuum ) to run every now and then and build everything. And if you're components are lously copuled from each other, you would reduce the risk of breaking the whole thing when you change one part.

Essentially, as soon as you break things up into multiple projects,
you take ownership of hand-crafting dependencies rather than just
letting the compiler do its job.    It's bad for exactly the same reasons
recursive make is bad.

That's usually not a problem ( I don't think I've encountered any serious compilation issues due to breaking things down into smaller projects).

Furthermore, it's easier to debug several small things instead of one big thing ( debugging - this I usually do, and because the group of sources that I look at is smaller, I've immediately isolated the problem to a smaller segment ).

How these compiled classes are *packaged* however, is a totally
orthogonal issue, so long as the build framework supports multiple
artifacts per project.   In  Maven I, that was an issue, but I don't
know what the deal is with Maven II.   Some stuff needs to go
into  server/lib while other stuff needs to be in common/lib,
so this actually is a hard requirement.   I don't want to be forced
into a different project  / javac invocation just because the build
framework thinks it knows better and refuses to generate multiple
artifacts in one project.   If that's the case with Maven II, it's a
non-starter in my view.

By default, Maven2 outputs one package per project. As for configuring the setup of binaries into specific location, that's usually handled by the specific plugin ( maven-war-plugin for wars, maven-ear-plugin for ear's, etc). And if all else fails, you can use ant tasks to fix Maven2's mess

Again, this isn't something we've had a chance to discuss a lot
internally, so I'm not summarizing a collectively held viewpoint,
nor am I even expressing misgivings regarding Maven II.
For all I know right now, I'd love it.   That said, this is at least
part of the checklist I'll use when I get around to forming my
own opinion on the topic.   I'm sure there will be other stuff
too, such as learning curve, interop with tools, how others
in the community & engineering feel about it all, and so forth.

Well, you asked for my opinion, and all you got was my meta-opinion!
I hope to do better than that soon.

  Cheers,
  -Jon

I am not sure as well whether Maven2 would be best for your use case since I am not familiar with your build.


Cheers,
Franz

[mindthegab]

Hi all, a bit delayed but wanted to leave my 2 cents on this *very interesting* topic.

I believe that independently from the technology that is used, which hardly depends upon the context you plan to adopt and deploy Alfresco in,
using Maven2 can be far more useful in structured enterprise environments where some of the mentioned requirements (strict dependency management and versioning or releasing policies / ownerships) which Ant is certainly the low key entry point for a test drive / evaluation or for very simple customizations for which maven would be overkill.

ATM, as no real news have happened in one year on the Alfresco side on the mavenization of the Alfresco sources,
we proceeded (in Sourcesense, http://www.sourcesense.com, alfresco partner) in using maven to build and customize Alfresco at our clients, typically big and potentially intricate enterprise environments.

To be clear: we use maven ATM to build alfresco extensions (in the future also AMP should be supported) and that does not mandatorily mean that Alfresco itself should build with maven, even if how it works now (basically a maven-webapp-archetype with a war dependency on the alfresco war) is not completely neat as it flattens the whole transitive dependency resolving magic of m2.

The process of improvement of this lifecycle management (more than just build) tool has been later on released on the alfresco forge (http://forge.alfresco.com/projects/m2alfresco/ and http://repository.sourcesense.com/maven2-sites/maven-alfresco-extension-archetype-1.0.0/) and provides a number of interesting features (like jetty embedded run, clean property filtering management, documentation, jboss/tomcat local/remote deploy, maven release support etc.).
For supporting this release, we made alfresco community artifacts (jar and war) available on Sourcesense repositories in order to have a zero conf startup for even junior developers (and to lower the maven learning curve). See http://repository.sourcesense.com/maven2/alfresco.

For enterprise customization instead, we happily use internal repositories (Sourcesense or clients') with the same approach.

About my 2 cents, it would be much better if Alfresco itself releases its artifacts on m2 repos (thus complete the m10n - mavenization - of source tree) for a number of reasons

- standardization of the naming/versioning
- easy startup
- Alfresco declares itself a ECM systems, as we know 'E' stands for "enterprise", and as we all (sadly) know "enterprise" stands for processes, release policies, bureaucracy, ownerships, lifecycle, development vs. maintenance. A m2 structured approach will ease to me integration in highly complex environments (and with many enterprise applications) in which is not just a matter of "taking and xml file, dropping it in the shared folder, and restart the server" , because it's *simply* not possible
- Having an archetype based approach can really gear community development and lower Alfresco "real" learning curve, as it's pretty much easier to have a list of properties to edit more than go-search-crawl-find on the wiki which is the Spring snippet to be modified. This is true for the extension projects (already in the forge) as well as AMP (let's say module build).
- Alfresco grows faster and faster, to me it's the moment (especially with the 3.0 release) to provide a real neat basis for the even greater growth alfresco expects from the great improvements they are doing: Apache maven is basis for most open source projects, and even tough and complex frameworks (e.g. Apache Cocoon) have completed (not without hassles) the m10n process, towards a more integrated and documented open source vision. The bigger it grows the more difficult will be to do this change and to me it may really make it an outstanding example of "enterprise opensource"
- I don't have to maintain a public m2 repository almost just for Alfresco

Ok, now you may think I'm a maven evangelist trying to find useful applications of that technology at all costs

On the other hand I believe that (especially being a partner) we should not impose technologies, while adapt the way we work to the open source (typically high) standards of development, and this is a characteristic that sometimes I see missing in Alfresco extension development: in this sense the archetype we've released builds *also* with Apache Ant (see http://forge.alfresco.com/forum/forum.php?forum_id=512), exploiting the tidy and standard maven project hierarchy and sharing the same property filtering policies.
Of course all the nice m2 features (like jetty run, jboss and tomcat deploy, release management, etc.) are not supported by this build, but what you just need is to checkout the jar, unzip it, and run ant to have a properly packaged, environment aware alfresco WAR build.

This was more a POC for the feasibility of this double approach, so I'm not planning to maintain the ant part while focusing on the interesting growth perspectives of the m2 part, but it can be useful to provide the double point of access I was mentioning before (enterprise vs. community).

So wrapping up:
I would be (and so my company Sourcesense) to see Alfresco move (and to help/support in this transition) to maven for building its source code, document and release it. But I understand it's a complex and risky process (but risks can be lowered supporting a bottom line Ant based build which provides continuity with current developments).

So ATM I'd love to have some feedback on the possibility of building extension (and AMP in a while) using archetypes / prototype based approach (even before an Alfresco official move to maven) and hopefully some contributions to the project.  I think that it's vitally important to have a (still agile but) sort of a process when developing and releasing and application. 
That's especially true/valuable for a partner but I think that will especially gear the professional open source community interest in Alfresco.

Hope this helps,
Gab

PS:
For some additional references on the maven topic you may be interested in (sorry for the spamming :

Library conventions - http://forums.alfresco.com/viewtopic.php?f=10&t=1017&p=39172#p39172
Maven repositories - http://forums.alfresco.com/viewtopic.php?f=12&t=10210&p=39175#p39175
Running outside a container - http://forums.alfresco.com/viewtopic.php?f=4&t=9317&p=39176#p39176
Maven4alfresco - http://forums.alfresco.com/viewtopic.php?f=32&t=9268&p=39179#p39179

[mindthegab]

On step forward here;: Following yesterday's announcement of the official Alfresco Artifact Repository (http://blogs.alfresco.com/wp/its-official-alfresco-artifacts-repository/), wehave Alfresco Community and Enterprise artifacts are hosted at http://artifacts.alfresco.com .

Please check:

https://wiki.alfresco.com/wiki/Alfresco_Artifacts_Repository
http://mindthegab.com/2012/06/05/introducing-the-alfresco-artifacts-repository-yes-with-alfresco-ent...

On the building with Maven topic follow:

https://issues.alfresco.com/jira/browse/ALF-14390