topic Re: new rest2 webapp in Alfresco Archive

new rest2 webapp

tombaeyens — Fri, 29 Oct 2010 13:42:20 GMT

Huh, what is the rest2 webapp?we're trying out a new rest style. one that gives us better control. we're not that happy with the indirect steps of the template rendering. lot of things need to be right in order for it to work and we believe it could be done simpler.so we're now trying out a simpl

Re: new rest2 webapp

frederikherema1 — Fri, 29 Oct 2010 17:27:35 GMT

There are 2 REST approaches we consider instead of using ftl-templates:

1) Using the existing spring surf webscripts, but replace the freemarker-templates with rendering done by a JSONObject, based on the model created in the existing ActivitiWebScript implementation which use the API.

Pros:
• Leveraging the rest-posibilities built in into spring surf to map url’s to java code
• Since we populate JSONObjects based on activiti domain model, it’s more easy to see where they are used (a simple Cmd+G will show where data is used), instead of templates which contain only text-references to these properties.
• Using existing WebScript implementations with some tweaks here and there.
• JSONObject generates valid JSON and escapes accordingly.

Cons:
• Dependency on spring-surf just to use rest-functionality

2) Create a custom REST servlet which handles the rest calls by delegating to the appropriate handlers, using JSONObject to create valid JSON output and stream it back to client. Large part of existing Webscript java code can be reused which calls activiti API.

Pros:
• No dependency at all, just plain old servlet spec
• Same advantages mentioned in 1) thanks to JSONObject (escaping + using domain model)

Cons:
• Creating parsing/delegating mechanism from scratch, which can takes a couple of days of development
•
________________________________________________________________________________________

Another thing we want to tackle is splitting up the REST-api in the folowing parts:

1) UI-independent REST, containing calls that map right onto the activiti java API, eg: Get list of deployments.

2) A single rest api for each of the UI’s we have (explorer, cycle, probe,…), since some rest methods are created for those. Examples:

• Get tasks summary. Returns number of tasks for all groups the user belongs to, assigned to him/her and unnasigned. This call is absolutely needed ion explorer to prevent calling a bunch of seperate rest-methods that contain too much data anyway. This is a valid usecase from a task/user oriented point of view, but isn’t really generic engine api stuff.

• Cycle-related rest calls, which also use cycle-related services/data which isn’t nessesairy always deployed alongside the activiti-rest-webapp.

The webapps (explorer, probe, cycle) will expose extra REST-services (if needed) in their own war. The spring surf services used in the components will use 2 rest-endpoints: the core api-rest and their own custom REST-services.

Only downside of this is that if other apps want to use the specific REST-services exposed this way by UI’s is that the UI webapp should be deployed to make use of the included rest-services. Offcourse, we can create seperate war for the UI-secific REST (eg. activiti-wabapp-cycle-rest), but we want to avoid a million war’s being deployed

When the custom REST is included in the cycle/explorer/probe the 'dependency on spring-surf' stated in the 'cons' of the first solution proposed in 'replace freemarker templates' are a non-issue.

___________________________________________________________

We want this change to be implemented in 3-4 months so we can provide the same backward-compatibility/stability guarantees on core rest api, as we do on the engine API.

To facillitate this, unit-tests should be added to test the rest api to signal that any engine refactorings may have broken rest API behaviour.

Major changes to rest-api after it’s marked “stable” (now it’s still marked as experimental) will use a deprecation-strategy. Depending whether or not the (deprecated) old rest service calls still work in future versions, they can be removed after being marked as deprecated after a certain amount of time.

Quite some material to read through above, some good lecture for a rainy sunday afternoon Please share any remarks, thoughts, suggestions or other solutions!!

Re: new rest2 webapp

falko_menge — Sat, 30 Oct 2010 11:11:30 GMT

Maybe, Restlet is worth a try.

It's API is much more suitable for REST than the Servlet API and we already have the dependency in Activiti, as we are using it as a REST client in Cycle. In a past project, I made some good experiences with it's server capabilities and it could save us the effort of building an own parsing/delegating mechanism on top of the old Servlet API.

Re: new rest2 webapp

frederikherema1 — Sun, 31 Oct 2010 08:33:17 GMT

Falko,

Restlet is also indeed a valid option. Since it supports using the JAX-RS annotations it's also future-proof for JEE6.
Apache CXF also implements this standard, I'll have a look at that as well.

thanks for the feedback, keep it coming

Re: new rest2 webapp

nils1 — Mon, 01 Nov 2010 13:16:57 GMT

Hi guys,
since I've been working with spring-surf and freemarker templates for a while now, implementing the activiti-cycle webapp, I thought I'd quickly share my thoughts on this discussion:

1) Data transformation calls for a template engine
I agree that it would be nice to have IDE support for refactorings, i.e. warnings when you rename a class or remove an attribute. However, transforming data is cumbersome in java. Annotations can be a solution, but aren't always flexible enough. So you need to define some kind of mappings, whether these are XML files, freemarker templates like in spring-surf, or custom java code like i.e. in Restlet 2.0. Also, JSON isn't the only format we need to support, we also stream binary content or process plain HTML forms that are posted to the server. Which ever framework you choose, it should support these things in an easy way.

2) "Don't re-invent the wheel"
Spring-surf currently takes care of a whole bunch of things you didn't mention in this discussion so far. Apart from doing data transformation with freeemarker templates, it also handles parameters, dispatches the content-type of HTTP requests and chooses the content representation accordingly, it takes care of session handling, authentication and authorization and provides the basic HTML scaffold for the web applications, which are mainly written in JavaScript/ YUI. Re-implementing all this with the "plain old servlet spec" would mean to be writing a whole new framework.

3) "Innovation is seldom hindered by platform"
I agree that a thorough review of the existing REST API is a good idea and it is essential to make it stable and consistent. However, the existing infrastructure is not the reason this isn't the case yet. The REST API simply grew according to UI requirements. Now that we have a fairly stable java API and know more about what we need in the UIs, we can concentrate on making the existing REST API stable and consistent. Spring-surf certainly won't get in the way!

Cheers,
Nils

Re: new rest2 webapp

frederikherema1 — Mon, 01 Nov 2010 13:52:46 GMT

Nils, thanks for sharing your thoughts.

You are right, there is more that meets the eye in the spring-surf REST running right now (authentication, dispatching, parameter parsing, …). Can I conclude that you would suggest to keep the spring-surf setup and try to use JSONObjects instead of the *.ftl to have a bit more safety, leaving all the other stuff (like streaming bytes instead of JSON) as is?

A quick note about your thought nr.3, the issue about reorganizing rest API methods to separate parts is indeed not related to the technology we use. Maybe I wasn't so clear in first my comment, but they are two separate issues.

Re: new rest2 webapp

davidcognite — Mon, 01 Nov 2010 15:45:32 GMT

My thoughts on changing the technology the REST API is based upon:

1) Separation of logic and presentation
As Nils says, there needs to be some layer between the application logic and the view the user gets - the question should be: Is FTL the best templating/markup language for the job?

2) Target audience
Who are we expecting to be doing API modifications, customisations and extensions? Are they Java developers? UI developers? Will the majority of them come from an Alfresco background (with pre-existing knowledge of Spring Surf)?

3) If it ain't broke…
As I understand it, from previous discussions with Tom, one of the main issues is that when bits of the core change, it can be tricky getting that bubbled up to the REST layer and into the UI - however as we're reach a release candidate, are things not going to become more stable, removing a lot of the need to be able to refactor the code easily? Minor changes will become easier with time as developers on both side understand the other better. Having just had the code freeze for RC1, is this the time for a major re-architecture?

4) Consistency
There's something to be said for using the same framework as the other webapps and keeping things consistant.

David.

Re: new rest2 webapp

nils1 — Mon, 01 Nov 2010 17:30:22 GMT

@Frederik:

JSON objects might seem like a nice solution if you only want to transform your data to JSON. However, a REST API should be able to respond with different content-types, according to the request.

I don't know what the best solution is, but I don't think JSON objects are the solution. As David said, we'll need some layer between the application logic and the view. I can add that I found it easy to work with spring-surf and freemarker.

Maybe a separate REST module to evaluate whether an alternative technology yields better results isn't a bad idea. I would suggest to have a look at how Restlet 2.0 deals with data transformation.

Cheers,
Nils

Re: new rest2 webapp

frederikherema1 — Mon, 01 Nov 2010 17:55:00 GMT

Nils, indeed. Actually, I meant using a custom "renderer" on top of the existing WebScripts' returned model hashmap, using JSONObjects. This way, any other format is still possible, only the actual rendering of JSON is done in java-code instead of freemarker which gives us the desired "safety".

This week I'll digg a bit deeper into the alternatives like the JAX-RS api (using restlet).

Re: new rest2 webapp

bernd_ruecker — Tue, 02 Nov 2010 07:10:03 GMT

Just a very quick opinion on this from me.

I don't think it is a good idea to make a big change in that area now, where we already have a lot of stuff and somehow understood how the current technology works. We already have to provide own stuff around myBatis, what e.g. Hibernate could do out of the box. I don't want to do the same in the are of REST. And I think it is still a good idea, that the web apps use the same technology then REST, less to know to understand the code. I agree with David, that changes in that API gets less frequent every day. IT's a bit annoying, yes. But it's the same on the DB level, where I have to change one attribute in several mapping SQL's and SQL scripts. But these 5 minutes here and there aren't worth the effort we need for such a refactoring I think

About splitting the REST-webapps: I don't see a big advantage here (unless we really want to aim for seperate installations of Cycle and Engine, which was blocked when I brought it up for database. Why for REST then?). And I see more Maven projects and especially much more complicated class loading. Currently just putting some stuff in the Tomcat lib directory already is a bit hacky in terms of classloading (and causes some trouble as for Cycle we would have to add more jars there, which we avoid by putting cycle in the REST/WEB-INF/Lib folder, where it truly belongs). The more web-apps, the more complicated class loading.

Cheers
Bernd

Re: new rest2 webapp

falko_menge — Tue, 02 Nov 2010 10:01:54 GMT

I'd like to add, that I clearly see value in having a generic REST API for remote interactions with the engine. From what I heard, the current REST API seems to be a bit too tailored to the three web applications shipped with Activiti.

Can we collect a list of changes that need to be made in terms of URL patterns and representations in order to make the REST API more generic?

Based on this set of changes, we could decide on one the following options:
1. Do the changes in the existing REST module, which would in turn require to adapt all our webapps. However, we are already quite busy with implementing the last features for the GA release in the webapps. Hence, there is a high risk of breaking existing features and not completing planned features.
2. If there are too many changes, a second REST module may avoid those risks. The existing REST module could then be marked as deprecated. But, I don't think that we will be able to switch all webapps to use a different REST API until the GA release, as there is an even higher risk of breaking existing features and not completing planned features.

Re: new rest2 webapp

nils1 — Tue, 02 Nov 2010 10:30:21 GMT

As I understood, the work on the new REST API (2) is planned for the next three to four months. As long as it doesn't break the existing web apps, I don't see a reason not to do it. I won't really be able to dedicate time to it myself, but I'm interested in the outcome. Once the API is there, we'll be able to judge whether we can easily switch the webapps to it.

We should just try to keep effort and actual benefits in balance. If it takes too much time to switch the webapps to the new API, the time might be better invested in making the existing API stable and consistent. But as long as you take the existing one as a starting point and make sure the new API provides everything the webapps need, the transition shouldn't be too difficult.

Cheers,
Nils

Re: new rest2 webapp

frederikherema1 — Tue, 02 Nov 2010 14:28:16 GMT

@Bernd:
I think i'll ditch the "roll our own" solution based on the feedback received
About the extra modules. The custom cycle rest services would be exposed in the activiti-webapp-cycle itself, so no extra jars/wars are needed. Since the UI (cycle, explorer, probe) will evolve much more than the engine API, it would be a bit harder to mark the whole REST (including custom UI stuff) as stable and guarentee the same level of stability and backwards-compatibility. A split will address this issue.

@Nils, Falko:
It's indeed a bit risky to do a whole architecture change right before a GA release. Anyway, first thing I will try is to create unit/integration tests for the existing REST-api, testing within webapp container to make it more stable.

Re: new rest2 webapp

tombaeyens — Tue, 02 Nov 2010 20:52:48 GMT

@Bernd: the motivation to split up the rest calls between UI specific and generic rest calls lies in our experience on how we create the rest calls now. Most of the view related rest requests are just based on how the needs for the current UI. Those calls group a number of data fetching api calls and build a data structure that contains all the data to render a particular screen. Those type of UI related calls are not generic, but UI specific. On the other hand, most of the action rest requests map 1-1 to api service calls. Those requests are more generic and can be reused by multiple applications. That's the motivation to split up the generic, API based rest requests from the specific UI based rest requests. We think it's best to include the UI specific rest requests as part of the specific .war

Re: new rest2 webapp

frederikherema1 — Fri, 05 Nov 2010 15:48:31 GMT

Hi,

We're sticking with the Spring-surf Webscript, using JSONObject instead of the *.ftl's for rendering JSON responses.

I've implemented a solution for this (which I'll checkin in a few minutes on trunk). It's already plugged-in into the webscript-context, but as long as the *.ftl template is available, the FreemarkerTemplateProcessor will be used. Once we are confident about the solution, the ftl's are removed and JSONObject stuff kicks in.

Curently, all rest-calls (identity, engine, process, repository, task) have new implementation on trunk, except for the api.cycle.* calls.

A quick overview of the implementation:

* Plugged in a TemplateProcessor into the webscript context, which is registered automatically when context starts/reloads:

<bean id="jsonObjectTemplateProcessor" class="org.activiti.rest.builder.JSONTemplateProcessor">
      <property name="templateProcessorRegistry" ref="webscripts.web.templateregistry" />
      <property name="defaultEncoding" value="UTF-8" />
      <property name="templateNamePrefix" value="org/activiti/rest/api/" />
      <property name="indentFactor" value="2" />
  </bean>

* For each webscript we want to have rendering for using JSONObject, we add a bean in spring-context, eg.

<bean id="org.activiti.rest.api.repository.deployments.get.json.builder" 
        class="org.activiti.rest.builder.repository.DeploymentsObjectBuilder">
    <property name="templateName" value="repository/deployments.get.json" />
  </bean>

The bean implements interface org.activiti.rest.builder.JSONObjectBuilder, which is picked up automatically by our TemplateProcessor:

public interface JSONObjectBuilder {

  String getTemplateName();
  JSONObject createJsonObject(Object model) throws JSONException;
  
}

* When for example webscript "org/activiti/rest/api/repository/deployments.get" is called, in format JSON, out TemplateProcessor will receive a request to render template 'org/activiti/rest/api/repository/deployments.get'. Since we have a JSONObjectBuilder regsitered with templateName "repository/deployments.get.json" (also note the 'templateNamePrefix' is used when matchin templates) , this bean will be called to create a JSONObject, which is written to the response by the TemplateProcessor.

private DeploymentJSONConverter converter = new DeploymentJSONConverter();

  @Override
  @SuppressWarnings("unchecked")
  public JSONObject createJsonObject(Object modelObject) throws JSONException {
    JSONObject result = new JSONObject();
    Map<String, Object> model = getModelAsMap(modelObject);
    List<Deployment> deployments = (List<Deployment>) model.get("deployments");

    JSONUtil.putPagingInfo(result, model);
    JSONArray deploymentsArray = JSONUtil.putNewArray(result, "data");

    if (deployments != null) {
      for (Deployment deployment : deployments) {
        deploymentsArray.put(converter.getJSONObject(deployment));
      }
    }
    return result;
  }

public class DeploymentJSONConverter implements JSONConverter<Deployment> {


  public JSONObject getJSONObject(Deployment deployment) throws JSONException {
    JSONObject json = new JSONObject();
    JSONUtil.putRetainNull(json, "id", deployment.getId());
    JSONUtil.putRetainNull(json, "name", deployment.getName());
    
    String deploymentTime = JSONUtil.formatISO8601Date(deployment.getDeploymentTime());
    JSONUtil.putRetainNull(json, "deploymentTime", deploymentTime);
    
    return json;
  }

…

The result model created by the existing DeploymentsGet is used to create a JSONObject. THe DeploymentJSONConverter is reusable, this calls our model's getters to get the data into JSONObject. Adding paging info can be done in a generic way.

* Only the rendering has been altered, the web script implementations haven't changed a single letter, so all the UI-guys' hard work hasn't gone to waste Other representations can still be plugged in using the normal webscript-way offcourse.
* JSONUtil has some convenience methods to add values/arrays and format dates using the spring-surf iso8601DateFormatter.

_____________________________

Currently, Exporer and Probe run fine with the new implementations. Although, we should have the REST covered by unit/integration tests. Looks like we have 2 options:

* Use embedded tomcat which enables us to sue the same ProcessEngine, so we can set up some test-data using the API and pull it out using Restlet (which provide one-liners to do a single rest-call)
* Use WebScriptTestServer which is present in /scr/test/java of spring-surf webscripts. I can't seem to find out how to get it running, but I hope Eric can give me a few pointers on this.

WDYT?

Re: new rest2 webapp

erikwinlof — Tue, 09 Nov 2010 10:12:45 GMT

Hi, here comes a bit of pro-freemarker feedback, since I i'm not 100% about the new solution 🙂

Please note that this response is only about response rendering, it has nothing to do with having a core api and additional ui rest apis.

As I've understood the complaints against templates are basically 2:

#1. An engine api change might break the rest api.
This is true and a good argument. It can be solved by the junit tests. Joram showed me some Alfresco tests that did it and seemed satisified with how it was done. Alfresco is using this for their rest api having thousands of lines of freemarkerjson responses and it works for them.

#2. Its hard for the engine team to maintain since you don't know freemarker.
Well I hope I'm not being hard on you know, but its like saying I don't want to use Activiti for processes since I already know BPM4.
And since we are planning to use freemarker in the forms, you will need to learn it anyway 🙂
Anyway Freemarker is extremely easy to learn and use, and in the bottom is a crash course containing all you need for Activiti development.

# Why I think programmatic rendering is a bad idea:

A. The overview of how the JSON response looks is totally lost.

B. The amount of code lines needed are much greater, just consider the condition examples below to be written in java.

C. The benefit of getting a toJSON method in all pojos and use them in the rest api is not realistic since the pojo model will never match the services methods in 100%, some examples:
- Task (resourceName isn't included anymore therefore needing a whole new class RestTask on some objects but not all, confusing an creates even more code to maintain)
- RestProcessDefinition (same story as line above)

D. More examples where the POJO doesn't match a response:
- In the /process-engine response ProcessEngineInfo doesn't cover the complete response and additional "version" attribute is required)

E. "modern" frameworks like rails, django use templates making it easier to use.

F. Freemarker is a robust framework with very good documentation: http://freemarker.sourceforge.net/docs/index.html

G. We don't use the webscripts framework in a way its not intended.

(H). Most of all alfrescans and alfresco users as well for that matter know how to use json in freemarker and the webscripts api making it easy for them to read and understand the code.

############################
# FREEMARKER CRASH COURSE: #
############################

# Examples
Hello ${user.firstName}!          // Prints firstName if it exists, if it doesn't it will crash.
Hello ${user.firstName!"Guest"} // Prints firstName if it exists, if it doesn't exist the defatul value 'Guest' will be printed
A number ${count}                // Will print and format ok as long as it doesn't exceed the number 1000, then it will be printed as "1 000"
A number ${count?c}              // will be printed as expeceted (i.e. "1000") for all values
A date ${iso8601Date(today)}     // This is how we print dates in the activiti rest so its formatted as iso8601 dates

To use macros string escaping and library files:

activiti.lib.ftl ( a file with a macro inside)
——————————————————

<#escape x as jsonUtils.encodeJSONString(x)>

<#macro printName name>
Name: ${name}
</#macro>

</#escape>

… can be used and imported as a lib in another file as:
————————————————————
<#import "../activiti.lib.ftl" as restLib>
<@restLib.printName "Erik Winlöf"/>

All strings will be json-escaped but the int & dates will need special formatted as explained above.

Conditions example that checks if the name is defined or not:
"name": <#if processDefinition.name??>"${processDefinition.name}"<#else>null</#if>,

List example that uses the "freemarked built in" of "_has_next" to decide if a comma sign shall be printed:
[
<#list processDefinitions as processDefinition>
{
    "id": "${processDefinition.id}",
    "key": "${processDefinition.key}",
    "name": <#if processDefinition.name??>"${processDefinition.name}"<#else>null</#if>
}<#if processDefinition_has_next>,</#if>
</#list>
]

Re: new rest2 webapp

erikwinlof — Tue, 09 Nov 2010 10:51:51 GMT

I did a quick count of the amount of code needed to support and write in the 2 implementations (and note that the numbers first numbers below include the whole cycle rest api implementation for the freemarker version but not in the java version).

# Number of lines of code in freemarker version
%> find resources/webscripts/org/activiti/rest/api/ -type f -name "*ftl" -exec wc -l {} \; | awk '{total += $1} END{print total}'
411 (actually 278 without the cycle implementations 133 rows)

# Number of lines of code in java version
%> find java/org/activiti/rest/builder/ -type f -name "*java" -exec wc -l {} \; | awk '{total += $1} END{print total}'
1475

# Number of characters of code in freemarker version
%> find resources/webscripts/org/activiti/rest/api/ -type f -name "*ftl" -exec wc -c {} \; | awk '{total += $1} END{print total}'
12613 (actually 7998 minus the cycle implementations 4615 characters)

# Number of lines of code in java version
%> find java/org/activiti/rest/builder/ -type f -name "*java" -exec wc -c {} \; | awk '{total += $1} END{print total}'
49819

So as long as I have typed these commands properly, apologies otherwise :-), it meant that when excluding the cycle rest api the java implementation requires:
5.3 times more lines
and
6.2 times more characters
than the freemarker implementation… at its current state!

Re: new rest2 webapp

davidcognite — Tue, 09 Nov 2010 11:32:35 GMT

Once we are confident about the solution, the ftl's are removed and JSONObject stuff kicks in.
[…]
WDYT?

I'm still not confident that this is the best approach. I think the above demonstrates how hard the new REST API would be for non Java developers to follow.

The templating layer (in addition to all the other benefits) gives visibility over the expected results,
e.g.: The FTL files generally contain something along the lines of this:

{
  "id": "${processDefinition.id}",
  "key": "${processDefinition.key}",
  "name": <#if processDefinition.name??>"${processDefinition.name}"<#else>null</#if>,
  "version": ${processDefinition.version},
  "deploymentId": "${processDefinition.deploymentId}",
  "resourceName": "${processDefinition.resourceName}",
  "startFormResourceKey": <#if processDefinition.startFormResourceKey??>"${processDefinition.startFormResourceKey}"<#else>null</#if>
}

Having the response spelled out like that gives developers an easily accessible reference as to what properties they can expect, where they come from and which ones might be null. If any properties were formatted (e.g. dates or numbers) it would show what format should be expected.

How can front end Activiti developers or developers of third party applications (who may not be working in Java and may not have a Java IDE) determine the same information with your JSONObjectBuilder approach?

Re: new rest2 webapp

erikwinlof — Tue, 09 Nov 2010 15:25:16 GMT

Any thoughts from the rest? ( sorry about that 🙂 )
Personally I think we shall stick with the freemarker, since I can't see the "win" of using the JSONObject approach (except for getting the possible refactor errors when running compile rather than running test).

Cheers,

:: Erik

Re: new rest2 webapp

jbarrez — Tue, 09 Nov 2010 17:09:49 GMT

I wasnt able to follow the discussion every line, but here's my opinion on some of the points:

B. The amount of code lines needed are much greater, just consider the condition examples below to be written in java.

* LOC's don't mean anything. With auto-completion, auto-import, etc that every IDE has, I dont think you claim that LOC = productivity. We're not the eighties anymore.

C. The benefit of getting a toJSON method in all pojos and use them in the rest api is not realistic since the pojo model will never match the services methods in 100%, some examples:
- Task (resourceName isn't included anymore therefore needing a whole new class RestTask on some objects but not all, confusing an creates even more code to maintain)
- RestProcessDefinition (same story as line above)

These were just hacks introduced as a last-minute fix in the last release. There are jira issues to replace these with proper handling.

E. "modern" frameworks like rails, django use templates making it easier to use.

* 'modern' frameworks such as (G)Rails *do not* have templating as their primary rendering method as you state.
They use a much more powerful concept called builders, which comes closer to the Java approach: http://www.grails.org/doc/latest/guide/6.%20The%20Web%20Layer.html#6.1.7%20XML%20and%20JSON%20Responses.
In fact, in Rails you could use templating for json. But the more accepted practice is to call .to_json() on whatever you want to expose.

F. Freemarker is a robust framework with very good documentation: http://freemarker.sourceforge.net/docs/index.html

So is Java.

G. We don't use the webscripts framework in a way its not intended.

Why not? Who decides on that? For me, it is a natural addition to webscripts.