About file encoding and freemarker (Cp1252 vs UTF-8)

[jmliege]

Hi,

I run into some troubles with the encoding when using email templates through the MailActionExecuter class even that everything (server, mailserver, … ) was set to UTF-8 !
My content also was written with UTF-8 encoding

After going down deeper and deeper the source, i get the following solution:

On creation of the freemarker.template.Configuration class, the system property "file.encoding" from the JVM is loaded as default encoding…

On JVM installation, sometimes, 'Cp1252' encoding is set automatically (for well-known but obscure reason…)  :?

To avoid this problem, add the following option to your JVM options:
-Dfile.encoding=UTF-8

Best regards,
JM

[kevinr]

Thanks for the tip, to me that looks like something that should be configurable as part of the FreeMarker setup configuration in Alfresco.

The FreeMarker engine doesn't know that it's processing templates for Emails, HTML, XML or whatever - so I don't think we should push the email encoding value down into the config. Iinstead I think it should be a separate config property that is part of the Alfresco FreeMarker Template processor bean config.

I have raised an issue for this:
http://issues.alfresco.com/browse/AR-1041

Thanks again,

Kevin

[unknown-user]

Hi,

I am using Alfresco 1.4.0 Enterprise and i run into the same encoding problem. I took a look at the template-services-context.xml and find a commented "defaultEncoding" parameter.

I tried to uncomment it, but it did not solve my problems. I also tried to to change the value of this parameter with no visible result.

Did I miss something?

PS : tried on Windows XP and Debian/Sarge

[rdanner]

I believe this also impacts webscripts when they call for content with remote.call / get / post etc

Say you have an Alfresco / Share setup and are returning XML containing UTF-8 from Alfresco via a webscript and a share webscript consuming that Alfresco repository service.

I've found on my mac that if I simply call the script with my browser it works fine (encoding handled properly etc.)

If I call the same service with a webscript, set the response to a freemarker template and in the template simply print the variable I see issues.
If I look at Java's default character set I see it is set to MacRoman.
What we determined with this is that while the Webscript connector seems to be assuming the encoding is the default encoding, the bytes are actually UTF-8.  This means that as far as java is concerned, any consumer down the line who asks for UTF-8 representation of this string needs an conversion. The bytes (which are actually already UTF-8 bytes) are mistakenly "converted."

There are two ways to prove this:
1.
    If I set the java default encoding on the JVM
    export JAVA_OPTS='BLAH BLAH -Dfile.encoding=UTF-8'
    I get the proper output on the client (share based) webscript

2. doing what would otherwise be non-sensical:
    (without the default encoding explicitly set with -D…)

     var callResponse = remote.call("/myXml-In-UTF-8");
     model.foo = new java.lang.String(callResponse.response.getBytes("macRoman"), "UTF-8");

     printing this in the FTL:
     {foo}

     gives you the proper response even though the code above is "crazy talk."

Hi,

I run into some troubles with the encoding when using email templates through the MailActionExecuter class even that everything (server, mailserver, … ) was set to UTF-8 !
My content also was written with UTF-8 encoding

After going down deeper and deeper the source, i get the following solution:

On creation of the freemarker.template.Configuration class, the system property "file.encoding" from the JVM is loaded as default encoding…

On JVM installation, sometimes, 'Cp1252' encoding is set automatically (for well-known but obscure reason…)  :?

To avoid this problem, add the following option to your JVM options:
-Dfile.encoding=UTF-8

Best regards,
JM