SSL and Indexing (SOLR?) Not Working

[vrothenberg]

Recently configured an SSL connector to encrypt access to my Alfresco Community 4.2.c installation.  I noticed that whenever I would first make a change to the connector, and restart the services, indexing would no longer work.  Searches would not bring anything up, and dashlets like My Documents / Site Content for displaying recent files would just display "Error loading items"

If anyone knows what exactly this problem is, and better yet how to solve it, I'd greatly appreciate it!

Thanks in advance,

Vince Rothenberg

[afaust]

Hello,

could you describe a bit more on how your Alfresco installation is set up (standard install with installer or separate Tomcat instances for Repository, Share and SOLR as is recommended) and what changes you made to the basic setup to enable your SSL transport encryption?

One possible cause I can think of are configuration conflicts caused by replacing the key- and truststores used by the Tomcat SSL connector with custom ones (which include your own server certificate) while not updating the key- and truststores used for holding client certificates of the internal Repository / SOLR communication. When the Tomcat connector can not verify the client certificates of Repositor / SOLR, indexing and searches will not work as authentication via client certificates is required for this part of Alfresco - even if it is just internal communication.

If this is the case, the problem should be easily rectifiable by issuing your own client certificates for Repository / SOLR and updating the client trust- / keystores with those certificates. (see <a href="http://wiki.alfresco.com/wiki/Alfresco_And_SOLR#Manually_Generating_New_SSL_Keys_Signed_by_a_Certifi...">Manually generating new SSL keys..</a> for information on the affected trust- / keystores - also see the two sections above it)

Regards
Axel

[vrothenberg]

Hello Axel,

Appreciate the response.  The server is running from the standard installer so most things should be set to default.  I have not configured multiple instances for the Repository / Share / SOLR.  In C:\Alfresco\tomcat\conf\server.xml I changed the default HTTPS connector from this:

 <Connector port="8443" protocol="org.apache.coyote.http11.Http11Protocol" SSLEnabled="true"
               maxThreads="150" scheme="https" keystoreFile="C:\Alfresco/alf_data/keystore/ssl.keystore" keystorePass="<pass>" keystoreType="JCEKS"
 secure="true" connectionTimeout="240000" truststoreFile="C:\Alfresco/alf_data/keystore/ssl.truststore" truststorePass="<pass>" truststoreType="JCEKS"
               clientAuth="false" sslProtocol="TLS" allowUnsafeLegacyRenegotiation="true" maxSavePostSize="-1" />  
‍‍‍‍‍‍

Then changed it to port 443, updating the keystoreFile with the keystore for my own certificates, and removing the truststoreFile:

<Connector port="443" protocol="org.apache.coyote.http11.Http11Protocol" SSLEnabled="true"
               maxThreads="150" scheme="https" keystoreFile="C:\Alfresco\java\bin\rapidssl.jks" keystorePass="<pass>" keystoreType="JKS" secure="true"
               clientAuth="false" sslProtocol="TLS" />
‍‍‍‍‍

In alfresco-global.properties solr.port.ssl was updated to 443, but that didn't seem to change anything yet.

I think you're right that the default ssl.keystore and ssl.truststore need to be configured to have indexing work properly.  However I'm a bit lost on how best to do this. 

Should my current rapidssl certificates have been generated with a .csr from the ssl.keystore? 

How would I currently reconcile the rapidssl.jks keystore with the ssl.keystore and ssl.truststore in my connector?

In the guide it uses openssl to have the Alfresco CA sign the .csr, should I already have openssl included with Alfresco?  It appears that I would have to download it.

For generating new SOLR SSL keystores it references the ssl.repo.client.keystore which exists in 4 folders in C:\Alfresco\alf_data\solr\ with 4 respective .truststores in those directories.  Which one is relevant?

If I were to use the default HTTPS connector with ssl.keystore and ssl.truststore, then generate my certificate request from there for an external CA, would that bypass these configurations?

Appreciate all the help so far! 

Best regards,

Vince Rothenberg

[afaust]

Hello,

ok, in this case, I think the following constellation would provide a working state without having to issue custom certificates for client authentification:

<ul>
<li>Update the ssl.truststore by importing the Rapid SSL Root CA certificate</li>
<li>Update the ssl-truststore-passwords.properties file by adding the approriate alias and password</li>

<li>Update the ssl.repo.client.truststore files (all of them are necessary) by importing the Rapid SSL Root CA certificate</li>
<li>Update the ssl-repo-client-truststore-passwords.properties files by adding the approriate alias and password</li>

<li>Modify your SSL connector to use the update ssl.truststore (as it did before)</lI>
</ul>

OpenSSL is actually not required for any of this, but is the tool-of-choice for many users / developers when it comes to create / transform keys and certificates. You should be able to do all of the required operations using just the Java Keytool. <a href="https://www.sslshopper.com/article-most-common-java-keytool-keystore-commands.html">This page</a> lists all the relevant commands. You need to add parameters to specify the password and type of the target truststores as they are not in JKS format - you can do this by adding "-storetype JCEKS" and "-storepass <pass>" to the command line. Whenever a keystore is passed, you should use the path of the truststore you are updating.

Regards
Axel

[vrothenberg]

Alright I updated all the truststores with the imported Root certificate and added the Root alias and rapidssl password in each of the respective truststore properties files.  Changed the connector in server.xml to the following:

<Connector port="443" protocol="org.apache.coyote.http11.Http11Protocol" SSLEnabled="true"
               maxThreads="150" scheme="https" keystoreFile="C:\Alfresco\java\bin\rapidssl.jks" keystorePass="<rapidsslpass>" keystoreType="JKS" secure="true" truststoreFile="C:\Alfresco/alf_data/keystore/ssl.truststore" truststorePass="<alfrescopass>" truststoreType="JCEKS"
               clientAuth="false" sslProtocol="TLS" />
‍‍‍‍‍

The previous errors are now gone and I can view recent uploads and comments in the My Activities / Site Activities dashlet.  However the My Documents / Site Content dashlets just show the default filler message "Keep track of content changes…" such as on a fresh installation with no content.  It appears there is still something missing.

Another strange issue I found is that searching for content that was in the system before I configured the SSL connector works fine.  However when adding new files they are not indexed and are not searchable.

In alfresco-global.properties SOLR indexing is set to:

### Solr indexing ###
index.subsystem.name=solr
dir.keystore=${dir.root}/keystore
solr.port.ssl=443

Not quite sure what settings I missed.  Your information worked, but there still seems to be a configuration problem. 

Anyway I appreciate all the great information you've already given.  If you know what could be causing the current indexing problem that would be excellent!

Best regards,

Vince Rothenberg

[afaust]

Hello,

ok, your search queries are passed and executed correctly, but your SOLR index tracking still fails to read and index any updates to the content repository. From personal experience, this means the modification to ssl.truststore, ssl-truststore-passwords.properties and the SSL connector have been correct, but there may still be something wrong with the ssl.repo.client.truststore / ssl-repo-client-truststore-passwords.properties (which are relevant primarily for index tracking).

Do you have any 401-errors in your logs (that seem to be related to SOLR) or anything pertaining to "certificate verification error", "PKIX" or similar?

Unfortunately, at some point it may be necessary to turn on SSL / authentication debugging to see what is going on and determine the problem. For example setting the

org.apache.catalina.authenticator.level = FINEST‍

in tomcat/conf/logging.properties should result in log output in Tomcat logs about SSL client certificate authentication between SOLR and Repository (SOLR => Repository is the interesting use case here). Setting

-Djavax.net.debug=all‍

in JAVA_OPTS of Tomcat will start to generate an insane amount of log output of the low-level SSL handshake / communication, which can be mined for clues (I once had similar problems with SOLR => Repository tracking and found that - due to some misconfiguration in JAVA_OPTS - the truststores of the SOLR client were not loaded correctly.

Just to make sure this is not related to the problem, you should add the maxSavePostSize and allowUnsafeLegacyRenegotiation attributes to the SSL connector as they were provided by Alfresco.

Regards
Axel

[vrothenberg]

Alright I changed the SSL connector with the added attributes, and imported the root certificate into the ssl.repo.client.keystores as well.  The aliases and passwords are in each passwords.properties file.  However that hasn't fixed the problem yet.

Searched through the localhost access, catalina, and alfrescotomcat-stderr logs and didn't find any certificate verification or pkix errors.  The only instances of 401 were on lines such as this:

63.87.61.77 - - [12/Jun/2013:16:07:14 -0700] "GET /share/feedservice/components/dashlets/activities/list?format=atomfeed&mode=user&site=&dateFilter=7&userFilter=all&activityFilter= HTTP/1.1" 401 -
‍‍‍

 

Tried doing a search for a file within Alfresco and the localhost access log lists the following:

127.0.0.1 - - [12/Jun/2013:16:03:15 -0700] "GET /alfresco/s/api/people/admin/preferences?alf_ticket=TICKET_1bc44e5ef623f64f8895d012724687b35b3a46d6 HTTP/1.1" 200 2154
127.0.0.1 - - [12/Jun/2013:16:03:15 -0700] "GET /alfresco/s/api/sites/filetransfer?alf_ticket=TICKET_1bc44e5ef623f64f8895d012724687b35b3a46d6 HTTP/1.1" 200 478
127.0.0.1 - - [12/Jun/2013:16:03:15 -0700] "GET /share/page/site/filetransfer/search?t=async.log HTTP/1.1" 200 36653
127.0.0.1 - CN=Alfresco Repository, OU=Unknown, O=Alfresco Software Ltd., L=Maidenhead, ST=UK, C=GB [12/Jun/2013:16:03:16 -0700] "POST /solr/alfresco/afts?wt=json&fl=DBID%2Cscore&rows=502&df=keywords&start=0&locale=en_US&fq=%7B%21afts%7DAUTHORITY_FILTER_FROM_JSON&fq=%7B%21afts%7DTENANT_FILTER_FROM_JSON HTTP/1.1" 200 103
127.0.0.1 - - [12/Jun/2013:16:03:16 -0700] "GET /alfresco/s/slingshot/search?site=filetransfer&term=async.log&tag=&maxResults=251&sort=&query=&repo=false&rootNode=alfresco%3A%2F%2Fcompany%2Fhome&alf_ticket=TICKET_1bc44e5ef623f64f8895d012724687b35b3a46d6 HTTP/1.1" 200 36
127.0.0.1 - - [12/Jun/2013:16:03:16 -0700] "GET /share/proxy/alfresco/slingshot/search?site=filetransfer&term=async.log&tag=&maxResults=251&sort=&query=&repo=false&rootNode=alfresco%3A%2F%2Fcompany%2Fhome HTTP/1.1" 200 25
127.0.0.1 - - [12/Jun/2013:16:03:16 -0700] "POST /alfresco/s/api/sites/query?alf_ticket=TICKET_1bc44e5ef623f64f8895d012724687b35b3a46d6 HTTP/1.1" 200 2380
127.0.0.1 - - [12/Jun/2013:16:03:16 -0700] "GET /share/service/modules/header/sites?htmlid=global_x002e_header_x0023_default-app_sites&favsites=%7B%22testtetestset%22%3Atrue%2C%22ats621%22%3Atrue%2C%22testsitesec%22%3Atrue%2C%22mail%22%3Atrue%2C%22swsdp%22%3Atrue%2C%22ats621bau%22%3Atrue%2C%22testsite%22%3Atrue%2C%22logs%22%3Atrue%2C%22filetransfer%22%3Atrue%7D&siteId=filetransfer HTTP/1.1" 200 2440
‍‍‍‍‍‍‍‍‍‍

Do you remember what you did to fix your problem?

[afaust]

Hello,

hmm.. I don't see any access by SOLR to the Repository here which should be there for the tracking. Of course, this is a very small sample. Even if there is a certificate issue, you should at least see failed calls to /alfresco/s/(api/)solr or something similar in the localhost access log.

Can you please check / provide more of the localhost access log? In a default configuration you should see SOLR access to the Repository every 15 seconds. Is your solr tracking enabled (solrcore.properties)?

Regards
Axel

P.S.: In my case of misconfiguration, there was a problem with the certificate authority's distinguished name in tomcat-users.xml - the order in which the attributes were reported to me by my certificate inspection client was reversed to what the OpenSSL component of Tomcat was verfying. But this should not affect you since you "only" have a custom server certificate, not a completely custom set of client certificates as I had created for the customer.

[vrothenberg]

Thanks for all your help Axel.

I decided to scrap using Alfresco's keystores to do external encryption and instead used Nginx as an encrypted reverse proxy server. 

For anyone in the same situation as I was, first get a new installation of Alfresco.  To export your current data from an old Alfresco database you must:

1. Stop the source server's Tomcat and PostgreSQL services.
2. In C:\Alfresco\alf_data copy the contentstore and postgresql folders.
3. Stop the target server's Tomcat and PostgreSQL services. 
4. Copy and overwrite the contentstore and postgresql folders with the previous installation.
5. On the target server reindex SOLR by deleting the files and subdirectories in the following folders:
Alfresco\alf_data\solr\archive\SpacesStore\
Alfresco\alf_data\solr\archive-SpacesStore\alfrescoModels\
Alfresco\alf_data\solr\workspace\SpacesStore\
Alfresco\alf_data\solr\workspace-SpacesStores\alfrescoModels\

To set up Nginx as an encrypted reverse proxy server follow this guide:

http://ezref.info/How+to+setup+Alfresco+with+Nginx+as+the+frontend+web+server.html

On Windows you must edit the nginx.conf file with the following server entries:

server {
      listen   80;
      server_name mydomain.net;
      server_name   *.mydomain.net;
      rewrite ^/$   https://mydomain.net/;

      location / {
      root   C:/alfresco/tomcat/webapps/share/;
      proxy_pass   http://localhost:8080;

      proxy_set_header   X-Real-IP   $remote_addr;
      proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header   host   $host;
      proxy_set_header   X-Forwarded-Server   $host;

      proxy_set_header        X-Forwarded-Proto $scheme;
      add_header              Front-End-Https   on;

      proxy_redirect     off;
      }

    }
   
      server {
      listen                  443;
      server_name             mydomain.net;
      server_name      *.mydomain.net;
      ssl                     on;
      ssl_certificate         C:/nginx/conf/mydomain_net-bundle.crt;
      ssl_certificate_key     C:/nginx/conf/myserver.key;
      rewrite ^/$      /share;


      location / {
      root   C:/alfresco/tomcat/webapps/share/;
      proxy_pass   https://localhost:8443;

      proxy_set_header   X-Real-IP   $remote_addr;
      proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header   host   $host;
      proxy_set_header   X-Forwarded-Server   $host;

      proxy_set_header        X-Forwarded-Proto $scheme;
      add_header              Front-End-Https   on;

      proxy_redirect     off;

      }
   }
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

After that you should be good to go!