Nuxeo 10.10-HF29 with Strimzi 0.18.0 Kafka 2.5 in OpenShift Code Ready Containers

[Eric_Ace]

Hello - I'm attempting to connect Nuxeo 10.10-HF29 to Strimzi 0.18.0 Kafka 2.5 in OpenShift Code Ready Containers.

Based on this: https://doc.nuxeo.com/nxdoc/nuxeo-server-release-notes/#sasl-and-tls-authentication-against-kafka, I believe I have configured nuxeo.conf correctly for SASL/PLAIN.

$ nuxeoctl showconf
Nuxeo home:          /opt/nuxeo/server
Nuxeo configuration: /etc/nuxeo/nuxeo.conf
Include template: /opt/nuxeo/server/templates/common-base
Include template: /opt/nuxeo/server/templates/common
Include template: /opt/nuxeo/server/templates/default
Include template: /opt/nuxeo/server/templates/docker
***** Nuxeo instance configuration *****
NUXEO_CONF: /etc/nuxeo/nuxeo.conf
NUXEO_HOME: /opt/nuxeo/server
** Distribution
- name: server
- server: tomcat
- version: 10.10
- date: 201901211253
- packaging: docker
** Packages:
- nuxeo-dam (version: 6.4.3 - id: nuxeo-dam-6.4.3 - state: downloaded)
- nuxeo-drive (version: 1.8.5 - id: nuxeo-drive-1.8.5 - state: downloaded)
- nuxeo-liveconnect (version: 1.3.3 - id: nuxeo-liveconnect-1.3.3 - state: downloaded)
- nuxeo-showcase-content (version: 1.3.3 - id: nuxeo-showcase-content-1.3.3 - state: downloaded)
- nuxeo-spreadsheet (version: 1.4.3 - id: nuxeo-spreadsheet-1.4.3 - state: downloaded)
- nuxeo-template-rendering (version: 6.8.3 - id: nuxeo-template-rendering-6.8.3 - state: downloaded)
- nuxeo-vision (version: 1.3.4 - id: nuxeo-vision-1.3.4 - state: downloaded)
- nuxeo-web-ui (version: 2.4.0 - id: nuxeo-web-ui-2.4.0 - state: started)
- nuxeo-10.10-HF01 (version: 1.0.0 - id: nuxeo-10.10-HF01-1.0.0 - state: started)
- nuxeo-10.10-HF02 (version: 1.0.0 - id: nuxeo-10.10-HF02-1.0.0 - state: started)
- nuxeo-10.10-HF03 (version: 1.0.0 - id: nuxeo-10.10-HF03-1.0.0 - state: started)
- nuxeo-10.10-HF04 (version: 1.0.1 - id: nuxeo-10.10-HF04-1.0.1 - state: started)
- nuxeo-10.10-HF05 (version: 1.0.0 - id: nuxeo-10.10-HF05-1.0.0 - state: started)
- nuxeo-10.10-HF06 (version: 1.0.0 - id: nuxeo-10.10-HF06-1.0.0 - state: started)
- nuxeo-10.10-HF07 (version: 1.0.0 - id: nuxeo-10.10-HF07-1.0.0 - state: started)
- nuxeo-10.10-HF08 (version: 1.0.0 - id: nuxeo-10.10-HF08-1.0.0 - state: started)
- nuxeo-10.10-HF09 (version: 1.0.0 - id: nuxeo-10.10-HF09-1.0.0 - state: started)
- nuxeo-10.10-HF10 (version: 1.0.0 - id: nuxeo-10.10-HF10-1.0.0 - state: started)
- nuxeo-10.10-HF11 (version: 1.0.0 - id: nuxeo-10.10-HF11-1.0.0 - state: started)
- nuxeo-10.10-HF12 (version: 1.0.0 - id: nuxeo-10.10-HF12-1.0.0 - state: started)
- nuxeo-10.10-HF13 (version: 1.0.0 - id: nuxeo-10.10-HF13-1.0.0 - state: started)
- nuxeo-10.10-HF14 (version: 1.0.0 - id: nuxeo-10.10-HF14-1.0.0 - state: started)
- nuxeo-10.10-HF15 (version: 1.0.0 - id: nuxeo-10.10-HF15-1.0.0 - state: started)
- nuxeo-10.10-HF16 (version: 1.0.0 - id: nuxeo-10.10-HF16-1.0.0 - state: started)
- nuxeo-10.10-HF17 (version: 1.0.0 - id: nuxeo-10.10-HF17-1.0.0 - state: started)
- nuxeo-10.10-HF18 (version: 1.0.0 - id: nuxeo-10.10-HF18-1.0.0 - state: started)
- nuxeo-10.10-HF19 (version: 1.0.0 - id: nuxeo-10.10-HF19-1.0.0 - state: started)
- nuxeo-10.10-HF20 (version: 1.0.1 - id: nuxeo-10.10-HF20-1.0.1 - state: started)
- nuxeo-10.10-HF21 (version: 1.0.0 - id: nuxeo-10.10-HF21-1.0.0 - state: started)
- nuxeo-10.10-HF22 (version: 1.0.0 - id: nuxeo-10.10-HF22-1.0.0 - state: started)
- nuxeo-10.10-HF23 (version: 1.0.0 - id: nuxeo-10.10-HF23-1.0.0 - state: started)
- nuxeo-10.10-HF24 (version: 1.0.0 - id: nuxeo-10.10-HF24-1.0.0 - state: started)
- nuxeo-10.10-HF25 (version: 1.0.0 - id: nuxeo-10.10-HF25-1.0.0 - state: started)
- nuxeo-10.10-HF26 (version: 1.0.0 - id: nuxeo-10.10-HF26-1.0.0 - state: started)
- nuxeo-10.10-HF27 (version: 1.0.0 - id: nuxeo-10.10-HF27-1.0.0 - state: started)
- nuxeo-10.10-HF28 (version: 1.0.0 - id: nuxeo-10.10-HF28-1.0.0 - state: started)
- nuxeo-10.10-HF29 (version: 1.0.0 - id: nuxeo-10.10-HF29-1.0.0 - state: started)
** Profiles:
Profile: 
** Templates:
Database template: default
Base template: docker
Base template: docker
** Settings from nuxeo.conf:
JAVA_OPTS= -Dfile.encoding=UTF-8 -Dmail.mime.decodeparameters=true -Djava.util.Arrays.useLegacyMergeSort=true -Xloggc:"/var/log/nuxeo/gc.log" -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.util.logging.config.file=/opt/nuxeo/server/conf/logging.properties -Djava.util.logging.config.file=/opt/nuxeo/server/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
audit.elasticsearch.enabled=true
elasticsearch.httpEnabled=true
kafka.bootstrap.servers=zzzzz-kafka-bootstrap.backing.svc.cluster.local:9092
kafka.enabled=true
kafka.sasl.enabled=true
kafka.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="zzzzzuser" password="S2WeOwff2JvP";
kafka.sasl.mechanism=SCRAM-SHA-512
kafka.security.protocol=SASL_PLAINTEXT
nuxeo.data.dir=/var/lib/nuxeo/data
nuxeo.db.host=localhost
nuxeo.db.name=nuxeo
nuxeo.db.password=********
nuxeo.db.user=nuxeo
nuxeo.force.generation=true
nuxeo.log.dir=/var/log/nuxeo
nuxeo.pid.dir=/var/run/nuxeo
nuxeo.redis.enabled=false
nuxeo.templates=default,docker,docker
nuxeo.wizard.done=true
server.status.key=********
****************************************

And:

$ grep kafka /etc/nuxeo/nuxeo.conf
kafka.enabled=true
kafka.bootstrap.servers=zzzzz-kafka-bootstrap.backing.svc.cluster.local:9092
kafka.sasl.enabled=true
kafka.security.protocol=SASL_PLAINTEXT
kafka.sasl.mechanism=SCRAM-SHA-512
kafka.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="zzzzzuser" password="S2WeOwff2JvP";

The only difference is that the docs mention SCRAM-SHA-256 but Strimzi doesn't support that, so SCRAM-SHA-512. Anyway after nuxeoctl start, then:

Server started with process ID 14349.
......................................................................................................................................................
Starting process is taking too long - giving up.

# and...
= Component Loading Status: Pending: 0 / Missing: 0 / Unstarted: 2 / Total: 506
  - service:org.nuxeo.runtime.stream.service
  - service:org.nuxeo.ecm.core.work.service

and then:

$ cat /opt/nuxeo/server/nxserver/config/kafka-config.xml
<?xml version="1.0"?>
<component name="org.nuxeo.kafka.defaultConfig">
  <require>org.nuxeo.runtime.stream.kafka.service</require>
  <extension point="kafkaConfig" target="org.nuxeo.runtime.stream.kafka.service">
    <kafkaConfig name="default" topicPrefix="nuxeo-">
      <producer>
        <property name="bootstrap.servers">zzzzz-kafka-bootstrap.backing.svc.cluster.local:9092</property>
        <property name="default.replication.factor">1</property>
        <property name="delivery.timeout.ms">120000</property>
        <property name="acks">1</property>
      </producer>
      <consumer>
        <property name="bootstrap.servers">zzzzz-kafka-bootstrap.backing.svc.cluster.local:9092</property>
        <property name="request.timeout.ms">30000</property>
        <property name="max.poll.interval.ms">7200000</property>
        <property name="session.timeout.ms">50000</property>
        <property name="heartbeat.interval.ms">4000</property>
        <property name="max.poll.records">2</property>
        <property name="default.api.timeout.ms">60000</property>
      </consumer>
    </kafkaConfig>
    <kafkaConfig name="bulk" topicPrefix="nuxeo-bulk-">
      <producer>
        <property name="bootstrap.servers">zzzzz-kafka-bootstrap.backing.svc.cluster.local:9092</property>
        <property name="default.replication.factor">1</property>
        <property name="delivery.timeout.ms">120000</property>
        <property name="acks">1</property>
      </producer>
      <consumer>
        <property name="bootstrap.servers">zzzzz-kafka-bootstrap.backing.svc.cluster.local:9092</property>
        <property name="request.timeout.ms">30000</property>
        <property name="max.poll.interval.ms">7200000</property>
        <property name="session.timeout.ms">50000</property>
        <property name="heartbeat.interval.ms">4000</property>
        <property name="max.poll.records">2</property>
        <property name="default.api.timeout.ms">60000</property>
      </consumer>
    </kafkaConfig>
  </extension>
</component>

Notice that the bootstrap.servers property is incorporated into kafka-config.xml but none of the sasl properties are included. Originally, I had been attempting this with 10.10 and no hotfixes. Under that configuration, all the the sasl properties were correctly making it through into the kafka-config.xml but the Kafka connection was failing. I surmised that hotfixes may be needed so I added them. Now the sasl properties dropped out so there's no possibility of a successful SASL negotiation.

I should add that I successfully connected Nuxeo 10.10 (no HF) to Strimzi Kafka in anonymous mode. So I know that the DNS, routing, etc. are all fine. And in fact the Kafka logs are active when Nuxeo initiates the connection, but all the Kafka entries are errors.

Any insight would be greatly appreciated. Thanks

Update: Just successfully tested with SSL

kafka.enabled=true
kafka.ssl=true
kafka.bootstrap.servers=strimzi-kafka-bootstrap:9093
kafka.truststore.type=PKCS12
kafka.truststore.path=/etc/nuxeo/strimzi/truststore.p12
kafka.keystore.type=PKCS12
kafka.keystore.path=/etc/nuxeo/strimzi/keystore.p12
kafka.keystore.password=ceXI3Ryv0fw0
kafka.truststore.password=uhVqjZ5tFlxz

As a follow-up, looking more carefully at: kafka-config.xml.nxftl:

...
<#if "${kafka.ssl}" == "true">
<#if "${kafka.sasl.enabled}" == "true">
        <property name="security.protocol">${kafka.security.protocol}</property>
        <property name="sasl.mechanism">${kafka.sasl.mechanism}</property>
        <property name="sasl.jaas.config">${kafka.sasl.jaas.config}</property>
<#else>
        <property name="security.protocol">SSL</property>
</#if>
...

It only supports SASL over SSL. Nuxeo folks -- If that's intentional, then the release notes should be changed, which presently read like SASL plain is supported. And even the ticket (NXP-25956) makes it seems like SASL plain is supported...