I have setup a cluster system on my local where I have three nuxeo instances. Out of the three instances, one is the interactive node(or producer node), from where I am uploading video, and the other two nodes(consumer node) are dedicated for video conversions. I have setup kafka, clustering, shared repository for binaries, and stream work manager for all this.
What I want is when I upload a video on the producer node, and if 4 automatic conversions are setup, then 2 of the conversions must happen on consumer node 1 and the other 2 conversion on the other consumer node.
I have contributed variants of the extensions shown below for each of the nodes:
<extension target="org.nuxeo.ecm.platform.video.service.VideoService" point="videoConversions">
<videoConversion name="MP4 480p" converter="convertToMP4" height="480" enabled = "true" rendition="true"/>
<videoConversion name="MP4 360p" converter="convertToMP4" height="360" enabled = "true" rendition="true"/>
<videoConversion name="MP4 720p" converter="convertToMP4" height="720" enabled = "true" rendition="true"/>
<videoConversion name="MP4 1080p" converter="convertToMP4" height="1080" enabled = "true" rendition="true"/>
</extension>
<extension target="org.nuxeo.ecm.platform.video.service.VideoService" point="automaticVideoConversions">
<automaticVideoConversion name="MP4 480p" enabled = "true" order="0"/>
<automaticVideoConversion name="MP4 360p" enabled = "true" order="0"/>
<automaticVideoConversion name="MP4 720p" enabled = "true" order="0"/>
<automaticVideoConversion name="MP4 1080p" enabled = "true" order="0"/>
</extension>
The producer node has all of the automatic video conversions enabled, where as the consumer 1 has the first two enabled and the consumer 2 has the last two enabled.
Kafka Settings: The producer node has only producer configuration while the two consumer nodes have both producer and consumer kafka settings as shown below.
<extension target="org.nuxeo.runtime.stream.kafka.service" point="kafkaConfig">
<kafkaConfig name="default" zkServers="localhost:2181" topicPrefix="nuxeo-">
<!-- Both producer and consumer settings are required for consumer instance-->
<producer>
<property name="bootstrap.servers">localhost:9092</property>
</producer>
<consumer>
<property name="bootstrap.servers">localhost:9092</property>
<property name="request.timeout.ms">65000</property>
<property name="max.poll.interval.ms">3600000</property>
<property name="session.timeout.ms">20000</property>
<property name="heartbeat.interval.ms">1000</property>
<property name="max.poll.records">50</property>
</consumer>
</extension>
As per my testing, the conversion work was picked up by the consumer 1 but not consumer 2, and consumer two could not create the other two renditions which were not enabled. Is the conversion work, i.e. creating 4 renditions of a video, queued as a single work? How can I distribute these tasks among other worker nodes?
