Is it possible to distribute different renditions of a video to different nuxeo instances in a cluster setup?

Ritesh_Kumar — Fri, 31 May 2019 10:04:30 GMT

I have setup a cluster system on my local where I have three nuxeo instances. Out of the three instances, one is the interactive node(or producer node), from where I am uploading video, and the other two nodes(consumer node) are dedicated for video conversions. I have setup kafka, clustering, shared repository for binaries, and stream work manager for all this.

What I want is when I upload a video on the producer node, and if 4 automatic conversions are setup, then 2 of the conversions must happen on consumer node 1 and the other 2 conversion on the other consumer node.

I have contributed variants of the extensions shown below for each of the nodes:

<extension target="org.nuxeo.ecm.platform.video.service.VideoService"  point="videoConversions">      
<videoConversion name="MP4 480p" converter="convertToMP4" height="480" enabled = "true" rendition="true"/>  
<videoConversion name="MP4 360p" converter="convertToMP4" height="360" enabled = "true" rendition="true"/>    
<videoConversion name="MP4 720p" converter="convertToMP4" height="720" enabled = "true" rendition="true"/>    
<videoConversion name="MP4 1080p" converter="convertToMP4" height="1080" enabled = "true" rendition="true"/>    
</extension>

<extension target="org.nuxeo.ecm.platform.video.service.VideoService"         point="automaticVideoConversions">   
 <automaticVideoConversion name="MP4 480p" enabled = "true" order="0"/>  
  <automaticVideoConversion name="MP4 360p" enabled = "true" order="0"/> 
 <automaticVideoConversion name="MP4 720p" enabled = "true" order="0"/>  
<automaticVideoConversion name="MP4 1080p" enabled = "true" order="0"/>

</extension>

The producer node has all of the automatic video conversions enabled, where as the consumer 1 has the first two enabled and the consumer 2 has the last two enabled.

Kafka Settings: The producer node has only producer configuration while the two consumer nodes have both producer and consumer kafka settings as shown below.

<extension target="org.nuxeo.runtime.stream.kafka.service" point="kafkaConfig">    
<kafkaConfig name="default" zkServers="localhost:2181" topicPrefix="nuxeo-">        
<!-- Both producer and consumer settings are required for consumer instance-->        
<producer>            
	<property name="bootstrap.servers">localhost:9092</property>        
</producer>       
<consumer>            
<property name="bootstrap.servers">localhost:9092</property>            
<property name="request.timeout.ms">65000</property>           
 <property name="max.poll.interval.ms">3600000</property>            
<property name="session.timeout.ms">20000</property>            
<property name="heartbeat.interval.ms">1000</property>           
 <property name="max.poll.records">50</property>       
 </consumer>     
</extension>

As per my testing, the conversion work was picked up by the consumer 1 but not consumer 2, and consumer two could not create the other two renditions which were not enabled. Is the conversion work, i.e. creating 4 renditions of a video, queued as a single work? How can I distribute these tasks among other worker nodes?

topic Is it possible to distribute different renditions of a video to different nuxeo instances in a cluster setup? in Nuxeo Forum

Is it possible to distribute different renditions of a video to different nuxeo instances in a cluster setup?