Issues with using arken/service-amb

[boardthatpowder]

I’ve been using your scripts at https://github.com/nuxeo/nuxeo.io-scripts as a guideline for configuring a Docker ambassador in core-os. I can honestly say these have been the best source of coreos/docker that I have come across!

I’m currently using core-os v353 as part of the beta channel (have also tried v361 on alpha, same result). I have a setup as follows:

(service-consumer) - (service-ambassador) - (service-publisher)

I originally had the (service-publisher) on a separate node to the others, but to help with debugging and rule out any network issues I reduced it down to 1 coreos node.

My (service-publisher) docker contains a java restful service that grabs a port, then registers with etcdtcl.

In the systemd unit file for (service-publisher) I define:

Environment='SERVICE_NAME=authentication-query-service' 
Environment='INSTANCE_NAME=authentication-query-%i-service'
Environment='INSTANCE=%i'

In ExecStart:

/usr/bin/docker run --rm -P \
	--name ${INSTANCE_NAME} \
	${REGISTRY}/adc2/authentication-query

In ExecStartPost:

PORT=`docker port $INSTANCE_NAME 8080 | awk -F : '{print $2}'`
/opt/data/tools/register-service.sh $SERVICE_NAME $INSTANCE $PORT

After running this, I can successfully retrieve the host and port from etcdctl:

core@core-01 ~ $ etcdctl ls /services/authentication-query-service --recursive
/services/authentication-query-service/1
/services/authentication-query-service/1/location
core@core-01 ~ $ etcdctl get /services/authentication-query-service/1/location
{"host":"172.17.8.101","port":49158}

(service-ambassador) is an instance of arken/service-amb, and is configured as follows:

In the systemd unit file I define:

Environment='INSTANCE_NAME=authentication-query-amb-%i-service'

In ExecStart:

/usr/bin/docker run --rm --name $INSTANCE_NAME \
	-P \
	arken/service-amb \
	-servicePath /services/authentication-query-service

I then configure (service-consumer) as follows:

In the systemd unit file I define:

Environment='SERVICE_NAME=workflow-public-service' 
Environment='INSTANCE_NAME=workflow-public-%i-service'
Environment='INSTANCE=%i'

In ExecStart:

AUTHENTCIATION_QUERY_AMB="authentication-query-amb-${INSTANCE}-service"

/usr/bin/docker run --rm -P --name ${INSTANCE_NAME} \
  --link ${AUTHENTCIATION_QUERY_AMB}:auth_query \
  ${REGISTRY}/adc2/workflow-public:latest

After running this, I can successfully retrieve the host and port of this service from etcdctl:

core@core-01 ~ $ etcdctl ls /services/workflow-public-service --recursive
/services/workflow-public-service/1
/services/workflow-public-service/1/location
core@core-01 ~ $ etcdctl get /services/workflow-public-service/1/location
{"host":"172.17.8.101","port":49161}

In my (service-consumer) application, I set the host and port of (service-publisher) to instead use the ambassador environment variables that have been defined as part of the docker link to (service-ambassador):

AUTH_QUERY_PORT_1337_TCP_ADDR=10.1.0.18
AUTH_QUERY_PORT_1337_TCP_PORT=1337

Now from my understanding of reading through all the nuxeo documents and scripts (which have been an invaluable source of help on coreos and docker by the way!), I would expect that when (service-consumer) makes a rest call to (service-ambassador), it uses the ‘-servicePath’ attribute that is set as part of ExecStart of that service to look up the available service locations of (service-publisher) registered in etcdctl. But instead, the following is output in the (service-ambassador) logs:

core@core-01 ~ $ docker logs authentication-query-amb-1-service
E0629 23:24:46.375219 00001 tcpproxy.go:36] No host found for service
E0629 23:24:46.396998 00001 tcpproxy.go:36] No host found for service
E0629 23:24:46.417416 00001 tcpproxy.go:36] No host found for service
E0629 23:24:46.437557 00001 tcpproxy.go:36] No host found for service

When starting up (service-ambassador), I’ve tried the following variations, each with the same result:

-servicePath /services/authentication-query-service
-servicePath /services/authentication-query-service/
servicePath /services/authentication-query-service
servicePath /services/authentication-query-service/

Any ideas? Any help will be much appreciated.

[boardthatpowder]

I found the issue with this one. Adding the details here should anyone else have a similar problem...

In the source code of etcd-netfw I noticed that the etcd url had a hardcoded default value (see https://github.com/arkenio/etcd-netfw/blob/master/config.go). Therefore, modified the ExecStart command of (service-ambassador) to pass in the etcd url as follows:

#!/bin/sh
ETCD_ADDRESS=http://$(/bin/ifconfig docker0 | awk '/inet /{print $2}'):4001
/usr/bin/docker run --rm --name $INSTANCE_NAME \
	-P \
	arken/service-amb \
	-servicePath /services/authentication-query-service \
	-etcdAddress ${ETCD_ADDRESS}

Now all working as expected 🙂