Setup a Docker Overlay Network on Multiple Hosts

We have seen many use cases where one fires up a few Docker containers on one single host. To accommodate the growth of data or complexity in business, we would need to consider running the containerized tasks on multiple physical hosts. One of the challenges was how to maintain the communications among the distributed tasks as if they were on the same host.

Fortunately Docker provides a mechanism called Overlay Networking, which basically creates an VXLAN layer 2 overlay tunnel on top of layer 3, i.e., TCP/IP. The details won’t be discussed here but interested readers can go here for more information. It is not hard to imagine that this would allow two containers, sitting on different hosts, to talk with each other. Cool!

This blog will walk through a simple example to create a Docker Swarm that spans two physical hosts, and we will create a Overlay Network to stitch together the distributed containers.

Since the version 1.12.0, Docker Engine natively includes Swarm mode, which makes bringing up a Swarm cluster much easier than using the previous standalone Swarm. Say now there are two nodes, node 1 and node 2. We decide to elect the node 1 to be the manager node. Note that one can have more than one manager nodes in a Swarm cluster but for the sake of simplicity, we just use node 1.

Initiate the Swarm

The following command on node 1 will initiate the Swarm and elect that node as the manager. This command will also spit out the command you would use on node 2 to join the cluster.

docker swarm init

Copy and paste the output from the above, run on node 2.

docker swarm join -- token ...

Come back on node 1 to verify 2 nodes are present in the cluster.

docker node ls

1gwudwxftloza3vldyr4p6p4y *  indocgubt103  Ready   Active        Leader
e9bcxw8vy1ow0jp80gopr2c58    indocgubt104  Ready   Active        

Create an Overlay Network

Now let’s create an Overlay Network called es. On node 1 run the following:

docker network create -d overlay es

To verify, run the following on node 1. Please note that es won’t show up on node 2 until there is actually a container uses the network.

docker network ls

NETWORK ID          NAME                DRIVER              SCOPE
43652a980910        bridge              bridge              local               
1ac35860a4cb        docker_gwbridge     bridge              local               
912wlikzt94x        es                  overlay             swarm               
5032a295b055        host                host                local               
1my3c1fbunaq        ingress             overlay             swarm               
7687da500317        none                null                local  

Attach the Service to the Overlay Network

Uses the example in this post, I would like to deploy an Elasticsearch service to es network.

docker service create \
               --network es \
               --name es-master \
               -p 9200:9200 \
               --mount type=bind,source=/data/es,destination=/usr/share/elasticsearch/data \

Now we bring up another test container to see if it can talk / ping the es-master.

docker service create \
               --name test \
               --network es \
               busybox sleep 300000

Run docker service ps to find out on which node the busybox is sent to, switch to that node, run

docker exec -it container_ID /bin/sh
ping es-master

If nothing goes wrong, the ping should return the results. More information about service communication on Overlay network is here