IP addresses are hard to remember and that's why we have DNS. But still, on most of the local systems, we don't have a DNS server and we have to remember the IP addresses of the systems. Even worse, let me say I have a cluster with 1000 nodes. Without a nameserver or auto-discovery tool, I should remember all the IP addresses of the cluster nodes.
One thing the modern cluster architecture supports is node
discovery. Now, you have the same cluster with 1000 nodes, you don't
have to remember all the nodes IP addresses but just one node and that
one node also acts as a service discovery to other nodes. For example,
if your application wants to talk to
node4 and you know only
node1's IP address, you can either talk to
node1 as if you are
node1 redirects the communication to
node4 or you can ask
node4's IP address.
But one major problem is that if
node1 goes down for some reason,
even though the cluster will still be working with the rest of the 999
nodes or one new node with different IP address gets added up, you
lose the access to the entire cluster because all you have got is
node1's IP address and
node1 is currently not in the cluster. For
this reason, we have a cluster IP address. What exists at the core of
the cluster system is a node discovery service. Your application
connects to the cluster IP, and you don't have to worry about some
nodes being down since the cluster IP is nowhere related to it and
only acts as a discovery tool for the nodes and other services in the
And now, we are at it again. We now have to remember the IP address of the cluster. If it is IPv6, it's nearly impossible to remember the address. The cluster system lacks a human-readable name, i.e., it lacks a name server. If the cluster exists in a cloud system, you don't have to worry about remembering since the external DNS comes into the picture and you can set up a domain name.
Most of the local systems don't have a name server. If you look up the network to know what all systems are connected to the machine, you get a bunch of IP addresses. You have a local cluster, printer, mobile phones, laptops, IoT etc.
Let us take the printer example. You are connected to 10 different printers, each one is of a different type. And if you need to print a file and you want to select a particular printer, and let's say you don't the name of the printers but just the IP addresses. So what you have to do is to look up a printer IP addresses table and take the right printer's IP address. Still, you are limited to type it out or copy-paste the IP address. To solve this, most of the private networks append .local at the end of the machine's hostname using the zeroconf technique. Now in the printer's example, if you look up the network for the connected printers, you get
1) my-epson-printer.local 2) my-canon-printer.local 3) my-hp-printer.local ...
Now you don't have to remember the IP address of the printer you want. All you have to remember is a human-readable name.
Same goes for the cluster too. You can reach your cluster at
my-awesome-cluster.local. Even better, you can access to a
node4.my-awesome-cluster.local. If say
node1 goes down and if the
cluster starts a new node with a different IP address to take up
node1's position, the new node will be reachable at
node1.my-awesome-cluster.local. This is called FQDN.
Why do we need a hostname resolver for local networks?
Earlier you used to connect to the cluster using the IP address of the
cluster and it worked. Now when you tell your application to connect
to the cluster using the
.local, it simply doesn't know how to
connect. The problem is that the application doesn't know how to
.local domain name to the cluster IP address. It lacks a
DNS server. And since the cluster is a local one, your DNS server
can't resolve it.
In the case of the cloud system, you would have set up a domain name and you can reach the cluster using the domain name. And the external DNS helps you to resolve the domain name to the cluster IP address.
To solve this problem we have mDNS. mDNS is a protocol that resolves
hostname to IP addresses within a small network that lacks the name
server. By default, mDNS exclusively resolves hostnames ending with
.local. But there will be a problem with the hosts that implement
.local doesn't support mDNS protocol and can be found via a
conventional unicast DNS sever. In those cases, necessary network
configuration should be changed.
Lets try out an example.
Why Redis Enterprise docker image?
The reason why I have chosen Redis Enterprise docker image is that it
aligns well with the previous cluster example. With Redis Enterprise
container, we can create a redis database cluster. And the cluster
will have a domain name with
my-awesome-cluster.local) and each node in the cluster receives a
To sum up the above paragraph, we get a cluster which has a
<my-cluster>.local address and nodes with
Now let's start the example.
You can start the container as follows:
docker run -it --cap-add sys_resource -p 12000:12000 -p 8443:8443 -p 9443:9443 redislabs/redis
9443 should be exposed compulsorily for the
container to do all its magic for services.
In the logs, you can find out that the container starts a mDNS server.
...more logs... 2020-05-30 05:10:03,048 INFO success: mdns_server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) ...more logs...
12000 we have exposed can be any free port. This is the
port where the database we create will be listening on. In the current
example, I'm creating just one node. But it can be many.
Once the container is up and running, you can navigate to
You don't have to edit anything and under "Cluster configuration" you can give your cluster FQDN like this.
You can skip entering the certificates to which the container creates the certificates with default options. And then you can set some credentials the redis enterprise container requires you to. Once you set the credentials your page will be refreshed and you'll again be told to enter the credentials. Once you sign in, you are ready to create a database.
You can select "Redis" as your database, and you'll be given a form to enter the configuration of the database.
I'm giving the name as
node1 and most importantly, you must set the
12000 since that's the only port that we have exposed. Click
on "show advanced options" and you can find the field "Endpoint port
number" where you can enter the port. Enter it as
that's the port I have exposed). Click "Activate" to create the
database. You'll then be redirected to your created DB's configuration
You can find the
.local address and IP address of the DB in the
Let us now create our sample application that simply connects to
node1 and pings the database.
FROM python:3.8-buster RUN pip3 install redis CMD ["python3"]
We have taken the python
buster image which is Debian 10 that has
the mdns packages which we will install later.
Let's build it the application.
$ docker build -t py-mdns . Sending build context to Docker daemon 3.072kB Step 1/3 : FROM python:3.8-buster 3.8-buster: Pulling from library/python 376057ac6fa1: Pull complete 5a63a0a859d8: Pull complete 496548a8c952: Pull complete 2adae3950d4d: Pull complete 0ed5a9824906: Pull complete bb94ffe72389: Pull complete 241ada007777: Pull complete be68aa7d1eeb: Pull complete 820ffc2e28ca: Pull complete Digest: sha256:ebe8df5c3e2e10a7aab04f478226979e3b8754ee6cd30358379b393ef8b5321e Status: Downloaded newer image for python:3.8-buster ---> 659f826fabf4 Step 2/3 : RUN pip3 install redis ---> Running in 1d31f5312e3e Collecting redis Downloading redis-3.5.2-py2.py3-none-any.whl (71 kB) Installing collected packages: redis Successfully installed redis-3.5.2 Removing intermediate container 1d31f5312e3e ---> c21a8bc34782 Step 3/3 : CMD ["python3"] ---> Running in 711a17a97211 Removing intermediate container 711a17a97211 ---> 27b0d68b69fc Successfully built 27b0d68b69fc Successfully tagged py-mdns:latest $ docker run --it py-mdns
Now lets start the application as follows:
$ docker run -it py-mdns
Remember that the
.local address of
redis-12000.mycluster.local, IP address is
172.17.0.2 and the port
Now lets try to ping the database using the
$ docker run -it py-mdns Python 3.8.3 (default, May 16 2020, 07:08:28) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import redis >>> redis.Redis(host='redis-12000.mycluster.local', port=12000).ping() Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 550, in connect sock = self._connect() File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 575, in _connect for res in socket.getaddrinfo(self.host, self.port, self.socket_type, File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.8/site-packages/redis/client.py", line 1378, in ping return self.execute_command('PING') File "/usr/local/lib/python3.8/site-packages/redis/client.py", line 898, in execute_command conn = self.connection or pool.get_connection(command_name, **options) File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 1183, in get_connection connection.connect() File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 554, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error -2 connecting to redis-12000.mycluster.local:12000. Name or service not known.
What happened is that your application lacks an mDNS service discovery
such that it doesn't know whom to ask and how to resolve the
hostname to IP address.
Connecting using the IP address works.
>>> redis.Redis(host='172.17.0.2', port=12000).ping() True
Now lets try to add mDNS.
Avahi is a free zero-configuration implementation of the mDNS protocol. Let's try to add it.
RUN set -ex \ && apt-get update && apt-get install -y --no-install-recommends avahi-daemon libnss-mdns
libnss by default can resolve up to two-label such that it can
resolve the IP address of
mycluster.local but what we need is
three-label. Let's try to configure add that configuration too.
RUN set -ex \ && apt-get update && apt-get install -y --no-install-recommends avahi-daemon libnss-mdns \ # allow hostnames with more labels to be resolved. so that we can # resolve node1.mycluster.local. # (https://github.com/lathiat/nss-mdns#etcmdnsallow) && echo '*' > /etc/mdns.allow \ # Configure NSSwitch to use the mdns4 plugin so mdns.allow is respected && sed -i "s/hosts:.*/hosts: files mdns4 dns/g" /etc/nsswitch.conf
Now, we have the configuration, we can start the
our container start using the
ENTRYPOINT script. Here's the
#!/bin/bash set -e # start avahi's dependency service dbus start # start avahi service avahi-daemon start exec "$@"
This starts dbus which is a dependency of
avahi-daemon. Now this is
Dockerfile looks like.
FROM python:3.8-buster WORKDIR /app COPY entrypoint.sh /app/ RUN set -ex \ && apt-get update && apt-get install -y --no-install-recommends avahi-daemon libnss-mdns \ # allow hostnames with more labels to be resolved. so that we can # resolve node1.mycluster.local. # (https://github.com/lathiat/nss-mdns#etcmdnsallow) && echo '*' > /etc/mdns.allow \ # Configure NSSwitch to use the mdns4 plugin so mdns.allow is respected && sed -i "s/hosts:.*/hosts: files mdns4 dns/g" /etc/nsswitch.conf \ && pip3 install redis ENTRYPOINT ["bash", "./entrypoint.sh"] CMD ["python3"]
Let's try to build the image and run the container.
$ docker run -it py-mdns [ ok ] Starting system message bus: dbus. [ ok ] Starting Avahi mDNS/DNS-SD Daemon: avahi-daemon. Python 3.8.3 (default, May 16 2020, 07:08:28) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import redis >>> redis.Redis(host='redis-12000.mycluster.local', port=12000).ping() True
As you can see, the avahi-daemon is being started. And it successfully
Running the container as non-root user
Now let's talk about the security.2 Many container platforms accept only non-root containers, Ex: openshift. If you want your application to be deployed at any container platform, you can't. Let us try to run the container as a non-root user.
$ docker run -it --user 1001 py-mdns mkdir: cannot create directory ‘/var/run/dbus’: Permission denied
The user argument takes a user UUID. It says to run as any other user
rather than root. `1001` is not a special user. It might just be
whatever UUID that doesn't match an existing user in the image. You
can also put
USER command inside the
It requires root permission to start the application. You can check the same by connecting to the container.
$ docker exec -it a701fb0d30e2 bash root@a701fb0d30e2:/app# ps -aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.8 0.0 13340 8276 pts/0 Ss+ 06:27 0:00 python3 message+ 19 0.0 0.0 8552 2368 ? Ss 06:27 0:00 /usr/bin/dbus-daemon --system avahi 46 0.0 0.0 7868 2564 ? S 06:27 0:00 avahi-daemon: running [a701fb0d30e2.local] avahi 47 0.0 0.0 7868 296 ? S 06:27 0:00 avahi-daemon: chroot helper root 71 7.0 0.0 5748 3636 pts/1 Ss 06:27 0:00 bash root 76 0.0 0.0 9388 3092 pts/1 R+ 06:27 0:00 ps -aux root@a701fb0d30e2:/app#
The reason why the application can't start as a non-root user is that
avahi's dependency, requires root permissions to start.
There's a way to run
dbus by adding
[server] section of avahi-daemon.conf file. The file exists
avahi by default requires root permissions to start. We can bypass
that by providing
flag.3 You can check out all the options
here. Let us remove
dbus. Also, let's start the process manually
# Dockerfile && sed -i "s/hosts:.*/hosts: files mdns4 dns/g" /etc/nsswitch.conf \ + && printf "[server]\nenable-dbus=no\n" >> /etc/avahi/avahi-daemon.conf \ && pip3 install redis
# entrypoint.sh - # start avahi's dependency - service dbus start # start avahi - service avahi-daemon start + avahi-daemon --daemonize --no-drop-root
Now with our configuration, let's try to run the container.
$ docker run -it --user 1001 py-mdns bash Timeout reached while wating for return value Could not receive return value from daemon process.
And the container exits. We still can't run the container as
non-root. The actual problem is with the files and folders the
avahi-daemon is accessing. These files should have either
avahi permissions to access it.
The files are:
Lets try to change the permissions of the files and folders.
# Dockerfile && printf "[server]\nenable-dbus=no\n" >> /etc/avahi/avahi-daemon.conf \ + && chmod 777 /etc/avahi/avahi-daemon.conf \ + && mkdir -p /var/run/avahi-daemon \ + && chown avahi:avahi /var/run/avahi-daemon \ + && chmod 777 /var/run/avahi-daemon && pip3 install redis
We change the permissions of the file
/etc/avahi-daemon.conf so that
avahi daemon can access the file. We also create
/var/run/avahi-daemon directory since the
avahi daemon requires
it. We also change the permissions of them after creating. Lets also
USER flag so by default it runs as the UUID specified.
FROM python:3.8-buster WORKDIR /app COPY entrypoint.sh /app/ RUN set -ex \ && apt-get update && apt-get install -y --no-install-recommends avahi-daemon libnss-mdns \ # allow hostnames with more labels to be resolved. so that we can # resolve node1.mycluster.local. # (https://github.com/lathiat/nss-mdns#etcmdnsallow) && echo '*' > /etc/mdns.allow \ # Configure NSSwitch to use the mdns4 plugin so mdns.allow is respected && sed -i "s/hosts:.*/hosts: files mdns4 dns/g" /etc/nsswitch.conf \ && printf "[server]\nenable-dbus=no\n" >> /etc/avahi/avahi-daemon.conf \ && chmod 777 /etc/avahi/avahi-daemon.conf \ && mkdir -p /var/run/avahi-daemon \ && chown avahi:avahi /var/run/avahi-daemon \ && chmod 777 /var/run/avahi-daemon \ && pip3 install redis USER 1001 ENTRYPOINT ["bash", "./entrypoint.sh"] CMD ["python3"]
#!/bin/bash set -e avahi-daemon --daemonize --no-drop-root exec "$@"
Now lets start the container. Now we don't have to pass
since we have put
USER command inside
$ docker run -it --user 1001 py-mdns Python 3.8.3 (default, May 16 2020, 07:08:28) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import redis >>> redis.Redis(host='redis-12000.mycluster.local', port=12000).ping() True
We can also see that the container is running as non-root.
$ docker exec -it c641eeb8559f bash I have no name!@c641eeb8559f:/app$ ps -aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 1001 1 0.4 0.0 27248 16220 pts/0 Ss+ 08:22 0:00 python3 1001 8 0.0 0.0 8012 2524 ? S 08:22 0:00 avahi-daemon: running [c641eeb8559f.local] 1001 10 11.0 0.0 5748 3612 pts/1 Ss 08:23 0:00 bash 1001 15 0.0 0.0 9388 3072 pts/1 R+ 08:23 0:00 ps -aux
We have reached the end of the post. You can find the code samples in this repository.
avahi-daemon.confhas many configurations and we don't need it in our application, so I'm just setting the