Microservices with Docker on netcup vServer using Traefik as Dynamic Reverse-Proxy

This guide is a small collection of configuration files and docker recipes to set up multiple webservices on a vServer or VPS from netcup GmbH. It uses Traefik v1.7 as a reverse proxy together with docker-compose.

Disclaimer: It shall be noted that there is no emphasis on security whatsoever. So all settings and configurations are educational only and not meant for production. However, we will discuss some security improvements to the system by usage of a docker socket proxy so that the daemon is not directly exposed by the Traefik container.

Furthermore, I am aware that there are major changes with Traefik v2. But since there is still room for improvement concerning guides for Version 2 we will stick to a well-known setup.

Overview

The following services will be configured exemplary to show the architecture and interaction with the reverse proxy.

• Traefik v1.7 + Docker Socket Proxy for indirect socket interfacing
• Nextcloud Stack (with mariadb)
• Wordpress Stack
• Grav Light-Weight Websites
• mstream Mediaserver
• Bookstack
• Jupyter Lab

The docker configuration of most of these services are inspired by the popular “linuxserver” Github repository: https://github.com/linuxserver

The main intent is to show how these services can be used together with a reverse proxy (Traefik in this case) and its advantage of automatic service discovery. The Traefik config was inspired by and modified for the present setup from: https://github.com/cbirkenbeul/docker-homelab

You can checkout this repository for additional info and documentation for things that I have might forgot.

For a proper overview of the architecture the following image shows all containers and connections for our system.

Microservices overview. The details of each service and its interaction with the proxy will be described in details.

Shown are all containers and their connections in-between annotated by the dashed circles. The circles are named by the individual network that is created for communication and isolation.

1. We can see that outside ingress is defined for the ports 80 (HTTP), 443 (HTTPS) and 8080 (Traefik Dashboard) in the default network (bridged).
2. The requests are forwarded from Traefik to the correct app container within the traefik_proxy network.
3. If the app stack consists of multiple containers (like databases etc.) there is an additional app network called xxx-internal.
4. The docker socket interaction is filtered by selection rules for the exposed remote API listening on 2375. It is connected to the Traefik container in the socket_proxy network.

In the following we will go through each individual service step by step and show how it is configured and set up.

Traefik Reverse Proxy

First we need to set up our reverse proxy for ingress into the system. There are many popular proxy systems each with pros and cons. Traefik is suited for dynamic forward rule definitions together with docker which is why I have chosen this software. I am aware that dynamic definitions are not very useful for this use case but the motivation to use Traefik came from another advantage which is the automatic generation and also renewal of the SSL certificates with Lets Encrypt.

I will start with the used docker-compose.yml:

---
version: '3.3'

services:
  traefik:
    image: traefik:1.7.6
    container_name: traefik
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - $PWD/config:/etc/traefik
    networks:
      - traefik_proxy
      - default
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    restart: always

networks:
  traefik_proxy:
    external:
      name: traefik_proxy
  default:
    driver: bridge

I will first show how everything is set up with direct mounting of the docker socket and describe how we can use a socket proxy at the end of the article.

We can see that most arguments and parameters are pretty straight forward. The outside world ports are forwarded explicitly. The container is part of two networks, the outside world network (default) as well as the internal container network (traefik_proxy). The flag “external” under networks tells docker-compose that this network was created upfront and therefore needs to be present. We can create this by:

docker network create traefik_proxy

There is another mount of a config folder. Within this folder we need another file for our Traefik settings called config/traefik.toml:

logLevel = "ERROR"
defaultEntryPoints = ["http", "https"]
insecureSkipVerify = true

[web]
address = ":8080"
  [web.auth.basic]
  users = ["DASHBOARD_USER:HASH"]

[docker]
domain = "YOURDOMAIN.COM"
watch = true
exposedbydefault = false

[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    entryPoint = "https"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]

# Let's encrypt configuration
[acme]
  email="email@email.com"
  storage="/etc/traefik/ACME/acme.json"
  entryPoint="https"
  acmeLogging=true
  OnHostRule=true
  caServer = "https://acme-v02.api.letsencrypt.org/directory"
    [acme.httpChallenge]
    entryPoint = "http"

There are four main parts in this config. The part [web] is just for the Traefik Dashboard. We have chosen a simple auth with one user. Place replace DASHBOARD_USER and HASH with the appropriate values. You can create a password hash with:

echo $(htpasswd -nbB <DASHBOARD_USER> "<PASS>") | sed -e s/\\$/\\$\\$/g

This will directly echo to the console the HASH. Please be aware that it could be possible to escape $ characters in the hash depending on the system.

The second part [docker] is used for custom docker configurations. For example the default domain or activation of watching changes during runtime. However, these settings can by overwritten by the individual app containers.

Under the part [entrypoint] are the settings for the inputs HTTP and HTTPS ports with the redirect of 80 to 443.

Lastly, there is the [acme] configuration for the Lets Encrypt SSL certificate. For the storage there needs to be an empty file called acme.json which we need to place into the config folder:

cd config
mkdir ACME
cd ACME
touch acme.json
chmod 600 acme.json

We have selected the HTTP challenge in this case but Traefik offers different methods for the SSL certification.

Now you can start the container and check the dashboard. From the main folder where your docker-compose.yml is located execute:

docker-compose up -d

And check if the container is running with docker ps. If you are not sure what is going on you can start the container without the flag -d.

Generally, if you are trying out settings please use the staging server from Lets Encrypt since there is a rate limit on how many requests you can send to the production server.

We can check the dashboard with your earlier defined credentials at https://your-domain.com:8080 where you should see in principle some overview of the proxy (without the front- and backends at the moment) just like in the following image:

Dashboard overview for Traefik.

Now when the reverse proxy is running we can add our first service…

Nextcloud

Our Nextcloud service is easily defined by simply one docker-compose.yml file and two folders on the host: app, and database for the persistent data. Lets start with the container configurations in the docker-compose.yml:

---
version: '3.3'

services: 
  nextcloud-db:
    image: mariadb
    container_name: nextcloud-db
    restart: always
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    environment:
      - MYSQL_ROOT_PASSWORD=SUPERSECRETPASSWORD
      - MYSQL_PASSWORD=ANOTHERPASSWORD
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
    volumes:
      - $PWD/database:/var/lib/mysql
    networks:
      - nextcloud-internal
    
  nextcloud-app:
    image: nextcloud
    container_name: nextcloud-app
    restart: always
    depends_on:
      - nextcloud-db
    labels:
      - "traefik.backend=nextcloud"
      - "traefik.enable=true"
      - "traefik.frontend.rule=Host:NEXTCLOUD.YOURDOMAIN.COM"
      - "traefik.port=80"
      - "traefik.docker.network=traefik_proxy"
    volumes: 
      - $PWD/app:/var/www/html
    networks:
      - traefik_proxy
      - nextcloud-internal
    
networks:
  nextcloud-internal:
  traefik_proxy:
    external:
      name: traefik_proxy

The stack consists of two services: nextcloud-db and nextcloud-app. The database uses mariadb for which you need to create two secure passwords and replace the placeholders: SUPERSECRETPASSWORD, and ANOTHERPASSWORD. I guess having the MySQL database passwords in clear text as the environment variables is not optimal so I suggest you should change these once the container is running with some admin db tooling.

For the nextcloud-app container we need to set the labels that are relevant for Traefik to create the correct forward rule since this container is connected with the traefik_proxy network. In my case I prefer to define the service by a sub-domain so please change NEXTCLOUD.YOURDOMAIN.COM.

We can see that there is a dedicated network between both containers that is called nextcloud-internal so that there is isolation of the database to the rest of the system.

Now that we have the docker-compose.yml and created the two folders app and database we can start the stack from the main folder with:

 docker-compose up -d

just like we did for the proxy. You should be able now to access the service by the URL defined by the forward rule and start setting up your Nextcloud…

Wordpress, Bookstack, grav, and mstream

The configuration of additional services like Wordpress, Bookstack, grav, and mstream are configured exactly like the Nextcloud service so you can use the recipes directly from my Github repository for this project:

trn84/vserver-docker-services

Please make sure to replace all the placeholders with the appropriate values and ensure that the required folder structure is there.

Jupyter Lab

Lastly, one recent service that I wanted to play around with my own Jupyter Lab on my vServer behind the proxy. Some might argue that possible issues with interactive widget using websockets (i.e. bokeh) in Jupyter Lab behind the proxy are more annoying than just using the port 8888 directly. But hey this project was supposed to be educational so nevermind ;-)

There are some things we need to consider when we want to access our Jupyter Lab with a password instead of a token. Here is the docker-compose.yml that I use:

---
version: "2"
services:
  jupyter:
    image: jupyter/datascience-notebook
    container_name: jupyter
    restart: unless-stopped
    environment:
      - TZ=Europe/Berlin
      - JUPYTER_ENABLE_LAB=yes
    labels:
      - "traefik.backend=jupyter"
      - "traefik.enable=true"
      - "traefik.frontend.rule=Host:JUPY.DOMAIN.COM"
      - "traefik.port=8888"
      - "traefik.docker.network=traefik_proxy"
    volumes:
      - $PWD/workspace:/workspace
    networks:
      - traefik_proxy
    command: start-notebook.sh --ip=0.0.0.0 --NotebookApp.password=sha1:HASH
networks:
  traefik_proxy:
    external:
      name: traefik_proxy

I am using the standard jupyter stacks docker images (datascience in this case). Like with all services we need to set the frontend forward rule: JUPY.DOMAIN.COM and insert the SHA-1 HASH of our password for the server. You can create the hash with Python using:

from notebook.auth import passwd
passwd()

Some people suggest to set the base_url of the notebook server when used behind a reverse proxy. I believe that this is more likely required for frontend rules that make use of PathPreFixes and PostPath settings?!

Indirect Docker Socket Exposure

In this experimental part we will try to make our system more resilient against security issues. One attack surface is the fact that the Traefik docker container need access on the hosts docker socket which is granted by mounting:

-v /var/run/docker.sock:/var/run/docker.sock

This means that if there are security issues in Traefik (bugs) that lead to access to this container the attacker has basically sudo rights on the whole system (also the case for Portainer etc.).

There are more details given in the following thread and also some approaches that possibly avoid this: https://github.com/containous/traefik/issues/4174

Please be aware that you should not do this on an important system since I can not guarantee that there are no big issues with this approach. I would be happy to get feedback on this.

We do this step-by-step:

# Create network for local docker host api access
docker network create socket_proxy

Now we need make changes to our Traefik docker-compose.yml and config/traefik.toml.

---
version: '3.3'
services:
  docker-socket-proxy:
    image: tecnativa/docker-socket-proxy
    container_name: docker-socket-proxy
    networks:
     - socket_proxy
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock
    environment:
     - SERVICES=1
     - NETWORKS=1
     - TASKS=1
     - CONTAINERS=1
traefik:
    image: traefik:1.7.6
    container_name: traefik
    volumes:
      - $PWD/config:/etc/traefik
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - socket_proxy
      - traefik_proxy
      - default
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    restart: always
networks:
  socket_proxy:
    external:
      name: socket_proxy
  traefik_proxy:
    external:
      name: traefik_proxy
  default:
    driver: bridge

I have highlighted the modified parts. We have added a service called docker-socket-proxy. You can check the following repo for details. It binds to the docker socket and provides the remote API to allow for limited access to the daemon.

Tecnativa/docker-socket-proxy

Finally we need to change the docker endpoint in the traefik.toml to the docker-socket-proxy:

logLevel = “ERROR”
defaultEntryPoints = [“http”, “https”]
insecureSkipVerify = true
[web]
address = “:8080”
 [web.auth.basic]
 users = [“DASHBOARD_USER:HASH”]
[docker]
domain = “YOURDOMAIN.COM”
endpoint=”tcp://docker-socket-proxy:2375"
watch = true
exposedbydefault = false
[entryPoints]
 [entryPoints.http]
 address = “:80”
 [entryPoints.http.redirect]
 entryPoint = “https”
 [entryPoints.https]
 address = “:443”
 [entryPoints.https.tls]
# Let’s encrypt configuration
[acme]
 email=”email@email.com”
 storage=”/etc/traefik/ACME/acme.json”
 entryPoint=”https”
 acmeLogging=true
 OnHostRule=true
 caServer = “https://acme-v02.api.letsencrypt.org/directory"
 [acme.httpChallenge]
 entryPoint = “http”

Finally we can restart the proxy stack and check if everything is working…

Remarks

You have seen that making your docker container “ready” to be used with Traefik as a reverse proxy is super easy by just setting the correct label during start.

We have tried to avoid mounting the docker socket directly by partial exposure of the docker remote API to handle GET calls from Traefik. You can also work with static configurations if you do not start container dynamically. However, since this is a big advantage of Traefik I wanted to make use of it for educational purposes.

I would be pleased to get feedback on this guide and further remarks on the setup.