Elasticsearch stack, Graylog and Traefik with Docker compose

The past several months (or perhaps the last one year) I have been trying to figure out a way to monitor our highly available distributed system. Over the past couple of year, the system have grown from around 20 virtual machines and now we have more than 100 virtual machines, running various applications and we’re getting tired of ssh or rdp-in to each vms to get our application logs. Enough is enough, I say.

• 

• 

One of our dev team recommended Elasticsearch stack while the other has already started using Graylog (which uses Elasticsearch for it’s database). So I decided, lets just use both.

• 

• 

To ease the deployment, I used docker and docker-compose, while for reverse proxy, I decided to use Traefik, to follow the footsteps of the dev team that uses Graylog and Grafana for their work.

For my setup I used the following versions:

• Elasticsearch/Logstash/Kibana 6.6.2
• Graylog 2.0
• Traefik 1.7.12
• Latest docker CE and docker-compose

The end objective is to have the following URLs:

Kibana: https://metrics.bekzilla.com
Graylog: https://logs.bekzilla.com

note that the above two urls does not exist and only written here to simulate two URLs

Traefik

So I start off by setting up the docker-compose.yml file. Below is the section for Traefik.

services:
  traefik:
    container_name: traefik
    restart: always
    command: --api --docker --logLevel=WARN
    image: traefik
    ports:
      - "80:80"
      - "443:443"
    labels:
      - "traefik.enable=false"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "./traefik.toml:/etc/traefik/traefik.toml:ro"
      - "./defaultssl/:/etc/traefik/defaultssl/:ro"
    networks:
      frontend:
    logging:
      driver: json-file
      options:
        max-size: "20m"
        max-file: "5"

This exposes port 80 and 443 and changes the logging driver to json. I also created and used docker network that we call frontend.

Traefik will also need some local volumes:

• traefik.toml for the traefik configuration to be located in the same folder as the docker-compose file
• defaultssl folder to store the public and private keys required for https

The traefik.toml file is configured to redirect port 80 traffic to port 443. Also to define the public and private keys

[entryPoints]
   [entryPoints.http]
   address = ":80"
     [entryPoints.http.redirect]
       entryPoint = "https"
   [entryPoints.https]
   address = ":443"
     [entryPoints.https.tls]
       [[entryPoints.https.tls.certificates]]
# Match the names of the cert and key file in the folder you're mounting.
       CertFile = "/etc/traefik/defaultssl/log.cer"
       KeyFile = "/etc/traefik/defaultssl/log.key"

[docker]
endpoint = "unix:///var/run/docker.sock"

MongoDB

Graylog runs on MongoDB and Elasticsearch. MongoDB will be using docker network called backend_graylog2

  graylog2_mongo:
    image: mongo:3
    container_name: graylog2_mongo
    restart: always
    labels:
      - "traefik.enable=false"
    volumes:
      - "graylog2_mongo:/data/db"
    networks:
      backend_graylog2:
    logging:
      driver: json-file
      options:
        max-size: "20m"
        max-file: "5"

Elasticsearch

Elasticsearch will be on a single container (not clustered) and the only non default config will be cluster name

  graylog2_elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.6.2
    container_name: graylog2elasticsearch
    restart: always
    command: elasticsearch -Ecluster.name='graylog'
    labels:
      - "traefik.enable=false"
    volumes:
      - "graylog2_elasticsearch:/usr/share/elasticsearch/data"
    networks:
      backend_graylog2:
    logging:
      driver: json-file
      options:
        max-size: "20m"
        max-file: "5"

Graylog

I need to configure an env file that will hold the default password, the api endpoint and elasticsearch url. This will be named graylog2.env and located in the same directory as docker-compose.yml

GRAYLOG_PASSWORD_SECRET=878511b7-23f4-4526-9c1e-5f2da8f8833a
# Generate an admin pass, which is SHA encrypted (hint: echo -n foobar | sha256sum)
GRAYLOG_ROOT_PASSWORD_SHA2=c3ab8ff13720e8ad9047dd39466b3c8974e592c2fa383d4a3960714caef0c4f2
#Match this with the api endpoint:
GRAYLOG_WEB_ENDPOINT_URI=https://logs.bekzilla.com/api
#Should be OK, it corresponds to the Elastic search container address.
GRAYLOG_ELASTICSEARCH_HOSTS=http://graylog2elasticsearch:9200

Then I start the graylog2 part of the docker-compose.yml with

 graylog2:
    image: graylog/graylog:2.5
    container_name: graylog2
    restart: always
    env_file:
      - ./graylog2.env

I need to configure traefik to point at port 9000 for graylog UI and it should be reachable via http/https and the URL will be logs.bekzilla.com.

    labels:
      - "traefik.backend=graylog2"
      - "traefik.docker.network=compose_frontend"
      - "traefik.port=9000"
      - "traefik.frontend.entryPoints=http,https"
      - "traefik.frontend.rule=Host:logs.bekzilla.com"

Port 12201 UDP will need to be exposed

    ports:
      - "12201:12201/udp"

I set links to both the mongodb and elasticsearch

    links:
      - graylog2_mongo:mongo
      - graylog2_elasticsearch:elasticsearch

and then define the volumes for graylog

    volumes:
      - "graylog2_graylog2-data:/usr/share/graylog/data/journal"
      - "graylog2_graylog2-config:/usr/share/graylog/data/config"

networks to be connected with the frontend and backend and make sure I use json-file driver

    networks:
      frontend:
      backend_graylog2:
    logging:
      driver: json-file
      options:
        max-size: "20m"
        max-file: "5"

Kibana

For kibana we start off with the following

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:6.6.2 

then define the traefik settings

    labels:
      - "traefik.backend=kibana"
      - "traefik.docker.network=compose_frontend"
      - "traefik.frontend.rule=Host:metrics.bekzilla.com"
      - "traefik.frontend.entryPoints=http,https"
      - "traefik.port=5601"

Then the networks and links

    networks:
      frontend:
      backend_graylog2:
    links:
      - graylog2_elasticsearch:elasticsearch

Logstash

Logstash was the tricky one for me as it has lots of configs for patterns and ports

  logstash:
    container_name: logstash
    image: docker.elastic.co/logstash/logstash:6.6.2
    volumes:
      - ./logstash/pipeline/:/usr/share/logstash/pipeline/
      - ./logstash/:/usr/share/logstash/config/
      - ./logstash/patterns:/etc/logstash/patterns
      - ./logstash/templates:/etc/logstash/templates
      - ./logstash/conf.d/:/etc/logstash/conf.d/
    ports:
      - "5044:5044"
      - "5000:5000"
      - "5001:5002"
      - "5002:5002"
      - "9600:9600"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      frontend:
      backend_graylog2:
    links:
      - graylog2_elasticsearch:elasticsearch

Then we finally define the volumes and networks

volumes:
  graylog2_mongo:
  graylog2_elasticsearch:
  graylog2_graylog2-data:
  graylog2_graylog2-config:
 
##Networks
networks:
  frontend:
  backend_graylog2:

All done

So there you have it, Graylog and Kibana on the front end. Elasticsearch and MongoDB on the back end. All for my first try at docker-compose.