The past several months (or perhaps the last one year) I have been trying to figure out a way to monitor our highly available distributed system. Over the past couple of year, the system have grown from around 20 virtual machines and now we have more than 100 virtual machines, running various applications and we’re getting tired of ssh or rdp-in to each vms to get our application logs. Enough is enough, I say.
One of our dev team recommended Elasticsearch stack while the other has already started using Graylog (which uses Elasticsearch for it’s database). So I decided, lets just use both.
To ease the deployment, I used docker and docker-compose, while for reverse proxy, I decided to use Traefik, to follow the footsteps of the dev team that uses Graylog and Grafana for their work.
For my setup I used the following versions:
- Elasticsearch/Logstash/Kibana 6.6.2
- Graylog 2.0
- Traefik 1.7.12
- Latest docker CE and docker-compose
The end objective is to have the following URLs:
note that the above two urls does not exist and only written here to simulate two URLs
So I start off by setting up the docker-compose.yml file. Below is the section for Traefik.
services: traefik: container_name: traefik restart: always command: --api --docker --logLevel=WARN image: traefik ports: - "80:80" - "443:443" labels: - "traefik.enable=false" volumes: - "/var/run/docker.sock:/var/run/docker.sock" - "./traefik.toml:/etc/traefik/traefik.toml:ro" - "./defaultssl/:/etc/traefik/defaultssl/:ro" networks: frontend: logging: driver: json-file options: max-size: "20m" max-file: "5"
This exposes port 80 and 443 and changes the logging driver to json. I also created and used docker network that we call frontend.
Traefik will also need some local volumes:
- traefik.toml for the traefik configuration to be located in the same folder as the docker-compose file
- defaultssl folder to store the public and private keys required for https
The traefik.toml file is configured to redirect port 80 traffic to port 443. Also to define the public and private keys
[entryPoints] [entryPoints.http] address = ":80" [entryPoints.http.redirect] entryPoint = "https" [entryPoints.https] address = ":443" [entryPoints.https.tls] [[entryPoints.https.tls.certificates]] # Match the names of the cert and key file in the folder you're mounting. CertFile = "/etc/traefik/defaultssl/log.cer" KeyFile = "/etc/traefik/defaultssl/log.key" [docker] endpoint = "unix:///var/run/docker.sock"
Graylog runs on MongoDB and Elasticsearch. MongoDB will be using docker network called backend_graylog2
graylog2_mongo: image: mongo:3 container_name: graylog2_mongo restart: always labels: - "traefik.enable=false" volumes: - "graylog2_mongo:/data/db" networks: backend_graylog2: logging: driver: json-file options: max-size: "20m" max-file: "5"
Elasticsearch will be on a single container (not clustered) and the only non default config will be cluster name
graylog2_elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:6.6.2 container_name: graylog2elasticsearch restart: always command: elasticsearch -Ecluster.name='graylog' labels: - "traefik.enable=false" volumes: - "graylog2_elasticsearch:/usr/share/elasticsearch/data" networks: backend_graylog2: logging: driver: json-file options: max-size: "20m" max-file: "5"
I need to configure an env file that will hold the default password, the api endpoint and elasticsearch url. This will be named graylog2.env and located in the same directory as docker-compose.yml
GRAYLOG_PASSWORD_SECRET=878511b7-23f4-4526-9c1e-5f2da8f8833a # Generate an admin pass, which is SHA encrypted (hint: echo -n foobar | sha256sum) GRAYLOG_ROOT_PASSWORD_SHA2=c3ab8ff13720e8ad9047dd39466b3c8974e592c2fa383d4a3960714caef0c4f2 #Match this with the api endpoint: GRAYLOG_WEB_ENDPOINT_URI=https://logs.bekzilla.com/api #Should be OK, it corresponds to the Elastic search container address. GRAYLOG_ELASTICSEARCH_HOSTS=http://graylog2elasticsearch:9200
Then I start the graylog2 part of the docker-compose.yml with
graylog2: image: graylog/graylog:2.5 container_name: graylog2 restart: always env_file: - ./graylog2.env
I need to configure traefik to point at port 9000 for graylog UI and it should be reachable via http/https and the URL will be logs.bekzilla.com.
labels: - "traefik.backend=graylog2" - "traefik.docker.network=compose_frontend" - "traefik.port=9000" - "traefik.frontend.entryPoints=http,https" - "traefik.frontend.rule=Host:logs.bekzilla.com"
Port 12201 UDP will need to be exposed
ports: - "12201:12201/udp"
I set links to both the mongodb and elasticsearch
links: - graylog2_mongo:mongo - graylog2_elasticsearch:elasticsearch
and then define the volumes for graylog
volumes: - "graylog2_graylog2-data:/usr/share/graylog/data/journal" - "graylog2_graylog2-config:/usr/share/graylog/data/config"
networks to be connected with the frontend and backend and make sure I use json-file driver
networks: frontend: backend_graylog2: logging: driver: json-file options: max-size: "20m" max-file: "5"
For kibana we start off with the following
kibana: container_name: kibana image: docker.elastic.co/kibana/kibana:6.6.2
then define the traefik settings
labels: - "traefik.backend=kibana" - "traefik.docker.network=compose_frontend" - "traefik.frontend.rule=Host:metrics.bekzilla.com" - "traefik.frontend.entryPoints=http,https" - "traefik.port=5601"
Then the networks and links
networks: frontend: backend_graylog2: links: - graylog2_elasticsearch:elasticsearch
Logstash was the tricky one for me as it has lots of configs for patterns and ports
logstash: container_name: logstash image: docker.elastic.co/logstash/logstash:6.6.2 volumes: - ./logstash/pipeline/:/usr/share/logstash/pipeline/ - ./logstash/:/usr/share/logstash/config/ - ./logstash/patterns:/etc/logstash/patterns - ./logstash/templates:/etc/logstash/templates - ./logstash/conf.d/:/etc/logstash/conf.d/ ports: - "5044:5044" - "5000:5000" - "5001:5002" - "5002:5002" - "9600:9600" environment: LS_JAVA_OPTS: "-Xmx256m -Xms256m" networks: frontend: backend_graylog2: links: - graylog2_elasticsearch:elasticsearch
Then we finally define the volumes and networks
volumes: graylog2_mongo: graylog2_elasticsearch: graylog2_graylog2-data: graylog2_graylog2-config: ##Networks networks: frontend: backend_graylog2:
So there you have it, Graylog and Kibana on the front end. Elasticsearch and MongoDB on the back end. All for my first try at docker-compose.