Docker, NGINX load balancing and SELinux

Been a while since I posted in this blog. Today I’m going to post about today’s “journey” of learning Docker. Totally different with the usual travel blog, but I need to write it down to make sure that it sticks to my mind. Probably will add a tech section in this site sometime in the future.

So I’m currently learning docker containerization in Linux Academy and the current topic is to deploy multiple httpd containers, use nginx as a load balancer.

I start with two containers running httpd. Each container hosts a website that I got for free in the internet. I forwarded port 8081 and 8082 to the local machine

[root@bekzilla]# docker run -itd --name=devweb1 -p 8081:80 -v /docker/dockerwww/:/var/www/html centos6:finalwebv1 /bin/bash
[root@bekzilla]# docker run -itd --name=devweb2 -p 8082:80 -v /docker/dockerwww/:/var/www/html centos6:finalwebv1 /bin/bash
[root@bekzilla]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
14a9eb5ea7e3 centos6:finalwebv1 "/bin/bash" 3 hours ago Up 3 hours 0.0.0.0:8082->80/tcp devweb2
ea75f7cdb4a6 centos6:finalwebv1 "/bin/bash" 3 hours ago Up 3 hours 0.0.0.0:8081->80/tcp devweb1

After confirming that both websites are accessible, I now proceed with setting up nginx and setting up the load balancer. The original guide instructs me to create a default.conf under /etc/nginx/sites-available/ then start nginx

[root@bekzilla]# yum install -y nginx
[root@bekzilla]# vim /etc/nginx/sites-available/default.conf
#define back end (docker containers)
upstream containerapp {
   server bekzilla:8081; #devweb1
   server bekzilla:8082; #devweb2
}

server {
   listen *:80;
   server_name bekzilla; #localip
   index index.html index.htm index.php

   access_log /var/log/localweb.log
   error_log /var/log/nginx/localerr.log

   location / {
      proxy_pass http://containerapp;
   }
}

[root@bekzilla]# systemctl start nginx

Note: I changed my ip to bekzilla instead.

This should have brought me to one of my dummy sites, unfortunately, it didn’t. It gave me nginx’s default website instead

I searched hi and lo for nginx load balancing guide, tested several guides and finally foundthis configure nginx load balancer guide page from linuxhelp. So it turns out that I only need to modify /etc/nginx/nginx.conf, comment out the existing configs and put the above config there. After restarting nginx, I met this page instead

Ugly… So, nginx is working, its trying to redirect requests to my containers, but something isn’t working. I checked the error log and found this

[root@bekzilla]# tail /var/log/nginx/error.log
2018/09/15 08:02:23 [crit] 18544#0: *1 connect() to bekzilla:8082 failed (13: Permission denied) while connecting to upstream, client: 192.168.0.10, server: bekzilla, request: "GET / HTTP/1.1", upstream: "http://bekzilla:8082/", host: "bekzilla"

Something seems to blocked. My first thought was selinux as I’m on CentOS7 and it’s not allowing port 8081 or 8082.

[root@bekzilla]# semanage port -l | grep 8081 #check port 8081
transproxy_port_t tcp 8081
[root@bekzilla]# semanage port -l | grep 8082 #check port 8082
us_cli_port_t tcp 8082, 8083
us_cli_port_t udp 8082, 8083

So it turns out those two ports are assigned for something else. Lets see what’s already assigned to http then

[root@bekzilla]# semanage port -l | grep http_port_t
http_port_t tcp 80, 81, 443, 488, 8008, 8009, 8443, 9000

Nice, there’s already port 8008 and 8009. So lets use those two ports instead.

First edit the back end in nginx

[root@bekzilla]# vim /etc/nginx/nginx.conf
#define back end (docker containers)
  upstream containerapp {
    server bekzilla:8008; #devweb1
    server bekzilla:8009; #devweb2
  }

Then stop and delete the containers and recreate the containers with the new ports

[root@bekzilla]# docker stop devweb1
[root@bekzilla]# docker stop devweb2
[root@bekzilla]# docker rm devweb1
[root@bekzilla]# docker rm devweb2
[root@bekzilla]# docker run -itd --name=devweb1 -p 8008:80 -v /docker/dockerwww/:/var/www/html centos6:finalwebv1 /bin/bash
[root@bekzilla]# docker run -itd --name=devweb2 -p 8009:80 -v /docker/dockerwww/:/var/www/html centos6:finalwebv1 /bin/bash

Verify that the container is running

[root@bekzilla]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1a9e37559cd8 centos6:finalwebv1 "/bin/bash" 7 seconds ago Up 5 seconds 0.0.0.0:8009->80/tcp devweb2
cca174d25b9c centos6:finalwebv1 "/bin/bash" 18 seconds ago Up 16 seconds 0.0.0.0:8008->80/tcp devweb1

Then tried again by accessing http://bekzilla

Awesome, no more error 502 and I could reach the websites and I could turn off any of the containers and still access the page.