The first thing I wanted to do upon setting up my Kubernetes cluster at home was to configure monitoring for it with Prometheus. Since I didn't really have any pods spun up at the time, I turned to what I did have -- this blog.

This blog runs on the Ghost blogging engine, which is served behind an Nginx reverse proxy. Nginx has a built in "stub_status" page, but it gives you basically no info. In order to get decent in-built monitoring/metrics support, you are directed to purchase NGINX Plus, which I'm obviously not the target audience for.

Luckily, there's a Github project to fill the void. The nginx-module-vts is available as an Nginx module that will provide host traffic statistics. However, it does involve compiling Nginx from source.

Instructions

Note: This guide is for Ubuntu systems and is a conglomeration of my experience and various resources I used, but the most useful one is located here if you want more info.

  1. First thing's first, you'll want to set up your workspace. So go ahead and make a directory that you can work out of. For the purposes of this demonstration, it will be ~/rebuildingnginx
    •   mkdir ~/rebuildingnginx
      
  2. cd to that directory and clone the VTS module from Github
    •   cd ~/rebuildingnginx
        git clone git://github.com/vozlt/nginx-module-vts.git
      
  3. Add the Nginx repository, if you haven't already
    •   sudo add-apt-repository -y ppa:nginx/stable
      
  4. Make sure that deb-src isn't commented out in the respository file
    •   cat /etc/apt/sources.list.d/nginx-ubuntu-stable-xenial.list
        
      deb http://ppa.launchpad.net/nginx/stable/ubuntu xenial main
      deb-src http://ppa.launchpad.net/nginx/stable/ubuntu xenial main
      
  5. Do package manager things
    •   # Update package lists
        sudo apt-get update
        
        # Install dpkg-dev to create package
        sudo apt-get install -y dpkg-dev
      
        # Get Nginx source files
        sudo apt-get source nginx
      
        # Install the build dependencies for nginx
        sudo apt-get build-dep nginx
      
  6. Edit the rules file in the source code
    •   vim ~/rebuildnginx/nginx-1.14.0/debian/rules
      
    • Note: the exact path may differ depending on what version of Nginx source your downloaded
    • Since we're going to install the nginx-full version, we're going to append the build flag to the full_configure_flags section of the file
    • Go ahead and add an --add-module=/root/nginx-module-vts to the end of that list of arguments
    • It should look something like:
      full_configure_flags := \
                         $(common_configure_flags) \
                         --with-http_addition_module \
                         --with-http_geoip_module=dynamic \
                         --with-http_gunzip_module \
                         --with-http_gzip_static_module \
                         --with-http_image_filter_module=dynamic \
                         --with-http_sub_module \
                         --with-http_xslt_module=dynamic \
                         --with-stream=dynamic \
                         --with-stream_ssl_module \
                         --with-stream_ssl_preread_module \
                         --with-mail=dynamic \
                         --with-mail_ssl_module \
                         --add-dynamic-module=$(MODULESDIR)/http-auth-pam \
                         --add-dynamic-module=$(MODULESDIR)/http-dav-ext \
                         --add-dynamic-module=$(MODULESDIR)/http-echo \
                         --add-dynamic-module=$(MODULESDIR)/http-upstream-fair \
                         --add-dynamic-module=$(MODULESDIR)/http-subs-filter \
                         --add-module=/root/nginx-module-vts
      
  7. Now we can start the actual build process
    •   # Again, this path may differ depending on your version
        cd ~/rebuildingnginx/nginx-1.14.0
        sudo dpkg-buildpackage -b
      
    • This takes a LONG time so go find something fun to do and come back later
  8. Now that it's built, we can actually install Nginx
    • The build process put a bunch of .deb files in our ~/rebuildingnginx directory, but the only one we need to care about is nginx-full_1.14.0-0+xenial1_amd64.deb or whatever the equivalent is for you depending on your Ubuntu & Nginx versions.
    •   cd ~/rebuildingnginx
        sudo dpkg --install nginx-full_1.14.0-0+xenial1_amd64.deb
      
  9. Nginx should be installed now! Now we need to configure the VTS module for our metrics
  10. Open your Nginx configuration file for some light editing
    •   sudo vim /etc/nginx/nginx.conf
      
    • I've ellipsed the parts of the configuration that aren't important to this with [...], but your configuration file should be updated to include the following info:
    •   user www-data;
        worker_processes auto;
        pid /run/nginx.pid;
        include /etc/nginx/modules-enabled/*.conf;
      
        events {
            worker_connections 768;
            # multi_accept on;
        }
      
        http {
      
            [...]
      
            ##
            # VTS Settings
            ##
            vhost_traffic_status_zone;
            vhost_traffic_status_dump /var/log/nginx/vts.db;
      
            server {
              listen 8080;
                  server_name wbhegedus.me;
      
              if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})") {
                set $year $1;
                set $month $2;
                set $day $3;
              }
      
              vhost_traffic_status_filter_by_set_key $year year::$server_name;
              vhost_traffic_status_filter_by_set_key $year-$month month::$server_name;
              vhost_traffic_status_filter_by_set_key $year-$month-$day day::$server_name;
      
              location /status {
                vhost_traffic_status_bypass_limit on;
                vhost_traffic_status_bypass_stats on;
                vhost_traffic_status_display;
                vhost_traffic_status_display_format html;
              }
            }	
           [...]
        }
      
  11. Once this is complete, you can reload your Nginx config
    •    # Check for a valid config first
         nginx -t
         
         # If that command returns fine, go ahead and reload
         sudo systemctl reload nginx
      
  12. If everything worked correctly, you should be able to access Prometheus-formatted metrics at localhost:8080/status/format/prometheus on your Nginx box
    • You can add this target to your Prometheus config the same as you would any other endpoint.
    • For example:
    - job_name: ghost_blog
     scheme: http
     metrics_path: /status/format/prometheus
     static_configs:
     - targets:
       - wbhegedus.me:8080
    

Enjoy the wonder of Prometheus and let me know in the comments if you run into any issues!