The first thing I wanted to do upon setting up my Kubernetes cluster at home was to configure monitoring for it with Prometheus. Since I didn't really have any pods spun up at the time, I turned to what I did have -- this blog.

This blog runs on the Ghost blogging engine, which is served behind an Nginx reverse proxy. Nginx has a built in "stub_status" page, but it gives you basically no info. In order to get decent in-built monitoring/metrics support, you are directed to purchase NGINX Plus, which I'm obviously not the target audience for.

Luckily, there's a Github project to fill the void. The nginx-module-vts is available as an Nginx module that will provide host traffic statistics. However, it does involve compiling Nginx from source.

Instructions

Note: This guide is for Ubuntu systems and is a conglomeration of my experience and various resources I used, but the most useful one is located here if you want more info.

First thing's first, you'll want to set up your workspace. So go ahead and make a directory that you can work out of. For the purposes of this demonstration, it will be ~/rebuildingnginx
- ```
  mkdir ~/rebuildingnginx
```

cd to that directory and clone the VTS module from Github

  cd ~/rebuildingnginx
  git clone git://github.com/vozlt/nginx-module-vts.git

Add the Nginx repository, if you haven't already

  sudo add-apt-repository -y ppa:nginx/stable

Make sure that deb-src isn't commented out in the respository file

  cat /etc/apt/sources.list.d/nginx-ubuntu-stable-xenial.list
  
deb http://ppa.launchpad.net/nginx/stable/ubuntu xenial main
deb-src http://ppa.launchpad.net/nginx/stable/ubuntu xenial main

Do package manager things

  # Update package lists
  sudo apt-get update
  
  # Install dpkg-dev to create package
  sudo apt-get install -y dpkg-dev

  # Get Nginx source files
  sudo apt-get source nginx

  # Install the build dependencies for nginx
  sudo apt-get build-dep nginx

Edit the rules file in the source code

  vim ~/rebuildnginx/nginx-1.14.0/debian/rules

Note: the exact path may differ depending on what version of Nginx source your downloaded
Since we're going to install the nginx-full version, we're going to append the build flag to the full_configure_flags section of the file
Go ahead and add an --add-module=/root/nginx-module-vts to the end of that list of arguments

It should look something like:

full_configure_flags := \
                   $(common_configure_flags) \
                   --with-http_addition_module \
                   --with-http_geoip_module=dynamic \
                   --with-http_gunzip_module \
                   --with-http_gzip_static_module \
                   --with-http_image_filter_module=dynamic \
                   --with-http_sub_module \
                   --with-http_xslt_module=dynamic \
                   --with-stream=dynamic \
                   --with-stream_ssl_module \
                   --with-stream_ssl_preread_module \
                   --with-mail=dynamic \
                   --with-mail_ssl_module \
                   --add-dynamic-module=$(MODULESDIR)/http-auth-pam \
                   --add-dynamic-module=$(MODULESDIR)/http-dav-ext \
                   --add-dynamic-module=$(MODULESDIR)/http-echo \
                   --add-dynamic-module=$(MODULESDIR)/http-upstream-fair \
                   --add-dynamic-module=$(MODULESDIR)/http-subs-filter \
                   --add-module=/root/nginx-module-vts

Now we can start the actual build process

  # Again, this path may differ depending on your version
  cd ~/rebuildingnginx/nginx-1.14.0
  sudo dpkg-buildpackage -b

This takes a LONG time so go find something fun to do and come back later

Now that it's built, we can actually install Nginx
- The build process put a bunch of .deb files in our ~/rebuildingnginx directory, but the only one we need to care about is nginx-full_1.14.0-0+xenial1_amd64.deb or whatever the equivalent is for you depending on your Ubuntu & Nginx versions.
- ```
  cd ~/rebuildingnginx
  sudo dpkg --install nginx-full_1.14.0-0+xenial1_amd64.deb
```
Nginx should be installed now! Now we need to configure the VTS module for our metrics

Open your Nginx configuration file for some light editing

```
  sudo vim /etc/nginx/nginx.conf
```
I've ellipsed the parts of the configuration that aren't important to this with [...], but your configuration file should be updated to include the following info:

  user www-data;
  worker_processes auto;
  pid /run/nginx.pid;
  include /etc/nginx/modules-enabled/*.conf;

  events {
      worker_connections 768;
      # multi_accept on;
  }

  http {

      [...]

      ##
      # VTS Settings
      ##
      vhost_traffic_status_zone;
      vhost_traffic_status_dump /var/log/nginx/vts.db;

      server {
        listen 8080;
            server_name wbhegedus.me;

        if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})") {
          set $year $1;
          set $month $2;
          set $day $3;
        }

        vhost_traffic_status_filter_by_set_key $year year::$server_name;
        vhost_traffic_status_filter_by_set_key $year-$month month::$server_name;
        vhost_traffic_status_filter_by_set_key $year-$month-$day day::$server_name;

        location /status {
          vhost_traffic_status_bypass_limit on;
          vhost_traffic_status_bypass_stats on;
          vhost_traffic_status_display;
          vhost_traffic_status_display_format html;
        }
      }	
     [...]
  }

Once this is complete, you can reload your Nginx config

   # Check for a valid config first
   nginx -t
   
   # If that command returns fine, go ahead and reload
   sudo systemctl reload nginx

If everything worked correctly, you should be able to access Prometheus-formatted metrics at localhost:8080/status/format/prometheus on your Nginx box
- You can add this target to your Prometheus config the same as you would any other endpoint.
- For example:
```
- job_name: ghost_blog
 scheme: http
 metrics_path: /status/format/prometheus
 static_configs:
 - targets:
   - wbhegedus.me:8080
```

Enjoy the wonder of Prometheus and let me know in the comments if you run into any issues!