By using the unless operator, you can return a set of labels only when a different matching metric does not exist. For example,

  group without (instance) (up{job="blackbox_http_2xx"})
unless
  count without (instance) (probe_http_status_code{job="blackbox_http_2xx"} == 200)

Only if there are no HTTP 200s for the label set that results from the group query will the alert would fire. The alert would fire with a label set that looks similar to this in my environment:

{job="blackbox_http_2xx", environment="production", cluster="clusterA", service="website"}

Having these extra labels can be extremely useful in your Alertmanager routing configs and any templating you do, which is why I strive to keep as many labels as possible when designing alerting rules.

One of the use cases I have for Vector is to have it aggregate multiple Prometheus exporters on a single host and expose them all under a single port/endpoint. Previously, I used a reverse proxy for this, which had its uses but was also overkill. However, it did provide me with the benefit of putting my exporters behind HTTPS, which I did not know was possible to do with Vector.

Vector's docs for the Prometheus Exporter sink do not mention it at my time of writing this, but Vector actually does support listening for SSL connections on the Prometheus exporter it exposes. To do so, simply add a tls object to your sink's configuration. Example below:

Thanos supports a variety of storage backends, including S3. Linode offers an object storage service with an S3-compatible API, and it's easy to get started with it and begin sending data from Thanos/Prometheus into it.

This presumes that you already set up a Thanos Sidecar to run alongside your Prometheus deployment and it has access to the directory in which the TSDB writes its blocks. Setting that up is outside of the scope of this post, but it is simple to get up and running.

On the Linode side, first login to your account and sign up for the Object Storage service. Please note that this comes at a flat rate of $5 in order to reserve your minimum storage space of 250GB. You can get signed up just by creating your first bucket.

After your bucket is created, at the home page for Object Storage, click on the "Access Keys" tab and create an Access Key. I recommend limiting access to just the bucket you just created, but that is up to you. Thanos Sidecar will need read and write in order to do its job.

Once that Access Key is created, make sure you save the secret key and the access key – we'll need to put them in a config file for Thanos Sidecar.

Go ahead and open your text editor of choice and create a file named something like linode.yml with the contents below. Modify endpoint based on your datacenter – inferable from the link displayed under your bucket name in the GUI. This is the least intuitive part – the URL presented is a full path directly to the bucket (e.g. thanos-snap.us-east-1.linodeobjects.com), but Thanos won't like that. So just strip out the bucket name from the URL for the endpoint and modify bucket based on your bucket name.

type: S3
config:
  bucket: "thanos-snap"
  endpoint: "us-east-1.linodeobjects.com"
  access_key: "YOUR_ACCESS_KEY_HERE"
  insecure: false
  secret_key: "YOUR_SECRET_KEY_HERE"
  signature_version2: false

Now you can point your Thanos Sidecar to this configuration file by adding the --objstore.config-file=linode.yml flag, and it should automatically start uploading TSDB blocks.

One last thing to note is to make sure that you followed the instructions in setting up the Thanos Sidecar so that Prometheus doesn't compact the blocks – you want another Thanos component, Thanos Compactor to do the compaction. The good news, though, is that you can reuse that linode.yml file when specifying the --objstore.config-file flag for Thanos Compactor (and any other components that connect to block storage).

The commands for the below presume that you are running Ubuntu 20.04 or higher, but the WSL2 specific configuration at the end is independent of which Linux distro you are using.

You can find the specific installation dependencies and commands for your distro in the Podman docs.

Basic Install

Here are the commands I used in order to install Podman on Ubuntu.

First, you must determine the version of Ubuntu you're running, if you don't already know. Then you export that to an environment variable, which I named VERSION_ID in order to be able to use that when adding repos from the Kubic project, which was necessary for my specific OS version.

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"
echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/Release.key | sudo apt-key add -
export VERSION_ID="20.04"
sudo apt-get update
sudo apt-get -y upgrade
sudo apt-get -y install podman

WSL2 Specifics

OK - now comes the WSL2 specific stuff that you need to change in order for Podman to work. Basically, you just have to change the systemd-specific stuff (default) to non-systemd stuff.

With version 3+ of Podman, this can all be done in one file.

First, make the requisite directory. Mine was not automatically created, but YMMV. Then, create a containers.conf file in that directory.

mkdir ~/.config/containers
vim ~/.config/containers/containers.conf

Inside that file, simply add the following:

[engine]
events_logger="file"
cgroup_manager="cgroupfs"

The default events_logger is journald and the default cgroup_manager is systemd, in case you were curious.

Now Podman should run for you with no issues, and you don't need to run that clunky Docker daemon in Windows anymore 🎉

When I first set up Grafana, I was unable to get nested group membership working but didn't care enough at the time to troubleshoot much – after all, only myself and a few others would be using it. Now that I've gone back and actually figured out how it works, I want to share what I've learned.

Note: these instructions presume that you are using Active Directory, but the concepts should be transferable to other LDAP providers.

Presumably, you've already configured your Grafana environment to use LDAP as your authentication provider with this bit in your configuration file:

[auth.ldap]
enabled = true
config_file = /etc/grafana/ldap.toml

The instructions on Grafana's site do a good job of getting you up and running with what you'll need in that /etc/grafana/ldap.toml file. However, we need to expand upon that by not just getting what groups a user is a member of, but also what groups those groups are a member of (nested group membership).

To do this, you need to specify a group_search_filter in addition to your plain search_filter. This is in supplement to search_filter, and not a replacement for it – it's required. Your group_search_filter is an LDAP query that essentially tells AD to find all groups that a user is a member of within the group_search_base_dns.

This group_search_filter looks like:

(member:1.2.840.113556.1.4.1941:=CN=%s,OU=Users,OU=FSA,DC=example,DC=domain,DC=com)

Those random looking numbers are an important OID to enable LDAP_MATCHING_RULE_IN_CHAIN, which is what lets us find nested group memberships, too.

When it's configured, it should look like:

## Group search filter, to retrieve the groups of which the user is a member (only set if memberOf attribute is not available)
group_search_filter = "(member:1.2.840.113556.1.4.1941:=CN=%s,OU=Users,OU=FSA,DC=example,DC=domain,DC=com)"
## An array of the base DNs to search through for groups. Typically uses ou=groups
group_search_base_dns = ["OU=Admin Groups,OU=Security Groups,OU=FSA,DC=example,DC=domain,DC=com"]
group_search_filter_user_attribute = "sAMAccountName"

As a bonus tip when configuring this, make your group_search_base_dns as specific as possible, because that is what Grafana is going to loop through looking for your group memberships. For example, it was taking 15-20s to log me in when I used "Security Groups" as my base OU, but when I got more specific to use the "Admin Groups" OU (which is where my "Grafana Admin", "Grafana Viewer", and "Grafana Editor" groups are), the results were nearly instant.

This particular solution uses Drone, as it is my CI/CD tool of choice, but you can use the same logic in Jenkins, Gitlab-CI, or whatever you utilize for your builds/deployments. I also use JSON as my file-format for file_sd but you can apply the same concepts if you use YAML.

First, we need a way to actually define what the file should look like. To do this, I chose to use JSON schemas. You can pop on over to jsonschema.net, paste in one of your valid JSON target files, and infer a schema based off of it. Personally, I changed it so that env, team, and service labels are required on targets in order to enforce uniformity in our targets regardless of what team is adding them. Go ahead and save that schema as schema.json in a drone folder in the top level of your repository.

With that schema in hand, now we need a way to compare our JSON files to it. To do this, I wrote a simple Python script:

from jsonschema import validate
from jsonschema.exceptions import ValidationError
from json.decoder import JSONDecodeError

import glob
import json

import sys

errors: {str: str} = {}
error_encountered = False

with open("drone/schema.json") as f:
    schema = json.load(f)
    f.close()

for f in glob.glob("files/targets/*.json"):
    with open(f) as inst:
        try:
            validate(json.load(inst), schema)
        except ValidationError as e:
            error_encountered = True
            errors[inst.name] = f"Schema error: {e.message}"
        except JSONDecodeError as e:
            error_encountered = True
            errors[inst.name] = f"JSON decoding error: {e.msg} on line {e.lineno}"

for f, err in errors.items():
    if err:
        print(f"[ ERROR in {f} ]")
        print("  |>", err)
        print()

if error_encountered:
    sys.exit(1)

print("All tests passed! Have a cookie. 🍪")

The script presumes the location of the schema file (drone/schema.json) and the target file(s) (files/targets/*.json) but you can adapt that as needed.

Now in your .drone.yml just add a step to your pipeline to run the validation script (and install its dependency):

  - name: validate JSON of targets
    image: python:3-alpine
    commands:
      - pip3 install --quiet jsonschema
      - python3 drone/validate.py

Easy peasy. Now you'll never have another malformed target file again! (err... at least not one that makes it to prod).

Personally, it took me a bit to wrap my head around it, which is why I'm writing this — in hopes of aiding someone else down the line.

I was working on a problem that required me to take a list of items and filter them by removing any items that matched a particular regex, then return the filtered list. Several hours of debugging and brainstorming later, I finally figured out how to do it right.

In my particular case, I did not care about the order of the elements in the slice so I could rearrange them any which way.

First, we start with the function declaration:

func filterExcluded(tl *[]task, excluded []string) []task {
	filtered := *tl
	lastIndex := (len(filtered) - 1)

The function accepts a pointer to a slice of task-typed objects, along with a slice of strings that will be interpreted as regex matchers, it then returns a slice of task-typed objects.

The filtered variable is a copy of all the items in the tl argument, which we will whittle down later. The lastIndex variable will be updated repeatedly later on and we'll use it to return a subset of data from the filtered slice.

Next, we loop over the regex matchers and make them *regexp.Regexp objects. Nothing fancy.

We'll also created a matched boolean that we'll use to log an error if it matches nothing, so that end users can make sure the regex is actually right and isn't just wasting compute time.

    for _, regexString := range excluded {
		matched := false
		regex, err := regexp.Compile(regexString)
		if err != nil {
			logrus.Error(err)
			continue
		}

Now we're gonna go 0 to 100 real quick, but don't panic — I've commented everything in-line so you can see what's happening:

		// Do not increment index on each loop.
		// Handle incrementing inside of the loop.
		for index := 0; index <= lastIndex; {

			taskName := &filtered[index].Name

			if regex.Match([]byte(*taskName)) {
				logrus.WithFields(logrus.Fields{
					"task_name": *taskName,
					"regex":     regexString,
				}).Debug("Excluding task")

				// Hooray! The provided regex matched something and wasn't a waste of time.
				matched = true

				// Replace the value at the current index with the value at
				// the last index of the slice.
				filtered[index] = filtered[lastIndex]

				// Set the value of the last index of the slice
				// to the nil value of `task`. The value that was previously
				// there is now at filtered[index], so we did not lose it.
				// We will just NOT increment `index` so that the new value gets checked, too.
				filtered[lastIndex] = task{}

				// Set the `filtered` slice to be everything up to
				// the last index, which we just set to a nil value.
				filtered = filtered[:lastIndex]

				// The last index will now be one less than before.
				// This is the same as if we just did
				// lastIndex = len(filtered)
				// everytime, except this should be slightly more performant.
				lastIndex--
			} else {
				// If no match was found, increment the index
				// so that we check the next value in the `filtered` slice
				index++
				logrus.WithFields(logrus.Fields{
					"task_name": *taskName,
					"regex":     regexString,
				}).Debug("Did not match regex")
			}
		}
		// Log an error if the regex didn't match any tasks.
		// This should warn users if they're providing a useless regex.
		if !matched {
			logrus.Error("No task found to exclude matching: ", regexString)
		}
	}

Now you just need to return the filtered list and you're golden.

	return filtered

Ta-Da! We just filtered a slice while incurring only a slight amount of pain.

TL;DR — Loop over slice; only increment index on non-matches; on match, set slice[i] = last item in slice; set last item in slice to nil (helps w/ GC); set slice = slice[:lastItem];  profit.

Further Reading

• SliceTricks (Golang Github wiki)

Bonus

Putting it all together, it looks like this:

func filterExcluded(tl *[]task, excluded []string) []task {
	filtered := *tl
	lastIndex := (len(filtered) - 1)

	for _, regexString := range excluded {
		matched := false
		regex, err := regexp.Compile(regexString)
		if err != nil {
			logrus.Error(err)
			continue
		}

		// Do not increment index on each loop.
		// Handle incrementing inside of the loop.
		for index := 0; index <= lastIndex; {

			taskName := &filtered[index].Name

			if regex.Match([]byte(*taskName)) {
				logrus.WithFields(logrus.Fields{
					"task_name": *taskName,
					"regex":     regexString,
				}).Debug("Excluding task")

				// Hooray! The provided regex matched something and wasn't a waste of time.
				matched = true

				// Replace the value at the current index with the value at
				// the last index of the slice.
				filtered[index] = filtered[lastIndex]

				// Set the value of the last index of the slice
				// to the nil value of `task`. The value that was previously
				// there is now at filtered[index], so we did not lose it.
				// We will just NOT increment `index` so that the
				// new value will get checked, too.
				filtered[lastIndex] = task{}

				// Set the `filtered` slice to be everything up to
				// the last index, which we just set to a nil value.
				filtered = filtered[:lastIndex]

				// The last index will now be one less than before.
				// This is the same as if we just did
				// lastIndex = len(filtered)
				// everytime, except this should be slightly more performant.
				lastIndex--
			} else {
				// If no match was found, increment the index
				// so that we check the next value in the `filtered` slice
				index++
				logrus.WithFields(logrus.Fields{
					"task_name": *taskName,
					"regex":     regexString,
				}).Debug("Did not match regex")
			}
		}
		// Log an error if the regex didn't match any tasks.
		// This should warn users if they're providing a useless regex.
		if !matched {
			logrus.Error("No task found to exclude matching: ", regexString)
		}
	}
	return filtered
}

Background

The service did not show up in the "Scheduled Downtime" page in the XI UI, nor did it show up in the nagios_scheduleddowntime table in our MariaDB instance. Yet, the service page still showed it as being in downtime.

According to the comment history on the service, the downtime was scheduled until 2042, so we couldn't really just wait and hope it expired.

This service has been scheduled for fixed downtime from 2019-04-30 00:39:16 to 2042-02-21 07:39:16.

Note: Before I talk about how the problem was resolved, if you're just here wondering how to remove scheduled downtime then you can do so in the aforementioned "Scheduled Downtime" page. It's at nagiosxi/includes/components/xicore/downtime.php in the XI UI, and under the "System -> Downtime" page in Nagios Core.

Technical Info

The issue that presented itself to us was that the entry in the nagios_servicestatus table had incorrect values in the acknowledgement_type and scheduled_downtime_depth columns.

The acknowledgement_type column can either 0, 1, or 2 which represent None, Normal, or Sticky, respectively.

The scheduled_downtime_depth column can be any smallint number, and it represents the number of downtimes that the service is in (since a service can be in multiple levels of downtime).

Fix

After determining the object ID of the service and verifying that no entries corresponding to it existed in nagios_scheduleddowntime, I used the following SQL to fix the service's status.

-- Fix acknowledgement_type, replace service_object_id with the appropriate ID
update nagios.nagios_servicestatus set acknowledgement_type='0' where service_object_id='8540';

-- Fix scheduled_downtime_depth, replace service_object_id with the appropriate ID
update nagios.nagios_servicestatus set scheduled_downtime_depth='0' where service_object_id='8540';

After doing this, you can also optionally remove the comment history for the downtime from the nagios_commenthistory table.

-- replace object_id and commenthistory_id with appropriate values.
delete from nagios.nagios_commenthistory where object_id = '8540' and commenthistory_id='2461761';

I use Selenium at work to run test cases that are called by Nagios checks. Since Nagios needs a response from a check within 60s, having predictable wait times is imperative. Rather than using the implicit waits for everything, I prefer to set explicit waits depending on what it is that I'm waiting on.

The default value in Selenium for an implicit wait is 0 seconds. This means that running driver.find_element_by_id("main-box") will fail if the element isn't there, even if the page hasn't fully loaded yet.

I solve this by using WebDriverWait and expected_conditions.

from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

# [...] (code ommitted)

WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "application")),
                                "Page did not load as expected")

This code calls WebDriverWait on the driver I set up, it waits a maximum of 10s or until the expected condition (EC) is met. If the EC is not met, then it will raise a TimeoutException with "Page did not load as expected" as the error message.

Not only does WebDriverWait allow me to explicitly set how long to wait for something (e.g. the presence of an element on a page), but it also allows me to set the error message to something meaningful that will help our on-shift team identify where the issue actually is – all without having to add an unnecessary try-catch block.

Explicitly specifying how long I want to wait for an element to load or until a page's title changes allows me to better guarantee that a site is actually broken when our test cases fail – not that it just happened to load slower than expected.

Note: code examples are in Python, but the same functionality is available in Java and other Selenium libraries.

In my Prometheus set up, most of my scrape targets come from a file_sd service discovery set up. This poses a challenge of deduplicating ICMP probes from the Blackbox Exporter for hosts with multiple scrape endpoints.

Most of my file_sd targets have a JSON files that looks like this:

[
  {
    "targets": [ "foo.example.com:9100","bar.example.com:9100" ],
    "labels": {
      "env": "prod",
      "job": "node",
      "service": "foobar"
    }
  },
  {
    "targets": [ "foo.example.com:9104" ],
    "labels": {
      "env": "prod",
      "job": "mysql",
      "service": "barfoo"
    }
  }
]

When setting up the Blackbox Exporter in a "standard" way, it uses relabel_configs to take the target's URL (including the port) and put that into the __param_target label. For example:

  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target

This means that foo.example.com would get scraped twice because it has two different __address__ labels (one on port 9100 and one on port 9104).

To prevent this, we can employ the use of regex to remove the port and prevent the creation of multiple ICMP metrics for a single host. Simply update the part of the relabel_configs configuration that sets the __param_target to look like this:

  relabel_configs:
    - source_labels: [__address__]
      regex: (.*?)(:[0-9]+)?
      target_label: __param_target
      replacement: ${1}

This regex will take everything up to the colon (the URL without the port) and save that in a capture group ( ${1} ), which we then use as the __param_target label.

Now you can update your config, reload Prometheus, and you'll see in the Targets page that it's no longer duplicating targets on your ICMP job.

The first thing I wanted to do upon setting up my Kubernetes cluster at home was to configure monitoring for it with Prometheus. Since I didn't really have any pods spun up at the time, I turned to what I did have -- this blog.

This blog runs on the Ghost blogging engine, which is served behind an Nginx reverse proxy. Nginx has a built in "stub_status" page, but it gives you basically no info. In order to get decent in-built monitoring/metrics support, you are directed to purchase NGINX Plus, which I'm obviously not the target audience for.

Luckily, there's a Github project to fill the void. The nginx-module-vts is available as an Nginx module that will provide host traffic statistics. However, it does involve compiling Nginx from source.

Instructions

Note: This guide is for Ubuntu systems and is a conglomeration of my experience and various resources I used, but the most useful one is located here if you want more info.

1. First thing's first, you'll want to set up your workspace. So go ahead and make a directory that you can work out of. For the purposes of this demonstration, it will be ~/rebuildingnginx

   •   mkdir ~/rebuildingnginx
     

2. cd to that directory and clone the VTS module from Github

   •   cd ~/rebuildingnginx
       git clone git://github.com/vozlt/nginx-module-vts.git
     

3. Add the Nginx repository, if you haven't already

   •   sudo add-apt-repository -y ppa:nginx/stable
     

4. Make sure that deb-src isn't commented out in the respository file

   •   cat /etc/apt/sources.list.d/nginx-ubuntu-stable-xenial.list
       
     deb http://ppa.launchpad.net/nginx/stable/ubuntu xenial main
     deb-src http://ppa.launchpad.net/nginx/stable/ubuntu xenial main
     

5. Do package manager things

   •   # Update package lists
       sudo apt-get update
       
       # Install dpkg-dev to create package
       sudo apt-get install -y dpkg-dev
     
       # Get Nginx source files
       sudo apt-get source nginx
     
       # Install the build dependencies for nginx
       sudo apt-get build-dep nginx
     

6. Edit the rules file in the source code

   •   vim ~/rebuildnginx/nginx-1.14.0/debian/rules
     

   • Note: the exact path may differ depending on what version of Nginx source your downloaded
   • Since we're going to install the nginx-full version, we're going to append the build flag to the full_configure_flags section of the file
   • Go ahead and add an --add-module=/root/nginx-module-vts to the end of that list of arguments
   • It should look something like:

     full_configure_flags := \
                        $(common_configure_flags) \
                        --with-http_addition_module \
                        --with-http_geoip_module=dynamic \
                        --with-http_gunzip_module \
                        --with-http_gzip_static_module \
                        --with-http_image_filter_module=dynamic \
                        --with-http_sub_module \
                        --with-http_xslt_module=dynamic \
                        --with-stream=dynamic \
                        --with-stream_ssl_module \
                        --with-stream_ssl_preread_module \
                        --with-mail=dynamic \
                        --with-mail_ssl_module \
                        --add-dynamic-module=$(MODULESDIR)/http-auth-pam \
                        --add-dynamic-module=$(MODULESDIR)/http-dav-ext \
                        --add-dynamic-module=$(MODULESDIR)/http-echo \
                        --add-dynamic-module=$(MODULESDIR)/http-upstream-fair \
                        --add-dynamic-module=$(MODULESDIR)/http-subs-filter \
                        --add-module=/root/nginx-module-vts
     

7. Now we can start the actual build process

   •   # Again, this path may differ depending on your version
       cd ~/rebuildingnginx/nginx-1.14.0
       sudo dpkg-buildpackage -b
     

   • This takes a LONG time so go find something fun to do and come back later
8. Now that it's built, we can actually install Nginx
   • The build process put a bunch of .deb files in our ~/rebuildingnginx directory, but the only one we need to care about is nginx-full_1.14.0-0+xenial1_amd64.deb or whatever the equivalent is for you depending on your Ubuntu & Nginx versions.

   •   cd ~/rebuildingnginx
       sudo dpkg --install nginx-full_1.14.0-0+xenial1_amd64.deb
     

9. Nginx should be installed now! Now we need to configure the VTS module for our metrics
10. Open your Nginx configuration file for some light editing

    •   sudo vim /etc/nginx/nginx.conf
      

    • I've ellipsed the parts of the configuration that aren't important to this with [...], but your configuration file should be updated to include the following info:

    •   user www-data;
        worker_processes auto;
        pid /run/nginx.pid;
        include /etc/nginx/modules-enabled/*.conf;
      
        events {
            worker_connections 768;
            # multi_accept on;
        }
      
        http {
      
            [...]
      
            ##
            # VTS Settings
            ##
            vhost_traffic_status_zone;
            vhost_traffic_status_dump /var/log/nginx/vts.db;
      
            server {
              listen 8080;
                  server_name wbhegedus.me;
      
              if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})") {
                set $year $1;
                set $month $2;
                set $day $3;
              }
      
              vhost_traffic_status_filter_by_set_key $year year::$server_name;
              vhost_traffic_status_filter_by_set_key $year-$month month::$server_name;
              vhost_traffic_status_filter_by_set_key $year-$month-$day day::$server_name;
      
              location /status {
                vhost_traffic_status_bypass_limit on;
                vhost_traffic_status_bypass_stats on;
                vhost_traffic_status_display;
                vhost_traffic_status_display_format html;
              }
            }	
           [...]
        }
      

11. Once this is complete, you can reload your Nginx config

    •    # Check for a valid config first
         nginx -t
         
         # If that command returns fine, go ahead and reload
         sudo systemctl reload nginx
      

12. If everything worked correctly, you should be able to access Prometheus-formatted metrics at localhost:8080/status/format/prometheus on your Nginx box
    • You can add this target to your Prometheus config the same as you would any other endpoint.
    • For example:

    - job_name: ghost_blog
     scheme: http
     metrics_path: /status/format/prometheus
     static_configs:
     - targets:
       - wbhegedus.me:8080
    

Enjoy the wonder of Prometheus and let me know in the comments if you run into any issues!