Thanos is an amazing tool for extending and scaling the functionality of Prometheus. One of the core features it provides is the ability to back up your TSDB data into an object storage bucket to be queried later on by the Thanos Store component.
Thanos supports a variety of storage backends, including S3. Linode offers an object storage service with an S3-compatible API, and it's easy to get started with it and begin sending data from Thanos/Prometheus into it.
This presumes that you already set up a Thanos Sidecar to run alongside your Prometheus deployment and it has access to the directory in which the TSDB writes its blocks. Setting that up is outside of the scope of this post, but it is simple to get up and running.
On the Linode side, first login to your account and sign up for the Object Storage service. Please note that this comes at a flat rate of $5 in order to reserve your minimum storage space of 250GB. You can get signed up just by creating your first bucket.
After your bucket is created, at the home page for Object Storage, click on the "Access Keys" tab and create an Access Key. I recommend limiting access to just the bucket you just created, but that is up to you. Thanos Sidecar will need read and write in order to do its job.
Once that Access Key is created, make sure you save the secret key and the access key – we'll need to put them in a config file for Thanos Sidecar.
Go ahead and open your text editor of choice and create a file named something like linode.yml with the contents below. Modify endpoint based on your datacenter – inferable from the link displayed under your bucket name in the GUI. This is the least intuitive part – the URL presented is a full path directly to the bucket (e.g. thanos-snap.us-east-1.linodeobjects.com), but Thanos won't like that. So just strip out the bucket name from the URL for the endpoint and modify bucket based on your bucket name.
type: S3
config:
bucket: "thanos-snap"
endpoint: "us-east-1.linodeobjects.com"
access_key: "YOUR_ACCESS_KEY_HERE"
insecure: false
secret_key: "YOUR_SECRET_KEY_HERE"
signature_version2: falseNow you can point your Thanos Sidecar to this configuration file by adding the --objstore.config-file=linode.yml flag, and it should automatically start uploading TSDB blocks.
One last thing to note is to make sure that you followed the instructions in setting up the Thanos Sidecar so that Prometheus doesn't compact the blocks – you want another Thanos component, Thanos Compactor to do the compaction. The good news, though, is that you can reuse that linode.yml file when specifying the --objstore.config-file flag for Thanos Compactor (and any other components that connect to block storage).