One of the reasons I stood up a Kubernetes cluster on Raspberry Pis in my house was because I wanted to gain savings by not running high-available, redundant infrastructure in the cloud. Kubernetes provides high availability by design. The possibilities that this capability offers are pretty awesome. Need a web server to run constantly? Build a container and throw it in the Kubernetes cluster. Need a service available all the time? Package it and ship it to the Kubernetes cluster.
Legacy systems
I have four old Raspberry Pi boxes doing various things here in the office. They do a single task fine, but I have over-extended one of the first-generation Raspberry Pis. I’d like to deprecate the older Raspberry Pis as they are no longer effective. To do that, I need to move the workloads (like a cloud migration). One thing I have no shortage of running on these Raspberry Pi boxes are cron jobs. I have a cron for almost everything I do: Monitoring, updating web apps, detecting changes in domain name configurations, etc.
k8s jobs and cron jobs
Kubernetes has the concept of jobs. To quote the official jobs documentation, "A job creates one or more pods and ensures that a specified number of them successfully terminate." If you have a pod that needs to run until completion, no matter what, a Kubernetes job is for you. Think of a job as a batch processor.
Kubernetes cron jobs are a relatively new thing. But I am ecstatic that this is a standard feature in modern Kubernetes clusters. It means that I can tell the cluster one time that I want a job to run at certain times. Since cron jobs build on top of the existing job functionality, I know that the job will be run to completion. The job will run on one of the six nodes I have in my Kubernetes cluster. Even if a pod is destroyed mid-job, it will spin up on another node and run there. High-available cron jobs have been a beast I’ve tried to slay many times. This problem is now solved, and all I have to do is implement it.
The implementation of Kubernetes cron jobs is like many other things with Kubernetes: YAML. There are a few projects to help with wrangling your YAML (ksonnet, for example), but that is a discussion for another article. For now, let’s put together a Dockerfile and a Kubernetes configuration file.
Use case
I moved my newsletter, DevOps'ish, off of Medium and onto Netlify with Hugo as a static site generator. This makes for a very fast and easy-to-manage website. But the one piece of functionality lost in the move is the ability to schedule posts. Netlify provides a build hook that will trigger builds when called. I can write the newsletter and set it to a date in the future. Hugo, by default, will not publish articles unless a build is completed after the specified date. Calling the build hook URL via curl with a cron job is a way to implement scheduled posts with Hugo on Netlify.
Dockerfile
The Dockerfile is pretty simple. Pull from alpine:latest, install curl, and run a curlcommand. But I don’t want the build hook URL exposed in the Dockerfile. Loading the URL as a variable via a Kubernetes secret is advisable. Do this so that the artifacts have no sensitive data and they can be shared publicly. Here is the Dockerfile:
FROM alpine:latest
LABEL maintainer="Chris Short <chris@chrisshort.net>"
RUN set -x \
&& apk update \
&& apk upgrade \
&& apk add --no-cache curl
ENTRYPOINT [ "/bin/sh", "-c" ]
CMD [ "/usr/bin/curl -vvv -X POST -d '' ${URL}" ]
Docker build
As my Kubernetes cluster runs on Raspberry Pi, I make sure to pull this Dockerfile down to a Raspberry Pi dev box and build it there:
docker build -t devopsish-netlify-cron .The name is whatever you want it to be. I will likely rename this netlify-curl or some other more appropriate name when I’m ready.
Docker registry
The next step is to add the image to a Docker registry. I thought about running a Docker registry in the Kubernetes cluster itself, but then I realized Google Container Registry (GCR) is a thing. Since I have a fair amount of stuff in Google Cloud, I decided to use GCR for simplicity and availability (also that whole “state inside Kubernetes” thing).
Heptio has a great guide titled Google Cloud Registry (GCR) with external Kubernetes. If you are going to use GCR with an external Kubernetes cluster, I highly recommend reading this first. Once GCR is configured, your Kubernetes cluster is configured to use GCR, and the container is built, you need to tag it for GCR:
docker tag devopsish-netlify-cron gcr.io/chrisshort-net/devopsish-netlify-cronThen push the newly tagged container image to GCR:
gcloud docker -- push gcr.io/chrisshort-net/devopsish-netlify-cron:latestKubernetes secret
As previously mentioned, the next piece will be the Kubernetes secret. There are a lot of ways to skin the k8s secret cat. Secrets can be loaded one time via command line or by applying a configuration file. I chose the configuration file method because I will save them in 1Password, then delete them. The secret file will look something like this:
apiVersion: v1
kind: Secret
metadata:
name: devopish-build-hook
type: Opaque
data:
url: [REDACTED]
The redacted URL string will be the Netlify build hook URL. As this is an opaque secret, the string will need to be base64-encoded:
echo -n "<SECRET>" | base64Once the base64 string is added to the file, apply it:
kubectl apply -f secret.ymlCron job configuration
Piecing together the Kubernetes cron job configuration file is relatively easy. 'Schedule' is a required field, and if you’re familiar with cron, it will look identical to the cron format string.
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: devopsish-netlify-cronjob
spec:
schedule: "1 2-14 * * 0-1,5-6"
jobTemplate:
spec:
template:
spec:
imagePullSecrets:
- name: gcr-secret
containers:
- name: devopsish-netlify-cronjob
image: gcr.io/chrisshort-net/devopsish-netlify-cron:latest
env:
- name: URL
valueFrom:
secretKeyRef:
name: devopish-build-hook
key: url
restartPolicy: OnFailure
One pitfall I experienced is Kubernetes uses UTC exclusively. Make sure you take that into account when you’re creating your schedule.
Here is what the Kubernetes configuration file is specifying:
- Create a cron job named devopsish-netlify-cronjob
- Schedule it to run the first minute of every hour from 0200 to 1400 UTC on Sunday, Monday, Friday, and Saturday
- Pull the image from gcr.io/chrisshort-net/devopsish-netlify-cron:latest using the provided secrets for gcr
- Set an environment based off the URL key in the secret named devopish-build-hook
- Run container on cron job schedule
Apply the configuration file, and you’re off to the races:
kubectl apply -f devopsish-netlify-cronjob.ymlConclusion
Voilà! You have built a Docker container, deployed the image to Google Container Registry, configured the Kubernetes cluster to pull images from GCR, created a secret to store the build hook, and created the cron job. If everything works okay, the following command should show an active cron job:
cshort@michiganjfrog ~> kubectl get cronjob
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
devopsish-netlify-cronjob 1 2-14 * * 0-1,5-6 False 0 8h 2d
Now go celebrate your high-availability, damn-near-guaranteed-to-run-every-time Kubernetes cron job. Congratulations!
This article was originally posted on Chris Short's blog. Reposted with permission.
2 Comments