Speed up administration of Kubernetes clusters with k9s

Usually, my articles about Kubernetes administration are full of kubectl commands for administration for your clusters. Recently, however, someone pointed me to the k9s project for a fast way to review and resolve day-to-day issues in Kubernetes. It's been a huge improvement to my workflow and I'll show you how to get started in this tutorial. 

Installation can be done on a Mac, in Windows, and Linux. Instructions for each operating system can be found here. Be sure to complete installation to be able to follow along.

I will be using Linux and Minikube, which is a lightweight way to run Kubernetes on a personal computer. Install it following this tutorial or by using the documentation.

Setting the k9s configuration file

Once you've installed the k9s app, it's always good to start with the help command.

$ k9s help
K9s is a CLI to view and manage your Kubernetes clusters.

Usage:
  k9s [flags]
  k9s [command]

Available Commands:
  help        Help about any command
  info        Print configuration info
  version     Print version/build info

Flags:
  -A, --all-namespaces                 Launch K9s in all namespaces
      --as string                      Username to impersonate for the operation
      --as-group stringArray           Group to impersonate for the operation
      --certificate-authority string   Path to a cert file for the certificate authority
      --client-certificate string      Path to a client certificate file for TLS
      --client-key string              Path to a client key file for TLS
      --cluster string                 The name of the kubeconfig cluster to use
  -c, --command string                 Specify the default command to view when the application launches
      --context string                 The name of the kubeconfig context to use
      --demo                           Enable demo mode to show keyboard commands
      --headless                       Turn K9s header off
  -h, --help                           help for k9s
      --insecure-skip-tls-verify       If true, the server's caCertFile will not be checked for validity
      --kubeconfig string              Path to the kubeconfig file to use for CLI requests
  -l, --logLevel string                Specify a log level (info, warn, debug, error, fatal, panic, trace) (default "info")
  -n, --namespace string               If present, the namespace scope for this CLI request
      --readonly                       Disable all commands that modify the cluster
  -r, --refresh int                    Specify the default refresh rate as an integer (sec) (default 2)
      --request-timeout string         The length of time to wait before giving up on a single server request
      --token string                   Bearer token for authentication to the API server
      --user string                    The name of the kubeconfig user to use

Use "k9s [command] --help" for more information about a command.

As you can see, there is a lot of functionality we can configure with k9s. The only step we need to take place to get off the ground is to write a configuration file. The info command will point us to where the application is looking for it.

$ k9s info
 ____  __.________
|    |/ _/   __   \______
|      < \____    /  ___/
|    |  \   /    /\___ \
|____|__ \ /____//____  >
        \/            \/

Configuration:   /Users/jess/.k9s/config.yml
Logs:            /var/folders/5l/c1y1gcw97szdywgf9rk1100m0000gn/T/k9s-jess.log
Screen Dumps:    /var/folders/5l/c1y1gcw97szdywgf9rk1100m0000gn/T/k9s-screens-jess

To add a file, make the directory if it doesn't already exist and then add one.

$ mkdir -p ~/.k9s/
$ touch ~/.k9s/config.yml

For this introduction, we will use the default config.yml recommendations from the k9s repository. The maintainers note that this format is subject to change, so we can check here for the latest version.

k9s:
  refreshRate: 2
  headless: false
  readOnly: false
  noIcons: false
  logger:
    tail: 200
    buffer: 500
    sinceSeconds: 300
    fullScreenLogs: false
    textWrap: false
    showTime: false
  currentContext: minikube
  currentCluster: minikube
  clusters:
    minikube:
      namespace:
        active: ""
        favorites:
        - all
        - kube-system
        - default
      view:
        active: dp
  thresholds:
    cpu:
      critical: 90
      warn: 70
    memory:
      critical: 90
      warn: 70

We set k9s to look for a local minikube configuration, so I'm going to confirm minikube is online and ready to go. 

$ minikube status
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

Running k9s to explore a Kubernetes cluster

With a configuration file set and pointing at our local cluster, we can now run the k9s command.

$ k9s

Once you start it up, the k9s text-based user interface (UI) will pop up. With no flag for a namespace, it will show you the pods in the default namespace.

If you run in an environment with a lot of pods, the default view can be overwhelming. Alternatively, we can focus on a given namespace. Exit the application and run k9s -n <namespace> where <namespace> is an existing namespace. In the picture below, I ran k9s -n minecraft, and it shows my broken pod

So once you have k9s up and running, there are a bunch of things you can do quickly. 

Navigating k9s happens through shortcut keys. We can always use arrow keys and the enter key to choose items listed. There are quite a few other universal keystrokes to navigate to different views:

• 0—Show all pods in all namespaces

• d—Describe the selected pod

• l—Show logs for the selected pod pod

You may notice that k9s is set to use Vim command keys, including moving up and down using J and K keys. Good luck exiting, emacs users :)

Viewing different Kubernetes resources quickly

Need to get to something that's not a pod? Yea I do too. There are a number of shortcuts that are available when we enter a colon (":") key. From there, you can use the following commands to navigate around there.

• :svc—Jump to a services view.

• :deploy—Jump to a deployment view.

• :rb—Jump to a Rolebindings view for role-based access control (RBAC) management.

• :namespace—Jump back to the namespaces view.

• :cj—Jump to the cronjobs view to see what jobs are scheduled in the cluster.

The most used tool for this application will be the keyboard; to go up or down on any page, use the arrow keys. If you need to quit, remember to use Vim keybindings. Type :q and hit enter to leave.

Example of troubleshooting Kubernetes with k9s

How does k9s help when something goes wrong? To walk through an example, I let several pods die due to misconfiguration. Below you can see my terrible hello deployment that's crashing. Once we highlight it, we press d to run a describe command to see what is causing the failure.

Skimming the events does not tell us a reason for the failure. Next, I hit the esc key and go check the logs by highlighting the pod and entering <shift-l>.

Unfortunately, the logs don't offer anything helpful either (probably because the deployment was never correctly configured), and the pod will not come up.

I then esc to step back out, and I will see if deleting the pod will take care of this issue. To do so, I highlight the pod and use <ctrl-d>. Thankfully, k9s prompts users before deletion. 

While I did delete the pod, the deployment resource still exists, so a new pod will come back up. It will also continue to restart and crash for whatever reason (we don't know yet).

Here is the point where I would repeat reviewing logs, describing resources, and use the e shortcut to even edit a running pod to troubleshoot the behavior. In this particular case, the failing pod is not configured to run in this environment. So let's delete the deployment to stop crash-then-reboot loop we are in.

We can get to deployments by typing :deploy and clicking enter. From there we highlight and press <ctrl-d> to delete.

And poof the deployment is gone! It only took a few keystrokes to clean up this failed deployment.

k9s is incredibly customizable

So this application has a ton of customization options, down to the color scheme of the UI. Here are a few editable options you may be interested in:

• Adjust where you put the config.yml file (so you can store it in version control)
• Add custom aliases to an alias.yml file
• Create custom hotkeys in a hotkey.yml file
• Explore available plugins or write your own

The entire application is configured in YAML files, so customization will feel familiar to any Kubernetes administrator.

Simplify your life with k9s

I'm prone to administrating over my team's systems in a very manual way, more for brain training than anything else. When I first heard about k9s, I thought, "This is just lazy Kubernetes," so I dismissed it and went back to doing my manual intervention everywhere. I actually started using it daily while working through my backlog, and I was blown away at how much faster it was to use than kubectl alone. Now I'm a convert. 

It's important to know your tools and master the "hard way" of doing something. It is also important to remember, as far as administration goes, it's important to work smarter, not harder. Using k9s is the way I live up to that objective. I guess we can call it lazy Kubernetes administration, and that's okay.