I'm a developer by nature, but I've been doing a lot of DevOps work lately, especially with Kubernetes. As part of my work, I've helped develop a tool called datree with the aim of preventing Kubernetes misconfiguration from reaching production. Ideally, it helps empower collaboration and fosters a DevOps culture in your organization for the benefit of people like me, who don't always think in DevOps.
A common scenario
The following scenario demonstrates a problem faced by many tech companies:
- At 3:46AM on a Friday, Bob wakes up to the sound of something falling onto his bedroom floor. It's his phone, showing 15 missed calls from work.
- Apparently, Bob had forgotten to add a memory limit in a deployment, which caused a memory leak in one of the containers, which led all Kubernetes nodes to run out of memory.
- He's supremely embarrassed about this, especially because the DevOps team had put so much effort into educating developers like him about Kubernetes and the importance of a memory limit.
How could this happen? Well, imagine that Bob works at Unicorn Rentals. Like many companies, they started as a tiny founding team of two developers, a CEO, and a CTO. Things were slow at first, but eventually everybody wanted to rent a unicorn, and when that happened, the company couldn't afford production outages.
A series of accidents like the one that woke Bob up at 3:46AM led the company to realize that something had to change.
If that mirrors scenarios in your own organization, then it could be that something needs to change for you, too.
The problem: scaling security policies
To avoid uncomfortable development issues and significant bugs in production, you need to educate your developers. They need to know about Kubernetes, how it works, how to develop it, and what they can do with it.
You also need to define policies so that if a resource doesn't match certain specifications on time, it doesn't enter the cluster. But what happens when there are hundreds of repos? How are those policies managed at scale? How can procedures be monitored and reviewed?
Datree is an open source command-line solution that enables Kubernetes admins to create policies and best practices they want the team to follow.
Datree allows admins to:
- Enforce policy restrictions on development: Enforce restrictions before applying resources to the cluster.
- Enable restrictions management: Flexible management of restrictions in a dedicated place across the entire organization empowers administrators to control their systems fully.
- Educate about best practices: Liberate DevOps from the constant need to review, fence, and future-proof all possible pitfalls on all current and future use cases which are part of the self-deployment.
Why Datree?
Datree aims to help admins gain maximum production stability with minimum time and effort by enforcing policies before misconfigured resources reach production.
- Education and best practices insurance: The CLI application simplifies Kubernetes deployment experience, so developers don't need to remember any rules governing development. DevOps developers are no longer forming a bottleneck. Datree's CLI application comes with Kubernetes best practices built-in, so there's no need to rely on human observation and memory.
- Enforcement on development: Developers are alerted early, as soon as a misconfiguration occurs in the PR. This way, they can catch mistakes before their code moves to production/collaborative environments.
- DevOps culture: Datree provides a mechanism similar to other development tools like unit tests. This makes it easier for developers because they are already used to these tools. Testing is the most common activity that developers carry out. Using familiar tools can be a great foundation for cultivating a DevOps culture.
How Datree works
The datree command runs automatic checks on every resource that exists in a given path. These automatic checks include three main validation types:
- YAML validation
- Kubernetes schema validation
- Kubernetes policies validations
$ datree test ~/.datree/k8s-demo.yaml >> File: .datree/k8s-demo.yaml
[V] YAML validation
[V] Kubernetes schema validation
[X] Policy check
X Ensure each container image has a pinned (tag) version [1 occurrence]
- metadata.name: rss-site (kind: Deployment)
!! Incorrect value for key `image` - specify an image version to avoid unpleasant "version surprises" in the future
X Ensure each container has a configured memory limit [1 occurrence]
- metadata.name: rss-site (kind: Deployment)
!! Missing property object 'limits.memory' - value should be within the accepted boundaries recommended by the organization
X Ensure workload has valid Label values [1 occurrence]
- metadata.name: rss-site (kind: Deployment)
!! Incorrect value for key(s) under 'labels - the vales syntax is not valid so the Kubernetes engine will not accept it
X Ensure each container has a configured liveness probe [1 occurrence]
- metadata.name: rss-site (kind: Deployment)
!! Missing property object 'livenessProbe - add a properly configured livenessProbe to catch possible deadlocks
[...]After the check is complete, Datree displays a detailed output of any violation or misconfiguration that it finds, which guides developers to fix the issue. You can run the command locally, but it's specially designed to run during continuous integration (CI) or even earlier as a pre-commit hook (yes, without losing any explanation for reasons behind the policy).
Along with the command-line application, Datree enables complete management of policies using the UI, like creating new customized policies, reviewing the full history of the invocations, and more.
(Noaa Barki, CC BY-SA 4.0)
How I've embraced the DevOps mindset
As a front-end full stack developer, I got trained to think solely about code, and I have always found DevOps technologies and thought processes to be a mystery. But recently, I was challenged to develop a CLI application at Datree and began to understand the importance and functionality of DevOps.
My mantra is, "Our job as developers isn't about coding—it's about solving real-life problems." When I started working on datree, I had to understand more than just the real-life problem. I also had to know how it became a problem in the first place. Why do organizations adopt Kubernetes? What's the role of the DevOps engineer? And most of all, for whom am I developing my application?
Now I can honestly say that through developing datree, I entered the world of Kubernetes and learned that the best way to learn Kubernetes is by embracing DevOps culture. Developing the datree command has taught me the importance of understanding my user persona. More importantly, it helped me gain fundamental knowledge about the ecosystem of an application and understand the product and user journey.
Summary
When Kubernetes is adopted, the culture of your development environment changes. DevOps isn't something that happens overnight, especially in a large organization. This transition can be aided with technology that helps developers catch their own mistakes and learn from them in the future.
With Datree, the gap between DevOps and developers has begun to shrink. Even diehard coders like me have started to take ownership of limitation policies. The code sent to production is of higher quality, saving time and preventing embarrassing mistakes.
Comments are closed.