Even as accepted standards on how to do it "right" remain elusive, DevOps is a crucial element of modern IT. Corey Quinn, director of DevOps at FutureAdvisor, has immense experience in operations and DevOps. I had an opportunity to talk to him ahead of his two talks at LinuxFest Northwest 2016: Terrible ideas in Git and Docker must die: Heresy in the church of Docker.
Read on for an insightful chat where we tête-à-tête on DevOps, Git, Docker, and open source—with a touch of Corey humor.
DevOps is a popular and sometimes confusing topic. How do you define it? What does it mean to you?
Five years ago, the term "DevOps" was such that you could ask 10 people to define the term and get 12 different answers. Today, that hasn't really changed—if anything, you'll get 15 answers that range from the aspirational ("Helping to break down cultural silos") to the descriptive ("A combination of culture and tools") to the cynical ("A way to add $30K to your salary with one simple title change"). Today, I'm going to go with: DevOps is a great way to start a semantic argument.
Can you cite an example of DevOps done right? Which tools were used?
It's hard to come back and say, "This one company really nailed DevOps, so everyone else should do things exactly as they did." That's not DevOps culture. That's cargo culting, and we see examples of that all the time. The best examples I've seen are companies that realize their 20-person engineering teams aren't Google's tens of thousands of engineers, or that things that work really well at Facebook just don't map to what they're working on today. So, they take the best aspects of other shops and forge them into something that's uniquely theirs. I wish I could say that "DevOps in a Box" existed—I'd love to sell it to you!—but it just doesn't.
Tell us about how we shouldn't do DevOps. Do you have a favorite horror story?
Probably the worst misunderstanding of the culture of DevOps that I've encountered was a shop where it was expected that folks would spend half of their day doing operational tasks, and the other half of the day doing development on the company's core application. That's not DevOps; that's two different jobs. Not surprisingly, everyone on the team failed at least one of them.
What trends are you seeing right now, particularly in the context of cloud and containerization?
A common theme is the rise of immutable infrastructure, but I prefer to think of this as ephemeral infrastructure: Things that exist right now won't exist in 10 minutes, by design. How do you monitor that without the pager going off constantly? How do you diagnose an intermittent issue that last occurred in a container that's been rescheduled to another node by the time the issue shows up on your radar? Containers and cloud require us to rethink a lot more than the delivery technologies we use; they force us to rethink how we're watching our environments and what we care about.
You've said Git is increasingly central to DevOps tooling. What makes it critical?
Git is important if for no other reason than that it's currently the de facto standard for version control in most shops. The larger realm of version control is important because it speaks directly to the "infrastructure as code" mindset. Knowing what the code looked like before you changed something, identifying when and why something was changed, providing an asynchronous workflow model—all of these are vital to a world where we're doing far more as operations people than manually configuring servers or switches. I'll go even further: Without Git or something like it, it would be functionally impossible for most environments to scale past a thousand nodes in any reasonable way.
What Git-integrated tools do you find useful?
When you say "Git tools," I think of things that help make Git less scary. To that end, Joey Hess's myrepos is great at simplifying the management and updating of large numbers of Git repos. Githug (a Ruby gem) is great at teaching how Git works through fun challenges. And of course, the Random Git Man Page Generator is scarily plausible for most of its auto-generated results; Git is a supremely simple tool that very quickly gives way to terrifying complexity.
In your talk, Docker must die: Heresy in the church of Docker, why do you say it is important to exercise caution?
There's a disturbing propensity in our sector to throw out tools that have stood the test of time in favor of this week's shiny new toy. It's important to make sure that you're optimizing for your own use cases rather than someone else's. If you wind up trying to solve for issues that crop up at a scale of 200,000 nodes when you yourself are a bit closer to 20 nodes, you may have lost the plot. The point of my talk isn't to say that Docker is terrible technology—it isn't. But it's important to maintain an awareness of what's going into your infrastructure, how it works, and what the failure cases look like before they hit you.
What are three of your top recommended best practices for containers?
My first best practice is don't use containers. When you come back and say, "That's ridiculous! It's the best solution to our use case because of reasons X, Y, and Z," there's a good chance you've put enough thought into this that you're not just blindly jumping onto a fad. Remember that best practices don't apply in every situation!
My second best practice is to make sure that everyone thoroughly understands your new architecture. When a developer gives you a container that's 3TB because it contains a complete copy of the database, there's been a miscommunication somewhere.
My third best practice is to think about the impact that containerization is going to have on your current environment. Something we discovered early on when moving to a more immutable model was that many of our stateless instances weren't nearly as stateless as we thought they were. Hardcoded IP addresses, writing out log files, "special" hosts that ran specific tasks that were forgotten—a lot of cruft accumulates in legacy environments.
As a contributor to SaltStack, what is the relevance of Salt among containerization technologies such as Docker?
That's a great question! We'll set aside the fact that Kubernetes (a popular container scheduler) was built on top of SaltStack, and instead talk about immutable infrastructure in the context of configuration management. Orchestration is still very much an unsolved problem, even in a world where all of your infrastructure is immutable. Additionally, very few shops would qualify under that definition—almost everyone winds up somewhere between pure config management and pure golden images. You're very likely to have at least a few nodes in your environment that aren't ephemeral or immutable.
Comments are closed.