In the DevOps community, we talk a lot about automated deployments, doing multiple deployments per day, and the need for culture. I want to share with you something that isn't talked about nearly as widely, but I think is just as important: the benefits of organizational learning.
Let’s take a moment to visualize what an organization that has fully adopted DevOps principles and practices might look like.
We are able to accommodate a high rate of change that allows us to satisfy our organization and out-experiment our competition. Our changes have short lead times, and we can make changes and deploy code at any time of the day (as opposed to only on Friday at midnight), without organization-paralyzing fear that it will cause massive chaos and disruption.
Furthermore, our code and environments are safe to change (and we can recover from mistakes quickly), ideally without even impacting the customer. We have created a high-trust environment where we can rely on our team members throughout the entire value stream, knowing that we are all working together to help the organization win.
When bad things happen—which entropy and Murphy’s Law ensure—we have sufficient monitoring in place to quickly find out what is going wrong, restore service, and resume normal operations. Because we have a culture of relentless improvement, we will figure out how to prevent it from happening again in the future, or if we can’t, at least enable quicker detection and recovery.
And because we know that more important than daily work is the improvement of daily work, we are constantly learning as an organization, and turning local discoveries into global improvements.
In his book, The Fifth Discipline, Peter Senge explains that “knowledge exists at the edges, not at the center,” and that we need organizational learning because it enables helping our customers, ensures quality, creates competitive advantage and an energized and committed workforce, and it uncovers the truth.
Therefore we must create a culture that rewards learning, which often comes from failure. Moreover, we must ensure that what we learn becomes embedded into our institutional memory so that future occurrences are prevented.
Encourage and celebrate learnings
No amount of command and control management can direct workers to fix each strand, one by one. Instead, we must create the organizational culture and norms so that everyone finds and fixes broken strands, all the time, as part of our daily work.
Our goal should be to maximize our organizational learnings from any accident, gain the best understanding of how the accident occurred, and empower everyone to create the most effective countermeasure to prevent it from happening again, or enable quicker detection and recovery. In addition, we must foster a culture where the entire organization learns from it, so that any local improvements can be turned into global improvements.
Intuit has a famous monthly ritual where the CEO of the company gives a ceremonial life preserver to the person who made the largest mistake. The recipient signs the life preserver, then tells the entire company what happened and what they can learn from it.
Make it easier to use standards than not
Standards, encompassing the sum of our organizational knowledge, should be easier to use than to not. One of the best places to put this knowledge is into a centralized source code repository that is shared throughout the organization, allowing the ability to quickly propagate knowledge. Some other characteristics of successful standards include:
- Shared source code repository and thorough documentation that can be searched and widely reused
- Internal discussion groups for each library and service (e.g. “github-users” or “puppet-users”); often people having questions will get responses from other users faster than from the developers
- Widely broadcasted, blameless postmortem reports
Justin Arbuckle, former chief architect of GE Capital once said, “The best architecture document is one that is implemented in code, in a shared source code repository, that anyone can pull from.”
Enable the organization to discover its way to greatness
By valuing learning, we create an organization where we no longer expect leaders to plan our way to greatness. Instead, leaders help foster and develop routines, test them in practice, recognize which don’t work, and reinforce those that do. Leaders do this by reinforcing the value of learning and ensure that obstacles are removed so that whatever got in our way yesterday and today won’t get in our way tomorrow.
What does organizational learning look like in a real DevOps journey?
I recently had a chance to hear about it from Jim Stoneham, CEO of Opsmatic. In 2009, he was the general manager of the Yahoo! Communities business unit, which Flickr became a part of. Stoneham shared:
The amount of our organizational learning went through the roof as we increased our deployment frequency at Yahoo! Answers from once every six weeks to multiple times per week. Suddenly, we were able to able to try things out and experiment in ways we hadn’t been able to do before. Our team became very much in tune with the numbers: we’d would look at them as a team on a daily and weekly basis, and use that to inform feature conversations and plans.
Instead of engineers talking about the product once every six weeks, we’d be talking about it daily. This was exactly the learning that we needed to win in the marketplace—and it changed more than our feature velocity. We transformed from a team of employees to a team of owners. When you move at that speed, and are looking at the numbers and the results daily, your investment level radically changes. This just can’t happen in teams that release quarterly, and it’s difficult even with monthly cycles.
I love how Jim Stoneham talks about the benefits about DevOps that sound very different than how we often talk about it as Dev or Ops. It's this capability of creating organizational learning that enables us to win in the marketplace.
This article is part of The Open Organization Guide to IT culture change.
1 Comment