Get the highlights in your inbox every week.
A case study in DevOps for teams
How DevOps best practices improve team dynamics
I've spent the past few months writing about the small, incremental behaviors that individuals can employ to be more successful. This month, I'd like to highlight team behaviors that I think are critical to having small successes at work. I spent time with one of the AtomicOpenShift (AOS) teams at Red Hat—the Cockpit project.
Although I spend a significant amount of my time with the AOS teams, I rarely get the chance to work directly with Cockpit. I was lucky enough to have the opportunity to sit with them for a while when we were all in Brno earlier this year. From an outsider's perspective, the team has an ease of speaking with each other—both on technical topics and personal ones—that makes you take notice. In fact, you might have assumed they all work together in the same office. However, all five engineers and the designer on the team are spread out across Europe and the United States.
I had the pleasure of speaking with Andreas Nilsson, Dominik Perpeet, Lars Uebernickel, Marius Vollmer, Peter Volpe, and Stef Walter earlier this month. We talked about many things, but three central themes developed during the interview:
- Understanding your goals
- Using feedback loops
- Committing to open and transparent communication
Over the next three months, my column will focus on those topics and what you can do with your teams to incrementally improve. This month, I invite you to spend time with the team working on the Cockpit project and get a glimpse at what they are doing that makes them successful together.
The Cockpit team has an extensive Continuous Integration/Continuous Deployment (CI/CD) system. How does it help you work together?
Dominik Perpeet (DP): First and foremost it allows us to focus more on the "fun stuff"—developing new and exciting features. When everything is tested before being merged, it is much easier to review something if you can tie errors to specific changes. Let's be honest: We flat out wouldn't be able to deliver at the pace we are delivering now without automation. Also, it is very motivating to tell others, "Hey, we just merged your stuff. You'll see it live on Fedora next week—along with everyone else!"
Stef Walter (SW): Depending on how you count, the Cockpit project takes anywhere up to 100 different system APIs, and integrates them into a coherent experience. Each of those is constantly changing, and integrating is critical. Without continuous automated integration, this would be the sole job of everyone on the team. It's a critical part of the project. Another thing to consider is regular and rapid delivery are a key part of open source and Agile. It loops in contributors. The integration testing allows this to happen.
Many of the teams in AtomicOpenShift participate in a heavily modified version of Scrum. You guys largely avoided the process, but now you are looking toward incorporating parts of into your workflow. What sparked the change?
SW: Recently we have adopted the sprint schedule without all of the aspects of scrum. The main goal was to shorten the feedback loop on our own development process. We wanted to be able to look back on tasks we worked on during a given sprint, have a clear delineation point for planning, retrospective, and iterating on both.
Andreas Nilsson (AN): One thing I kept running into was that I got a lot of different tasks from all kinds of places without a good way to be able to prioritize that. The sprints helps me keep focused on the things that are important and give a better way of estimating. If another thing wants to go in, something else needs to go out from the priority list.
Stef talks frequently about design driven development. How does it help you achieve your goals?
DP: When we take the time to take things "from the top" (design perspective), we waste less time overall, benefit most from others on the team, and the entire development process feels less chaotic.
AN: It allows us to have a better agreement on exactly what we're building during the entire development process, starting from writing down what we're building and for whom, creating provisional personas based on those requirement,s and making wireframes based on those personas' workflow. Compared to wasting time on expensive iterations in code, it allows us to experiment and iterate quicker.
SW: It also helps us coordinate across projects. When having a design and story first, it makes it easy to work with other projects to achieve a given goal. The target is very clear. Without it, it's often futile and frustrating discussing implementation details without a goal.
DP: Open communication on the goal—whether it is ours, theirs, or joint goals—and the willingness to scrap work if it was going nowhere or otherwise flawed. I think with pretty much every cross-community work, all sides have improved. We aren't isolated—quite the contrary, actually—so every interaction with others means more calibration of internal and external views.
SW: Design driven development, user stories, and wireframes have made a massive difference. Other communities are far more willing to engage when a clear user story is targeted, and they can provide feedback and iteration on that story.
How does communication impact the team's work?
DP: Communication is a huge part of our work. We're all remote from each other and thus communication feels more conscious and directed. We use whatever mode is available and works for what we want to achieve—face-to-face either in person or on video chat for when we want to see each other, voice chat for brainstorming, IRC if we need a public discussion, email to keep a record and to be asynchronous, GitHub for design or code discussions. I think all of this also depends on the mood and who is involved.
Time zones are a challenge, but manageable. They also lead to quiet periods of work, which aren't that bad. However, I believe we wouldn't function this well as a team if we hadn't met up in person.
SW: Open source work is distributed by nature. Whenever implementing a tool or system that is distributed, there's a non-zero overhead. This is true of code, micro services, or open source. However, the key goal there is that the distributed can scale out, but I also agree with Dominik that we're pragmatic about this, and there are times when discussion occurs on IRC during times when most people are not around. Meeting up and talking about the overall project goals has helped us build fundamental understanding of what we're trying to accomplish and kept us in sync.
Not everything can be great. What are the current challenges you face as a team?
Marius Vollmer (MV): We started to specialize, which increases context switch costs and the risk of breaking things when working on unfamiliar sections of code. We get more feedback, which is great, but perhaps we are not doing a good enough job of reacting to it.
SW: Getting the open source community further involved. We do have contributions from outside the team, but it's a challenge to try and help that to grow.
AN: Prioritizing between the different products where our stuff will end up. I also feel we're missing the customer feedback loop. What are the exact needs and exactly for whom—once things are implemented, did we solve those needs in a good way that works for our end users?
If you had any advice for anyone starting from scratch, what would it be?
SW: Never leave integration testing for later. If Marius hadn't started with this early on, it would have been tempting to make it a future goal, and that would have been the death of the project. Also, release early, release often, and release everywhere, from the very beginning. We waited too long to distribute Debian and Ubuntu packages, and it hindered project adoption.
AN: Get a design workflow started day one. If not, your competition will keep running circles around you.
DP: Always have clear long-term goals, but never be afraid of changing them for good reasons.