Coffee Shop DevOps: How to use feedback loops to get smarter

8 readers like this.
Coffee shop photo

Florida Memory. Modified by CC BY-SA 4.0

This month let's look at how to break the cycle of doing the same thing repeatedly and expecting different results.

Do you think git blame is the only feedback loop you need? Or hg annotate -u -n. Or svn -x -b...well, you get the picture.

What is a feedback loop, anyway? Why is it important? If you haven't read Harnessing the Power of Feedback Loops by Thomas Goetz, I highly recommend it. Thomas writes that feedback loops have four stages: the collection of evidence, understanding its emotional relevance, the consequences of what happened, and then take action to improve the next cycle. It's in repeating the cycle that we gain the most from feedback.

In general, technologists find technology comfortably easy to work with, and easier to fix than people or relationships. It is no surprise that when I ask teams about feedback loops in the context of DevOps, they immediately start talking about system monitoring, getting feedback on a submitted PR, or more commonly, the dreaded standup meeting.

My experience tells me most teams stumble during the relevance stage of feedback loops because of a lack of connection. We understand what it means when a production server is down, but it is a rare team that will process the event emotionally much further than to never want to experience it again, even though it's all but inevitable. This lack of connection during the relevant stage translates feedback activities—like a daily sync meeting—into another exercise you aren't particularly interested in. Too often, we spend time participating in feedback loops because someone told us to and not because we derive value from them. Let's face it: with the state of mobile technology, social media, and our general habit of multitasking, sometimes we are in a position where we have way more feedback available to us than we can use.

So how does someone learn how to emotionally relate to the wall of incoming feedback they receive on a daily basis at work? As an example, let's look at the feedback loops I have at home to understand how to increase the value of the feedback.

Feedback loops at home

I tried to categorize most of the feedback I receive from my home life and came up with three general buckets: friends & family, my spouse & best friend (or confidant), and the hobbies I participate in. My friends and family provide me with an unending stream of feedback: who to vote for, what to eat, what to watch on TV, things I did they didn't like, and things I did they liked. You get the picture. We all have incoming feedback in some form in these categories. However, we probably don't slow down enough to take time to think about the information coming, and we don't notice we are participating in the stages of a feedback loop.

To illustrate this point, let's look at the hobbies section of my home life. Hobbies are the things I do outside of work that I am passionate enough to pursue in a meaningful way, like watercoloring, swimming and getting hassled by my cats. For my husband, it's video gaming. Alex is a consummate Dark Souls player. This doesn't always mean he is good at it, but he is tenacious. Recently, he was telling me he amassed 450k souls (for those of you not versed in Dark Souls, it's the monetary system). Knowing my husband, I told him, "You should be extra careful with all of those souls and spend them before you lose them." He laughed me off. Sure enough, just a day later the despairing "Nooooo!" echoed throughout our house. He accidentally fell off a cliff, died again without recovering them, and lost all of his souls for good.

home image

Typically the experience would pass without much introspection, but because I have been thinking about feedback loops for over a month now, I asked him to participate in an exercise with me. We classified the initial experience and conversation we had using Goetz's feedback loop stages to see how we reacted:

Evidence: You have a lot of souls and you die a lot in this game. When you die twice in a row, you lose your souls.

Emotional Resonance: You should spend your souls before you lose them.

Consequence: If you don't spend your souls now and die, you lose them all.

Action: Don't die in the future.

Well, duh, right? However, based on my experience with Alex and his gaming habits, I know that he's going to die and lose souls again in this game. I know this because it's happened about 1,000 times in the past year. How can we make the feedback more emotionally relevant so he changes his behavior?

Evidence: You have a lot of souls and you die a lot in this game. When you die twice in a row, you lose your souls.

Emotional Resonance: There is so much stuff for sale, don't you want a cool new sword? What is in your list that you don't already have? There is no reason to save souls here, especially when you have that many. How about you spend half now?

Consequence: If you don't buy it now and you die, you'll be disappointed with yourself. Also, you'll have to play forever before you can buy a cool new sword.

Action: Spend the damn cash and get a better sword. O\r better yet, don't die. ;)

Using feedback loops to make improvements

Why talk about personal feedback loops in reference to DevOps? For me, it's straightforward. When I take time to understand the data flowing in a way that is meaningful to me, I am more likely to make a functional adjustment and improve the situation. Reflecting on personal problems gives me a place to practice because it is typically something I want to do rather than have to do. Frankly, it's easier to build a habit around feedback that doesn't have the same implications as a production outage.

And I'll admit to being stubborn. My perception of change is different when I take an active role in defining what the action is going to be. It also feels easier for me to accomplish. As a result, being able to see the incremental changes definitely impacts my morale and my overall sense of well-being increases.

Are you someone who is currently in a situation where you are required to participate in a mandatory feedback loop? I feel your pain! Try disrupting the negative feelings by going through the exercise my husband went through for Dark Souls. Also ask yourself these questions:

  • What are the data you are receiving and sharing? Is it enough? Is it useful? Is it timely?
  • What is your perception of the data? Is your perception correct? How are others perceiving the interaction?
  • How can you change your perception so you can potentially get more out of the feedback loop?
User profile image.
Jen Krieger is Chief Agile Architect at Red Hat. Most of her 20+ year career has been in software development representing many roles throughout the waterfall and agile lifecycles. At Red Hat, she led a department-wide DevOps movement focusing on CI/CD best practices. Most recently, she worked with with the Project Atomic & OpenShift teams.


In Jeff Sutherland's book "Scrum", he talks very briefly about a military theoretician, John Boyd, that identified a decision making cycle (placing high value on feedback loops) that is explicitly and implicitly used by anyone that needs to make rapid decisions and adjust to the consequences of those decisions quickly. Before he came up with the decision making framework, Boyd wrote a paper while in the Air Force called "Destruction and Creation" that I believe applies directly to what you are talking about here. Here's the link, if you scroll down to the bottom in the Source List you can see a link to the paper.

I picked up that book a couple of weeks ago, and it is on my queue to read. Thanks Matt for the recommendation. I'll take a read! :)

In reply to by Matt

Understanding feedback loops is key to systems thinkings, and systems thinking is key to learning organizations. I've just about finished the new edition of "The Fifth Discipline" by Peter Senge, and the same sort of feedback diagrams are all through out.

One of the other things that making the loops real (i.e. drawing them) is you can uncover links to other processes that influence your loop without your explicit understanding. There's a great example in the book that talks about two vehicle engineering teams whose changes were affecting the other teams goals, but neither team knew it until someone else pointed it out.

@crowsfly: Boyd's OODA loop is one of my favorite constructs. Better than Deming Cycles :) Orient -> Observe -> Decide -> Act, short circuted at any point, success comes from executing "inside" the time the opposition takes to complete a cycle. Highly applicable to software building.

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.