Engineers know that there’s much more to a successful software project than meets the eye. Refactoring and testing are as important as functionality when it comes to managing the complexity of programs that inevitably evolve over time. But when deadlines get tight, the management tends to focus on the functionality. But what if this technical debt starts piling up?
I had a chance to interview Caskey L. Dickson about his upcoming LISA15 talk, Why Your Manager LOVES Technical Debt and What to Do About It. Caskey is a software engineer with an MBA and more than 20 years of experience in online services development. He has seen both sides of the barricade and will talk about how engineers and managers can work together toward better software products.
What are the most common reasons that lead to creation of technical debt?
I want to say the most common reason is the pressure to release, but that's only half the story. We are pressured to show progress and results in a way that is visible to customers. Even our best meaning and most effective methods of producing software rapidly—I'm looking at you Agile—puts more emphasis on delivery of user-visible features over the things that are necessary to make sustainable software. I'm not saying that agile methods can't include backlog elements that address software quality, just that without extra discipline, it's easy to overlook those items. That, in fact, is the other half of the story. Your backlog should include things that relate to software quality in a quantifiable fashion. "Module X cannot be unit tested without an instance of module Y, which is expensive to create (or mock)." Fixing something like that belongs in your backlog, but they only appear if developers are putting them there. If product management owns the backlog, then development and operations concerns are never going to magically appear.
This, of course raises the question of where does this pressure come from? It's easy to go straight for the pat answer of "product management", but it is more complex than that. No engineer wants to deliver a sub-standard product. But as a species, software engineers are impatient and forward looking; we want to move on to the next cool thing and don't want to "eat our vegetables." We need to realize as a developer community the number of times someone from management has come and said directly "don't do that test coverage, let's ship it now" in those plain terms is vanishingly rare. Instead, we let quality slip a day at a time because of pressure we put on ourselves. We compromise on a day-to-day basis over exactly how disciplined we are going to be. I freely admit I have looked at something, said I should refactor this, and then decided I just didn't have the emotional energy to take that on because the needed change was not that big, and so I took the easy route just to ensure the feature didn't slip to the next sprint. These decisions add up rapidly across a development team.
What are the best ways of managing technical debt in a project? Is it possible to avoid it entirely?
I don't pretend to know the best way, but for me it comes down to finding proxy measures for what we call technical debt and then using those to drive specific actions into the backlog. What does technical debt mean to you? As developers we have this implicit, vague idea in the back of our minds, but it covers a bunch of things we call the -ilities: maintainability, manageability, extensibility, and so forth.
I like that old saying that you should write your code like the next person to work on it is a homicidal sociopath and knows where you live. Once you come up with a set of things that you value in your code, finding a way to measure that and observe changes over time is what it takes to manage debt. We have an easy measure of the features: bulleted lists of new shiny on every sprint, product release announcements, new capabilities and compatibilities handed over to our users at great fanfare. These are all measures of the progress of our product. We also need measures for the missing elements in our software development process—those missing elements that create product risk.
As for avoiding debt entirely: Absolutely not. Risk is a continuum. Everything has some measure of risk and the absolute value is never zero. Furthermore, the cost of driving it down goes up exponentially and it would be foolish to try. As engineers, it is incumbent upon us to strike a balance between an acceptable amount of risk and value delivered for effort expended. We are an active participant in that process and need to take responsibility for the decisions made. And let's not kid ourselves—some engineers are willing to accept larger amount of risks than others. Sometimes our frustration is not with management, but with our fellow developers who have different ideas as to what risk is acceptable. This is why being able to come up with explicit measures of risk and an agreed upon set of thresholds is so important. I feel the dialog that needs to happen is not just us to management, but among ourselves as well.
Why do you think managers don’t understand the risks of accumulated technical debt in a code base?
I think it is a stretch to say they don't understand it; it's more appropriate to say they aren't being given the information they need to make an informed judgement. Part of management's job is to make informed decisions on risk vs. results. I think they are not given tools to objectively evaluate what we feel in a visceral way—namely the actual risk being taken due to quality failures in our code.
I have no problems with management taking on informed risks. Frankly, that's their job. My concern is that right now the vast majority of organizations are taking uninformed risks as to the actual quality of their product. The risk I want my email provider or my health insurance company to take with regard to product quality is entirely different than the risk I want my favorite online game to take. Of course everyone has a different weight they might give to those products. Developers need to find ways to express that risk in quantifiable and actionable ways. Then we can take proactive steps to achieve and maintain a level of quality that we all agree upon.
What can programmers do to make the management appreciate the importance of refactoring amid a neverending backlog of urgent tasks?
Programmers can take the time to quantify their frustration and put that into the backlog. Quality is a feature, too. As my talk will try to point out, bemoaning an abstraction like technical debt is too vague. What isn't vague are actionable ideas, like "our build process is insufficiently repeatable," "our unit test coverage is too low," "smoke tests take too long to run," and so forth. Sometimes our own enlightened solutions get in the way of clean testable code. If you can't smoke test a module because there is no easy way to inject a mocked dependency, then maybe it's time to refactor the component and its dependency to make it more testable. These are actionable items that can be put into the backlog on the basis of agreed upon standards of quality.
The other thing developers need to do is take a principled stand. You have more power than you think, and as long as you can provide a rational argument about the nature and degree of code quality issues, your manager will be your ally. Your manager doesn't want your product to fail any more than you do, but they are often not given quantified information as to the quality of your product. Furthermore, don't be surprised when your manager looks at your assessment and disagrees. I'll never forget the consulting gig I had where I reviewed a code base, demonstrated some basic SQL injection on their production service, and management decided the cost of fixing it wasn't worth (the risk it presented). As a young developer, I was shocked; over a decade later I see that their decision was broader than just the software in question.
Management's job is to assume risk. So, let's give them the information they need to make an informed decision.
Do you think that open source projects are less prone to accumulating technical debt? Or do they suffer from the same problems?
That's a tricky question. Any project is capable of becoming burdened with technical debt. The difference is that it's rare for an open source project to accumulate much debt once set into the wild. It's only when you have a population of captive developers that are tasked with adding features to a code base with no choice in their participation that you can achieve the truly abysmal levels of code quality that is out there. When a project is open source, developers simply move on when it becomes too much to deal with and the project dies.
I feel that successful open source projects manage their technical debt through aggressive refactoring, an adherence to quality-inducing behaviors like code reviews, unit test coverage standards, and an ecosystem where bad code bases are starved of willing developers. There is a limited population of people who are willing to contribute to open source, and given the choice of a nice, clean code base and one riddled with complexity and obscure spaghetti, most will work on the higher quality projects. It's a virtuous cycle for the good open source projects and one that implicitly punishes poor quality projects. One of the gold standards of a good OSS project is a contribution guide. Does your internal org's project have a contributors guide? If another engineer in your company wanted to fix a bug for you, how long would it take before they had a working dev environment?
That said, the worst code I see on a regular basis in non-open source projects are in siloed development environments. Just as in politics, sunlight is the best disinfectant. I know I have some side and toy projects I would be ashamed for anyone to see. This is why I am absolutely in love with the movement toward "internal open source". Using open source methodologies to make code visible and widely editable within an organization can do nothing but improve things. Not only do good internal projects serve as exemplars of what should be done, bad projects get visibility and more voices are brought to bear on the concerns that lie within.
Of course, combine internal open source with the ability for developers to freely move among projects, and you've hit upon the holy grail. Then again, woe the product manager who is left holding the bag on the business-critical spaghetti project.
If you’re attending LISA15 in Washington, D.C. this month, you can catch Caskey's talk on Friday, November 13, 2015 at 11:45AM.
1 Comment