“Too much technical debt” is one of the most common complaints you will hear from software developers working on any long lived project. In this article, we will talk about technical debt and technical investment and learn that both can lead to positive, or negative outcomes depending on how they are managed.
Technical Debt describes any suboptimal implementation of technology which requires the maintainers to pay “interest” in the form of additional work fixing problems and dealing with bad performance.
Technical debt is a powerful term because it’s so easy to understand even amongst non technical stakeholders. The analogy to financial debt is so powerful that it’s easy to imagine the software team burdened by tech debt feeling like the cash strapped individual who has just maxed out their final credit card.
The problem with most analogies is that they get pushed way beyond their applicability, but in the case of technical debt the similarity is so strong it often doesn’t get pushed far enough. Technical debt is mostly talked about in purely negative terms, but just as taking on financial debt in the form of a mortgage can be a very sound decision so too can be taking on technical debt provided it is done for the right reasons.
It may surprise you that Ward Cunningham the person who coined the term “Technical debt” (and also the inventor of the Wiki) did so in a positive context rather than a purely negative way:
“With borrowed money, you can do something sooner than you might otherwise, but then until you pay back that money you’ll be paying interest. I thought borrowing money was a good idea, I thought that rushing software out the door to get some experience with it was a good idea, but that of course, you would eventually go back and as you learned things about that software you would repay that loan by refactoring the program to reflect your experience as you acquired it.”
Before we look at some of the ways in which technical debt can be a benefit let’s consider some of the classic technical debt anti patterns:
Accidental complexity is the kind of technical debt that occurs when code is written by people who lack the skills necessary to produce an efficient solution. This can be when code is written by inexperienced developers, but can also happen when working on novel, or cutting edge projects for which established patterns and tools are not easily available.
Accidental complexity can be an especially nasty kind of technical debt because the people creating it may not know that they have created it, or what to do about it.
In the software industry it's very common for a project to go into a “crunch” before an important deadline where everyone is working especially hard and focussed on delivery. A team in crunch mode will typically not have much time for planning or elegant solutions and as a result technical debt will increase. Ideally crunch should be avoided, but if it happens occasionally and the technical debt that was created is noted and paid down in a timely manner then quality will not suffer too much. The problem occurs if a team is in constant crunch and no time is allocated to paying down the debt. This usually happens if the team is under-resourced, or the expectations placed on it are too high. A visible example of this is the games industry where crunch before a release is very common and patching after release day to fix major bugs is a frequent occurrence. The recent failed launch of the game Cyberpunk 2077 shows what happens if there is too much crunch in a project.
This is another area where the analogy to financial debt is very apt. When paying off a loan part of the monthly payment goes to servicing the interest and part of it to paying off the principal. If the interest becomes too high it becomes impossible to pay back the debt.
The same is true of technical debt. As technical debt increases more of the effort going into the project is spent dealing with the problems caused by the problems in the codebase.
The common experiences of a code base like this are that even simple changes require modifying code in many classes (or over multiple services in a microservice environment), or that fixing one thing breaks something else. This is especially problematic if the code lacks sufficient tests.
Eventually progress grinds to a halt and more time is spent dealing with problems than developing new functionality.
We’ve considered some cases where technical debt can hurt an organization, but let’s think about some occasions where it makes sense to create some technical debt.
Often the most critical question isn’t how a piece of software should be made, but if it should be made at all. In those circumstances it makes sense to build something as quickly as possible to support the product discovery process. When prototyping you will typically take any shortcuts necessary to get to working software and learn more about the product before committing significant resources.
Where something is extremely technically challenging and the right mechanism to implement the solution isn’t clear, an engineering prototype can also be used for technical derisking of the project. “Mythical Man Month” author Fred Brooks described this as “Writing one to throw away”.
Engineers are often very skeptical about creating proof of concept prototypes with high technical debt because they are concerned that they may be required to productionize them without suitable refactoring time being allocated. To paraphrase Milton Freidman “There is nothing more permanent than a temporary solution”.
Taking this into account it is very important that stakeholders understand that a high fidelity prototype they have just seen at a demo won’t be production ready in a week’s time.
This differs from the prototype case because in this instance you are sure of the value of the software you are building, but you are prepared to take some shortcuts to get there quicker. This is the sense that was referred to by Ward Cunningham in the original coining of the term “Technical Debt”.
This might be because you are writing an application in a very competitive market where there is a strong element of “winner takes all” and launching a few weeks earlier offers a significant advantage, or because the software must be prepared for a specific date that can’t move for example a Black Friday sale, or the launch of a movie.
In this situation it can make sense to deliberately take on technical debt to release more quickly. When doing this it’s essential that any technical debt is tracked and logged and time is set aside to pay it down later.
The opposite of technical debt is technical investment. This is a less commonly used term, but means making technical improvements to support the future needs of the application. On the surface this would appear to be an activity with no downsides, but done without care this can have negative consequences.
Creating optimisations which don’t reflect real performance problems, but make the code more complex and hard to support
Less talked about than Premature Optimisation, but equally damaging this means turning straightforward functionality into a generalized framework where the flexibility is never used, but the additional complexity makes the application harder to understand.
Adding features to an application that doesn’t help the users, but adds complexity to the interface and more functionality to test and support.
Clearly, some kinds of technical investment can be harmful, but what kinds can be relied upon to deliver benefits?
This is especially powerful if combined with refactoring the application to be more testable. A highly testable application is almost always a flexible and extensible application due to the layering and modularity that is required for effective tests.
Simplifying applications and streamlining a complex application helps users and engineers to better understand the product. Using another financial analogy, unused code should be considered as a liability rather than an asset.
In the talk Software That Fits in Your Head Dan North talks about the concept of “Software Halflife”. That code that’s existed for a while should either be hardened or replaced. Adoption of a microservice architecture provides clearly defined boundaries between systems which allows a problematic or underperforming component to be replaced without risking a knock on effect to the system as a whole.
Hopefully this piece has highlighted the power and subtlety of “Technical Debt” and how with pragmatism it can be used to derive maximum value from the software delivery process.
At Curvestone we love helping our customers to deliver value fast without sacrificing quality. If you have a problem we can help you with please get in touch.
