// ThomasWeller

---

I am practicing Test Driven Development (TDD) now for some two years or so, and soon this technique of writing software felt so natural, that I hardly could imagine doing it another way or even imagine a reason why I should do so. But on the other hand, I know that not questioning something anymore and not being self-critical from time to time is a certain recipe for running into a disaster sooner or later. So I asked myself: What makes TDD such a natural way of writing software? and What are your main conceptions about TDD? This post is the result of my reflections.

---

Disclaimer

I should preventively make some remarks regarding this post:

• The cited studies are the first ones that came to my mind and seemed relevant for the subject, I did not do any extra investigation to find other studies specifically relevant for the subject.
• The derived figures are rough guesses that serve only to underpin my reasoning and to give an idea about the magnitudes in question.

---

1. TDD automatically produces good design

I am currently reading ASP.NET MVC in Action (which is btw. highly recommendable and a very good read even if you think you know everything about ASP.NET MVC), and one sentence almost blew me out of my chair – not so much because of what it’s saying but primarily because of it being said as if it were the most natural thing in the world:

It’s nearly impossible to test-drive code that ends up with a bad design.

In other words: If you practice TDD, then your application design will most likely be of high architectural quality in the end. I never thought about that specific aspect in such clarity, but it’s obviously true: How likely is it to end with a Big ball of mud, when you do your coding test-first? Next to zero. The result may occasionally be wrong or problematic in some respect, but it almost never will be a mess.

Vice versa, this means: If you have problems to test-drive a feature, then it’s time to look at your design (and maybe at your design skills). The regular complaint, that TDD significantly slows down production without providing sufficient equivalence,  is often due to the fact that the introduction of TDD unravels latent problems with a software project’s architecture, that otherwise might surface differently (mostly through increased maintenance costs) or even go unnoticed. So attributing these costs to TDD is at least questionable, although it might be difficult to draw the line in a specific case.

A 2006 study ([1]) examined the effects of TDD on internal software design quality, providing empirical evidence for the fact that TDD actually raises this quality. From the abstract:

Results from this study indicate that TDD can be an effective software design approach improving both code-centric aspects such as object decomposition, test coverage, and external quality, and developer-centric aspects including productivity and confidence.

2. TDD produces business value

There are not so many empirical studies on the Test driven development in the real (business) world issue, partly because software development in general is hard to measure objectively, partly because TDD is a very young phenomena. So, if you were asked by a business executive: Why should the dev team do TDD? It slows down the development process! What do I get from it in compensation?, your answer is usually a variation of: Well, it feels right, it produces good code, it makes for high quality, and I’m convinced of it. This certainly is not very compelling, especially if you talk to the person who is responsible for your project’s monetary resources. A business person does and should not care about good vs. bad code quality – he or she has to care about costs. So you have to argue from a business perspective with such a person, i.e. you have to present numbers and hard enough empirical evidence for your point.

Two relevant studies come into my mind regarding this subject:

The first one ([2]) looks at the total costs of ownership for a software system over a five-year period under three different scenarios: low, average and high quality control. The size of the software project is assumed to be 1.000 function points, the figures depict dollars per function point, the term Maintenance here means anything that is done after initial development (defect repairs, support, enhancements etc.). These are the original numbers from this study:

	low	average	high
Development	1.200	1.000	800
Maintenance	1.116	860	512
Total	2.316	1.860	1.312

As we can immediately see, the internal quality of a software system has a massive impact on its total costs, in most cases the maintenance costs will by far outweigh the initial development costs. (The here shown figures cover only the first five years of a software system’s life, but most systems actually have a total lifetime somewhere between 10 and 30 years. Also, the average costs usually grow bigger with the system’s age.) The study puts it like this:

[…], if you build applications properly at the start, you can get many years of useful service. If you build them poorly at the start, you can expect high initial maintenance costs that will grow higher as time passes.

The second research paper ([3]) conducted some case studies with development teams that have adopted TDD (three at Microsoft, one at IBM). It comes to this conclusion:

The results of the case studies indicate that the pre-release defect density of the four products decreased between 40% and 90% relative to similar projects that did not use the TDD practice. Subjectively, the teams experienced a 15-35% increase in initial development time after adopting TDD.

So you buy a 40%-90% lower defect density with 15-35% higher development time…

Now if we combine the figures from the two studies, we get something like this (due to the fact that the one study is measuring defect density, whereas the other speaks in terms of total maintenance costs, the defect density ratios are applied only to 75% of the total maintenance costs):

Worst case scenario (35% higher development time, 30% lower maintenance costs)

	low	average	high
Development	1.620	1.350	1.080
Maintenance	781	602	358
Total	2.401	1.952	1.438

Average case scenario (25% higher development time, 49% lower maintenance costs)

	low	average	high
Development	1.500	1.250	1.000
Maintenance	569	439	261
Total	2.069	1.689	1.261

Best case scenario (15% higher development time, 68% lower maintenance costs):

	low	average	high
Development	1.380	1.150	920
Maintenance	357	275	164
Total	1.737	1.425	1.084

If we compare these total costs to the original ones, we get the following:

	low	average	high
original	2.316	1.860	1.312
worst case	2.401 (+4%)	1.952 (+5%)	1.438 (+10%)
avg. case	2.069 (-10%)	1.689 (-9%)	1.261 (-4%)
best case	1.737 (-25%)	1.425 (-23%)	1.084 (-17%)

Phew, quite a lot of figures, but in the end it roughly boils down to the following:

When applying Test driven development, an average software project will normally have around 10% lower total costs within the first five years of its lifecycle compared to not doing TDD, with actual values ranging from about +10% to -25% in the extremes.

I know that this is only a very rough guess, but it’s grounded on empirical findings, and it’s more on the conservative side of things – software systems usually live far longer than five years, maintenance costs tend to increase significantly with the age of a system, and I didn’t see a system yet that actually has a very high quality. Furthermore, the statement is speaking in terms of business value ($, €…) rather than in the usual geek speak, that developers so often assume to be understood by other (should I say normal) people…

3. Test are materializations of the natural thought process

Tests are not about testing, at least not in the first place. Rather, they are a materialization of the normal software development thought process, which roughly goes something like this:

1. You have an (initially possibly vague) idea of what’s needed.
2. You implement (and then maybe refine) the required behavior.

TDD is nothing but the executable formulation of this process, and therefore shouldn’t be regarded as something special. It naturally follows from above, whereas coding and then writing tests, or not writing tests at all, is simply the wrong order of things in this respect and should be seen as the exception from the TDD-rule in everyday developer’s work.

4. Tests are developer-documentation

Most programmers don’t read the written documentation (sometimes simply because there is none or it is unreliable), instead they prefer to work directly with the code.  When trying to understand a piece of code (class, method, assembly…), most programmers will instinctively look for sample code that already invokes it and gives a typical, intention-revealing usage sample.  Good unit tests do exactly this – they provide a working specification of your functional code – and as a result unit tests effectively become a significant portion of a software project’s technical documentation and a valuable source of knowledge. A nice and helpful side-effect is that the test code is a good source of code samples for the more formal parts of documentation (e.g. xml code comments).

To test or not to test, if it becomes seemingly trivial ?

Many argue, that there is no value in testing something trivial like e.g. a simple property (see for example this blog post: Testing only the code of value). This argument makes the implicit assumption that unit tests are something negotiable, you might have them or not, and you can decide upon it. This may be true if you’re instrumenting a legacy codebase with unit tests, or you write your tests in an after-the-fact manner, but again it turns around the natural thought process, in that you first produce the result, and then decide upon producing the requirement for it. The hidden concept behind that rationale is: The main purpose of a test is testing, i.e. verifying the correct functioning of a piece of code. Instead, a test should be seen as a statement of the initial requirement for a certain piece of code – and as a side-effect (a very important one), it verifies its correct functioning.

So while to test or not to test is not a valid alternative from the psychological point of view, there’s also strong support for the TDD methodology from the business numbers, as seen above. Sure, if you have a group of good developers, who write good to perfect code instinctively, they won’t profit much from applying TDD – making them use TDD will raise costs, but it won’t raise quality all too much, since it is already very high. That’s the case against TDD – regularly heard and in almost every case just plain wrong. What if your enlightened developers are no more present for some reason or the other? Relying on a person or a few persons is just not acceptable in most cases – imagine the production of your car (and the availability of spare parts) would depend on a few persons in the car manufacturer’s company… While that may be good for quick hit-and-run business successes, it’s just not sustainable, a recipe for mid-to-long-term disasters, and is clearly an organizational anti-pattern.

If you follow a strict TDD-approach, the above question does not even come to mind. It’s just not thinkable to produce something when there exists no need for it…

Is TDD agile ?

While TDD inarguably became popular as part of Agile Development and XP Programming, I don’t think it is something that naturally belongs there. The aforementioned development techniques are to a certain extent subject to contemporary trends and fashions, and there sometimes is an almost religious excitement around them. I’m not especially happy with the fact that TDD is discussed primarily in this context, because it is much more fundamental in my eyes.

(Don’t get me wrong here: I think that Agile/XP is the most important innovation in software development since its beginnings. And of course TDD can be seen as an agile technique in the sense that Agile/XP itself is fundamental to software development. It’s just that IMHO the TDD approach should not be perceived in the context of anything that could be regarded as a buzzword or a movement or the like…)

Conclusion

Summarizing the above, I’d say that

• TDD has nothing to do with testing in the first place, but is a general development technique.
• TDD makes you develop better and more reliable software.
• TDD reduces costs.
• TDD is the natural way of developing software, other approaches (code-first) are exceptional.
• TDD shouldn’t be seen in an agile context.

I think that TDD would greatly benefit from being renamed. Getting the Test out of the name would far better reflect its true nature, and it would hopefully make the name something that doesn’t suggest it may be somehow negotiable.

This would hopefully trigger a cultural shift in the software development industry towards a world where developers don’t need to justify the practicing of TDD anymore, and where business executives don’t see TDD as something that can be subject to cost-cutting.

And maybe there will be a day when TDD simply is named after what it actually is: Programming.

---

References

[1]	Janzen D. & Saiedian, H., 2006. TOn the Influence of Test-Driven Development on Software Design. In: Proceedings of the Conference on Software Engineering Education and Training, pp. 141-148.
[2]	Jones C., 2006. The Economics of Software Maintenance in the Twenty First Century.
[3]	Nagappan, N., Maximilien, E. M., Bhat, T. & Williams, L., 2008. Realizing quality improvement through test driven development: results and experiences of four industrial teams.. In: Empirical Software Engineering 13 (3/2008), pp. 289-302.