iTKO | Software Testing Quality Assurance Testing Tools

Why Unit Testing Alone Fails 

More unit testing can fix mistakes, yet fail to prevent real problems

Article appeared in May 2005 issue of Software Test & Performance magazine
by John Michelsen and Jason English, iTKO

The Current Quality Situation

Unit testing as a quality strategy? It won’t fly.
Yes, unit testing solves problems. Yes, developers should continue to practice it. But is unit testing still a relevant frontier for improving the quality of enterprise applications? Brand me a heretic, but I think it is only solving the small problems.

Testing during development is a quality discipline you need to enforce. Therefore, you’ve invested in training and tools for your programming staff. More unit tests are helping your development teams find structural bugs more efficiently. Churning through even more unit tests, and looking at charts that show how your byte code is 99.5% covered is reassuring. Then you realize while using the finished project at deployment time that the developers executed flawless code, but built an application that fails to meet business requirements in production.

In our experience, only 1 in 100 critical software failures that enterprises encounter in production could be prevented by better developer unit testing.

Increasingly in the enterprise application development field, quality improvement initiatives have become oriented around either 1) testing the front-end UI of a nearly finished application to tell you "it's broken," or 2) promoting the practice of unit testing as a discipline reserved for developers. We believe these forces open up a serious quality gap.
This article addresses the second assumption: that code coverage (or "white box") developer unit testing can prevent most errors in production. Relying on unit testing alone to improve software quality is extremely dangerous when you are trying to uncover problems due to resources, business rules, logic errors, performance hang-ups, data quality, user behavior, security, long-term system stress and integration.