Everyone that wants to get serious with unit testing should follow the lead of Edgeware and dedicate half a day or so of their developers’ and testers’ time to identify strengths and weaknesses within the development organization. I was invited to present best practices of unit testing. We also touched upon continuous integration and future directions in terms of development excellence. It was fun and interesting and we had good discussions. :) Perhaps the slides (pdf) can give you some inspiration to improve your unit testing and development process.
When you read unit test code, have you ever wondered why some tests tell a story (easy to read) while others just feel mechanical (more computer readable than human).
While I have no definitive answers on the human/non-human readable unit tests subject, I have an insight that came to me and a colleague some time ago: Some tests are written as a specification, and some tests are written to fixate. Let me explain what I mean.
When you write a test as a specification, the test code is normally written upfront or together with the code. The purpose of the test code is to document the behavior of the code. Thus, the developer makes an effort to make the test code human-readable. The tests are often used as an acceptance criteria, to verify a piece of functionality was implemented correctly. The fact that you get an executable specification is a great bonus, since the test code would be valuable just as an example.
Other tests are written to fixate the existing behavior of the code. Typical examples are tests written long after the production code in order to cover legacy code with tests. Legacy code can be hard to test, so the test code is difficult to write in human-readable way and tends to be more “mechanical” in nature. The focus of these tests is execution, since the purpose is to make sure refactoring does not break existing behavior.
Naturally, we want our test code to lie in the specification category. But if we have a piece of code that is really hard to test, we can’t just start refactoring blindly. So we write the best tests we can, and refactor the code and tests in parallel. After some work, the production code is in better shape and we have moved our test code into the specification category.
When you write test and production code from scratch, you have every opportunity to write good tests. Make sure you make your code human-readable. The computer will do just fine. :)
At work, I have been challenged with the question “What are the most important characteristics of a software developer?“. This is a tough question, and no matter what you decide to include, you have to leave something out.
I’ve been part of software development projects in various companies. The successful projects teach you invaluable lessons. The dysfunctional projects even more so. Inevitably, a list of characteristics would include qualities I value and desire in my fellow colleagues, as well as characteristics I’d expect them to want in me. (So this is also a long todo list for me. ;)
To get some kind of structure, I decided on three main categories: Professionalism, Long-term code and Quality mindset.
You are a professional developer, and professionals act responsibly. If things go wrong,
take responsibility. Understand why things went bad. Learn, adapt and make sure it
never happens again. When faced with a difficult choice, “do the right thing”. Optimize for the long run even if it results in more work today. Be a team player, even if this means saying no. Speak the truth.
Know your product
We’re part of a business. Without successful products, there will be no business.
Know your users and their needs. Use your product, use competitor’s products,
visit customers and watch them use your product (from purchase/download to
installation to day-to-day usage to upgrade to uninstall and so forth).
Be humble and practice relentless inquiry. Question everything (since everything has potential for improvement), embrace questions on your code, realize others’ suggestions might improve your work, be happy other developers change “your” code
(it’s good enough to understand!). Improve yourself and others. Become a better developer by reading books, papers, blogs etc. Watch instruction videos, listen to podcasts, try new technologies, participate in open source development, discussion groups etc. Discuss your code and what you read with your peers. Talk to developers of other specialities. Teach others what you know well.
You write a line of code once, but it is read hundreds of times. Invest time to write code easy to read for others. Write code at the right level of abstraction, abstract enough for expressiveness but without hiding necessary detail. Adhere to design principles as
they capture proven ways to high-quality code. Apply design patterns to better communicate your intent.
Decouple the different parts of your software, on every level – sub-system, module, class and function. Write extendable code, so that you can add functionality with minimal change to existing code. Avoid technical debt, and repay debt as soon as possible. Interest has to be payed on all debt.
Never deliver code unless you’ve proven it works. If you don’t test it, it will be faulty.
Write testable code. Make it testable from the start, later it will be too expensive. Automate your tests to run before check-in, after check-in and nightly. If the tests are not executed automatically, they will not be updated and soon be obsolete. Without automated tests, no-one will dare change any code. Write fast and reliable unit tests. Write realistic integration tests. For each new feature, write acceptance tests. Automate everything.
Quality is your responsibility
You, as a developer, is responsible for the quality of the product. Other roles can help you spot problems or give you more time, but they cannot affect quality. Never ship anything without being certain of its correctness.
Find bugs early
Find bugs early in the development process. If a bug can be found by the developer, it should be. If you need tools, get them. If you need hardware, get it. If a bug is found late, understand why it was not found earlier. Fix the process so that bugs of this kind never slips through. Automate.
Fix it now
If you find a bug, fix it now instead of filing a bug report. Ask your colleagues to help out. You will save time. File bug reports on things that couldn’t be solved in half a day.
Do things properly the first time. If you don’t have time to do it right today, when are you ever going to find time? Give time estimates that allow you to produce quality products. Think about what is stopping you from being more productive. Fix it, and then move on to the next thing stopping you.
If you’re interested in these things, you should have a look at Poppendiecks’ “Lean software development” books, Senge’s “The fifth discipline“, Martin’s “Clean coder“, McConnell’s “Code complete“, McConnell’s video “World class software companies” and others. I also provide some resources in previous blog posts (e.g. videos and books).
What characteristics do you value in a software developer?
Automatic compilation as you type is useful, but can we take it further? JUnit Max automates the execution of your unit tests.
In the good old days, you would build your software by running make from the command line. The output would tell you where the errors were, and it would be up to you to find the file and line to fix the error. Nowadays, most IDEs are somewhat better. Normally, you press a key to build. The IDE then helps you navigate the errors by taking you to the file and line. This still involves a context switch: you will have to stop writing code in order to press a key, and subsequently to look at the error messages. Doing this 50 times per day, and the disruption of the context switches becomes noticeable.
Modern IDEs offer another improvement. It will run the compiler automatically in the background for you. For example, Eclipse will compile the code while you’re typing and show the errors underlined in red or as a red icon on files that fails to compile. This removes much of the context switch mentioned above. This, combined with the immediate feedback, leads to less interruptions. This is an improvement.
One role of your compiler is actually that of a test tool. In a sense, it tests your code for a set of very specific problems, such as type mismatches, undefined functions etc. It might even warn you about some null pointer issues that would normally surface during execution. The compiler knows a lot about you code, and this is is also why it makes sense turn on “treat warnings as errors”. Without it, warnings will go unnoticed, and you will never have a clean build.
After the compiler tests, the next level of tests are the unit tests. Most IDEs allow you to run your unit tests by the press of a button. Again, you will have the task switch described above. Furthermore, the results are often not as integrated into the IDE as the compiler output. But if the code compiles automatically when we write our code, why can’t the unit tests run too, nice and clean and fully integrated?
This is probably the question Kent Beck asked himself when he came up with the idea of JUnit Max. JUnit Max is an Eclipse plugin designed to help test-driven development. When you save a file, JUnit Max executes the unit tests for you. Results are shown as red icons on failed tests. Since big projects can have thousands of unit tests, JUnit Max contains some extra logic to order the execution of the tests. First of all, it will run the fastest of your unit tests first. This will provide you with fast feedback. Second, it will run those of your tests that failed recently. Those tests are more likely to fail than those failing a long time ago. Using JUnit Max for test-driven development is awesome. Just save your files and it will provide you with instant feedback. I use it when I do “acceptance unit test-driven development” (see post) and it really speeds things up.
So, what is the next level of tests after unit tests? What is the next level of tests that would be beneficial to execute when you press “save”? How about running some functional acceptance tests automatically? It would probably require a full and successful build. The same prioritization between tests as above would be helpful: run the fast and the shaky tests first. Running acceptance tests automatically could work, and could be useful.
What’s next? Load tests? Full integration tests of a system? And after that, deploy automatically? Imagine, you change a single line of code, save, and minutes later, the changes are deployed live. Now that is what I call continuous deployment! Ok, I agree, deploying without committing to a source repository is probably not what you want. :) (I guess for deployment, doing it 50 times per day is cool enough.)
I just finished reading the book “Implementing Lean Software Development – From Concept to Cash” by Mary and Tom Poppendieck. In essence, “lean” means “reduce waste”, and “waste” means “everything that does not make your customers happier”. Many concepts in the book was pioneered by Toyota while applied to manufacturing. Toyota then moved on and introduced lean in product development (including software).
When a customer ask you for a feature, how long does it take until they get it (calendar time)? While implementing it, how much effective time have you spent on the feature? The difference between calendar time and implementation time is waste (since waiting makes your customer unhappy). This waste can be analyzed using Value Stream Mapping (example). Typical things that takes time for no good reason is waiting for approval (solution: approve within 24h instead of in the bi-weekly meeting) and long fixed releases. I have come across really long release cycles in previous work places and when talking to friends at other companies. (Half a year, a year is not unusual, which should be contrasted with the book’s claim that Toyota developed and released the Prius within a year and a half!). The solution is to release and/or deploy more often. In order to do this, we need a relentless focus on quality (= no bugs).
According to the book, having bugs in a bug tracking system is waste. Software with bugs in it is not finished work, and piling up unfinished work is waste (since it cannot be released to your customer until it’s finished). Also, someone has taken the time to enter the information, and someone must look at it to resolve the bug. Meanwhile, someone else might stumble across the same bug. All this takes time. Lean software development says you should solve the bugs right now. In order to make changes to solve the bug right away, you need tests to make sure you don’t break something else. Good quality software starts with rigorous and automated tests on acceptance and unit level (see my previous posts: test-driven development done right and edge-to-edge unit tests). Reducing the extra work from bugs will free up resources to implement new features.
Lean software development is about continuous improvements. Every day, we should ask questions like: How do I make my customer happier? How do I reduce waste in order to release faster? But who should answer these questions and implement the improvements? Lean says it’s the people closest to the problems. Inevitably, in software development, it will be the programmers! I think this is the most appealing part of the contents of the book.
This is a fantastic book and I am sure I will read it again pretty soon. Read it!
The term unit test implies at least two different things. First, it means testing your code at the smallest unit, which is a function or a class (well, technically, you test the methods of a class, but they would make little sense in isolation). Second, it means writing test code in a language to test functionality in the same language.
Normally, when I write C++ code to test my C++ functionality, I tend to stay away from the “unit” level. Instead, I like tests that exercise the system edge-to-edge, resembling the interactions with the outside world as much as possible. Now, full edge-to-edge testing is normally not possible since the peripheral parts of a system are often hard to control. For example, let’s say I have a system with network on one side and a GUI on the other side. A realistic test case would have traffic over the network and a GUI reflecting that. But taking the network as an example, it complicates testing due to issues like slow response times, need for a remote side and network failures. So I settle for testing up to the interfaces of the network and GUI: you would inject network “traffic”(or time-outs) on the network interface, verify that the GUI interface is told to show something, do some user input on the GUI interface and watch outgoing network traffic being generated.
As with everything, there are pros and cons when testing at this level. To me, the main benefits compared to low-level unit tests are:
- Having tests at that level gives me confidence that there’s a reasonably low probability of faulty interaction with the outside world.
- The boundaries of your system are much less likely to change than the internals. This means you are less likely to spend time changing your tests.
- It is easy to argue for the business value of the tests. They correspond well to what the customer expects and are used to guarantee the quality.
- The tests are a decent measure of progress. Having a passing test means you are close to something to show to your customer.
- It’s fun! And you can test-run your system before you have a network and a GUI.
The downsides I’ve experienced compared to testing at the unit level are:
- Tests like these can make it harder to achieve decent code coverage. For example, your code might involve randomness, timing issues or use of the current time. You will have to make sure these can be controlled from the test context.
- High-level testing can be hard to introduce late in the development process. For this to succeed, the whole system must be designed for testability. See the previous item.
- The tests become monolithic. I’ve come across the situation where parts of my system were broken out to form a new shared component. The new component has to have tests of its own (or someone changing it won’t notice it’s broken until they run your tests). Your tests use the classes of your system, which are not suitable to use in a shared component since dependencies would go the wrong way.
- It might be overkill for testing some parts of the system. For example, if you have some deep-down string manipulation code, you should go ahead and unit test it (in the true sense of the word). It’s all about choosing the proper tools for the problem.
- Due to complexity in the lower levels of your software, you might be facing a combinatorial explosion of different test cases. You will have to select a few representative test cases and resort to normal unit testing of test the low-level parts. See the previous item.
- Testing on this level poses sort of a communication problem. If I call my tests “unit tests”, most people think only of tests on the lowest level. If I call them “acceptance tests” or “functional tests”, someone will inevitably assume I have properly tested the system from the outside, edge-to-edge (which is definitively necessary, even with the tests described above). Calling them e.g. “functional unit tests” only adds to the confusion. (“What do you mean? Is it a unit test? Is it a functional test? Surely, it can’t be both.”) If you know of terminology that could help, let me know. Until I hear from you, I will just call them Edge-to-Edge Unit Tests.
As I said before, it’s about choosing the proper tool for your problem. If at all possible, I resort to “unit testing” at the highest possible level. If you haven’t done so, you should give it a try.