The Random Sustainable Coder

Looking for good books? Here are two on software development and one on… randomness.

The Clean Coder: A Code of Conduct for Professional Programmers” by Robert Martin addresses what it takes to be successful in the long term as a software professional. First and foremost, a software professional takes responsibility for himself, his peers and his product. For example, he speaks the truth. He voices his opinion if a deadline will not be met. Martin has a series of conversations between different roles (such as programmer, project leader) which are pretty entertaining. There are lots of topics in this book that are important if you want to be long-lived and respected as a software professional. If you don’t have time to read the whole book, at least read the first chapter, which is gold.

Sustainable Software Development: An Agile Perspective” by Kevin Tate addresses what it takes to be successful in the long term when developing a software product. It is not unusual to see software products released after a year and then maintained for ten more years. This sets some specific requirements on the software, the development environment and the mindset of the developers. For example, the software must be able to accommodate change at reasonable cost and without breaking. In order to be able to deliver fixes and new features, the product must always be in good shape. Preferably good enough to be delivered every day, if so desired. This means the development process and the mindset of the developers must be heavily focused on quality. We must shift from defect detection (finding bugs) to defect prevention (avoiding bugs). Root cause analysis is key. Read the book, the message is great.

Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets” by Nassim Nicholas Taleb is a very interesting book! It is about randomness and how it affects our life. For example, how much of the success in your own career is due to chance? According to the author, if you’re a stock broker, it’s probably close to 100% (Taleb does not go easy on his fellow finance colleagues! :). If you’re a software developer, my gut feeling is that luck is much less significant. Re-live your life a million times, and your average life salary would probably still be around your current. Not so for the stock broker.

Taleb also describes an interesting scam. Send one million emails saying “the stock market will rise next week”. Send another million emails saying “the stock market will fall next week”. Next week, if the stock market rises, keep only the first million email addresses. To half of them, send an email saying “stocks will rise next week”, and to the second half “stocks will fall”. Every week, keep only the half where you predicted correctly. After 10 weeks you will have hundreds of people thinking you have super powers… This is called “survivor’s bias”. Taleb applies the exact same reasoning when a stock broker asks him for a job. Taleb won’t ask “How did you manage to do so well in the stock market in the past?”, but rather “How many like you were there from the beginning?”! This is a fantastic book, read it.

Sustainable Software Development, Part 2: Managing Complexity

As the code base grows, the complexity of your code increases. The pace of software development often slows down as the product matures. Why is this? What can we do to manage complexity?

Sustainable software development is about retaining software development velocity over the life-time of your product. In part one, I covered the technical debt side of sustainable software development. In essence, when you postpone refactoring, bug fixing, testing etc., you accumulate debt. As all debt, it has to be re-payed. But even worse, you will pay interest. Every time you think about, swear about, discuss, plan or anything (but fix) your known problems, you pay.

In this second part, we address another challenge to a sustainable pace of development: the increased complexity of the software. In order to avoid letting the complexity make your life miserable, we need to manage the complexity. For brevity, we will touch only upon these four areas:

  • Simplicity
  • Dependencies
  • Hierarchy
  • Abstraction

Fred Brooks makes the distinction between essential and accidental complexity (also discussed in his book “The Mythical Man-Month”). Essential complexity comes from your problem domain. The problem will have an inherent complexity you cannot get away from. Accidental complexity comes from the solution you’ve chosen. The “Keep it simple, stupid!” (KISS) design principle encourages developers to prefer simple solutions over those more complex. Thus, in order to manage complexity, it is important to keep the accidental complexity to a minimum. For example, dividing your code using the Single Responsibility Principle is one way, so that different pieces of functionality are not tangled up in a single class.

Another example of accidental complexity is the introduction of unnecessary dependencies in your software. Every subset of your code will have connections and communication with other subsets of your code. In a naive software implementation, the number of connections would grow wildly with the size of the software. The interconnectedness of your code is one aspect of complexity that needs to be managed. Decomposition and decoupling will help you, and they should be applied to all levels of your system: service, sub-system, module, class, function etc. Various design principles (e.g. Dependency Inversion Principle) and patterns can guide you.

Steve McConnell discusses hierarchy and abstraction in his article “Keep it simple” (and in his book “Code Complete”). Structuring your software as a hierarchy is a means of decomposing your solution into more manageable pieces. Normally, different kinds of hierarchies exist within your code. For example, think about the difference between the relation between objects (“which object creates/destroys/contains/depends on/owns the memory of which”) and the relation between functions (“which function calls which”). One architectural pattern that can help the high-level structure of your code is Layer (defined in the book “Pattern-Oriented Software Architecture: A System of Patterns” by Buschmann et al.). One upside of a structured approach like Layer is that tooling, such as automatic mapping which classes depend on which, can help you enforce your structure over time.

Abstraction is all about what level of detail is visible at what level (see also my post on abstraction). For example, the file system is an abstraction hiding the hard-drive with its tracks and sectors. When writing a program, you refer only to “file x”, and not to “track y, sector z”, which helps tremendously. Abstraction will help you reason about your program. Abstractions are largely domain dependent. Your domain will decide what is a good level of abstraction, hiding unnecessary detail while not constraining you.

Last words from McConnell’s “Keep it simple” article: “Neither hierarchies nor abstractions reduce the total number of details in a program — they might actually increase the total number. Their benefit arises from organizing details in such a way that fewer details have to be considered at any particular time.”

How do you fight complexity in your daily work?

Sustainable Software Development, Part 1: Managing Technical Debt

Software projects sometimes go bad. The pace of development is not sustainable. To achieve sustainable software development, we need to keep our focus on what’s important: the long-term health and maintainability of our source code.

Robert Martin has written a nice article on Scrum projects, and how applying Scrum often start out with hyper-productivity. However, most projects slow down when code size increases. After a while, we might even experience that progress approaches zero as time approaches infinity (if the project is not discontinued before it :).

Why is that? Martin lists a number of activities that are well-known to the experienced developer, such as unit testing, continuous integration and code coverage measurements. Most developers would even agree that applying them will help retaining productivity over the long run. Still, the law of least resistance often takes us down a different path. A strong focus on feature growth is very seductive, as it makes everybody happy in the short term: the customer is happy about their new features, the product owner is happy when he can satisfy the market and the team is happy when everybody commends them for their work. At least for a short while. The lack of focus on quality (such as a lack of automated unit tests and a poor continuous integration setup) will catch up with you.

Technical debt are the things you know you must do but haven’t done yet: fixing a bug, adding more unit tests, refactoring, cleaning up etc. Just as normal debt, it has to be payed back. No surprise there. Unfortunately, just like a normal debt, technical debt also come with an interest rate. You pay interest every time you say “aargh, this would have taken me ten minutes had I refactored this module” or “aargh, I just spent half a day tracking down a problem caused by a known bug”. Discussing, planning or thinking about what must be fixed is paying interest. Soon, your house belongs to the bank. Software development like this is not sustainable – it will come to a halt.

To avoid this, and to achieve sustainable software development, we need to keep our focus on what’s important: the long-term health and maintainability of our source code. Just like the debt analogy, there’s a savings analogy. Invest some money in a savings account (or the stock market), and you will receive interest instead of paying. As a start, it is necessary to make an investment in good practices: fix things now, and use tools and techniques to ensure quality. You will be able to deliver new features for a long time and your velocity will stay high, or even increase over time.