Sustainable Software Development, Part 2: Managing Complexity

As the code base grows, the complexity of your code increases. The pace of software development often slows down as the product matures. Why is this? What can we do to manage complexity?

Sustainable software development is about retaining software development velocity over the life-time of your product. In part one, I covered the technical debt side of sustainable software development. In essence, when you postpone refactoring, bug fixing, testing etc., you accumulate debt. As all debt, it has to be re-payed. But even worse, you will pay interest. Every time you think about, swear about, discuss, plan or anything (but fix) your known problems, you pay.

In this second part, we address another challenge to a sustainable pace of development: the increased complexity of the software. In order to avoid letting the complexity make your life miserable, we need to manage the complexity. For brevity, we will touch only upon these four areas:

  • Simplicity
  • Dependencies
  • Hierarchy
  • Abstraction

Fred Brooks makes the distinction between essential and accidental complexity (also discussed in his book “The Mythical Man-Month”). Essential complexity comes from your problem domain. The problem will have an inherent complexity you cannot get away from. Accidental complexity comes from the solution you’ve chosen. The “Keep it simple, stupid!” (KISS) design principle encourages developers to prefer simple solutions over those more complex. Thus, in order to manage complexity, it is important to keep the accidental complexity to a minimum. For example, dividing your code using the Single Responsibility Principle is one way, so that different pieces of functionality are not tangled up in a single class.

Another example of accidental complexity is the introduction of unnecessary dependencies in your software. Every subset of your code will have connections and communication with other subsets of your code. In a naive software implementation, the number of connections would grow wildly with the size of the software. The interconnectedness of your code is one aspect of complexity that needs to be managed. Decomposition and decoupling will help you, and they should be applied to all levels of your system: service, sub-system, module, class, function etc. Various design principles (e.g. Dependency Inversion Principle) and patterns can guide you.

Steve McConnell discusses hierarchy and abstraction in his article “Keep it simple” (and in his book “Code Complete”). Structuring your software as a hierarchy is a means of decomposing your solution into more manageable pieces. Normally, different kinds of hierarchies exist within your code. For example, think about the difference between the relation between objects (“which object creates/destroys/contains/depends on/owns the memory of which”) and the relation between functions (“which function calls which”). One architectural pattern that can help the high-level structure of your code is Layer (defined in the book “Pattern-Oriented Software Architecture: A System of Patterns” by Buschmann et al.). One upside of a structured approach like Layer is that tooling, such as automatic mapping which classes depend on which, can help you enforce your structure over time.

Abstraction is all about what level of detail is visible at what level (see also my post on abstraction). For example, the file system is an abstraction hiding the hard-drive with its tracks and sectors. When writing a program, you refer only to “file x”, and not to “track y, sector z”, which helps tremendously. Abstraction will help you reason about your program. Abstractions are largely domain dependent. Your domain will decide what is a good level of abstraction, hiding unnecessary detail while not constraining you.

Last words from McConnell’s “Keep it simple” article: “Neither hierarchies nor abstractions reduce the total number of details in a program — they might actually increase the total number. Their benefit arises from organizing details in such a way that fewer details have to be considered at any particular time.”

How do you fight complexity in your daily work?

Characteristics of a Software Professional

At work, I have been challenged with the question “What are the most important characteristics of a software developer?“. This is a tough question, and no matter what you decide to include, you have to leave something out.

I’ve been part of software development projects in various companies. The successful projects teach you invaluable lessons. The dysfunctional projects even more so. Inevitably, a list of characteristics would include qualities I value and desire in my fellow colleagues, as well as characteristics I’d expect them to want in me. (So this is also a long todo list for me. ;)

To get some kind of structure, I decided on three main categories: Professionalism, Long-term code and Quality mindset.

Professionalism

Take responsibility

You are a professional developer, and professionals act responsibly. If things go wrong,
take responsibility. Understand why things went bad. Learn, adapt and make sure it
never happens again. When faced with a difficult choice, “do the right thing”. Optimize for the long run even if it results in more work today. Be a team player, even if this means saying no. Speak the truth.

Know your product

We’re part of a business. Without successful products, there will be no business.
Know your users and their needs. Use your product, use competitor’s products,
visit customers and watch them use your product (from purchase/download to
installation to day-to-day usage to upgrade to uninstall and so forth).

Continuous learning

Be humble and practice relentless inquiry. Question everything (since everything has potential for improvement), embrace questions on your code, realize others’ suggestions might improve your work, be happy other developers change “your” code
(it’s good enough to understand!). Improve yourself and others. Become a better developer by reading books, papers, blogs etc. Watch instruction videos, listen to podcasts, try new technologies, participate in open source development, discussion groups etc. Discuss your code and what you read with your peers. Talk to developers of other specialities. Teach others what you know well.

Long-term code

Communication

You write a line of code once, but it is read hundreds of times. Invest time to write code easy to read for others. Write code at the right level of abstraction, abstract enough for expressiveness but without hiding necessary detail. Adhere to design principles as
they capture proven ways to high-quality code. Apply design patterns to better communicate your intent.

Maintainability

Decouple the different parts of your software, on every level – sub-system, module, class and function. Write extendable code, so that you can add functionality with minimal change to existing code. Avoid technical debt, and repay debt as soon as possible. Interest has to be payed on all debt.

Proven functionality

Never deliver code unless you’ve proven it works. If you don’t test it, it will be faulty.
Write testable code. Make it testable from the start, later it will be too expensive. Automate your tests to run before check-in, after check-in and nightly. If the tests are not executed automatically, they will not be updated and soon be obsolete. Without automated tests, no-one will dare change any code. Write fast and reliable unit tests. Write realistic integration tests. For each new feature, write acceptance tests. Automate everything.

Quality mindset

Quality is your responsibility

You, as a developer, is responsible for the quality of the product. Other roles can help you spot problems or give you more time, but they cannot affect quality. Never ship anything without being certain of its correctness.

Find bugs early

Find bugs early in the development process. If a bug can be found by the developer, it should be. If you need tools, get them. If you need hardware, get it. If a bug is found late, understand why it was not found earlier. Fix the process so that bugs of this kind never slips through. Automate.

Fix it now

If you find a bug, fix it now instead of filing a bug report. Ask your colleagues to help out. You will save time. File bug reports on things that couldn’t be solved in half a day.
Do things properly the first time. If you don’t have time to do it right today, when are you ever going to find time? Give time estimates that allow you to produce quality products. Think about what is stopping you from being more productive. Fix it, and then move on to the next thing stopping you.

If you’re interested in these things, you should have a look at Poppendiecks’ “Lean software development” books, Senge’s “The fifth discipline“, Martin’s “Clean coder“, McConnell’s “Code complete“, McConnell’s video “World class software companies” and others. I also provide some resources in previous blog posts (e.g. videos and books).

What characteristics do you value in a software developer?

Design Patterns By Example: Implementing a State Machine

How do you implement the State pattern, while separating the different concerns? We use an example to discuss how to write code easy to understand and maintain.

When writing code, our classes often go through a series of transformations. What starts out as a simple class will grow as behavior is added. If care is not taken, your code will become difficult to understand and maintain. For example, assume you’re implementing a telephone. First, you support only the simplest of usages: your phone is not connected to the telephone jack. Thus, the phone can be either on-hook or off-hook.

You implement this as a class Telephone with a single boolean member offHook. Your phone has two operations pickUp() and hangUp() which manipulate the boolean. Great. Now you jack up your phone and want to implement the next use-case: the user picks up the phone (after which he should hear a tone) and presses a key (after which the tone should stop). You introduce a new boolean hasPressedFirstKey and a function pressKey(int key).

Boolean-Oriented Programming

Already, you have a couple of problems with your code. First, you have modeled three states (on hook, off hook while waiting for first key to be pressed, off hook while waiting for more keys to be pressed) with two boolean variables. Thus, there is one combination of values that does not correspond to a valid state for your telephone: offHook = false and hasPressedFirstKey = true. The state of a bigger system might involve more variables and might even be scattered over many classes. In that case, “boolean-oriented programming” like this makes the code very hard to understand.

Second, member variables live for as long as the object does. In our telephone example, the variable hasPressedFirstKey makes little sense as soon as the full phone number has been dialed and someone on the other side has answered. So the actual lifetime of hasPressedFirstKey is shorter than its “physical” lifetime. If we strive for self-documenting code, this is pretty far from it. And of course, this is even worse in the case of a complex system. So what can we do about it? State design pattern to the rescue.

The State Design Pattern

The Wikipedia page on the State design pattern says that the purpose of State is to “represent the state of an object”. In our telephony example, we would create three state classes: e.g. OnHookState, OffHookWaitForFirstDigitState, OffHookWaitForMoreDigitsState. State classes will only model valid states of the telephone, removing the first problem from above. Also, since there is always a valid state for our telephone, we remove the lifetime problem from above.

I use the State pattern for two reasons: First, it captures the behavior of the code in a single place. This will make your code easy to understand. Second, it makes it easy for you to separate what the system does (the behavior) from how it’s done (the implementation). This will make your code easy to maintain and test. Let’s go through an example.

States, Events and Actions

We have already mentioned the state classes. They all inherit from a common interface ITelephoneState. The state interface defines the events that the system accepts (here in Java):

interface ITelephoneState {
    void pickUp();
    void hangUp();
    void pressKey(int key);
}

As said earlier, we want the state class to be explicit about the system behavior, but without involving implementation details. Instead of letting the state class contain implementation details, we delegate to an action interface ITelephoneAction (shown later). Let’s implement the OffHookWaitForFirstDigitState class:

class OffHookWaitForFirstDigitState implements ITelephoneState {
    OffHookWaitForFirstDigitState(ITelephoneAction action) {
        this.action = action;
    }
    void pickUp() { /* do nothing */ }
    void hangUp() {
        action.stopTone();
        action.changeState(new OnHookState());
    }
    void pressKey(int key) {
        action.stopTone();
        action.changeState(new OffHookWaitForMoreDigitsState());
    }
    private ITelephoneAction action;
}

We see that the state conforms to the ITelephoneState interface. ITelephoneAction is defined like this:

interface ITelephoneAction {
    void stopTone();
    // ... more telephone specific functions here ...
    void changeState(ITelephoneState newState);
}

Thus, the responsibility of the state class is to implement the behavior (what the system does). The responsibility of the action class is to provide the implementation (how things are done). This makes the state code very easy to read.

The ITelephoneAction interface is normally implemented by the Telephone class:

class Telephone implements ITelephoneAction {
    Telephone() {
        state = new OnHookState(this);  // start state
    }

    // public interface
    public void pickUp() { state.pickUp(); }
    public void hangUp() { state.hangUp(); }
    public void pressKey(int key) { state.pressKey(key); }

    // implements ITelephoneAction
    void stopTone() { /* do something */ }
    void changeState(ITelephoneState newState) {
        state = newState;
    }
}

Note that the state never talks directly to the Telephone class. This ensures that the state uses only what’s needed from Telephone, and not the full range of public functions in Telephone. Furthermore, talking to an interface will allow you to unit test the logic of the state machine without using Telephone. To summarize the interactions between Telephone and its state: the Telephone class talks to the state class through the ITelephoneState interface; the state class talk to the Telephone class through the ITelephoneAction interface.

Trace Logging

As an added bonus, having a clear separation between state, event and action will make it easy for you to implement nice trace logs. If we trace every state change, event function call and action function call, we can use indentation to show the flow through the state machine:

OnHookState
    offHook                      // event
        changeState              // action
OffHookWaitForFirstDigitState    // new state
    pressKey
        stopTone
        changeState
OffHookWaitForMoreDigitsState
    ...

As you see, states are not indented, events are indented one level and actions resulting from the event are indented two levels.

The Truth Behind Design Patterns

Design patterns help you write good software. How can you master all 23+ of them? You are probably better of focusing on the underlying truth: the design principles.

The book “Design Patterns: Elements of Reusable Object-Oriented Software” is a classic and a must-read (see post). When I first came across the book, I read about all those 23 fantastic patterns. They were all different and applicable in different situations. As a computer scientist, I hate special cases, so I wanted to see some kind underlying truth. At the time, I did not see a pattern (ironic, huh? :).

The book “Implementing Lean Software Development” (see post) makes the following definitions:

  • Principles are underlying truths that don’t change over time […]”
  • “[…] practices are the application of principles to a particular situation”

If the design patterns are 23 different applications of some underlying truth, this will make them object-oriented software development practices. Then what are the principles of object-oriented software development?

Here are my favorites among the design principles:

  1. Decoupling: Strive for loosely coupled designs between objects that interact.
  2. Open/Closed Principle: Classes should be open for extension, but cOpen/Closedlosed for modification (= classes should be easy to extend without needing to change them).
  3. Abstraction: Depend on abstractions, do not depend on concrete classes.
  4. Single Responsibility: A class should have only one reason to change.
  5. DRY (Don’t Repeat Yourself): Store information in a single place, do not scatter it throughout your code.

To exemplify, let’s look at design patterns that support some of the above principles. Two of the patterns that are closely related to Decoupling is Observer and Factory Method. Observer lets a class notify other classes without knowing about them. Factory Method lets one class create objects without knowing which type they are. Not knowing about each other typically favors a loosely coupled design.

Two of the patterns that support the Open/Closed Principle is Strategy and Template Method. Strategy lets you vary the behavior of an object as if the type was not fixed at compile-time. You determine the behavior by supplying a strategy object e.g. during construction. Template Method will let you implement parts of an algorithm. Typically, you subclass the algorithm and implement specialization in the subclass. Both let you change the behavior without changing the code.

If you adhere to the object-oriented principles above, design patterns come up naturally. I think this is also the best way to learn how to apply design patterns. Instead of actively looking for places where your design patterns could fit, use the principles and the patterns will follow.

Some of the design principles are hard to match with any of the design patterns. For example, Abstraction is typically something expressed in the domain language of the problem you’re trying to solve. Thus, it helps you to write code that relates better to your domain. The same goes for Single Responsibility, since the reason for change is domain specific. If you look at all design principles, it is clear that they are themselves special cases that can be divided into groups of related responsibility. It makes you wonder what is the underlying truth of the design principles?

Design Patterns: Elements of Reusable Object-Oriented Software

Duplication-Driven Development?

What is the primary force that drives you when doing test-driven development? Surprisingly, it might just be code duplication.

In the book “Test-Driven Development by Example“, Kent Beck uses an implementation of money as an example of TDD. At one point, a test case says “assert(someFunc(5) == 5 * 2)” and to satisfy the test case, someFunc is implemented simply by returning a hard-coded 10. “Now that is dirty” you say (and that’s what I said :). According to the book, the reason for hard-coding is that we want the test case to pass as soon as possible. After that, the refactoring phase can take over and we can clean up the code.

The Petronas Twin TowersHere comes the interesting part: hard-coding is a terrible habit, right? And we don’t want our bad habits to show up in the code. This sign of bad taste should be reason enough to fix it, right? However, there is a more fundamental reason for getting rid of the constant: code duplication. The constant 2 * 5 = 10 occurs both in the test case and in the code. To remove the duplication, change someFunc(param) to return “param * 2” (later, the book also gets rid of the 2 by replacing it with a member variable). To sum up, instead of acting on “taste”, we use the well proven design principle of (avoiding) code duplication.

A fascinating insight from the book is that the design of the code, too, can grow naturally from code duplication. For example, let’s say we have a class that encapsulates some algorithm. When another class with similar behavior is required, the book suggests we can accept some code duplication (even extensive copy-paste) to pass the tests. After getting the tests to pass, we refactor. Lets say the difference between the two classes is the fact that they use different algorithms to carry out their work. Apart from that, they are identical. To avoid code duplication, we combine the two classes. Quite naturally, this results in us isolating the algorithms behind an interface. By following the path of least resistance for removing code duplication, we have implemented the strategy pattern. To summarize, this suggests that code duplication helps us drive the high-level design of our code. If true, it is kind of a comforting feeling that refactoring and code duplication naturally leads us to good design.

All and all, “Test-Driven Development by Example” is well-written, easy to read and quite a good book, albeit maybe more suitable for the aspiring test-driven designer than the seasoned on.

Decoupling Starts with an Interface, but Where Does It End?

Writing decoupled software components means writing them so that they depend on each other as little as possible. If one of your classes uses another class directly, they will inevitably be coupled together. Use one, and you will have Couplingto use the other. To decouple components, we normally think of letting our classes talk to interfaces instead of concrete classes. Any subclass can hide behind the interface, so the coupling between the classes is reduced. One good example of “talking to interfaces” is the Factory method design pattern. Instead of using new on a concrete class, you will ask an interface to create the object for you. But decoupling is more than just talking to interfaces.

When I first started programming Java, I came across Spring and its inversion of control containers (IoC). Compared to the Factory method above, Spring IoC takes object creation decoupling a step further. By using XML files, you specify which object to create and which arguments to give to the constructor (e.g. other objects). Coming from C++, it was pure magic the first time I saw it. Nowhere in the code do your classes refer to each other. Now, that is decoupling!

Later, I started working with OSGi. It is a Java framework which allows for independent life-cycles of modules (called bundles). For example, you could replace a bundle with a newer version during run-time. Other bundles won’t notice they are communicating with something new. Bundles communicate with each other by publishing and consuming services, which are just plain interfaces. XML is used to specify what services are published and consumed by a bundle. Similar to above, individual modules never refer to each other in the code. Again, that is decoupling.

Web APIs, such as a REST or SOAP API over HTTP, are often used to provide access to web or cloud services. Also, more and more enterprises expose their internal sub-systems as web services. To assemble software using web APIs, you can combine software components that may execute anywhere in the world. The pattern here seems to be that the larger the software component, the more decoupling is possible (or required?). Decoupled components through interfaces needed to be compiled together. Decoupled components through web APIs need not even be in the same time zone. The downside is that more powerful decoupling techniques require a lot of work.

Combining decoupling on different levels will make your system more resilient to change and open up for use in new and unexpected ways. The next time you think of ways to make your code more loosely coupled, remember that decoupling is not only about interface classes!