Sustainable Software Development, Part 2: Managing Complexity

As the code base grows, the complexity of your code increases. The pace of software development often slows down as the product matures. Why is this? What can we do to manage complexity?

Sustainable software development is about retaining software development velocity over the life-time of your product. In part one, I covered the technical debt side of sustainable software development. In essence, when you postpone refactoring, bug fixing, testing etc., you accumulate debt. As all debt, it has to be re-payed. But even worse, you will pay interest. Every time you think about, swear about, discuss, plan or anything (but fix) your known problems, you pay.

In this second part, we address another challenge to a sustainable pace of development: the increased complexity of the software. In order to avoid letting the complexity make your life miserable, we need to manage the complexity. For brevity, we will touch only upon these four areas:

  • Simplicity
  • Dependencies
  • Hierarchy
  • Abstraction

Fred Brooks makes the distinction between essential and accidental complexity (also discussed in his book “The Mythical Man-Month”). Essential complexity comes from your problem domain. The problem will have an inherent complexity you cannot get away from. Accidental complexity comes from the solution you’ve chosen. The “Keep it simple, stupid!” (KISS) design principle encourages developers to prefer simple solutions over those more complex. Thus, in order to manage complexity, it is important to keep the accidental complexity to a minimum. For example, dividing your code using the Single Responsibility Principle is one way, so that different pieces of functionality are not tangled up in a single class.

Another example of accidental complexity is the introduction of unnecessary dependencies in your software. Every subset of your code will have connections and communication with other subsets of your code. In a naive software implementation, the number of connections would grow wildly with the size of the software. The interconnectedness of your code is one aspect of complexity that needs to be managed. Decomposition and decoupling will help you, and they should be applied to all levels of your system: service, sub-system, module, class, function etc. Various design principles (e.g. Dependency Inversion Principle) and patterns can guide you.

Steve McConnell discusses hierarchy and abstraction in his article “Keep it simple” (and in his book “Code Complete”). Structuring your software as a hierarchy is a means of decomposing your solution into more manageable pieces. Normally, different kinds of hierarchies exist within your code. For example, think about the difference between the relation between objects (“which object creates/destroys/contains/depends on/owns the memory of which”) and the relation between functions (“which function calls which”). One architectural pattern that can help the high-level structure of your code is Layer (defined in the book “Pattern-Oriented Software Architecture: A System of Patterns” by Buschmann et al.). One upside of a structured approach like Layer is that tooling, such as automatic mapping which classes depend on which, can help you enforce your structure over time.

Abstraction is all about what level of detail is visible at what level (see also my post on abstraction). For example, the file system is an abstraction hiding the hard-drive with its tracks and sectors. When writing a program, you refer only to “file x”, and not to “track y, sector z”, which helps tremendously. Abstraction will help you reason about your program. Abstractions are largely domain dependent. Your domain will decide what is a good level of abstraction, hiding unnecessary detail while not constraining you.

Last words from McConnell’s “Keep it simple” article: “Neither hierarchies nor abstractions reduce the total number of details in a program — they might actually increase the total number. Their benefit arises from organizing details in such a way that fewer details have to be considered at any particular time.”

How do you fight complexity in your daily work?

Characteristics of a Software Professional

At work, I have been challenged with the question “What are the most important characteristics of a software developer?“. This is a tough question, and no matter what you decide to include, you have to leave something out.

I’ve been part of software development projects in various companies. The successful projects teach you invaluable lessons. The dysfunctional projects even more so. Inevitably, a list of characteristics would include qualities I value and desire in my fellow colleagues, as well as characteristics I’d expect them to want in me. (So this is also a long todo list for me. ;)

To get some kind of structure, I decided on three main categories: Professionalism, Long-term code and Quality mindset.

Professionalism

Take responsibility

You are a professional developer, and professionals act responsibly. If things go wrong,
take responsibility. Understand why things went bad. Learn, adapt and make sure it
never happens again. When faced with a difficult choice, “do the right thing”. Optimize for the long run even if it results in more work today. Be a team player, even if this means saying no. Speak the truth.

Know your product

We’re part of a business. Without successful products, there will be no business.
Know your users and their needs. Use your product, use competitor’s products,
visit customers and watch them use your product (from purchase/download to
installation to day-to-day usage to upgrade to uninstall and so forth).

Continuous learning

Be humble and practice relentless inquiry. Question everything (since everything has potential for improvement), embrace questions on your code, realize others’ suggestions might improve your work, be happy other developers change “your” code
(it’s good enough to understand!). Improve yourself and others. Become a better developer by reading books, papers, blogs etc. Watch instruction videos, listen to podcasts, try new technologies, participate in open source development, discussion groups etc. Discuss your code and what you read with your peers. Talk to developers of other specialities. Teach others what you know well.

Long-term code

Communication

You write a line of code once, but it is read hundreds of times. Invest time to write code easy to read for others. Write code at the right level of abstraction, abstract enough for expressiveness but without hiding necessary detail. Adhere to design principles as
they capture proven ways to high-quality code. Apply design patterns to better communicate your intent.

Maintainability

Decouple the different parts of your software, on every level – sub-system, module, class and function. Write extendable code, so that you can add functionality with minimal change to existing code. Avoid technical debt, and repay debt as soon as possible. Interest has to be payed on all debt.

Proven functionality

Never deliver code unless you’ve proven it works. If you don’t test it, it will be faulty.
Write testable code. Make it testable from the start, later it will be too expensive. Automate your tests to run before check-in, after check-in and nightly. If the tests are not executed automatically, they will not be updated and soon be obsolete. Without automated tests, no-one will dare change any code. Write fast and reliable unit tests. Write realistic integration tests. For each new feature, write acceptance tests. Automate everything.

Quality mindset

Quality is your responsibility

You, as a developer, is responsible for the quality of the product. Other roles can help you spot problems or give you more time, but they cannot affect quality. Never ship anything without being certain of its correctness.

Find bugs early

Find bugs early in the development process. If a bug can be found by the developer, it should be. If you need tools, get them. If you need hardware, get it. If a bug is found late, understand why it was not found earlier. Fix the process so that bugs of this kind never slips through. Automate.

Fix it now

If you find a bug, fix it now instead of filing a bug report. Ask your colleagues to help out. You will save time. File bug reports on things that couldn’t be solved in half a day.
Do things properly the first time. If you don’t have time to do it right today, when are you ever going to find time? Give time estimates that allow you to produce quality products. Think about what is stopping you from being more productive. Fix it, and then move on to the next thing stopping you.

If you’re interested in these things, you should have a look at Poppendiecks’ “Lean software development” books, Senge’s “The fifth discipline“, Martin’s “Clean coder“, McConnell’s “Code complete“, McConnell’s video “World class software companies” and others. I also provide some resources in previous blog posts (e.g. videos and books).

What characteristics do you value in a software developer?

Object-Oriented Programming Lecture at KTH: Slides etc.

Thank you all that participated in my lecture at KTH October 26, 2011. I had a lot of fun, and we had some good discussions. For you who were not there, it was about object-oriented programming and how to write good code. We used some example code to discuss OO principles and testing. Here’s the presentation (pdf)!

A couple of posts of relevance:

Software Architecture Built to Survive Change

How do you create an object-oriented architecture that can survive change? We discuss how to isolate the parts of your system that never changes, while still making it easy to add new functionality.

What is it that makes some software architectures fragile, wavewhile others are solid as a rock? Recently, I have come across a very appealing idea from multiple sources. First, a colleague of mine brought it up (hi, D!), then I read about it in a book “Lean Architecture for Agile Software Development” by James Coplien and Gertrud Bjørnvig. The book introduced me to the DCI software architecture, which I might cover in more detail in a later post (what I describe here is just parts of it). Learning a new idea is like learning a new word: once you know it, you see it everywhere.

Let’s walk through a simple (and highly contrived :) example. Assume we have a bank system with users, accounts, transactions, transaction logs etc. We have a use case to implement: pay interest to user’s account. Without giving it much thought, we could add a function addInterest to the Account class which calculates and deposits interest and updates the transaction log. Given a few hundred of these types of features, your Account class will be cluttered with functions. As you can imagine, this will eventually lead to poor readability, increased risk of breaking working code and problems parallelizing work. Over time, your code might complex enough to prohibit any kind of progress. Actually, this is the path many projects go.

Let’s try a different take. In the terminology of the “Lean Architecture” book, we want to separate “what the system is” (e.g. accounts, users) from “what the system does” (e.g. pay interest, transfer money). We create classes Account, TransactionLog etc. and define simple interfaces IAccount and ITransactionLog. The interfaces expose only administrative operations like getters/setters and add/remove functions (but no behavior or use case logic!). Among other functions, IAccount might expose addMoney while ITransactionLog might expose addLogEntry. We then implement our use cases as “algorithms” that operate on the interfaces. For example:

interface IUseCase
{
    void execute();
}

class PayInterestUseCase implements IUseCase
{
    PayInterestUseCase(IAccount account,
        ITransactionLog transactionLog) {...}

    void execute()
    {
        double interest = calculateInterest(account);
        account.addMoney(interest);
        transactionLog.addLogEntry(account.getId(),
            interest);
    }
}

Later, when we add a “money transfer” use case, we already have the addMoney and addLogEntry functions. We create a new use case class TransferMoneyUseCase. We withdraw money from one account (e.g. by providing a negative number to addMoney), deposit the money in the other account and update the log. The Account/TransactionLog objects and interfaces do not change at all, but we can still extend the system! After a few use cases, we have a set of stable interfaces on which we can build almost anything. Changes and additions to the interfaces will be fewer and fewer.

To me, the concept of separating domain objects (what-the-system-is) from business logic (what-the-system-does) is very appealing. One changes slowly, if at all. The other will evolve and change all the time. I like it. What drawbacks/benefits do you see? Have you tried it? Leave your comments below.

My Favorite Interview Question

Do you interview to hire a software developer? Maybe you’re being interviewed. In any case, this is my favorite question to determine how skilled you are.

What is the ultimate interview question? What question reveals whether or not a person is the one you’re looking for? Clearly, it depends on what you want. But even if you have a job advertisement outlining very specific technical skills, I argue in my previous post that, what you first and foremost need is someone that has a good sense on software structure. In object-orientation, structuring a program is really about understanding and applying object-oriented design principles. Now that we know what skills we’re looking for, it is easier to formulate a question.

I like evidence based interviews where we discuss what the potential hire did in the past. Thus, my favorite question is “Can you give me an example of good object-oriented code from your working career?”question answer Sometimes I even add “it doesn’t have to be code that you wrote”. You would certainly expect a professional programmer to be able to recall at least one good example from his working career. The point is not really to test whether or not he can remember code he wrote or read. Rather, it is to make sure he has an opinion. But most importantly, to have something to discuss. Anyone can do namedropping, so we need a concrete piece of code or a drawing to talk about. The upside with something that I didn’t choose is that the problem and solution domain (its terminology, background etc.) is known to my potential hire. That makes the interview situation less stressful. And I get to see if he is good at communicating ideas.

Surprisingly, this seems to be a tough question and I sometimes have to settle without an answer. Maybe it’s the pressure of the interview situation (hey, maybe it’s me!) but it reflects badly on the interview subject. If he can’t recall or don’t have an opinion on what is good code, how could I expect him to write any? So I have to move on. “How about bad examples of object-oriented programming?” This is a little bit easier. Everybody has opinions on other people’s code, and we can pick up from there: “Why is it bad? How can it be improved? What is the difference between bad and good?”. With luck, we get some examples to discuss.

I recently looked for decent examples as a basis for discussions around object-oriented programming. I came across this blog post by Reginald Braithwaite discussing the game of Monopoly. The main ideas are that Monopoly is well-known, but still complex enough to require the interview subject to discuss requirements. It also have some non-local properties, e.g. you can only build a house if you own all streets in that neighborhood. I like it. If the above fails, it will open up the discussion on object-oriented design principles in a nice way.

I think I will take a summer break from the writing now. A few weeks at least. See you!

 

Skills That Will Get You Hired

What is it that companies look for when they hire programmers? Surprisingly, they probably want something else than what’s in the job ad.

Job advertisements often focus on technologies (network programmingfor hire, windows programming, XML, you name it) or expert skills in a specific language. As a consequence, most CVs focus on the same thing. This is only natural, since people want to get hired. With otherwise equally strong candidates, the technology skills listed in the CV might affect who’s hired. But highlighting technology skills in a job ad is probably misguided, since I think these skills are not what companies really need. Let me explain what I mean.

In order to get a software system to do what we want, we need the above-mentioned technology knowledge. So, it’s definitively required, don’t get me wrong. But for the project to be successful over time, technologies are worth nothing unless the structure of the code is good enough for maintenance. (Somebody said that your code is in maintenance mode after the first line of code has been checked in. There is some truth in that.)

Also, acquiring a new piece of technology is not rocket science. Give it a couple of months, and you will have good enough knowledge. You will not become an expert, but there is no need for everyone to be an expert. To me, the skill to properly structure software is an essential one. Nevertheless, it is a rare one. Could it be that it is a skill hard to acquire? My point is this: avoid hiring people for knowledge that can be taught, without making sure he/she has what is essential. More specifically, what is that?

Here it is: I value people that write code that are testable, modular, readable and extensible. Without testability, I dare not touch your code. So I would have to throw it all away and start over, risk breaking it while making it testable or just dive in with my eyes closed. Nether is acceptable. With good modularity, the worst case scenario is that modules with unreadable code can be replaced or wrapped with tests. Modularity will also help readability and understanding. If you can understand the interfaces, understanding the internals become less important. Once we get down to internals of a module, readability becomes crucial. Only if I can read it (= understand it), can I maintain and extend it. Extensibility allows your software to meet customer’s needs and survive over time.

In object-orientation, there are design principles that will help you write testable, modular, readable and extensible software (see e.g. these three posts). If someone showed up with a good sense of how to write well-structured software using these design principles, he would certainly be a good hire. And conversely, if someone wasn’t convincing enough on his skills in writing structured software, he would be hard to hire. Regardless of his skills in other areas.

The Truth Behind Design Patterns

Design patterns help you write good software. How can you master all 23+ of them? You are probably better of focusing on the underlying truth: the design principles.

The book “Design Patterns: Elements of Reusable Object-Oriented Software” is a classic and a must-read (see post). When I first came across the book, I read about all those 23 fantastic patterns. They were all different and applicable in different situations. As a computer scientist, I hate special cases, so I wanted to see some kind underlying truth. At the time, I did not see a pattern (ironic, huh? :).

The book “Implementing Lean Software Development” (see post) makes the following definitions:

  • Principles are underlying truths that don’t change over time […]”
  • “[…] practices are the application of principles to a particular situation”

If the design patterns are 23 different applications of some underlying truth, this will make them object-oriented software development practices. Then what are the principles of object-oriented software development?

Here are my favorites among the design principles:

  1. Decoupling: Strive for loosely coupled designs between objects that interact.
  2. Open/Closed Principle: Classes should be open for extension, but cOpen/Closedlosed for modification (= classes should be easy to extend without needing to change them).
  3. Abstraction: Depend on abstractions, do not depend on concrete classes.
  4. Single Responsibility: A class should have only one reason to change.
  5. DRY (Don’t Repeat Yourself): Store information in a single place, do not scatter it throughout your code.

To exemplify, let’s look at design patterns that support some of the above principles. Two of the patterns that are closely related to Decoupling is Observer and Factory Method. Observer lets a class notify other classes without knowing about them. Factory Method lets one class create objects without knowing which type they are. Not knowing about each other typically favors a loosely coupled design.

Two of the patterns that support the Open/Closed Principle is Strategy and Template Method. Strategy lets you vary the behavior of an object as if the type was not fixed at compile-time. You determine the behavior by supplying a strategy object e.g. during construction. Template Method will let you implement parts of an algorithm. Typically, you subclass the algorithm and implement specialization in the subclass. Both let you change the behavior without changing the code.

If you adhere to the object-oriented principles above, design patterns come up naturally. I think this is also the best way to learn how to apply design patterns. Instead of actively looking for places where your design patterns could fit, use the principles and the patterns will follow.

Some of the design principles are hard to match with any of the design patterns. For example, Abstraction is typically something expressed in the domain language of the problem you’re trying to solve. Thus, it helps you to write code that relates better to your domain. The same goes for Single Responsibility, since the reason for change is domain specific. If you look at all design principles, it is clear that they are themselves special cases that can be divided into groups of related responsibility. It makes you wonder what is the underlying truth of the design principles?

Design Patterns: Elements of Reusable Object-Oriented Software

Design Principles by Example: Talk to an Interface or an Abstraction?

What is the relation between design principles “Talk to an interface, not an implementation” and “Talk to an abstraction, not a concrete”? When you apply them, you want to achieve different goals.

Two important design principles for writing good software are

  1. “Talk to an interface, not an implementation” and
  2. “Talk to an abstraction, not a concrete”.

Admittedly, they sound very much alike, so what is the difference between them?

communications towerAssume you are writing a client implementation that needs to communicate with a server somewhere. You have chosen to use a web socket for sending messages over the wire. To that end, you will use a class ClientWebSocketSenderImplementation. Now, the “Talk to interface, not an implementation” design principle suggests that talking to the web socket implementation class directly is inappropriate. Instead, you should talk to an interface ClientWebSocketSender.

Following the first design principle have several upsides. First, it will make your code easier to test. In this case, using the web socket implementation directly would use the network. That would make your unit tests slow and unreliable and might require a complicated setup phase. Second, talking to the web socket implementation directly would couple your client code to that specific implementation. If needed, changing web socket implementations would be difficult. Also, your code would not be reusable without shipping the web socket implementation.

We have chosen to send messages using a web socket. But there is really no need for our client application to know how messages are sent over the network. The second design principle says “Talk to an abstraction, not a concrete”. The concrete here is a web socket. A suitable abstraction in this context could be the ability to send messages over the web without specifying how. So we introduce an interface ClientWebSender. Depending on the application, we could take it one step further. It might make sense to abstract away the fact that we’re sending messages over the internet (for example, it could be over an IPC channel). We would end up with an interface ClientSender.

The second design principle will make your application more resilient to change. Without the abstraction, the web socket details might propagate throughout your code. For example, functions, arguments and return types could be specific to web sockets. If you would like to change your application to support message sending over e.g. HTTP or your own proprietary protocol over TCP, you would have to chase down all references to web sockets.

Last, the “Talk to an abstraction, not a concrete” does not require us to talk to an interface. You might have a class that represents the “abstraction” part and hides the “concrete” part (e.g. by delegating to a web socket implementation). So our two design principles serve different purposes and does not necessarily overlap. That said, they work very well in combination to write decoupled, testable and change resilient software.

Object-Orientation is not Really About Objects

How do we write good object-oriented programs? Despite the name, it is not by focusing on objects.

OrientationObject-oriented (OO) programming presents several concepts to the programmer not present in a procedural language. Given the name “object-orientation”, objects is clearly a central concept. Objects are the basic building blocks of our program and they carry out all the important work of our program. This should be contrasted with procedural programs, where functions are global and data is passed as arguments.

From time to time, you come across OO programs that has a procedural flavor to them. There are few, if any, interfaces and classes are primarily used to organize related functions with data. Admittedly, this is a slight administrative improvement over procedural programming with structs and global functions. The problem is, OO programs like this have the same problems as procedural programs: coupling.

Martin Fowler illustrates the concept very well in his article Design Principles and
Design Patterns
(Figure 2-17 on page 13). Assume you have a main function. Main calls three other functions, which in turn calls two other functions. Main now depends on all its descendants! High-level logic should not depend on low-level implementation.

The same applies to the “procedural style OO program” described above. Main might use three objects and call a function in each. Those functions might call two functions on some other classes and so forth. Main will depend on all classes and functions. Before you know it, main depends on your whole system.

The point is, even though objects do all the important work, they do not take us very far when trying to write good, decoupled software. Instead, have a look at Figure 2-18 in the article. Assume that main calls functions on three different interfaces. Main would not know or care which class implements each interface. Main depends only on the interfaces.

Thus, when writing object-oriented software, think primarily about abstractions and interfaces. After that, you can think about objects.