The Random Sustainable Coder

Looking for good books? Here are two on software development and one on… randomness.

The Clean Coder: A Code of Conduct for Professional Programmers” by Robert Martin addresses what it takes to be successful in the long term as a software professional. First and foremost, a software professional takes responsibility for himself, his peers and his product. For example, he speaks the truth. He voices his opinion if a deadline will not be met. Martin has a series of conversations between different roles (such as programmer, project leader) which are pretty entertaining. There are lots of topics in this book that are important if you want to be long-lived and respected as a software professional. If you don’t have time to read the whole book, at least read the first chapter, which is gold.

Sustainable Software Development: An Agile Perspective” by Kevin Tate addresses what it takes to be successful in the long term when developing a software product. It is not unusual to see software products released after a year and then maintained for ten more years. This sets some specific requirements on the software, the development environment and the mindset of the developers. For example, the software must be able to accommodate change at reasonable cost and without breaking. In order to be able to deliver fixes and new features, the product must always be in good shape. Preferably good enough to be delivered every day, if so desired. This means the development process and the mindset of the developers must be heavily focused on quality. We must shift from defect detection (finding bugs) to defect prevention (avoiding bugs). Root cause analysis is key. Read the book, the message is great.

Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets” by Nassim Nicholas Taleb is a very interesting book! It is about randomness and how it affects our life. For example, how much of the success in your own career is due to chance? According to the author, if you’re a stock broker, it’s probably close to 100% (Taleb does not go easy on his fellow finance colleagues! :). If you’re a software developer, my gut feeling is that luck is much less significant. Re-live your life a million times, and your average life salary would probably still be around your current. Not so for the stock broker.

Taleb also describes an interesting scam. Send one million emails saying “the stock market will rise next week”. Send another million emails saying “the stock market will fall next week”. Next week, if the stock market rises, keep only the first million email addresses. To half of them, send an email saying “stocks will rise next week”, and to the second half “stocks will fall”. Every week, keep only the half where you predicted correctly. After 10 weeks you will have hundreds of people thinking you have super powers… This is called “survivor’s bias”. Taleb applies the exact same reasoning when a stock broker asks him for a job. Taleb won’t ask “How did you manage to do so well in the stock market in the past?”, but rather “How many like you were there from the beginning?”! This is a fantastic book, read it.

Sustainable Software Development, Part 2: Managing Complexity

As the code base grows, the complexity of your code increases. The pace of software development often slows down as the product matures. Why is this? What can we do to manage complexity?

Sustainable software development is about retaining software development velocity over the life-time of your product. In part one, I covered the technical debt side of sustainable software development. In essence, when you postpone refactoring, bug fixing, testing etc., you accumulate debt. As all debt, it has to be re-payed. But even worse, you will pay interest. Every time you think about, swear about, discuss, plan or anything (but fix) your known problems, you pay.

In this second part, we address another challenge to a sustainable pace of development: the increased complexity of the software. In order to avoid letting the complexity make your life miserable, we need to manage the complexity. For brevity, we will touch only upon these four areas:

  • Simplicity
  • Dependencies
  • Hierarchy
  • Abstraction

Fred Brooks makes the distinction between essential and accidental complexity (also discussed in his book “The Mythical Man-Month”). Essential complexity comes from your problem domain. The problem will have an inherent complexity you cannot get away from. Accidental complexity comes from the solution you’ve chosen. The “Keep it simple, stupid!” (KISS) design principle encourages developers to prefer simple solutions over those more complex. Thus, in order to manage complexity, it is important to keep the accidental complexity to a minimum. For example, dividing your code using the Single Responsibility Principle is one way, so that different pieces of functionality are not tangled up in a single class.

Another example of accidental complexity is the introduction of unnecessary dependencies in your software. Every subset of your code will have connections and communication with other subsets of your code. In a naive software implementation, the number of connections would grow wildly with the size of the software. The interconnectedness of your code is one aspect of complexity that needs to be managed. Decomposition and decoupling will help you, and they should be applied to all levels of your system: service, sub-system, module, class, function etc. Various design principles (e.g. Dependency Inversion Principle) and patterns can guide you.

Steve McConnell discusses hierarchy and abstraction in his article “Keep it simple” (and in his book “Code Complete”). Structuring your software as a hierarchy is a means of decomposing your solution into more manageable pieces. Normally, different kinds of hierarchies exist within your code. For example, think about the difference between the relation between objects (“which object creates/destroys/contains/depends on/owns the memory of which”) and the relation between functions (“which function calls which”). One architectural pattern that can help the high-level structure of your code is Layer (defined in the book “Pattern-Oriented Software Architecture: A System of Patterns” by Buschmann et al.). One upside of a structured approach like Layer is that tooling, such as automatic mapping which classes depend on which, can help you enforce your structure over time.

Abstraction is all about what level of detail is visible at what level (see also my post on abstraction). For example, the file system is an abstraction hiding the hard-drive with its tracks and sectors. When writing a program, you refer only to “file x”, and not to “track y, sector z”, which helps tremendously. Abstraction will help you reason about your program. Abstractions are largely domain dependent. Your domain will decide what is a good level of abstraction, hiding unnecessary detail while not constraining you.

Last words from McConnell’s “Keep it simple” article: “Neither hierarchies nor abstractions reduce the total number of details in a program — they might actually increase the total number. Their benefit arises from organizing details in such a way that fewer details have to be considered at any particular time.”

How do you fight complexity in your daily work?

Characteristics of a Software Professional

At work, I have been challenged with the question “What are the most important characteristics of a software developer?“. This is a tough question, and no matter what you decide to include, you have to leave something out.

I’ve been part of software development projects in various companies. The successful projects teach you invaluable lessons. The dysfunctional projects even more so. Inevitably, a list of characteristics would include qualities I value and desire in my fellow colleagues, as well as characteristics I’d expect them to want in me. (So this is also a long todo list for me. ;)

To get some kind of structure, I decided on three main categories: Professionalism, Long-term code and Quality mindset.

Professionalism

Take responsibility

You are a professional developer, and professionals act responsibly. If things go wrong,
take responsibility. Understand why things went bad. Learn, adapt and make sure it
never happens again. When faced with a difficult choice, “do the right thing”. Optimize for the long run even if it results in more work today. Be a team player, even if this means saying no. Speak the truth.

Know your product

We’re part of a business. Without successful products, there will be no business.
Know your users and their needs. Use your product, use competitor’s products,
visit customers and watch them use your product (from purchase/download to
installation to day-to-day usage to upgrade to uninstall and so forth).

Continuous learning

Be humble and practice relentless inquiry. Question everything (since everything has potential for improvement), embrace questions on your code, realize others’ suggestions might improve your work, be happy other developers change “your” code
(it’s good enough to understand!). Improve yourself and others. Become a better developer by reading books, papers, blogs etc. Watch instruction videos, listen to podcasts, try new technologies, participate in open source development, discussion groups etc. Discuss your code and what you read with your peers. Talk to developers of other specialities. Teach others what you know well.

Long-term code

Communication

You write a line of code once, but it is read hundreds of times. Invest time to write code easy to read for others. Write code at the right level of abstraction, abstract enough for expressiveness but without hiding necessary detail. Adhere to design principles as
they capture proven ways to high-quality code. Apply design patterns to better communicate your intent.

Maintainability

Decouple the different parts of your software, on every level – sub-system, module, class and function. Write extendable code, so that you can add functionality with minimal change to existing code. Avoid technical debt, and repay debt as soon as possible. Interest has to be payed on all debt.

Proven functionality

Never deliver code unless you’ve proven it works. If you don’t test it, it will be faulty.
Write testable code. Make it testable from the start, later it will be too expensive. Automate your tests to run before check-in, after check-in and nightly. If the tests are not executed automatically, they will not be updated and soon be obsolete. Without automated tests, no-one will dare change any code. Write fast and reliable unit tests. Write realistic integration tests. For each new feature, write acceptance tests. Automate everything.

Quality mindset

Quality is your responsibility

You, as a developer, is responsible for the quality of the product. Other roles can help you spot problems or give you more time, but they cannot affect quality. Never ship anything without being certain of its correctness.

Find bugs early

Find bugs early in the development process. If a bug can be found by the developer, it should be. If you need tools, get them. If you need hardware, get it. If a bug is found late, understand why it was not found earlier. Fix the process so that bugs of this kind never slips through. Automate.

Fix it now

If you find a bug, fix it now instead of filing a bug report. Ask your colleagues to help out. You will save time. File bug reports on things that couldn’t be solved in half a day.
Do things properly the first time. If you don’t have time to do it right today, when are you ever going to find time? Give time estimates that allow you to produce quality products. Think about what is stopping you from being more productive. Fix it, and then move on to the next thing stopping you.

If you’re interested in these things, you should have a look at Poppendiecks’ “Lean software development” books, Senge’s “The fifth discipline“, Martin’s “Clean coder“, McConnell’s “Code complete“, McConnell’s video “World class software companies” and others. I also provide some resources in previous blog posts (e.g. videos and books).

What characteristics do you value in a software developer?

Object-Oriented Programming Lecture at KTH: Slides etc.

Thank you all that participated in my lecture at KTH October 26, 2011. I had a lot of fun, and we had some good discussions. For you who were not there, it was about object-oriented programming and how to write good code. We used some example code to discuss OO principles and testing. Here’s the presentation (pdf)!

A couple of posts of relevance:

Software Architecture Built to Survive Change

How do you create an object-oriented architecture that can survive change? We discuss how to isolate the parts of your system that never changes, while still making it easy to add new functionality.

What is it that makes some software architectures fragile, wavewhile others are solid as a rock? Recently, I have come across a very appealing idea from multiple sources. First, a colleague of mine brought it up (hi, D!), then I read about it in a book “Lean Architecture for Agile Software Development” by James Coplien and Gertrud Bjørnvig. The book introduced me to the DCI software architecture, which I might cover in more detail in a later post (what I describe here is just parts of it). Learning a new idea is like learning a new word: once you know it, you see it everywhere.

Let’s walk through a simple (and highly contrived :) example. Assume we have a bank system with users, accounts, transactions, transaction logs etc. We have a use case to implement: pay interest to user’s account. Without giving it much thought, we could add a function addInterest to the Account class which calculates and deposits interest and updates the transaction log. Given a few hundred of these types of features, your Account class will be cluttered with functions. As you can imagine, this will eventually lead to poor readability, increased risk of breaking working code and problems parallelizing work. Over time, your code might complex enough to prohibit any kind of progress. Actually, this is the path many projects go.

Let’s try a different take. In the terminology of the “Lean Architecture” book, we want to separate “what the system is” (e.g. accounts, users) from “what the system does” (e.g. pay interest, transfer money). We create classes Account, TransactionLog etc. and define simple interfaces IAccount and ITransactionLog. The interfaces expose only administrative operations like getters/setters and add/remove functions (but no behavior or use case logic!). Among other functions, IAccount might expose addMoney while ITransactionLog might expose addLogEntry. We then implement our use cases as “algorithms” that operate on the interfaces. For example:

interface IUseCase
{
    void execute();
}

class PayInterestUseCase implements IUseCase
{
    PayInterestUseCase(IAccount account,
        ITransactionLog transactionLog) {...}

    void execute()
    {
        double interest = calculateInterest(account);
        account.addMoney(interest);
        transactionLog.addLogEntry(account.getId(),
            interest);
    }
}

Later, when we add a “money transfer” use case, we already have the addMoney and addLogEntry functions. We create a new use case class TransferMoneyUseCase. We withdraw money from one account (e.g. by providing a negative number to addMoney), deposit the money in the other account and update the log. The Account/TransactionLog objects and interfaces do not change at all, but we can still extend the system! After a few use cases, we have a set of stable interfaces on which we can build almost anything. Changes and additions to the interfaces will be fewer and fewer.

To me, the concept of separating domain objects (what-the-system-is) from business logic (what-the-system-does) is very appealing. One changes slowly, if at all. The other will evolve and change all the time. I like it. What drawbacks/benefits do you see? Have you tried it? Leave your comments below.

Outliers: The Story of Success(ful Development Careers)

Do you want to be an expert in your field? Do you want to be a superb programmer? If so, the good news is that there’s a simple plan that takes you there.

outliersI recently read Malcolm Gladwell’s book “Outliers: The Story of Success“. To me, the main message from the book is that hard work can make anyone into an expert in his field, no matter if it’s violin playing, ice hockey or… programming. The hard work is the bad news. According to the book, you cannot find a single “expert” (whatever that means), that has not spent at least 10.000 hours practicing. For example, not one of the top violinists have gotten to where they are by talent only. The fact that you don’t have to be born with a certain talent is kind of comforting. With 20 hours of practice per week and 52 weeks per year, anyone can be an expert in only ten years. :)

Unfortunately, working as a software developer for ten years will not automatically give you expert skills. Going to work and doing a good job is not sufficient. According to Gladwell, there needs to be a conscious and focused effort to improve. As a programmer, what can we do to improve?

First of all, I suggest you take every opportunity for practice during your workday. Make sure every day counts. You have the perfect setting: you work with code many hours per week, you have skilled colleagues to interact with, you have people from different fields for cross-pollination (management, sales, graphics design etc or different fields of engineering) and so forth. Make sure that you constantly think about your code and how to improve the way you write it. Seek out those that are better than you in whatever aspect. Make other people review your work. Discuss your design with others.

Work will only take you so far. Apart from practice, you also need theory. My favorite source of inspiration and knowledge is books. Most books I read challenge the way I think about code. Each author has his view on what constitutes good software. Be critical and incorporate the things you like into your code. Set aside an hour or a few minutes every day to read books, online articles and blogs (maybe even this one). Do not constrain yourself to just your field; problems and solution in other fields will also help.

Speaking of other fields: you should read the “Outliers” book, or you would not know why

  • Being born in January, February or March maximizes your chances to become a successful ice hockey player.
  • Asian children are better at math than western children.
  • Poor children’s reading abilities degrade over the summer while wealthy children’s reading ability improves.
  • There’s a school in Bronx where young children study from 7 am to 5 pm (and have 3 hours of homework every day) with very little summer vacation, and still, lots of children want to go there.

In short, I like the book.

Great Books for a Software Developer

I have received questions about good books, so I thought I would share some titles with you. First, my absolute favorites, should be mandatory reading for everybody (so go ahead and read them right away):

  • Growing Object-Oriented Software Guided by TestsGrowing Object-Oriented Programs Guided by Tests” by Steve Freeman and Nat Pryce (Amazon). My bible. :) A fantastic book about test-driven development. I touched upon it in my post Test-Driven Development Done Right.
  • Implementing Lean Software Development: From Concept to Cash” by Mary and Tom Poppendieck (Amazon). An amazing book on continuous improvement and customer focus in your software development organization. See also my post Lean Software Development.
  • Code Complete, 2nd edition” by Steve McConnell (Amazon). A great and practical book on how to write good code.

Really good books (read them when you finish the above :):

  • The Fifth Discipline: The Art & Practice of the Learning Organization” by Peter Senge (Amazon). A fantastic book on how to make people work together so that your organization becomes greater than the sum of the individuals. Not really a software development book, that’s the only reason why it didn’t make it to the “mandatory reading” list. :)
  • Design Patterns: Elements of Reusable Object Oriented Software” by Erich Gamma et. al. (Amazon). The Book on design patterns. A must-have.
  • Peopleware: Productive Projects and Teams” by Tom DeMarco and Timothy Lister (Amazon). A book on what is required to create an productive software development organization. Pretty interesting facts in there.
  • How to Win Friends and Influence People” by Dale Carnegie (Amazon). A classic about how to build good relationships and how to handle people.
  • One Minute Manager” by Kenneth Blanchard and Spencer Johnson (Amazon). A classic about how to manage people in order to get them as happy and productive as possible.
/www.amazon.com/Minute-Manager-Ph-D-Kenneth-Blanchard/dp/0425098478

Lean Software Development

I just finished reading the book “Implementing Lean Software Development – From Concept to Cash” by Mary and Tom Poppendieck. In essence, “lean” means “reduce waste”, and “waste” means “everything that does not make your customers happier”. Many concepts in the book was pioneered by Toyota while applied to manufacturing. Toyota then moved on and introduced lean in product development (including software).

Implementing Lean Software DevelopmentWhen a customer ask you for a feature, how long does it take until they get it (calendar time)? While implementing it, how much effective time have you spent on the feature? The difference between calendar time and implementation time is waste (since waiting makes your customer unhappy). This waste can be analyzed using Value Stream Mapping (example). Typical things that takes time for no good reason is waiting for approval (solution: approve within 24h instead of in the bi-weekly meeting) and long fixed releases. I have come across really long release cycles in previous work places and when talking to friends at other companies. (Half a year, a year is not unusual, which should be contrasted with the book’s claim that Toyota developed and released the Prius within a year and a half!). The solution is to release and/or deploy more often. In order to do this, we need a relentless focus on quality (= no bugs).

According to the book, having bugs in a bug tracking system is waste. Software with bugs in it is not finished work, and piling up unfinished work is waste (since it cannot be released to your customer until it’s finished). Also, someone has taken the time to enter the information, and someone must look at it to resolve the bug. Meanwhile, someone else might stumble across the same bug. All this takes time. Lean software development says you should solve the bugs right now. In order to make changes to solve the bug right away, you need tests to make sure you don’t break something else. Good quality software starts with rigorous and automated tests on acceptance and unit level (see my previous posts: test-driven development done right and edge-to-edge unit tests). Reducing the extra work from bugs will free up resources to implement new features.

Lean software development is about continuous improvements. Every day, we should ask questions like: How do I make my customer happier? How do I reduce waste in order to release faster? But who should answer these questions and implement the improvements? Lean says it’s the people closest to the problems. Inevitably, in software development, it will be the programmers! I think this is the most appealing part of the contents of the book.

This is a fantastic book and I am sure I will read it again pretty soon. Read it!

Test-Driven Development Done Right

A couple of years ago, I had at least two misconceptions about Test-Driven Development: (1) you should write all your tests up-front and (2) after verifying that a test case can fail, you should make it pass right away. To better understand TDD, I got a copy of the book “Growing Object-Oriented Software Guided by Tests” by Freeman and Pryce (now one of my absolute favorites). Although the book does a great job explaining the concepts, it took me ten chapters to admit I had been wrong. Let’s never do that again. :)

Let me walk you through my misconceptions so that you don’t have to repeat my mistakes:

Misconception 1: write all tests up-front. Thinking about potential test cases up-front is not a bad thing. It will exercise your imagination and with some luck, many of them will still be applicable after the code is written. But don’t waste energy trying to compile an exhaustive list of tests. At least for me, this approach didn’t work since my imagination appears too limited to come anywhere near the final list. But foremost, I would like to get going writing some test code!

You are better off writing a few happy-path test cases. Filling in the test code will get you started working on the user interface of your classes. When the test code starts acting as a “user” of your interface, it will be obvious to you whether the API is okay or awkward to work with. The tests will drive you to improve your user interface. Creating the user interface will invariably make you think about error cases and how the API can be abused or misunderstood. You will come up with more test cases, and implement these. With some effort, but surprisingly little so, you will grow your test suite.

Misconception 2: fail the test, then make it pass right away. When you have written your test code, filled in the production code to get it all to compile and seen the test fail, it is very tempting to just fix things. Make the changes to have the test pass. Actually, you can do that. But there are at least two reasons not to.

First, I strongly prefer an incremental approach. I fix only the problem reported by the test! If the test says “null pointer exception”, I will fix it. Running the test again, you will get another failure and fix that. This is the convenient/lazy approach, you just let the test drive you. Also, it will result in minimal increments, which is very helpful if another test case would break while changing the production code.

Second, fixing the failed test case right away will throw away a lot of information in the process. When the test fail, it will provide you with valuable information on what went wrong. If you cannot immediately understand what the problem is, maybe you should improve your test or production code? For example, if you get a “null pointer exception”, maybe error handling or an assert earlier in the production code could make sure your program never gets into that kind of a corrupted state. Alternatively, you could improve your test code with all kinds of helpful diagnostics. The idea is, if it takes time for you to understand what went wrong today, imagine how much time will be wasted when the same test case fails in six months. “Growing Object-Oriented Software Guided by Tests” says you should “let the tests talk to you”. There you go, you are test-driven.

Correlation Between Experience and Programming Skills

In my daily work, I have the privilege to work with talented people. But finding these people in the first place is not always that easy, it seems. I’ve conducted lots of interviews with potential hires over the last five years. Preferably, you would like to learn from the CV whether or not a programmer is good, but this proves to be a tough challenge. Maybe I’ve just had a skewed set of samples, but I’ve made a few observations:

  • There seems to be little correlation between experience and programming skills. In my experience, you are as likely to find an experienced developer (10+ years of experience) with poor skills as you are finding one with excellent skills. On the bright side (and further “evidence”), I’ve come across lots of very talented newly graduated students. It seems some people “get it” right away, while some struggle.
  • Also, there seems to be little correlation between education and programming skills. I’ve seen self-taught programmers that really humbled me, as well as PhDs with poor skills. (On this topic, check out self-taught James Bach’s book “Secrets of a Buccaneer Scholar“.)

If my experiences are not just due to bad luck, it seems programming skills correlate neither with education nor experience. Then what does correlate with good programming skills?

To be honest, I have no answer. But research published in the article “The Camel Has Two Humps” presents an interesting observation. Maybe there’s a correlation between between consistency and programming skills. True or not, it is an appealing idea: it’s easy to imagine that a consistent person (i.e. a person that repeatedly makes the same decisions in similar situations) should be able to become a good programmer. Correct his flawed thinking once, and he will use his newly found knowledge in many places. An inconsistent person would need correction on a case-by-case basis.

Now if (big if) consistency is key, the question is just how do we device a test for it? I’m not sure how revealing the tests from “The Camel Has Two Humps” would be on real programmers.  (They are on the form “int a = 10; int b = 20; int c = 30; a = b; b = a; c = b; What are the values of variables a, b and c?” :))