Correlation Between Experience and Programming Skills

In my daily work, I have the privilege to work with talented people. But finding these people in the first place is not always that easy, it seems. I’ve conducted lots of interviews with potential hires over the last five years. Preferably, you would like to learn from the CV whether or not a programmer is good, but this proves to be a tough challenge. Maybe I’ve just had a skewed set of samples, but I’ve made a few observations:

  • There seems to be little correlation between experience and programming skills. In my experience, you are as likely to find an experienced developer (10+ years of experience) with poor skills as you are finding one with excellent skills. On the bright side (and further “evidence”), I’ve come across lots of very talented newly graduated students. It seems some people “get it” right away, while some struggle.
  • Also, there seems to be little correlation between education and programming skills. I’ve seen self-taught programmers that really humbled me, as well as PhDs with poor skills. (On this topic, check out self-taught James Bach’s book “Secrets of a Buccaneer Scholar“.)

If my experiences are not just due to bad luck, it seems programming skills correlate neither with education nor experience. Then what does correlate with good programming skills?

To be honest, I have no answer. But research published in the article “The Camel Has Two Humps” presents an interesting observation. Maybe there’s a correlation between between consistency and programming skills. True or not, it is an appealing idea: it’s easy to imagine that a consistent person (i.e. a person that repeatedly makes the same decisions in similar situations) should be able to become a good programmer. Correct his flawed thinking once, and he will use his newly found knowledge in many places. An inconsistent person would need correction on a case-by-case basis.

Now if (big if) consistency is key, the question is just how do we device a test for it? I’m not sure how revealing the tests from “The Camel Has Two Humps” would be on real programmers.  (They are on the form “int a = 10; int b = 20; int c = 30; a = b; b = a; c = b; What are the values of variables a, b and c?” :))

The Myths of Innovation

I just finished reading The Myths of Innovation by Scott Berkun. It’s a decent book, and probably required reading nowadays when innovation (whatever it means) is the coolest buzzword of all.

The moral of the book is: there’s no shortcut to creating the next Facebook or Google. You will need an idea, some skills, but most of all, lots and lots of hard work. Newton didn’t invent physics when he got hit by that apple (assuming it really happened), he worked on the problem for years. Unfortunately, after putting in all that hard work, you will still need a lot of luck in order to succeed (adoption, timing, press coverage, competition and so forth). Tough.

Edge-To-Edge Unit Tests

The term unit test implies at least two different things. First, it means testing your code at the smallest unit, which is a function or a class (well, technically, you test the methods of a class, but they would make little sense in isolation). Second, it means writing test code in a language to test functionality in the same language.

Normally, when I write C++ code to test my C++ functionality, I tend to stay away from the “unit” level. Instead, I like tests that exercise the system edge-to-edge, resembling the interactions with the outside world as much as possible. Now, full edge-to-edge testing is normally not possible since the peripheral parts of a system are often hard to control. For example, let’s say I have a system with network on one side and a GUI on the other side. A realistic test case would have traffic over the network and a GUI reflecting that. But taking the network as an example, it complicates testing due to issues like slow response times, need for a remote side and network failures. So I settle for testing up to the interfaces of the network and GUI: you would inject network “traffic”(or time-outs) on the network interface, verify that the GUI interface is told to show something, do some user input on the GUI interface and watch outgoing network traffic being generated.

As with everything, there are pros and cons when testing at this level. To me, the main benefits compared to low-level unit tests are:

  • Having tests at that level gives me confidence that there’s a reasonably low probability of faulty interaction with the outside world.
  • The boundaries of your system are much less likely to change than the internals. This means you are less likely to spend time changing your tests.
  • It is easy to argue for the business value of the tests. They correspond well to what the customer expects and are used to guarantee the quality.
  • The tests are a decent measure of progress. Having a passing test means you are close to something to show to your customer.
  • It’s fun! And you can test-run your system before you have a network and a GUI.

The downsides I’ve experienced compared to testing at the unit level are:

  • Tests like these can make it harder to achieve decent code coverage. For example, your code might involve randomness, timing issues or use of the current time. You will have to make sure these can be controlled from the test context.
  • High-level testing can be hard to introduce late in the development process. For this to succeed, the whole system must be designed for testability. See the previous item.
  • The tests become monolithic. I’ve come across the situation where parts of my system were broken out to form a new shared component. The new component has to have tests of its own (or someone changing it won’t notice it’s broken until they run your tests). Your tests use the classes of your system, which are not suitable to use in a shared component since dependencies would go the wrong way.
  • It might be overkill for testing some parts of the system. For example, if you have some deep-down string manipulation code, you should go ahead and unit test it (in the true sense of the word). It’s all about choosing the proper tools for the problem.
  • Due to complexity in the lower levels of your software, you might be facing a combinatorial explosion of different test cases. You will have to select a few representative test cases and resort to normal unit testing of test the low-level parts. See the previous item.
  • Testing on this level poses sort of a communication problem. If I call my tests “unit tests”, most people think only of tests on the lowest level. If I call them “acceptance tests” or “functional tests”, someone will inevitably assume I have properly tested the system from the outside, edge-to-edge (which is definitively necessary, even with the tests described above). Calling them e.g. “functional unit tests” only adds to the confusion. (“What do you mean? Is it a unit test? Is it a functional test? Surely, it can’t be both.”) If you know of terminology that could help, let me know. Until I hear from you, I will just call them Edge-to-Edge Unit Tests.

As I said before, it’s about choosing the proper tool for your problem. If at all possible, I resort to “unit testing” at the highest possible level. If you haven’t done so, you should give it a try.

“We Train You to Be the Best Programmer You Can Be”

What could a company do to differentiate itself in order to attract talents? Many companies take pride in their innovative work environment. Benefits and salaries might be another way. But we are yet to see a company with an official policy saying “Come work for us. We will train you to become the best programmer you can be.“? You know of such a company? I would be happy to be proven wrong.

I’m not talking about having the employees take courses or read books all day long. I would rather see a collaborative effort to improve (yourself and the organization), such as a dedicated mentor for everyone, expert brain picking sessions, study circles (discuss a book, a bug, a piece of code or a paper together), knowledge transfer (weekly presentations of interesting topics), pair programming with an expert, software architects to discuss your design with, cross-functional training, panel discussions, required reading, you name it. All in the context of your current work assignment. Your personal development would be part of the company targets.

Many companies train their staff, but I would like to see it as an official policy. Such a company would certainly be able to attract talented people. The collaborative effort to improve would also benefit the company (e.g. improved co-operation, better code, fewer bugs, increased knowledge transfer). For the individual, working there would be a sign of quality.

I had the pleasure to have a coffee break discussion about this with James Coplien the other day. He said he both loved and hated the idea. It might be a bad idea since it does not bring any business value. But that aside, we both agreed it would be a fantastic place to work in.


Update (May 7, 2011): In the book “Implementing Lean Software Development” (see post), the authors talk about river companies: “The purpose of a river company, […] is to keep on flowing, that is, to stay in business and provide jobs over the long term.” and “The individual will deliver care and commitment in exchange for the fact that the company will try to develop each individual’s potential to the maximum.” This actually sounds a lot like what I described above. The term seems to have been defined in this article.

Welcome to My Blog!

I finally got around setting up a blog using WordPress. The old edit-four-different-html-pages-to-get-anything-published was just too tiresome. Setting it up was not hard at all. Hm, makes me wonder why I didn’t do this a long time ago? Scott Berkun says in Myths of innovation that change is scary. That must be it.

I will write about stuff that is of interest to me. With any luck, you will find something of interest to you.