Fast build turnaround time is essential

Being able to make a code change and quickly see the result is critical. The longer we have to wait, the more code quality suffers, and the more productivity drops. Though often neglected, our build systems need to be well tuned to maintain rapid iterations.

The problem of slow turnaround

I measure the turnaround time as how long it takes from making a change in the code to seeing that change live in the program. Live could be visually in the UI, the result of a test case, or some other observable effect. I include the entire process of saving the file, invoking the build system, and running the resulting program.

The longer this process takes the more difficult it becomes to make exploratory code changes.

To give a hard number, I feel that I should be able to make a code change and see the result less than 10 seconds later. Anything longer than 30 seconds is unbearable and has a chance of making me task switch.

One of the first things to suffer are unit tests. I tend to find they are best written as an interactive exploration of the software, and thus require many builds. I write the main test, then I add several sub checks, or variants to the test, to improve the coverage. If I can’t quickly see the results of my new tests then I’ll be significantly less inclined to actually write them.

I consider test driven development to be a good approach. If long build times are getting in my way it basically breaks the way I think software should be written.

I actually subscribe to the idea of use driven development. It needn’t be a unit test, but I always think from a top-level feature, or use-case, down towards the internals. I implement things piecewise in small iterations to complete the high-level use.

The second aspect to suffer is debugging. Debuggers are useful, but I find that code instrumentation yields the most consistent and useful results. Adding print statements and small code variations produces a wealth of information. The quicker I can add and remove such statements the quicker I can narrow down the source of the defect. Long turnaround times directly impinge on my ability to debug code.

Abandoned thoughts

A third problem is that slow turnaround tends to derail my train of thought. The longer I watch the build the more likely it is my mind drifts to something else. Even if short, these critical lapses can result in massive losses in productivity.

And that’s just the short lapses. If the build is long enough that I have time to switch to my browser, then we have a major problem. Not only have I lost my train of thought, I’ve gone off on a tangent. We all know situations, in the afternoon, or working on a nasty defect, where these tangents just take over for the next hour, or till the end of the day.

Incremental delays

It is incremental building and running that is the issue here. I don’t expect a fresh clone to build within 10 seconds. What I want is the typical turnaround from code change to running. Whether this involves a partial rebuild, new library, or some runtime loading doesn’t matter.

My change doesn’t need to be reflected through the whole project either. If I’m working on a unit test, all I really want to see is that unit test running. If I’m writing a single dialog, then I just need to see that dialog working, perhaps not even with real data, maybe it’s just a mock.

It’s these various partial requests that make a fast turnaround time possible. Even in slow to compile languages, like C++, one can always structure the libraries, and tests to reduce these incremental build times. In slow to start languages, like Java, we can use live class reloading.

Most languages and frameworks allow some way to get fast turnaround time. If they don’t then they’re broken, inadequate technologies.

Architecture of a project also plays a role here. Large monolithic apps tend to suffer the most. Rapid turnaround favours a modular architecture. Good modularity is actually a recurring theme in programming for many reasons.

Appeal of dynamic languages

The desire for fast turnaround is perhaps a major driver in dynamic and interpreted languages. Using JS, CSS and HTML for web development is perhaps the best example here. We can avoid the compile-stage, and usually don’t even need to restart the server. Just press reload in the browser and we can see the results instantly.

On the server, Python and NodeJS are appealing for the same reason. On my own projects I’ve been known to implement rough scripts and mini-languages to improve the turnaround time. It is can often be valuable to trade some runtime performance for improved development speed.

Keep it fast

Maintaining a well tuned build system that allows us to rapidly iterate and test our code is essential. It’s should be easy to justify investment in the build system; each improvement helps everybody on the team, and for the remainder of the project. Don’t forget about this amplified affect while planning.

A slow turnaround directly hurts the quality of the product and morale of the team. It makes development harder. It makes debugging harder. It makes it harder to focus. It makes us annoyed and angry.

Give your build system the care it deserves. You’ll be happier for it.

6 replies »

  1. I agree with all of your post. My day job has a one minute turnaround with a heavyweight Java web application whenever I’m manually testing changes, so whenever feasible I work with unit tests instead. Java has live class reloading, but unless you use third party tools like JRebel the reloading only works if method signatures (return types and parameter types) does not change. JRebel isn’t compatible with some of the other Java libraries core to our application, so we’re stuck with a slow restart for all but trivial code changes during development.

    I’ve had a few opportunities to work with Python or Node.js applications, and at least for small ( < 5000 lines of code ) applications the benefit of nearly instant turnaround completely offsets the lack of a static type system. Typographical errors and missing function parameters blow up in your face because of the missing type checks, but after 20 seconds of typing, Ctrl+S,F5 and the problem is fixed and you're moving on to the next feature. It may be a nightmare for sufficiently large applications – I wouldn't know, I only worked with it on small intranet projects. But for small projects, it's a dream and the return to Java is especially painful.

  2. “Give your build system the care it deserves.”

    This is fantastic.

    I work on short-term embedded projects and often the build system doesn’t get much attention, but this is short-sighted in my view.

    To reduce the dependencies on hardware, I’m increasingly using unit tests and TDD, but this requires fast turnaround times as you’ve discussed.

    A little care up front is going to ease the developer pain and increase productivity in the long run.

  3. Couldn’t agree more. As a part of my day job, I get to hear of many stories about developers and software companies that in some point just couldn’t keep on working due to slow build times.

    Slow turnaround, unfortunately, is a rather ignored problem. But it’s one of the most deadly silent killers there is. Many teams choose to ignore their testing because running even incremental builds takes too long. I think that many project managers, team directors, CTOs tent to overlook slow turnaround. As you mentioned, take a good care of the build system is an important investment that many ignore because they are pressed by the management to concentrate in the product itself, and the programmers don’t have any reason to set other priority.

    When your builds are getting ridiculously slow and you find yourself waiting more than a minute to see the slightest change, or you have to wait more than half an hour for a full rebuild, most of the times it’s too late to change the build process dramatically. People skimp on testing and other processes in order to speed up their builds – at the expense of quality.
    Although there is no replacement to a neat and efficient build system, not just because of build speed but from the need to work correctly, there are tools such as IncrediBuild for cutting turnaround times by parallelizing the process across the local network. Therefore, you don’t need to make the tradeoff between quality and speed. With the proper tools for the job, you can still reduce waiting time to a minimum.

    Note that I work for IncrediBuild

    • The fact that there are companies like yours dedicated to improving build speed it is testament to the degree of the problem. As you said, slow builds can really kill products. People just stop following good engineering practices and bit by bit the quality crumbles.

      For my own language Leaf I’ve also taken the approach of allowing parallel building (even within a single source file). OF course, I’ve also made sure it has limited parsing/build dependencies to just limit the overall time. Of course, building the source is just one part of total turnaround time.

  4. I’m very grateful for the very fast turn-around times in web-development, and having previously worked in traditional languages with the much longer “change-compile-test” cycle, I can attest to the massive benefit that quick turn-around brings.

    These days I generally end up using Meteor.js as my go-to platform, and it positively spoils me rotten, as on my dual monitor setup, I have on one screen my editor and on the other, the browser showing my app. A quick ctrl+s in the editor and within 2-3 seconds Meteor’s hot-reload feature updates the browser on it’s own, ultimately providing an almost real-time live view of changes. It’s bliss.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s