On Complex Software

Tracer Bullets

One of my favourite pieces of software engineering literature is “The Pragmatic Programmer”, similar to a lot of software enthusiasts. It has a lot of value on both initial and subsequent reads and I constantly find myself going back to it to see if I can pick up anything that I may have missed the last time. On my most recent foray into the depths of pragmatism, the concept of “tracer bullets” really stood out.

The principal logic behind tracer bullets is to get some small, usable, testable chunk of the final system implemented quickly and visbily. A rather agile approach to development, one might think. It is generally agreed that complex systems evolve from simple systems that work, Gall’s law, and as such when developing software we should be aiming to implement small working chunks that eventually evolve into a more complex feature or workflow.

Horizontal Development

Horizontal development removes a little of the ambiguity, and warlike connotations, from tracer bullets. When building software we typically have various verticals that require implementing:

• User interface
• Authorization
• View logic
• Application logic
• Data model
• Database

As a developer you could chunk up your work into these verticals, so you have one pull request (PR) for your data models, one for your application logic, one for your views etc. This would be “vertical development”. I argue that this method of development is very restricting and even damaging, as your feedback loop becomes unnecessarily long and code reviews are more difficult as the reviewer cannot see how the data model will actually be used or how application logic ties into a view. A more horizontal approach would be instead to build a simple, working, end-to-end flow that users, QA, product etc. can begin testing right away. Subsequent PRs can then introduce more complicated functionality on top of the already tested, simple base.

So what?

At Ben developers some developers were already using a horizontal development approach, however many PRs would be produced that fit into one of two categories:

• Vertical development, i.e. PR 1: Models, PR 2: Application logic, PR 3: View logic
• Complex feature, i.e. 1-2k line PR with a full, complex feature with partially tested edge cases

The main problem with vertical development is as outlined above, it’s hard to see how the different pieces link together until you reach the final PR which “turns on” the feature. The problem with big PRs is that they are difficult to get all the relevant context into your head which makes reviewing more difficult and makes it far easier for edge cases to slip through. They also take more time to create, more time to review and delay getting the thing into the hands of the person they are supposed to help.

Now, we are trying to instill the mentality of horizontal development into the engineering team. Even within a couple of weeks we have seen noticeable impact on our productivity:

• Lower average PR size
• Shorter duration in review
• Lower regression rate
• Higher feature uptake

I really believe that this should be a core tenet of any development team that wants to build quality software, and whenever one is in doubt about how to implement a new feature or try to improve legacy code think back to Gall’s law.