The Tyranny of closed source

27 Nov 2013


Horatio Alger, Strive and Succeed
"Work expands so as to fill the time available for its completion."-- Parkinson's law


The following are all things that have been said to me in a professional software environment:

“It’s not that hard. We just need to build a globally distributed key-value store and put some application logic in it.”

”[That open-source project] didn’t have a feature we wanted so we forked it 3 years ago and now maintain this broken 3-year old version.”

“ActiveRecord validations aren’t flexible enough, let’s build a framework for database triggers.”

“The problem with [an enormous repo] is that we haven’t finished building all the parts of it yet.”

To a person each one of the engineers quoted was a capable developer and had many years of experience. And at the time the decision to do these things made perfect sense. Not only did it solve an immediate problem – it was fun engineering.

We engineers like to build large things and nothing feels like it needs a large solution quite like a large problem. The process of actually solving a technical problem is quite methodical and requires great discipline: You have to develop a strategy, you have to diligently get feedback on your strategy, and you have to implement a small version of your proposed solution so you can determine quickly if it’s actually going to work.[1]

Strategy

The best guide around on strategy is Richard Rumelt’s Good Strategy, Bad Strategy. Rumelt teaches that a strategy has three essential parts: a diagnosis of a problem, a guiding policy for how you will solve the problem, and specific actions taken in line with that policy.

As an example let’s pretend somebody comes up to you and says “The postgres database drives are full.” That’s a presenting problem but is it really the issue? Rather than jumping to a “shard more!” or “buy bigger drives!” solution Rumelt would suggest we identify if this is a result of massive surge of new customers or a rogue logger or if “full” actually only means 50%. That’s diagnosing the problem.

If, say, it’s an influx of new customers then you gotta develop a policy. So you’d not just fix the postgres boxes because most of your infrastructure is probably going to start creaking under the load. What you need is a proper capacity plan that identifies bottlenecks and can give you a rough estimate of how many and what kind of servers you’ll need.

Lastly you’d need to actually provision new machines. This problem-solving step is the one we engineers tend to be good at but we often arrive here too early. If, in this contrived scenario, we merely reclaim 20% of disk space through an epic compression hack and call the problem solved then we’ve not only wasted time but fooled ourselves.

Feedback

Some Xooglers at Square introduced our formal design review process and it’s proven very effective at improving engineering concepts before they get committed to code. Our procedure is that you write up a 1-page document saying what problem your new code is solving, what your code will do, what it will explicitly not do, what resources is consumes and what it produces.

This is small enough that it’s not onerous to either write or read. And as long as your coworkers have worked at at least one other company you’ll be surprised by the helpful ideas.

Prototype

So you’ve got a strategy, you’ve got a design that has its most obvious flaws patched, and now you have to build it. If you value your career and you should probably not just spend the next 9 months perfecting this new application. You might find that it’s so complex that staffing it is more costly than the problem and the company will hate you for building it. You might find that your company has no experience operating your chosen database at scale and you need to switch to a datastore with a totally different API. You might find that you built it in technology X but everybody at your company and everybody you’re able to hire only knows technology Y. An MVP will let you flush out these issues quickly.

No more software

The goal isn’t to write good software but to write as little software as possible. If you’ve ever mentored a very junior engineer then you know that they can produce vast quantities of code. They also devote little time to maintaining. Veterans not only write clean-ish, maintainable-ish code but do it as rarely as possible.

“In-house software expands so as to require the maintenance of the entire engineering team.” – Jack Danger’s law of not-invented-here

The job of your company is to produce a product that is worth hard cash to somebody. And your job is to build that product. Not to build a database, not to build a javascript framework, and not to build a new testing library. Those are great hobbies and if your company has the in-house skill to build a best-of-breed tool and release it then absolutely do it. But that’s rare.

It’s better to build a simple solution than a complex one, even better to use an open-source solution, and if you see a path to do it without software at all then take it.

 

  1. You could replace ‘feedback’ with ‘market testing’ and this is also the formula for effectively driving a startup.

Please if you found this post helpful or have questions.