The Tyranny of closed source

27 Nov 2013


Horatio Alger, Strive and Succeed
"Work expands so as to fill the time available for its completion."-- Parkinson's law


The following are all things that have been said to me in a professional
software environment:

“It’s not that hard. We just need to build a globally distributed
key-value store and put some application logic in it.”

”[That open-source project] didn’t have a feature we wanted so we
forked it 3 years ago and now maintain this broken 3-year old version.”

“ActiveRecord validations aren’t flexible enough, let’s build a
framework for database triggers.”

“The problem with [an enormous repo] is that we haven’t finished
building all the parts of it yet.”

To a person each one of the engineers quoted was a capable developer and
had many years of experience. And at the time the decision to do these
things made perfect sense. Not only did it solve an immediate problem
– it was fun engineering.

We engineers like to build large things and nothing feels like it needs
a large solution quite like a large problem. The process of actually
solving a technical problem is quite methodical and requires great
discipline: You have to develop a strategy, you have to diligently get
feedback on your strategy, and you have to implement a small version of
your proposed solution so you can determine quickly if it’s actually
going to work.[1]

Strategy

The best guide around on strategy is Richard Rumelt’s Good Strategy,
Bad
Strategy
.
Rumelt teaches that a strategy has three essential parts: a diagnosis of
a problem, a guiding policy for how you will solve the problem, and
specific actions taken in line with that policy.

As an example let’s pretend somebody comes up to you and says “The
postgres database drives are full.” That’s a presenting problem but is
it really the issue? Rather than jumping to a “shard more!” or “buy
bigger drives!” solution Rumelt would suggest we identify if this is a
result of massive surge of new customers or a rogue logger or if “full”
actually only means 50%. That’s diagnosing the problem.

If, say, it’s an influx of new customers then you gotta develop a
policy. So you’d not just fix the postgres boxes because most of your
infrastructure is probably going to start creaking under the load. What
you need is a proper capacity plan that identifies bottlenecks and can
give you a rough estimate of how many and what kind of servers you’ll need.

Lastly you’d need to actually provision new machines. This
problem-solving step is the one we engineers tend to be good at but we
often arrive here too early. If, in this contrived scenario, we merely
reclaim 20% of disk space through an epic compression hack and call the
problem solved then we’ve not only wasted time but fooled ourselves.

Feedback

Some Xooglers at Square introduced our formal design review process and
it’s proven very effective at improving engineering concepts before they
get committed to code. Our procedure is that you write up a 1-page
document saying what problem your new code is solving, what your code
will do, what it will explicitly not do, what resources is consumes and
what it produces.

This is small enough that it’s not onerous to either write or read. And
as long as your coworkers have worked at at least one other company
you’ll be surprised by the helpful ideas.

Prototype

So you’ve got a strategy, you’ve got a design that has its most obvious
flaws patched, and now you have to build it. If you value your career
and you should probably not just spend the next 9 months perfecting this
new application. You might find that it’s so complex that staffing it is
more costly than the problem and the company will hate you for building
it. You might find that your company has no experience operating your
chosen database at scale and you need to switch to a datastore with a
totally different API. You might find that you built it in technology X
but everybody at your company and everybody you’re able to hire only
knows technology Y. An
MVP will let you
flush out these issues quickly.

No more software

The goal isn’t to write good software but to write as little software
as possible. If you’ve ever mentored a very junior engineer then you
know that they can produce vast quantities of code. They also devote
little time to maintaining. Veterans not only write clean-ish,
maintainable-ish code but do it as rarely as possible.

“In-house software expands so as to require the maintenance of the
entire engineering team.” – Jack Danger’s law of not-invented-here

The job of your company is to produce a product that is worth hard cash
to somebody. And your job is to build that product. Not to build a
database, not to build a javascript framework, and not to build a new
testing library. Those are great hobbies and if your company has the
in-house skill to build a best-of-breed tool and release it then
absolutely do it. But that’s rare.

It’s better to build a simple solution than a complex one, even better
to use an open-source solution, and if you see a path to do it without
software at all then take it.

 

  1. You could replace ‘feedback’ with ‘market testing’ and this is also
    the formula for effectively driving a startup.

Please if you found this post helpful or have questions.