In Pursuit of Production Minimalism

While working at Lockheed during the cold war, Kelly Johnson was reported to have coined KISS (“keep it simple, stupid”); a principle that suggests glibly that systems should be designed to be as simple as possible.

While complexity is never a conscious design goal of any project, it arises inherently as new features are pursued or new components are introduced. KISS encourages designers to actively counteract this force by making simplicity an objective in itself, and thus produce products that are more maintainable, more reliable, and more flexible. In the case of jet fighters, that might mean a plane that can be repaired in the field with few tools and under the stressful conditions of combat.

During his tenure, Lockheed’s Skunk Works would produce planes like the U-2 and SR-71; so notable for their engineering excellence that they’ve left a legacy that we reflect on even today.

Minimalism in technology

Many of us pursue work in the engineering field because we’re intellectually curious. Technology is cool, and new technology is even better. We want to be using what everyone’s talking about.

Our news sources, meetups, conferences, and even conversations bias towards shiny new tech that’s either under active development or being energetically promoted. Older components that sit quietly and do their job well disappear into the background.

Over time, technologies are added, but are rarely removed. Left unchecked, production stacks that have been around long enough become sprawling patchworks combining everything under the sun. This effect is dangerous:

• More parts means more cognitive complexity. If a system becomes too difficult to understand then the risk of bugs or operational mishaps increases as developers make changes without understanding all the intertwined concerns.

• Nothing operates flawlessly once it hits production. Every component in the stack is a candidate for failure, and with sufficient scale, something will be failing all the time.

• With more technologies engineers will tend to become jacks of all trades, but masters of none. If a particularly nefarious problem comes along, it may be harder to diagnose and repair because there are few specialists around who are able to dig deeply.

Even knowing this, the instinct to expand our tools is hard to suppress. Oftentimes persuasion is a core competency of our jobs, and we can use that same power to convince ourselves and our peers that it’s critical to get new technologies into our stack right now. That Go-based HA key/value store will take our uptime and fault resilience to new highs. That real-time event stream will enable an immutable ledger that will become foundational keystone for the entire platform. That sexy new container orchestration system that will take ease of deployment and scaling to new levels. In many cases, a step back and a moment of dispassionate thought would reveal that their use could be withheld until a time when they’re known to be well vetted, and it’s well understood how they’ll fit into the current architecture (and what they’ll replace).

Through ephemeralization

In his book Nine Chains to the Moon (published 1938), inventor R. Buckminster Fuller described the idea of ephemeralization:

Do more and more with less and less until eventually you can do everything with nothing.

It suggests improving increasing productive output by continually improving the efficiency of a system even while keeping input the same. I project this onto technology to mean building a stack that scales to more users and more activity while the people and infrastructure supporting it stay fixed. This is accomplished by building systems that are more robust, more automatic, and less prone to problems because the tendency to grow in complexity that’s inherent to them has been understood, harnessed, and reversed.

For a long time we had a very big and very aspirational goal of ephemeralization at Heroku. The normal app platform that we all know was referred to as “user space” while the internal infrastructure that supported it was called “kernel space”. We want to break up the kernel in the kernel and move it piece by piece to run inside the user space that it supported, in effect rebuilding Heroku so that it itself ran on Heroku. In the ultimate manifestation of ephemeralization, the kernel would diminish in size until it vanished completely. The specialized components that it contained would be retired, and we’d be left a single perfectly uniform stack.

Realistic? Probably not. Useful? Yes. Even falling short of an incredibly ambitious goal tends to leave you somewhere good.

In practice

Here are a few examples of minimalism and ephemeralization in practice from Heroku’s history:

• The core database that tracked all apps, users, releases, configuration, etc. used to be its own special snowflake hosted on a custom-built AWS instance. It was eventually folded into Heroku Postgres, and became just one more node to be managed along with every other customer DB.

• Entire products were retired where possible. For example, the ssl:ip add-on (providing SSL/TLS terminate for an app), which used to be provisioned and run on its own dedicated servers, was end-of-lifed completely when a better (and cheaper) option for terminating SSL was available through Amazon. With SNI support now widespread, ssl:endpoint will eventually follow suit.

• All non-ephemeral data was moved out of Redis so that the only data store handling persistent data for internal apps was Postgres. This had the added advantage of stacks being able to tolerate a downed Redis and stay online.

• After a misguided foray into production polyglotism, the last component written in Scala was retired. Fewer programming languages in use meant that the entire system became easier to operate, and by more engineers.

• The component that handled Heroku orgs was originally run as its own microservice. It eventually became obvious that there had been a time when our microservice expansion had been a little overzealous, so to simplify operation, we folded a few services back into the hub.

To recognize the effort that went into tearing down or replacing old technology, we created a ritual where we symbolically fed dead components to a flame called a burn party. The time and energy spent on some of these projects would in some cases be as great, or even greater, as it would for shipping a new product.

In ideas

Practicing minimalism in production is mostly about recognizing that the problem exists. After achieving that, mitigations are straightforward:

• Retire old technology. Is something new being introduced? Look for opportunities to retire older technology that’s roughly equivalent. If you’re about to put Kafka in, maybe you can get away with retiring Rabbit or NSQ.

• Build common service conventions. Standardize on one database, one language/runtime, one job queue, one web server, one reverse proxy, etc. If not one, then standardize on as few as possible.

• Favor simplicity and reduce moving parts. Try to keep the total number of things in a system small so that it stays easy to understand and easy to operate. In some cases this will be a compromise because a technology that’s slightly less suited to a job may have to be re-used even if there’s a new one that would technically be a better fit.

• Don’t use new technology the day, or even the year, that it’s initially released. Save yourself time and energy by letting others vet it, find bugs, and do the work to stabilize it. Avoid it permanently if it doesn’t pick up a significant community that will help support it well into the future.

• Avoid custom technology. Software that you write is software that you have to maintain. Forever. Don’t succumb to NIH when there’s a well supported public solution that fits just as well (or even almost as well).

• Use services. Software that you install is software that you have to operate. From the moment it’s activated, someone will be taking regular time out of their schedule to perform maintenance, troubleshoot problems, and install upgrades. Don’t succumb to NHH (not hosted here) when there’s a public service available that will do the job better.

It’s not that new technology should never be introduced, but it should be done with rational defensiveness, and with a critical eye in how it’ll fit into an evolving (and hopefully ever-improving) architecture.

Nothing left to add, nothing left to take away

Antoine de Saint Exupéry, a French poet and pioneering aviator, had this to say on the subject:

It seems that perfection is reached not when there is nothing left to add, but when there is nothing left to take away.

Most of us can benefit from architecture that’s a little simpler, a little more conservative, and a little more directed. Only by concertedly building a minimal stack that’s stable and nearly perfectly operable can we maximize our ability to push forward with new products and ideas.