Here’s me evangelizing the idea of an ephemeral database one more time.

When we first put ours in, I wasn’t totally convinced that it was the right decision. Keeping high volume data out of the core path seemed like a good idea in a conceptual sense, but another migration line and another database to look after would make development and operations harder.

Months later, I’m certain it was the right decision. Our core database (accounts, clusters, subscriptions, billing, audit log, etc.) is stable at less than 1 GB of data, while the ephemeral DB has ballooned to 300 GB by virtue of the kind of raw, voluminous data it holds (metrics). Having the two operationally independent in case of a major catastrophe helps me sleep better at night. If our ephemeral DB were to go hard down, metrics pages wouldn’t load, but everything else would work just fine. Maintaining a separate migration line for the ephemeral DB is mildly annoying, but that annoyance is so mild that I’ve stopped noticing. A single make target bootstraps both databases for development/testing, and 95%+ of database changes happen on the main line.

Software engineering is tradeoffs abound. But I’m giving ephemeral DB the platinum award of computation – major benefits, and with downside so miniscule that it’s hardly noticed.