Heroku: Core Impact

Herald Square in New York

It’s been a rough month for Heroku. On April 15th, they opened a security notice that’s been active ever since – the status page, designed for incidents measured in hours rather than days or weeks, dutifully reads off a duration of “478 hours, 9 minutes” 1.

According to their latest update, an attacker gained access to Heroku’s main database (called core-db back in my day) and exfiltrated its contents, including hashed passwords and secrets used for GitHub integrations. The latter were used to iterate the private GitHub repositories of Heroku users, and it seems to have been a stroke of luck that the hacker was aggressive about doing so, because while no one at Heroku/SFDC noticed the intrusion, GitHub flagged the unusual activity and tipped them off.

Heroku indicates that although the attacker accessed the raw application environments where secrets are stored, they didn’t have the key necessary to decrypt them. This implies that they weren’t able to access the secrets to the API itself, which in turn implies that they weren’t able to get the decryption key for secrets stored in the main database. This is where things get a bit odd because we know they were able to use OAuth secrets, which would imply that those weren’t encrypted. Odd that no one had thought to encrypt those secrets in particular, but especially in software, oversights do happen.

As a current Heroku customer, it’s been hard not to notice the disruption. Not only were all GitHub integrations and deploys disabled immediately after – they’ve stayed disabled ever since (exactly a month as of today). I’m a happy git push heroku user most of the time anyway so it hasn’t affected my habits much, but things like my auto-deploying docs set up have been MIA for weeks.

I’ve been enjoying reading comments from ex-employees posted to HN under throwaway accounts. For example, on Project Periwinkle:

They’re working on something called “Project Periwinkle” that is intended to remove all Heroku branding and make everything Salesforce branded. Periwinkle being a colour between blue (salesforce) and purple (heroku). No more Heroku signups, you’ll need a Salesforce account to use it. No more free tier either.

Heroku has been in the process of being sunset for years now. New features have been banned for years. Only “keep the lights on” projects are allowed. Not that they could do anything with the skeleton crew they have running the platform.

Another insider throwaway notes that the product is frozen in all but name, and has been for years:

You’ve gotta go way back on the Heroku Changelog to find anything that isn’t a language version upgrade or feature removal: https://devcenter.heroku.com/changelog

I think the feature freeze happened in 2018

As to why GitHub integration is still disabled:

GitHub is not allowing them to turn it back on yet (source: multiple people on both sides)

(Keep in mind that these are from random anonymous accounts on the internet, so exercise your usual judicious skepticism for such things.)

One of the things Patrick would assiduously list as an existential threat to the success of Stripe was a major security incident – the absence of a major security incident in the past doesn’t rule out the possibility of one in the future, and one may be all that’s needed for catastrophic reputational harm. Based on the number of mentions I’ve seen of Fly.io and Render over the last couple weeks (up about 100x from before), Heroku may be about to become poster child incarnate proving this fear.

For better or worse, it’s been the most publicity the company’s had in years. And with sympathies to my ex-colleagues who’ve spent nearing a month working on response, what a crazy time.

Relatedly, a few weeks ago one my colleagues tweeted the not-so-simple question of, “was Heroku a success or failure?”

It got me thinking. I remember a few years back having a conversation with one of my ex-colleagues from the company where I said something (probably unwisely) that implied that Heroku had been a failure, and which was met with open astonishment as the other party had internalized precisely the opposite impression – to him, it as an unmitigated success. Reaction to the Twitter question was similarly split, with lots of engagement in both directions.

Selling to Salesforce for $212 million was an obvious win, but that’s balanced by a product that’s broadly been on cruise control since 2012, and which hasn’t seeded much in terms of product longevity or lasting industry technology.

Most obviously, with an acquisition of that size, some people got rich, and even for those who didn’t make anything off the sale (yours truly included), it still gave us a couple of great years working in an environment that was startup-like in terms of feel and innovation, but with the benefits of big tech salaries and largesse.

There’s also the matter of Heroku being surprising sticky. Given a product that’s gone largely unchanged for years now, along with a host of new entrants in the market, I would’ve assumed that Heroku would’ve been resigned to obscurity by broader cloud competition years ago, but to this day, it’s still plausible as a platform. At my company, we made the decision to host on it as little as a year ago – we know the product well, there’s minimal lock-in in case we need to evacuate, and it keeps us generally hands-free on operations/infrastructure not core to the services that we’re selling. Every major cloud provider’s introduced new services aimed at satisfying that PaaS tier (and in cases like AWS, more than one), but few so far have been able to match Heroku’s streamlined workflows and simplicity.

Beyond that, a lot of great things fell out of the company:

That’s a pretty impressive list – even one or two of these would be more of a mark on the world than most tech companies ever make. But there’s also a common undercurrent to most these items – although they’re all great ideas and will make a lasting impression into the future in how services are deployed, none of them resulted in lasting residuals for the Heroku product itself – other platforms captured the concepts and took the proceeds, and even with commercial aspect aside, no specific technologies will be attributed back to Heroku. Even though Docker as a company might be doomed to failure, it’ll be remembered as the progenitor of container-based deployment for decades. Future history about the 2010s will talk about the evolution of Docker to OCI, but Heroku will be a footnote at best.

Heroku was the ultimate ideas factory for the cloud – concepts like 12-factor, erosion resistance, and DX will stand the test of time, but few of their beneficiaries will recognize their lineage back to Heroku.

Not much lasting product or technological impact is one side of the coin, the other being disappointment around an epic vision with so much potential that never materialized.

The Cedar stack was a work of true genius (I had no hand in its creation, so this isn’t as self-serving as it might sound). The previous Aspen and Bamboo stacks were far more limited, supporting only specific versions of specific stacks, and with a lot of special-casing necessary. Cedar made Heroku the platform that could run everything – users could bring their own stacks with buildpacks and Procfiles, and its sophisticated internal state machine and routing layer made apps running on it impressively robust. Even after learning how the sausage was made, I never stopped being impressed by how well it worked, or how the platform could be pushed to do things that its creators never imagined thanks to great primitives and the flexibility that fell out of them.

Back in 2012 the momentum coming off shipping Cedar was so great that despite its success, it was only considered the first step of a much more ambitious project. Before long, it’d be extended to handle programs of all shapes and sizes, with the current 512 MB container just an incidental first option. Even the biggest data-crunching apps would be deployable on containers with 10s or 100s of GBs of memory, and all the way down to tiniest one-off cloud grep runs needing only just a couple of megabytes. So fast and easy that it’d crazy not to run on Heroku.

It’d become modular. The shared router fleet was an adequate option for most uses, but large users might want to bring their own routing implementations to avoid the party cloud, or to provide their own highly customized routing configurations. Even in the Heroku “kernel” would be swappable so that you could still have Heroku building, orchestrating, and monitoring your apps, but those apps would be running on your own dedicated, single-tenant servers.

The Heroku cloud would become so extensible and so robust that just like a self-bootstrapping language compiler, it’d be able to host itself. Core components like the platform API, dyno state machine, and router would run as Heroku apps and gain all the DX ergonomics and robustness of such. This optimistic and ambitious vision was coined, The Self-hosting Singularity.

It’d be the anti-AWS. Whereas AWS exposes every possible primitive to a new user the first time they log in – thousands of confusing and overlapping concepts – the Heroku vision would’ve been to expose none of it to a new user. They’d be started with a basic git push heroku master and single dyno app, but as their software developed and their requirements became more complicated, new primitives would’ve been progressively exposed as they were needed – VPCs with ingress/egress rules, configurable hosts with alternative base images or architectures, SSH access, static IPs, etc. An onion that could be peeled back layer by layer.

There were some other things too. 12-factor factor no. IV (“backing services”) describes “attachable resources”, persistent services like databases that live as isolated resources which can be attached and detached to and from more ephemeral apps at will. It took us years to ship this feature, and although it was pretty neat when we did, by then Heroku’s golden days of product leadership were behind it, and we didn’t make much in the way of inroads into convincing anyone else as to why it was a good idea.

Pricing was another elusive beast. The cost of jumping up from the free tier to a paid app was a big step up which users had complained about it since the first day the product launched. Eventually, a new pricing model did ship, but didn’t do much to address those original concerns.

So what happened? All the foundations for success were in place, so not achieving its ambitious vision wasn’t an inevitability.

All in all, especially in light of recent security concerns, Heroku as a self-sustaining product was a failure. Heroku as a prolific progenitor of ideas, and direct ancestor of countless current and future tools and platforms, was a grand success.

A few links this week:

Greenwich St intersection

Apropos of nothing, I had the opportunity to visit New York a few weeks back. I’d enjoyed my time when I went last, but can’t say I was hugely impressed – Central Park and the Met were neat, but walk by enough tall buildings and they all start to blend together after a while. This time it left more of an impression on me.

The city’s had some notable upgrades since my last visit. Times Square is more friendly than it used to be for plebians like myself trying to navigate it on foot, and there were an impressive amount of new bike infrastructure. But the biggest win is similar to San Francisco’s – parklets. They’re everywhere, and some restaurants now must have as much space outdoors as they do in. Bike couriers have replaced fixed gear with electric, and a new hallmark of the city is seeing these guys whizzing by on every street, running reds more often than not.

Brooklyn seems to be about as cool as people say it is. I was told that it’d be full of trendy, eclectically-dressed young people, and to my surprise, it was full of trendy, eclectically-dressed young people (in my experience, the romantic elements of cities tend to be exaggerated more often than not). See photo of this cafe with “no laptop” signs on every table, which despite working from a cafe on a laptop right now, is an idea that I’m entirely supportive of. There was even one guy in there who was, *gasp*, reading a book.

A Brooklyn cafe

Most importantly, I saw a little bit of that infamous New York grit. Despite its own long darkness, the city’s rebounded, and feels vibrant and alive. Contrast to SF, which to this day is more reminiscent of a bad set for a shoestring zombie apocalypse film than anything else.

I made one mistake of taking a redeye into town, arriving in the city around 7 AM, and assuming I’d be good to walk around until check-in around three that afternoon, not realizing that turns out, New York is pretty cold, and that east coast wind chill sure doesn’t help. Four hours later, after having made my way from the center of the city down to around Wall Street further south, and realizing that I could no longer operate my camera because my hands were so numb, I figured it was a good time to make a tactical retreat into the first Starbucks I could find.

I can feel my way around the subway now, but for the life of me, still can’t seem to get the regional trains right. After getting to Penn station, only to realize that it was apparently the wrong Penn station (Moynihan) with no signs pointing to the right Penn station, then walking up to the Port Authority after a stupendous misreading of Google Maps, before making my way back to the other part of Penn and making a few rapid fire guesses as to the right platform, I caught my train with roughly 13 seconds to spare – an almost perfect repeat of my misadventure the last time I left NY.

Until next week.

The Highline

1 Although it seems like they stopped the clock at 19 days. Double that number would be more apt now.