Since my last post complained about things that haven’t changed in Germany, I’ll follow up with some that have:

• Deutsche Bahn now has an app that can purchase or import tickets, and it works well. Ticket checkers can scan a QR code to verify your ticket, where previously they’d only take a paper printout, which was looking very old fashioned in 2010, and a huge pain for visitors that couldn’t manage to fit a printer in their carry-on. It can also purchase tickets for local rail systems like MDV in Leipzig.

• The app has real-time platform information for each station. You can check your train’s platform before arriving, and use the information to immediately go to exactly the right place. It sends accurate push notifications, and alerted me to a last minute (literally) platform change in Berlin.

• Onboard wifi is now free, and it works.

This was all a major relief after having just experience the comparative hellscape which is the UK rail system.

It’s been years since I was in Germany last, and I was curious to see what’s changed. Particularly:

• Germany is a cash-heavy society. This might not be a problem except that no one has any change, no one wants to take card, no one wants to break €50 notes, and ATMs won’t dispense anything but €50 notes. Has anything changed?

• The first couple times I went to Berlin, bars were like a time warp back to North America in the 80s and 90s, with an all-pervasive, ever-present cloud of smoke in every one of them. And even outside of bars, you’d find people smoking in every stoop and alley. Has the country stopped smoking yet?

I was optimistic on the first point at least, after having successfully spent two weeks in the UK without having withdrawn a single pound. Surely conventions between European countries can’t be too dissimilar.

But alas, they are. The answers are “no” and “no”. Maybe next time.

After a flight cancellation that snowballed into a logistics nightmare, I took the train from London to Berlin.

Four hours on the Eurostar London to Amsterdam (the fast train best known for making Channel Tunnel trips between London and Paris), and then eight hours on a slower train across Germany’s countryside to Berlin. Wake at 4 AM (to preclear customs for Eurostar), arrive at 8 PM, with a one hour forward time difference. I wouldn’t recommend it, but hey, it’s cool that it’s possible.

Published sequence 052, the Flying Scotsman and locomotive 35018.

Published fragment PGX + sqlc v4 to v5 upgrade notes. Somewhat painful, with 114 files and ~800 LOCs changed.

Nanoglyph 037 is published, on speed as a UX feature.

Discourse sees Ruby rendering speeds decrease 16-17% after enabling YJIT in Ruby 3.2. A nice performance boost for a minimal time investment that amounts to tweaking the RUBY_YJIT_ENABLE env var and doing some testing.

Published sequence 052, La Grande Arche.

Published sequence 051, Musée de l’Armée.

Published sequence 050, Rue Claude Monet.

Nanoglyph 036 is published, on Atlanta, job queues, batch-wise operations.

Published sequence 048, under construction.

32 consecutive days of running at least 5 km (average 7-8 km).

Observations: Get it done early, habit is everything. For the lazy amongst us (me), it may require bullying yourself out the door. I hate eating my vegetables, but I know that I should.

Tomorrow: Flight to Paris wherein I time travel half a day forward. I’ll do a morning San Francisco run, but one after landing in Paris at 4 PM on the day after, exhausted from the hell that is a transatlantic flight in economy, will be hard. Is this the end? Find out next week.

Published sequence 047, venue.

Published sequence 046, whale shark(s).

Published fragment PG advisory locks in Go with built-in hashes.

Today I discovered the power of Postgres’ DISTINCT ON syntax, a useful feature that did exactly what I wanted it to with my first try.

Everyone knows the common distinct function (e.g. SELECT distinct(name)), which gets all unique values for a given field.

DISTINCT ON is similar, but it groups rows by a distinct field, and then returns whatever you want from one of them based on an ORDER BY clause.

Example: we have saved queries that have one or more query runs each. Return the latest query run (finished_at DESC) that either failed or succeeded for each saved query:

-- name: QueryGetFailedOrSucceededBySavedQueryIDMany :many
SELECT DISTINCT ON (saved_query_id) *
FROM query_run
WHERE status = 'failed' OR status = 'succeeded'
    AND saved_query_id = any(@saved_query_id::uuid[])
ORDER BY saved_query_id, finished_at DESC;

Easy to use and concise.

A piece from LWN on the early days of Linux at the University of Helsinki.

On Linus taking down the university’s fragile Sun machines:

At some point, Linux gained support for Ethernet and TCP/IP. That meant one could read Usenet without having to use a modem. Alas, early Linux networking code was occasionally a little rough, having been written from scratch. At one point, Linux would send some broken packets that took down all of the Sun machines on the network. As it was difficult to get the Sun kernel fixed, Linux was banned from the university network until its bug was fixed. Not having Usenet access from one’s desk is a great motivator.

On the version 1.0 release event (complete with press) in 1994:

In the spring of 1994 we felt that Linux was done. Finished. Nothing more to add. One could use Linux to compile itself, to read Usenet, and run many copies of the xeyes program at once. We decided to release version 1.0 and arranged a release event. The Finnish computer press was invited, and a TV station even sent a crew. Most of the event consisted of ceremonially compiling Linux 1.0 in the background, while Linus and others spoke about what Linux was and what it was good for. Linus explained that commercial Unix for a PC was so expensive that it was easier to write your own.

And it was recorded, complete with a young, handsome university student version of Linus.

After a year of inactivity and (more recently) some strange trouble with Mailgun mailing lists, Nanoglyph 035 is published, on Go generics, and our endeavor to build a safer API framework with them. This newsletter may be struggling, but it’s not completely dead, yet.

Linked from HN today, Future Blues, a 90s era Cowboy Bebop fansite, complete with Photoshop layout, shareable banner, and guestbook.

I miss the old internet. What’s available today is infinitely more powerful compared to what we had back then, but in many ways, so much worse. In the old days people built sites on subjects they were passionate about, and doing so was its own reward – this was before the web became monetized, and long before the pursuit of social media clout. There was no Squarespace, so people did their own design and coding, even if they weren’t technologists. You might’ve been hosting on Geocities, but it was also common to be running a web server on the computer under your desk.

Today, 95% of the internet has converged on the same bland big-fonts-in-center-column-with-lots-of-whitespace design (of which this site is guilty), and slathered with paywalls, cookie selectors, login modals, toast notifications, trackers, and so many other dark patterns that they’re hard to keep track of. And that’s the good news. The bad is that most people are skipping the website completely and retreating into the walled gardens of Instagram, TikTok, YouTube, or Twitter.

A hat tip to the few out there who still do it the old way.

And at long last, finished redesigning the main articles pages themselves, for example.

I’ve been intending to do this for at least two years, and for once am glad that I procrastinated on it. If I’d done it even mid-last year, it would’ve been in vanilla CSS instead of Tailwind, and over the subsequent years after that all the same problems around maintainability would’ve reoccurred. The site’s now close to being 100% Tailwind, and I’m more confident that I’ll be able to change things in the future without having to page a bunch of site-specific CSS context back into my brain, and without accidentally breaking anything as styles, you know, cascade.

The redesign is far from perfect (e.g. the tables of contents that were there before are MIA for now), and I’ll be tweaking it as I go, but I wanted to get something out the door instead of continuously delaying it into infinity. I’m in Europe all next month, and while I hope to do many writing dispatches from there, the likelihood is that no major design projects will be forthcoming.

Redesigned my newsletters page. It’s a sad day, but I’m officially calling Passages & Glass defunct. Nanoglyph very well may be too unless I can get one out the door soon.

A Berlin-based prompt artist wins a category of Sony World Photography Awards, and because he’d submitted to make a point, refuses the prize.

This is the tip of iceberg in two ways:

• This conscentious objector will be the rare exception that reveals what they did. Most AI-generated content will (and already has) win the prize and take the money without saying anything, or pass generated work off as if it was human-created, or take their spot on the front page of HN.
• This is just the beginning. It’s going to get more widespread and much worse.

Why does it matter? This is going to cheapen content in a fundamental way. News outlets, social media platforms, and Google search results are going to be flooded with this stuff. Even employers trying to pay for original writing or art will have a hard time telling if the person is lying and just turning in GPT output. There will be job losses. Arguably, they’re here already.

Long term, there will be a loss in human aspiration. Remember how your parents or grandparents knew how to build things in their garage or be able to make/repair clothes, and now almost no one does as we replaced those skills with fast fashion and Amazon.com? How many kids can we expect to spend the thousands of hours learning to write, draw, or make music when they can accomplish better results with instant gratification by prompting GPT? Many would contend that this is fine, humans don’t need to be doing that stuff. But to that I would ask two questions:

1. In a hyper competitive world, what exactly are your kids going to have that separates them from everyone else?
2. What exactly are we doing with all that saved time? More YouTube? More TikTok? The Star Trek: The Next Generation future where people master the violin in their spare time does not appear to be forthcoming.

A few years ago I wrote Tweeting for 10,000 years, a thought experiment in how to improve the longevity of software, which is infamously prone to breakage by way of bugs or changes in its underlying infrastructure. It explores ideas like choosing platforms likely to be long-lived, stable languages, minimizing dependencies, and deploying self-contained binaries.

In it I list two of the top risk factors for its eventual demise as:

• Changes in Twitter’s API could spell the end. This would take the form of a backwards-incompatible change like a new required parameter, change in the structure of responses, or adjustment to how applications authenticate.

• Relatedly, changes in *Twitter’s product are also dangerous. They could move to a new pricing model, remodel the product’s core design, or fold as a company.

Well, if it’d made it 10,000 years you probably wouldn’t be reading this. Today I got this email:

This is a notice that your app - 10000-years - has been suspended from accessing the Twitter API.

Please visit developer.twitter.com to sign up to our new Free, Basic or Enterprise access tiers.

The experiment’s final longevity was 4 years, 8 months. Not half bad, but a little short of the stated goal.

It may be for the best. What I put in those time capsule messages probably wouldn’t have aged well anyway. But the article’s thesis was correct – writing software that withstands time and entropy on all fronts sure isn’t easy.

Today I learned that Rack servers like Puma can run an operation after a request has been completed with rack.after_reply. There’s been various ways to accomplish this since the early 2010s, but I hadn’t realized that it’d become standardized.

This can be a great way to offload more expensive operations like network calls out-of-band from user requests to improve latency (even if incompatible with use of request-level transactions). In the linked GitHub describes using rack.after_reply for emitting statistics, and they’d previously used a similar technique to perform Ruby garbage collection between requests, although later stopped doing that.

At Stripe we did something similar to make generating “events” (for use in sending webhooks) out-of-band. Generating events involved rendering API resources, which was expensive, making generating events also expensive, so taking them out-of-band shaved somewhere in the neighborhood of 10-100 ms in latency (can’t remember the exact numbers) off some of our most critical API paths.

Here’s me evangelizing the idea of an ephemeral database one more time.

When we first put ours in, I wasn’t totally convinced that it was the right decision. Keeping high volume data out of the core path seemed like a good idea in a conceptual sense, but another migration line and another database to look after would make development and operations harder.

Months later, I’m certain it was the right decision. Our core database (accounts, clusters, subscriptions, billing, audit log, etc.) is stable at less than 1 GB of data, while the ephemeral DB has ballooned to 300 GB by virtue of the kind of raw, voluminous data it holds (metrics). Having the two operationally independent in case of a major catastrophe helps me sleep better at night. If our ephemeral DB were to go hard down, metrics pages wouldn’t load, but everything else would work just fine. Maintaining a separate migration line for the ephemeral DB is mildly annoying, but that annoyance is so mild that I’ve stopped noticing. A single make target bootstraps both databases for development/testing, and 95%+ of database changes happen on the main line.

Software engineering is tradeoffs abound. But I’m giving ephemeral DB the platinum award of computation – major benefits, and with downside so miniscule that it’s hardly noticed.

Published fragment Lean = fast, on the unreasonable effectiveness of small teams and sharp tools for getting features out the door.

Updated now with recent features shipped and a new run commute.

Following up from yesterday, redid the fragments index. More consistency. More Tailwind. Less hand-written CSS that I’d given up on being able to maintain years ago. Only like three quarters of the site left to go.

I redesigned the layout for fragments to bring it more in line with the “V2” redesign used for atoms and my now page. I hadn’t intended to do this today, but started prototyping this morning, and it got far enough along that it got to a point where although far from perfect, I was just like “why not”.

See for example, Ephemeral DB.

Glaciers flow faster than this redesign is coming along, but it’s seeing slow progress forward.

Two weeks into a WeWork subscription. An aspect I love is the gear. Most people are just using a pretty vanilla laptop set up like me, with a large minority using a laptop stand and external keyboard.

But some go all the way, with large form factor notebooks, tablets, and external screens, sometimes multiple of each. They sit down, start pulling things out of their bag, and within minutes a fully operational battlestation appears. They can be creative, but unlike a traditional desk set up, there are constraints – it all has to be portable. Calibrated in the morning. Collapsed in the afternoon. No trace left behind.

Pragmatic, minimalist, and with a dash of Marie Kondo.

Published fragment Stay mainline, on GitHub’s practice of keeping their code synchronized with Rails main.

Published sequence 044, indomitable.

Okta is considered to be the gold standard in the identity management front, the sleek Silicon Valley entrant to a market of long-toothed competitors.

I was looking into building an Okta integration for inclusion in the Okta Integration Network (OIN), and something that I found incredibly surprising is how manual the whole process it. Intuitively, I would’ve thought that an Okta admin would browse the catalog, install an app, and be automatically granted access to its features as machines cooperated behind the scenes to make it happen.

Here’s how it actually works: an Okta admin browses the catalog, installs an app, rummages around in Okta dashboards extracting a set of URLs, IDs, and secrets, then manually transmits them to the app owner to facilitate integration. CloudFlare provides an Okta-specific Dashboard to enter them. Brex will send you an email where you copy them in. Others suggest visiting their sites and opening a support ticket with the magic values.

I thought I (and all of them) must be missing something, but I don’t think so. Here’s Okta developer support confirming this is how it’s done.

Does this seem, like, crazy, to anybody else? Of course there’s still going to be some manual inputs on the web, but for this to be the normal process for most preeminent IDM on the internet? Again, surprising.

A few weeks ago Tailscale announced they’d be allowing anyone to bring their own OIDC provider. My first reaction to this was that it seemed like overkill more likely than not to create unforeseen problems. But after looking into Okta, my new train of thought is that if we were to build that, we’d not only provide the openness of OIDC standard, but it could act alternatively as our Okta onboarding portal for free. Is this the way?

For the last year, absent an even semi-reasonable work-from-home set up, I’ve been working in and out of cafes across town. It’s okay, but it leaves you feeling like a vagabond, always thinking about whether you’ve bought a coffee recently enough for the workers/owners to not be metastasizing latent resentment towards you, and always negotiating for wi-fi and bathroom codes.

Last week I finally got sick of it and got WeWork. Paired with a shiny new gym membership, it’s had the major benefit of letting me build in a morning run commute, guaranteeing daily exercise, and leaving me with an intense mental acuity for the first couple hours of the day, an effect now attributed to endocannabinoids that are part of the runner’s high.

Modern technology is great. I have WeWork All Access, meaning I can’t leave anything at the office. My pack contains deoderant, a change of clothes, and three pieces of technology:

• M1 laptop, to work on.
• AirPods Pro, for podcasts while running, and to take calls.
• iPhone, for everything else.

The M1 (paired with an eye on Activity Monitor for power offenders) gives me all-day power and then some. I don’t carry charging cables or any other peripherals. The grotesqueries which were the butterfly keyboard and Touch Bar are finally dead, so no external keyboards required. With built-in retina display, fast Cmd-Tab, and good window management I don’t miss an external monitor.

This seems obvious, but even five years ago it would’ve been a pipe dream. All this technology is cool, but I have to give the tech-of-the-decade award to the M1, which came out of left field, and has fundamentally changed the expectations around responsiveness and power requirements in computation.

A few weeks back I’d mentioned in passing that we’d migrated off Keycloak.

I got a question about why, so I’m putting down some rationale here. I respect open-source projects a lot, so this isn’t meant to throw shade on Keycloak in any way, but I believe in being transparent about these things so people can read alternative arguments and make better-informed tech decisions.

A few reasons why we moved away from Keycloak:

• It has an API for integration, but its API reference offers precious little beyond the names of API resources and fields. There was never enough documentation to integrate a new feature, always necessitating trial and error.

• Core authentication-related flows were the most likely to regress because writing tests against a remote API requires a lot of stubbing. With heavy stubbing involved, you really only end up writing tests that verify that your stubs are set up the way you expected them to be, and that made it very easy to break things because mistakes would not be caught by CI.

• Keycloak pages like login or email verification are customized using a template language that was divergent from our main Frontend stack, and which no one was every particularly excited to write. They’d languish without updates and look awful compared to the rest of the site. This is also why some features were slow to happen, like multi-factor support, which we’ll finally be shipping next week.

• Operationally, it was a black box wherein if something went wrong, no one knew how to fix it. We had a very near miss where a Heroku regression was crashing Keycloak on start up with an inscrutable backtrace, and which luckily manifested in our staging environment first, but which we knew was coming for prod on the next 24 hour dyno cycle. None of the people who’d chosen and integrated Keycloak had even the faintest idea how to diagnose the problem or fix it. In the end we were able to debug it by digging deeply in Keycloak source code, but were within hours of our login system being hard down with no fix in sight.

• I wasn’t super confident of its security bonafides given the glimpses into its internals I’d seen. For example, it defaults to PBFKD2 for password hashing at 27,500 iterations, far below the 2023 recommendation from OWASP of 600,000 iterations. I admit to having only this one example and maybe everything else is above bar, but who knows.

• It’s a huge Java app and not that cheap to run. It needed a performance dyno on Heroku due to high memory requirements, and between that and a database in staging and prod it was $800/month, or about half our total bill.

I’m usually against the NIH mindset, but sometimes it’s right. Looking up best practices for storing passwords is easier than looking up how Keycloak does it, the answer to which isn’t documented and only exists in Discourse forums or by reading source code. Integrating SSO isn’t that hard anymore thanks the widespread standardization around OIDC. Implementing flows for email verifications and password resets is a little more work, but you end up having to do most of it under Keycloak anyway if you want a custom theme.

DPReview.com is shutting down on April 10th.

G’damn. This is one of maybe three to five websites that I visit every day. It’s well known for its in-depth reviews, but was also one of the few sites out there that hadn’t degenerated into the labyrinth of clickbait and dark patterns that is the modern web. If I wanted to see which lenses Canon’s released this year or last, DPreview is the only place I’d think about getting that information, far more usable than even Canon’s official site.

I’d forgotten that Amazon had bought them – a shame. An underperforming division of the trillion-dollar everything-store company might be a reasonably sustainable independent business on its own terms.

Published fragment on last week’s layoffs from Meta, on Mark Zuckerberg’s words on manager-heavy organizations and remote work.

The crypto crowd is at it again, with the latest being Balaji saying he’d bet $1M that Bitcoin will be worth $1M within 90 days (current price is $26k). Multiple high net worth individuals have committed to taking the bet. I assume what happens next is that the idea is memoryholed, but this could be an epic win for crypto-skeptics if it takes.

Published fragment Policy on util packages.

Stated simply, general util packages aren’t allowed because they tend to become dumping grounds. But, specifically targeted util packages with narrow, focused domains, are.

util/
    assetutil/
    cookieutil/
    cryptoutil/
    emailutil/
    jsonutil/
    maputil/
    passwordutil/
    ptrutil/
    randutil/
    signingutil/
    sliceutil/
    stringutil/
    testutil/
    timeutil/
    uuidutil/

Published fragment CL.THROTTLE implemented in DragonflyDB, in which the rate limiting command from redis-cell is implemented in DragonflyDB.

CL.THROTTLE user123 15 30 60 1
               ▲     ▲  ▲  ▲ ▲
               |     |  |  | └───── apply 1 token (default if omitted)
               |     |  └──┴─────── 30 tokens / 60 seconds
               |     └───────────── 15 max_burst
               └─────────────────── key "user123"

Published fragment RFCs and review councils, prompted by Squarespace’s write up on their RFC and review process, which sounds very close to Stripe’s.

Published sequence 043, outpost.

Downskill skiing is a very uneven sport. Trips are booked months in advance, and the time you get there, you’re at the mercy of the weather gods. Occasionally you can be buried in snow, like the Tahoe region was this last week with 140”+ of fresh powder. Other times you get nothing, and have to find ways to make your own fun under scratchy conditions.

After years of bad luck skiing BC’s interior, we finally got a good one. A couple big dumps last week before we arrived to establish a base, and fresh snow Sunday, Monday, Tuesday, and Thursday while we were here. What a year.

A small CI improvement we made last week: when running Go workflows we’d generally define a variable for GO_VERSION near the top and pass it into each actions/setup-go below:

env:
  GO_VERSION: 1.20

steps:
  - name: Install Go
    uses: actions/setup-go@v3
    with:
      cache: true
      check-latest: true
      go-version: ${{ env.GO_VERSION }}

GitHub Actions don’t allow steps to be extract and moduralized in any way, so with many jobs in the workflow, the actions/setup-go step needs to be duplicated many times over. Defining GO_VERSION meant that when upgrading to a new version of Go, we only had to change one value in the file.

But it still wasn’t optimal because when upgrading to a new version of Go, we’d have to change the version in go.mod (twice actually, because we’re on Heroku and Heroku’s dumb +heroku goVersion 1.20 magic comment is mandatory), and then also in every GitHub Actions workflow .yaml file. It’s easy to forget one and break something.

There’s an easy solution. It turns out that actions/setup-go supports a go-version-file directive that reads a version out of a go.mod file:

steps:
  - name: Install Go
    uses: actions/setup-go@v3
    with:
      cache: true
      check-latest: true
      go-version-file: "go.mod"

Now when we upgrade to a new Go, only one file is changed.

From Amazon Route 53: Workload isolation using shuffle sharding.

In short, Route 53 domains are sharded by so they map to one of 2,048 virtual servers. The sharding process means that some domains share servers with other domains, and under a normal sharding algorithm, domains would share all their servers with those same domains on their shard. This isn’t great because if I’m an unlucky customer on the same shard as github.com who’s getting DDOSed all the time, if their shards are taken down that means I’m down too. “Shuffle sharding” defines sharding such that even if I share servers with github.com, some of the servers I map to are “shuffled” so that I’m also guaranteed to not share some with github.com. Even if every server hosting github.com goes down, some of mine are still up.

The link above explains it in more detail and with diagrams.

Last week, in a change presumably driven by LastPass’ recent compromise, Bitwarden released support for Argon2. We’d moved to Argon2id just two days before. Some good news about LastPass’ loss is that it’s been everybody else’s gain as other products take notice and shore up their own security.

After a three week odyssey involving cascading dependency failures and 24-hour build loops, Go 1.20 is released on Homebrew.

Published fragment Findings from six months of running govulncheck in CI.

Vulnerability #1: GO-2023-1571
  A maliciously crafted HTTP/2 stream could cause excessive CPU
  consumption in the HPACK decoder, sufficient to cause a denial
  of service from a small number of small requests.

  More info: https://pkg.go.dev/vuln/GO-2023-1571

  Module: golang.org/x/net
    Found in: golang.org/x/net@v0.6.0
    Fixed in: golang.org/x/net@v0.7.0

    Call stacks in your code:
Error: client/awsclient/aws_client.go:156:34: awsclient.Client.S3_GetObject
    calls github.com/aws/aws-sdk-go-v2/service/s3.Client.GetObject,
    which eventually calls golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip

A follow up from yesterday’s post on GitHub being close to having ended up as a Postgres stack, somebody pulled Tobi from Shopify into the conversation, and it turns out Shopify was a near miss on Postgres too:

Yea. Shopify beta was on Postgres. But in our case it was the poor state of replication (in 2005) that made me switch to mysql before launch.

Published fragment Honest health checks that hit the database.

errGroup, ctx := errgroup.WithContext(ctx)

errGroup.Go(func() error { return checkDatabase(ctx, svc.Begin) })
errGroup.Go(func() error { return checkDatabase(ctx, svc.BeginEphemeral) })

if err := errGroup.Wait(); err != nil {
    return nil, apierror.NewServiceUnavailableErrorf(ctx, "Health check error: %v.", err)
}

In a surprise twist, Chris Wanstrath (@defunkt), one of the founders of GitHub, has taken back to Twitter after a long dormancy. Better yet, the subject matter is entirely about engineering, product, and occasionally metal. For example:

Last tidbit on DMs: We almost moved from MySQL to Postgres early on, and even had a branch going, but some of the horrible SQL queries I wrote for the private messaging feature didn’t work in Postgres. So, MySQL stayed.

You have to wonder about the counterfactual there, in which GitHub switched to Postgres and had to scale it, possibly having to develop a Vitess-like layer on top.

GitHub may just be the only decent candidate for the Apple of our generation, a company that did product well, continues to do it well, didn’t fail or turn evil, and whose history is something you’d actually want to read about. Chris, maybe do a book next?

The latest in the accidental culture war: Twitter is sunsetting SMS as a 2FA method.

As usual, the I-hate-whoever-WaPo-tells-me-to crowd immediately lost their minds, flying into spectacular fits of performative outrage, dramatically falling to the ground and beating their tiny fists against the floor wailing about Elon compromising security for society’s most vulnerable, like the downtrodden legacy blue-check elite, whose refusal to pay $8 puts them on moral parity with the conscientious objectors who fled north to dodge their draft for the Vietnam War.

For anyone who doesn’t follow this closely, SMS isn’t just the worst way to do second factor auth, it’s the worst by far:

1. Being monopolies that don’t actually have to be good at what they do, the security posture of telcos is total garbage, and simjacking isn’t just common, it’s terrifyingly common. The Twitter account of @Jack himself was taken over by this method as recently as 2019.
2. Lapsed phone numbers get reused, and new subscribers can often pick from a pool. Fish in a barrel.
3. UX-wise, SMS deliverability sucks, especially in countries without modern infrastructure, and especially in our age of ubiquitous SMS spam.
4. Sending SMS is expensive. So not only do you get reduced security, but you get to pay extra for the privilege.

If you’re a user, you shouldn’t use it. If you’re a provider, you shouldn’t provide it.

Sure, Musk’s motivation for retiring it is probably mostly (4), but that doesn’t make it a bad idea. Sure, it’s weird that Twitter Blue users can still use it, but that’s Twitter saying, “if you pay money, you have a license to do whatever you want, even opt into reduced security (at your own risk)”. You also have to think that paying users are < 1% of the platform, so their continued option to use SMS has negligble effect on the whole.

One more note on password migrations: damn it, we did find one bug. Luckily, an internal user ran into it early and it was never experienced publicly.

We run our Postgres service on Postgres, and develop like we prescribe others to, with a core tenet being data consistency. I’d added these CHECK constraints:

ALTER TABLE account
    ADD CONSTRAINT pbkdf2_not_null_check CHECK (
        (algorithm <> 'argon2id-pbkdf2-sha256' AND algorithm <> 'pbkdf2-sha256')
        OR ((algorithm = 'argon2id-pbkdf2-sha256' OR algorithm = 'pbkdf2-sha256')
            AND (pbkdf2_hash_iterations IS NOT NULL))
    );
ALTER TABLE account
    ADD CONSTRAINT pbkdf2_null_check CHECK (
        (algorithm = 'argon2id-pbkdf2-sha256' OR algorithm = 'pbkdf2-sha256')
        OR ((algorithm <> 'argon2id-pbkdf2-sha256' AND algorithm <> 'pbkdf2-sha256')
            AND (pbkdf2_hash_iterations IS NULL))
    );

They’re a little awkward to read, but the first guarantees we have a value for PBKDF2 hash iterations where the password involves PBKDF2. The second guarantees we don’t value a value where it doesn’t.

There was a case in the password reset flow where we’d upgrade a newly reset password to Argon2id, but if the old password was PBKDF2, forgot to empty its PBKDF2 hash iterations. The CHECK constraint failed along with the password reset.

We got a fix out quickly, but there’s an argument to be made that it wasn’t worth breaking a critical path (even temporarily) for data consistency.

The cost/benefit of this case is close enough that I might agree, but would still form a counterargument. A 500 is bad, but when this sort of thing would happen at Stripe (on Mongo, no CHECKs), instead of some 500s you’d end up with a pile of inconsistent data written that someone would have to manually repair afterwards. A small error that was in production for only a minute or two would lead to a multi-week cleanup effort, with many a bespoke migration and outbound email written.

When CHECKs (or FKs, data type validations, etc.) fail, no tainted state makes its way to the database. There’s user impact until the problem is fixed, but no impact afterwards. Also, even if a user request succeeds in the non-CHECK alternative with inconsistent data written, it doesn’t mean that things are actually right, and often leads users having to find out for themselves that something’s broken thanks to the inconsistencies.

We’ll probably keep doing the CHECKs.

Follow up from yesterday’s write up on password hash migration. We’re fully migrated:

=> SELECT distinct(password_algorithm) FROM account;
   password_algorithm
------------------------
 (null)
 argon2id
 argon2id-pbkdf2-sha256

(Notice: No vanilla pbkdf2-sha256 left. (null) is for password-less SSO-based accounts.)

I captured a backup beforehand just in case, but otherwise went for a pretty aggressive migration strategy of ovewriting hashes without intermediary fields or other major hedges. It went smoothly. Here’s a log line showing a hash upgrade to Argon2id-only, implying the user was able to sucessfully login via the nested hashing scheme:

password_hash_upgrade_line: Upgraded from "argon2id-pbkdf2-sha256" to "argon2id"
    [account: 03c97908-32a3-445b-a52e-03aa9fdec5f8] [hash time: 0.024923s]

Some guards I put in:

• Wrote tests for the migration. 95% of programmers have the attitude that tests are for code, but they’re unneeded for migrations, even complex ones. IMO, scary.
• Wrote the critical section of code to hash the hash in triplicate. Once to do the work, once to check the work from tests, and again in the migrator to hash again to check its own work. Writing it manually three times reduced the likelihood of me accidentally flipping the hash or parameter order by accident.
• Rolled the migrator out with a dry run flag, prompting it to do everything it’d normally do in a complete run, but then roll back the transaction afterwards, producing useful logging we can verify, but discarding results. (We use this transactional dry run trick a lot.)
• Used a compiled language with explicit, static types so my code is helping prevent bugs instead of creating them (cough Ruby).
• Manually tested the migration of my own passwords in a production-like-but-not-production environment before rolling out to prod.

Published a follow up to my write up on password hashing: Migrating weaker password hashes by nesting them in an outer hash.

Well worth reading: It’s so sad when old people romanticize their heydays, also the 90s were objectively the best time to be alive.

Don’t get me wrong, I’d hate to give up all the great stuff I’ve gotten access to over the last twenty years: access to all information ever produced by humanity a pocket’s reach away, portable computers with functionally infinite computing power and battery life, AirPods, ANC, Google Maps, cameras/lenses with better low light performance than my eye, Amazon Prime, and many more (notably, all of which are advancements in consumer electronics rather than hard science …).

But while these things have indisputably made things more convenient, they haven’t made things better.

The wins are small and nominal and the losses are large and profound. Community and social cohesion is at all time lows. Low key meeting spaces like malls, music venues, rundown bookstores, and dive bars are mostly gone – real estate draws such a premium that no one can afford to have it go to waste. Everywhere you go, every man, woman, and child buried in their personal glass.

Check out the article. It’s more convincing than I am.

Published fragment adventures in password hashing, in which we inherit PBKDF2 hashes from Keycloak, and after a couple intermediate steps, migrate to Argon2id.

Published sequence 042, a wall alive.

Seymour Hersh writes about how the US destroyed the Nord Stream pipeline, and although it’s built around the testimony of only a single source, it contains impressively specific details, like how the C-4 was planted by divers under the cover of the BALTOPS22 naval exercise, and was to be triggered by a sonar buoy that’d later be dropped by plane.

As every Maddow-loving, Fauci-fearing, big-D Democrat knows, this is a filthy lie. It was Russia that blew up Nord Stream for … reasons. They didn’t want to make money selling gas so badly that simply pressing the “off” button on the pipeline which they own was insufficient. No, it had to be a bolder, and shall we say, more incendiary statement. KA-BOOM! That Biden went on record a year ago saying he’d destroy it makes no difference. It’s Russia, stupid.

An interesting Twitter thread documents a Wikipedia edit war in real-time in which Mr. Hersh changes from “journalist” to “conspiracy theorist”, a favored new term of art of the illiberal, institutional left that translates roughly to English as “person who disagrees with me”. Most notable is how when the change was at risk of being reverted, powerful editors with special moderation privileges emerge from the woodwork to defend the presence of the new appellation.

On the day of GitHub announcing total closure of all physical offices, I published sequence 041, chez GitHub.

Published sequence 040, on a rainy evening in San Francisco.

This site uses AWS Certificate Manager (ACM) for automated TLS certificate managment, and I got a request via email today to approve a renewal through 2024. Validating ownership of a domain can either happen via email or by a CNAME record installed to its DNS, and all those years ago when I first set this up I apparently chose email and have been manually approving a renewal every year since.

Validation by CNAME is better because it’s fully automatic. As long as ACM sees that the CNAME is still installed, it can validate ownership of the domain without asking the user about it. So today, after many years of procrastination, I finally went into ACM to generate a new cert validated by CNAME instead of email.

While doing so, I noticed that as of mid-2021, ACM supports non-RSA certificates using ECDSA elliptic curves, which I opted into. They’re widely supported, and superior to RSA in every respect that matters: less computationally expensive to verify, shorter keys that mean reduced network traffic, and superior cryptography that was developed more in the open compared to RSA.

But as complete of a product as AWS is, sometimes it has some fairly mistifying edges. I was given the choice between the P256 and P384 curves, opted for P384, but then found that I couldn’t select my new certificate over in CloudFront (AWS’ CDN). Upon further reading I found CloudFront only supports P256. This limitation is not explained or justified. Just a vague “sorry, P256 only even though our other product generates P384”.

Amazon is an absolute powerhouse beyond all doubt, but is the weirdest of the trillion dollar cohort.

Published fragment PartialEqual, on a custom Go assertion helper that we’ve been using for a few months now to good effect:

resp, err := apitest.InvokeHandler(svc.Create, ctx, req)
require.NoError(t, err)
prequire.PartialEqual(t, &apiresourcekind.Cluster{
		Environment:  ptrutil.Ptr(dbsqlc.EnvironmentProduction),
		IsHA:         ptrutil.Ptr(req.IsHA),
		MajorVersion: req.PostgresVersionID.Int32,
		PlanID:       req.PlanID,
		RegionID:     req.RegionID,
		Storage:      req.Storage,
		TeamID:       req.TeamID,
}, resp)

An HN discussion on unlimited PTO.

We ran under unlimited PTO at Heroku the whole time I was there. At Stripe we started with unlimited PTO before moving to fixed. Stripe’s PTO allowance wasn’t generous, but it worked out okay in the end as I’d had some built up when I left which they ended up having to pay out.

A lot of commenters are so cynical that they view unlimited PTO policies as ill-intentioned, designed to avoid a PTO payout like mine. Maybe that’s the case for some companies, but in the ones I worked for, it was good-intentioned with suboptimal outcomes.

You might intuit that the problem with unlimited PTO would be that it’d get abused, and I’m sure that’s happened, but in my experience, the types of people who’d abuse such a system don’t tend to be hired at the types of companies that’d offer it. I’m entirely sure that unlimited PTO worked against me the entire time I was under it, and the same was true for most of my colleagues. It wasn’t entirely the fault of the policy. To succeed under such a system, you really need to be looking out for number one it, tracking your own vacation target carefully, and championing for yourself.

But of course it’s difficult, especially when you see your colleagues working so hard, which makes you feel bad about taking time off. And unless you’ve got a vocally pro-vacation manager, which is unlikely, you always have to wonder what they think about someone taking more than average.

A lot of companies have moved back in the direction of fixed PTO, which is probably for the best. Years ago Travis CI implemented a minimum vacation policy which is an idea I liked, but things didn’t turn out so well for that company, and I haven’t heard of much about minimum PTO since.

Go 1.20 is released. From the notes:

Work specifically targeting compilation times led to build improvements by up to 10%. This brings build speeds back in line with Go 1.17.

Excellent. New features are nice, but it’s the boring fundamentals like build times that really make a difference for everyday users. The addition of generics in Go 1.18 had an impact on build times, but those losses have been recouped. Generics are good. Fast builds are good. We can have both.

HBO’s The Last of Us televisation is our latest cultural zeitgeist, and I’m watching it like everybody else. Last night’s episode (ep 3) was the best so far, although it’s notable what they changed to appeal to a broader audience.

— SPOILERS —

Without going into the specifics, sufficed to say that Bill and Frank’s life in the game version is more antagonistic and stark. You can spoil yourself here, but the last piece in Bill’s ark is grim, and it’s not because he dies or is injured. He’s also more abrasive, and exchanges some great one-liners with Ellie as they insult each other. The TV version portrays a far more idyllic life and very gentle redemption ark. It plays on your emotions well and I liked it better overall, but it does some to detract from driving home the brutal edge of The Last of Us’s world.

By the way, I love how they kept the game’s military-style angle-head flashlights looped onto their backpack straps (see Ellie’s left shoulder in the image below). You can actually buy these things.

See: PEP 703 – Making the Global Interpreter Lock Optional in CPython.

Fascinating stuff, and the detail and attention that goes into these PEPs is just incredible. The GIL (global interpreter lock) has been the age-old Achilles Heels of Python and Ruby, and given the advanced age of these languages, I’d assumed they were problems that weren’t going to be solved as an increasingly intricate implementation makes more fundamental changes harder to tackle every year. But an increasingly parallel world of computing seems to have created demand to attract people with the necessary gumption to pull this off.

Ruby gave up on the GIL in favor of Ractors which suggest the use of many parallel environments, each with their own GIL. They might’ve worked, but were extremely backward incompatible, and two years later, nothing supports them.

Python’s optional GIL will be opt-in via compiler flag, but provides a path for the ecosystem to migrate incrementally so it could be default in a not-so-distant future.

A couple things that stood out to me. The implementation will move to a stop-the-world GC to freeze necssary state:

When running without the GIL, the implementation needs a way to ensure that reference counts remain stable during cycle detection. Threads running Python code must be paused to ensure that references and reference counts remain stable. Once the cycles are identified, other threads are resumed.

And because the GC becomes stop-the-world, its generational GC is disabled:

The existing Python garbage collector uses three generations. When compiling without the GIL, the garbage collector will only use a single generation (i.e., non-generational). The primary reason for this change is to reduce the impact of the stop-the-world pauses in multithreaded applications.

Single-threaded performance takes a 10% hit, but there’s already suggestions for changes to recoup some of that. Over time most of that likely gets optimized away.

Published sequence 039, on the site of the bird site.

Two years ago in the depths of San Francisco Covid-mania, I wrote up some predictions on what would happen to the city over the medium term. Today I published Revisiting my two-year SF predictions, and by my count, got 7 out of 9 right.

Some of these seem obvious in retrospect, but I wrote them down because they were starkly contrarian compared to claims made by the city’s commentariat. A popular opinion at the time was that tech needed San Francisco, so the longest and hardest lockdown in the nation would be perfectly okay as a closed downtown would spring right back to life on a word from London Breed, like a dog waking on command. Policy apologists guffawed with dramatic bemusement at the very idea that any of these departures could be permanent. Today, we see that almost every one of them was.

Network effects in cities are powerful, and that indeed has been the story of SF and the Bay Area writ large over the decade as companies swooped in soak up innovative energy, capital, and talent, which further compounded and cascaded. My prediction today is the city will continue to observe a reverse network effect as more companies give up expensive leases which have continually lessening ROI.

Published sequence 038. You look nice today.

Posted today: Stripe Sets One-Year Timetable to Decide on Going Public. Along with this classic Stripe press “leak”, the company also sent a simultaneously email to alumni, which honestly, was nice of them.

Predictably, there was a lot of carthasis from ex-employees, with even more chatter than during the 14% layoff a few months ago, and a hundredfold above ambient Slack levels. With the first batch of RSUs set to expire early 2024, there’d been a lot of consternation over the last couple years to say the least, so it was a big event.

It’s unambiguously a good thing, but I couldn’t help but notice that even with this BEST NEWS EVER message, the company is still tacking into its usual non-commitalism. Instead of unambiguously planning an IPO, it’s “either an IPO or private market transaction”, leaving huge error bars and uncertainty. Maybe something cynical, or maybe just unavoidable given the unpredictable market conditions of 2023. Hopefully, elite 4-D chess in pursuit of profitable ends that lowly grunts like myself aren’t privvy to, and couldn’t possibly understand.

Is this a food picture blog now? I hope not. I’ll try not to make a habit of it.

My curiosity was piqued when I found out there’s a Mensho subsidary in the ground floor of the Twitter building. Mensho’s a ramen company out of Japan that opened a shop called Mensho Tokyo a few years ago in San Francisco, notable for having the best ramen on this continent, and an omnipresent 50+ person lineup out the door and down the street, something I thought would clear with time, but never did. In 2021, they expanded into Twitter’s ground floor with a new installment – Jikasei Mensho.

Unlike the original, Jikasei Mensho’s more of a fast lunch stop sort of deal, and as far as I can tell, not great? You walk up to a counter, order from a computer McDonald’s-style, and your minimum tip option is 15% on a $22 bowl of fast food. It arrives in a plastic bowl. The chashu was good, the eggs were good, but the noodles and broth subpar, and some of furnishings questionable, like sliced lemons. Maybe I happened across it on an off day, or maybe ramen tastes worse out of plastic.

Post-Elon return-to-office Twitter is a bit more lively. There were quite a few people down in the cafeteria area where Jikasei Mensho is, where only a few months ago it all sat empty.

Published sequence 037 on the stairs at Oyster Point.

Witnessed a laptop snatching this morning. Woman is sitting next to window in a crowded cafe. Thief comes crashing in, grabs woman’s computer, and bolts. Five people run out after him, but as with a lot of criminal activity in San Francisco these days, it was organized, and a getaway car was waiting at the corner. Thief slams car door shut and rides off into the sunset (not literally; this was 11 AM).

It sounds blasé described post-hoc, but even “just” a theft with nobody hurt (luckily, it could easily have gone the other way) is hectic in real time. Downshifted mentally at the time in computing mode, it took a good ten seconds to process what’d even happened. People yelling. Tables knocked over. Coffee cups in pieces on the floor.

Whenever something like this comes up, San Francisco apologists are quick to make platitudinal statements like “report it to the police”, as if this is some kind of deep insight that only an enlightened California pro(re)gressive could have imagined. The situation was a perfect microcosm for why most people have stopped bothering. The SFPD was called immediately, and I waited around an hour without a single officer showing up, despite being within six blocks of a station, a cruiser rolling by every few minutes, and a situation that easily could’ve ended with someone seriously hurt. The crime had dozens of witnesses and clear HD footage from two separate cameras, but even if the PD did eventually appear, no one will be caught, let alone see a day in prison.

I started chatting with the guy next to me, who having just moved to the city a week previous was somewhat surprised. (But not that surprised, having gotten an accurate taste of the city’s culture from various viral videos.)

I told him that this wasn’t unknown, but not common. San Francisco, intent to preserve its reputation as deteriorating municipal hell stemming from the worst mismanagement of wealth in a thousand years, made sure to prove me wrong. Five minutes later, a woman walks in, grabs a handful of bills from the tip jar, and before anyone can react, leisurely strolls back out.

Published sequence 036 on 510 Townsend St.

I finally finished God of War Ragnarök this weekend. It’s a great game, with creative interpretations of Thor and Odin that turned out very well. Thor’s a lumbering giant who’s highly able, even if overly loyal to his father. Odin’s got mob boss vibes, which is unconventional, but a gamble that paid off. The game is huge, and I was surprised multiple times thinking I’d gotten to the end only to find a whole new world to explore.

I 100%‘ed it unlike the 2018 God of War, which meant that I wasted a ton of time following YouTube guides to find artifacts and ravens. I downgraded the difficulty to “give me grace” for the last two optional bosses and don’t feel bad about it. They spam attacks like no tomorrow, have a criminal number of unblockables, and the target lock system kind of sucks, especially in multi-enemy fights.

It’s notable that although very good, for all intents and purposes it’s the same game as the one released in 2018. There’s new story and new areas (although with some reuse), but the game engine, game mechanics, upgrades/skills system, and combat are all practically identical. Development for 2018’s started in 2014, which means that back then it took five years to build a full game and whole new engine, wherein Ragnarok was six for a game plus some minor updates. I’m sure some of that was lockdown delay, but I kind of suspect the game industry is hitting a plateau for large projects in the same way NASA did for space missions, Lockheed Martin for fighter jets, or Oracle for databases. Horizon Forbidden West (another triple-A title) was the same – great game, but practically indistinguishable from the original.

I’ve been writing Go professionally for something like a year and a half now, and compared to my previous daily driver Ruby, almost everything is better. Readability, speed of runtime, speed of tests, speed of refactoring, IDE insight, tooling – I could go on all day.

But when building out a larger-scale app with a non-trivial amount of domain logic and objects, it’s got holes. Some are as wide as Jupiter, and I can’t believe how little I see about them online.

The biggest I’m looking for an answer to is data loading, or more specifically, how to do data loading without thousands of lines of extraneous boilerplate. We solved our SQL-in-Go problem by moving to sqlc, but that in itself isn’t enough. I still write code like this daily:

team, err := queries.TeamGetByID(ctx, uuid.UUID(req.TeamID))
if err != nil {
    return nil, xerrors.Errorf("error getting team: %w", err)
}

if team.OrganizationID.Valid {
    org, err := queries.OrganizationGetByID(ctx, team.OrganizationID.UUID)
    if err != nil {
        return nil, xerrors.Errorf("error getting organization: %w", err)
    }

    if org.MarketplaceID.Valid {
        marketplace, err := queries.MarketplaceGetByID(ctx, org.MarketplaceID.UUID)
        if err != nil {
            return nil, xerrors.Errorf("error getting marketplace: %w", err)
        }

        return nil, apierror.NewBadRequestErrorf(ctx,
            errMessageTeamDeleteMarketplace, marketplace.DisplayName)
    }
}

In Ruby (or any language with some dynamicism and a good ORM), that whole block compacts comfortably to a single line of code:

raise ... if Team[req.team_id].organization.marketplace

And the Go version is only as short of it is because our foreign keys mean that we can skip handling some types of user-facing errors. i.e. We don’t have to worry about translating a missing organization to a 404 because foreign keys guarantee its existence. Sqlc also saves hundreds of lines – before the Go code for every one of those queries like TeamGetByID had to be written by a human and tested. Now we write the SQL and let sqlc do the work, but using Go built-ins database/sql you still get to do all of it.

Another bad-but-unavoidable Go pattern is preloading objects to avoid N + 1 queries, but then having to manually map them into structures that your code can actually use:

teams, err := queries.TeamGetByIDMany(ctx, sliceutil.Map(unsentInvoices,
    func(i dbsqlc.Invoice) uuid.UUID { return i.TeamID }))
if err != nil {
    return xerrors.Errorf("error getting teams: %w", err)
}

teamsMap := sliceutil.KeyBy(teams,
    func(t dbsqlc.Team) uuid.UUID { return t.ID })

Again, with an ORM this is:

Invoice.load(..., eager: [:team]).each { |i| i.team.name }

And the Go code was >2x longer just one release of Go ago. Notice the functions sliceutil.Map and sliceutil.KeyBy which were impossible without generics. Before, these were an initialization and a for loop – 4-5 lines each.

I’ve experimented with custom data loading frameworks that sit a layer above sqlc to reduce boilerplate:

err = dbload.New(tx).
    Add(dbload.Loader(&loadBundle.Cluster, req.ClusterID)).
    Add(dbload.LoaderCustomID(&loadBundle.Provider,
       req.ProviderID)).
    Add(dbload.LoaderCustomID(&loadBundle.Plan,
        dbsqlc.ProviderAndPlan(req.ProviderID, req.PlanID))).
    Add(dbload.LoaderCustomID(&loadBundle.Region,
        dbsqlc.ProviderAndRegion(req.ProviderID, req.RegionID))).

    // must be loaded after cluster
    Add(dbload.LoaderFunc(&loadBundle.PostgresVersion, func() *uuid.UUID {
        return &loadBundle.Cluster.PostgresVersionID
    })).
    Add(dbload.LoaderFunc(&loadBundle.Team, func() *uuid.UUID {
        return &loadBundle.Cluster.TeamID
    })).
    Load(ctx)
if err != nil {
    return nil, err
}

But so far it’s nowhere near enough – this one makes point loading by IDs a lot more succinct, but doesn’t handle anything less trivial like associated objects or one-to-many relationships.

I’m fairly sure that there’s nothing approaching an answer to this problem in the Go ecosystem. But aren’t there millions of lines of production Go out there by now? How aren’t more people running into this?

An article from 37 Signals on cloud spend.

They’re concerned enough about their AWS spend that they’re moving to their own hardware, a path paved before them by the likes of Dropbox and GitHub. It’s also always interesting when companies are transparent about their cloud spend. Hey (their email service) for example, costs $89k/month or $1.1M/year in AWS, with the biggest component being RDS, which eats a quarter of that.

The AWS bill is a creeping concern that tends to trend slowly upward over time without anyone really noticing, then suddenly jumps out of a bush to knife you in jugular. It in the beginning it’s “wow, look at all this stuff we’re getting for a few bucks a month”. Then a few years later it’s “what?! how many f* millions?? holy shit!!” It’s amazing how much money can be spent at $0.023 a gigabyte. At Heroku and Stripe we were eventually forced to engage in major AWS cost reduction projects, and in both cases the engineers involved ended up paying their own salaries many times over.

[3 months since day zero. The day Elon Musk bought Twitter, and the world ended.]

Know anybody who loudly quit “the bird site” in anger to go to Mastodon, and has stuck with it?

Me neither.

An essay: There’s no planet B, making the case that making another planet like Mars fit humanity is going to be so impossibly hard that it’s not going to happen, and by extension that the bulk of our energies should be focused on planet A, the one we’re already living on.

I’m reminded of Christopher Nolan’s Interstellar, in which he makes an effort to keep the movie relatively grounded, famously bringing on physicists as consultants, and even using contemporary shuttles and rockets in some parts. But even this “grounded” movie necessitates the sudden appearance of a nearby wormhole, and the invention of anti-gravity technology for the plot to work.

As cool as SpaceX is, it’s not even one percent of one percent of what humanity needs to colonize another planet, and appears to be right up against the envelope in terms of scale we’re able to tackle. Planet B isn’t happening, but its hope is the policy equivalent of deus ex machina, letting us rationalize current direction under the premise of a nebulous future savior.

Twitter’s holding an auction for assorted office furnishings that closes tomorrow at 10 AM Pacific.

Mildly entertaining: many items from Ohio Designs, a boutique San Francisco manufacturer, and the same one that we used to buy from at old Heroku to add to our office’s raw “loft” look. Disgustingly overpriced, but some of the most solid furniture ever created, with heavy steel frames that I wouldn’t bet against surviving a nuclear blast. And by extension, also impossible to move/lift. Their HQ are next door to Southern Pacific Brewing Company on Treat. I know that because at one point I became obsessed with these things and went over to inquire buying one for home, before settling on an alternative at CB2 with similar build quality at a quarter of the price.

A Twitter bird statue and Twitter neon light are sitting at $18k and $20k respectively, 10x their worth, and probably < 1x what Twitter paid for them. A far shot from $44B, but a tiny consolation prize, and notable for their part in the symbolism of selling off the worst excesses of the old era of tech.

In the off chance this helps someone else: I started building my own ImageMagick after the project changed their static binary to an AppImage, which doesn’t work very well.

As with all programs from that age, every feature is configurable, and it’s non-trivial to get it compiled with support for more unusual formats like HEIC. I thought I’d succeeded, but have for months been using a build that couldn’t read HEIC, and it’d only been working by virtue of me not uploading a HEIC in a while.

My ./configure invocation looked right:

./configure --with-heic=yes --with-webp=yes ...

But it’d been silently failing:

Delegate Library Configuration:
  ...
  Ghostscript lib   --with-gslib=no	 no
  Graphviz          --with-gvc=yes	 no
  HEIC              --with-heic=yes  no
  ...

Yes to HEIC, but actually … no.

There’s GitHub issues around with the same problem. It can have a variety of causes, but probably means that configure couldn’t load one of the dependencies for HEIC properly (libde265 and libheif).

ImageMagick’s build process silently swallows problems, so it’s necessary to go to configure.log to find out what happened. Digging into mine, I found that I was missing another dependency called dav1d, which turns out to be an AV1 decoder from the VideoLAN project.

configure:31871: $PKG_CONFIG --exists --print-errors "libheif >= 1.4.0"
Package dav1d was not found in the pkg-config search path.
Perhaps you should add the directory containing `dav1d.pc'
to the PKG_CONFIG_PATH environment variable

I fixed the problem with:

apt-get install libdav1d-dev

The FAA’s recent trouble with NOTAM caught my eye for a couple reasons:

• It happened 12 hours after my flight out of Calgary. I hadn’t been too happy about our 4 hour delay, but it goes to show that things can always be worse.

• FAA traced the outage to a damaged database file. Mongo, is that you?

I’m still holding out hope for a full Silicon Valley-style outage postmorterm, but from what we can glean, it sounds less exciting than you might hope. “Database” can mean a lot of things, and from the sound of it, this is more like a flat file.

A finding from yesterday was that the file had been corrupted by manual manipulation by a pair of contractors. There were supposed to be safeguards to prevent that from happening, but they didn’t. We also found out that in line with long tradition in operations, there had been backups, but when brought online, the backups were also found to be corrupted. Absolutely classic.

I watched Black Adam (2022). I don’t know why – mild curiosity? Masochism? Latent Dwayne Johnson fandom?

Let’s start with character vitals:

• Strengths: Invulnerable, power of flight, super strength, super speed, teleportation, unassailable combat skills, shoots lightning bolts, literally a demigod.
• Weaknesses: 5,000 years old, so occasionally makes endearing anachronistic comments that are lightly funny but not really.

It’s never clear what exactly the conflict is or why the viewer is supposed to care. As Black Adam wakes, a group of superheroes is sent to confront him, but it’s never in question that they’re on the same side, and they only fight it out to prove the narcissism of small differences. Later there’s a token bad guy, and as you might be able to imagine, it’s a huge mystery as to whether the combined might of Black Adam and his new friends at the Justice Society will be enough to take him down.

There was a long period of MCU golden age over the last ten years where I thought that Hollywood had figured it out – the perfect formula of fan service, special effects, and quippy dialog that was reliably reproducible to make a film that fans liked and would land a solid ~8 on IMDB. Apparently not, or the at least the DC studios never got the memo.

Don’t worry, this hasn’t become a movie blog. Tomorrow, back to your regularly scheduled programming.

A light movie recommendation: Triangle of Sadness (2022).

I didn’t know much of anything about it going in beyond “a couple, both young models, go on a cruise with wealthy elites”, which is the exactly the right amount to know about it. The movie takes some major turns throughout, and it’s very enjoyable if you don’t know they’re coming at all (I didn’t, but have since read some synopses that would’ve spoiled them). It’s a bit artsy and a black comedy, so it won’t be everyone’s cup of tea, but it punches well above its IMDB 7.5 and distantly above its 63% RT (can anyone trust film critics anymore?).

This one’s a few years old, but a nice essay on Where to live, a question I’m struggling a lot with right now (and for the last five years). A stranger will never be able to answer something so important for you, but while searching for insight, cast your net wide.

Anyone remotely adjacent to the tech industry will know about the Sam Bankman-Fried saga by now: the collapse of FTX as the biggest Ponzi scheme of all time, subsequent world podcasting tour by Sam to broadcast his innocence, arrest in the Bahamas, extradition to the US, release on $250M bail, and return to his parents’ home in Stanford, back on Twitter and League of Legends until October.

Consuming pre-exposure material on Sam qualifies as its own genre of entertaining dark comedy. The broad themes are always the same – hosts laud his genius, ask no real questions, and make dozens of grand assertions with absolute confidence.

Highlights of Acquired’s FTX episode:

• In the hosts’ own words, FTX’s 10x growth in valuation for 2021 was quote “unbelievable”. Literally correct.

• Sam is not shy about congratulating his own brilliance, but the hosts do most of the work for him, unprompted. He’s compared to Jeff Bezos 3-4 times, and not in succession. The comparison is resurrected again and again to drive it home.

• A hearty laugh is shared as Sam jokes “all you have to do to beat Mt. Gox is not lose everyone’s money”. So on the nose.

• He was thinking about leaving his suit at his brother’s in DC because except for wearing it for congress, he didn’t know when he’d use it again. Good thing he didn’t, having since gotten good mileage out of it in the Bahamas and NYC.

The greatest moment is when Sam is bestowed the appellation “the zeitgeist artist of our time”, meaning that he understands today’s confluence of culture and technology better than anyone else on Earth.

See also Sam on Unusual Whales, post-exposure. He’s typically evasive, but the last 13 minutes (after Sam leaves) is frustrated editorial from panelists, and excellent.

This is simply excellent: Production Twitter on One Machine? 100Gbps NICs and NVMe are fast.

The article isn’t suggesting that Twitter actually do this, but explores how possible it’d be to run Twitter on one really big server. It shows its work, and contains dozens of Fermi estimates based on best available public data:

Now we can use this to compute some sizes for both historical storage and a hot set using fixed-size data structures in a cache:

tweet avg size = tweet content avg size + metadata size => 176 byte
tweet storage rate = avg tweet rate * tweet avg size in GB/day => 88 GB/day
tweet storage rate * 1 year in TB => 32.1413 TB
tweet content fixed size = 284 byte
tweet cache rate
    = (tweet fixed size + metadata size) * max sustained rate in GB/day 
    => 251.7647 GB/day

Doesn’t every programmer secretly love the idea of a mainframe? One giant machine that runs everything and has its own redundancy internally. Ensuring scalability through processes designed to be run in parallel is obviously more practical and more robust nowadays, but if you were to try and run Twitter on one machine, you might be able to get results that aren’t too much worse, and with 1000x less infrastructure.

It wasn’t too long ago we were really trying to do this. At iStock circa 2011 where the ops team was running the asylum, right around the end of my tenure we purchased a huge mainframe-esque box that advertised being able to stay online even if one of its CPUs failed. I was never sworn in on how much it cost, but undoubtedly the price tag would’ve made my eyes bleed. That was right around the period where misguided attempts to scale vertically and racking your own specialized hardware was already starting to look pretty silly, so I’m curious in retrospect how far it made it into production.

Published sequence 035.

This turned out to be a bit of a yakshake as I found out that despite a lot of work adding support to this site for various types of image formats over the years, it still wasn’t well set up for .heics that come out of an iPhone. ImageMagick resizes them, but browser support sits comfortably at 0% so it’s not a format you want to use on the web. I added some code to the build process convert them to WebP (96% support), but that in turn required sizable refactoring to support outputs with an extension different than their input.

I should investigate using WebP as a general target regardless of the input format since it overall seems like a better default that produces better results by default compared to mucking around with JPGs and PNGs. But it’s not super pressing because although WebPs will produce considerably better compression than most JPG exports, they’re not too much better than a JPG that’s been optimized via MozJPEG, which this site does.

Published Content authored by ChatGPT front pages, wherein we see empirical proof that an article generated by ChatGPT is good enough to get all the way to the front page of Hacker News before anyone notices.

A guilty pleasure is reading other peoples’ multi-thousand word reviews on Leica cameras. From today: Fujifilm to Leica which goes into detail in rationalizing buying an M11.

I hate that I love these cameras. At Heroku we had a few vocal Leica users. One went as far as to sell all his equipment to simplify his life on a single Leica rangefinder and lens (an M9). But like the Leica brand itself, the move wasn’t about practicality or technical superiority, it was about romance. Leica doesn’t sell cameras, it sells lifestyles, and fans do 80% of the work for them. See also sh*t Leica photographers say.

Years later I’d pick up a Q, an amazing camera. The best that Leica’s ever produced. But once you’ve dipped your toes in, you’re forever haunted by the siren call of the M-series.

M cameras are a rip off ($9k for an M11 body + $6k for a lens). Everybody knows they’re a rip off. No honest person even tries to argue they’re not a rip off. But they’re well-designed, and again, supremely romantic.

I check in every so often, and luckily, have held the line. Besides the price tag:

• Rangefinders were always questionable, but EVFs made them obsolete. This might be debatable, except we have proof that Leica knows it too. The camera doesn’t come with an EVF, but Leica won’t hesitate to sell you one. For $750.

• $15k. Not weather sealed.

• Questionable software. Slow start up time.

On the plus side, the M11 did away with the bottom plate, a feature that’d been left in until the M10 for historical reasons, but completely impractical in every way. You can’t beat full frame in a body that size, or the minimalist controls.

Leica must send $100M/yr Fuji’s way to keep cameras on APS-C and with at least six unnecessary dials each. It’s the only explanation for such magnanimity in staying out of the competition.

Marc Andreessen:

The most common event in our times: An expert with power/authority/influence is wrong in full view, yet suffers no consequences.

Contrary to popular belief, it doesn’t rain that much in San Francisco. A few dozen times a year it does. Very occasionally, it rains an unusual amount. Today it rained 0.83”, which is kind of a lot, I guess.

Local media and Twitterati (now newly christened armchair meteorologists), not ones to let a kinda-maybe-almost-but-not-really catastrophe go to waste, dramatically dubbed this THE BOMB CYCLONE. In a flashback to her greatest hits of 2020, London Breed heroically ordered residents to stay home.

By comparison, during an actual adverse weather event like hurricane Ian in Florida last year, the state saw rainfalls of 10 to 20” over a four-day period, ~3 to 6x daily what SF saw today. Tokyo’s rainiest month of October averages 9.24” over ~12 days of rain. In other words, the poor Japanese are dealing with twelve THE BOMB CYCLONES just in the month of October, every year.

See also “our new era is characterised by an unending self-reinforcing cyclone of hyperventilation journalism, the likes of which we’ve never seen before. […] we will careen from one crisis to the next.”

Gap puts their corporate HQ in San Francisco up for lease or sale, notable due to the size of the lease (162k sqft), but also because it’s historically one of the city’s preeminent companies, founded, headquartered, and proudly operating out of San Francisco.

Only months ago San Francisco’s pundit class decried “misinformation” that Gap had shuttered operations in SF, despite having clearly closed their flagship store on Powell (which remains an iconic SF empty storefront to this day) and corporate office for Old Navy. Gap, they said, is still very much in the city, you stupid conspiracy theorists. But as the facts become indisputable, the strategy shifts from attack to denial – none of this is happening. Downvote, misdirect, ignore.

Gap’s departure is the latest blow in a long series of insults to San Francisco’s bottom line. Office vacancy is at a 30-year high, large real estate holdings are requesting reassessments to reflect decreased value, and even other SF darlings continue to flee (Salesforce announced a 10% layoff round this morning along with further office space reduction).

The deficit, currently a mere $728M, grows.

I’m trying to read James Joyce’s masterpiece Ulysses, again, fast becoming an annual tradition in humiliation and failure. I’ve tackled my fair share of literature – Dostoevsky, Dickens, Kafka, Vonnegut, Hemingway – and for my money, Joyce and Ulysses are on a level of their own, with allusions, puns, and classical references so rich they’ll leave you reeling, and wordplay so clever that it’s a wonder a human could write it.

But I’ve never in my life come across a book that’s so thoroughly and so totally unreadable. The prose is heavier than a pallet of bricks. A few years ago I tried to cheat my way to success by getting it in audiobook form, only to be foiled by the narrator’s dramatic Dubliner accent, which while appropriate, made things even worse. I gave up after less than an hour.

Back to written form, and 10% through. I’ll make it, some day.

English language Stack Exchange bans ChatGPT-generated answers, taking a step beyond what the network as a whole has committed to.

For years the concerns from public intellectuals (few of whom are AI or computer specialists) over AGI have felt overblown to me, but ChatGPT’s got me officially worried. Not that AGI’s going to take over the world and enslave us all, but rather that we’re about to smash headlong into a creative wall in communication and writing as ChatGPT becomes the perfect remixer, able to endlessly recycle new permutations of the existing bulk of human output in a way that’s convincing enough for us to accept. Never able to produce anything interesting or new, but with few people who notice since we spend our days consuming human-regurgitated remixes anyway.

I’m expecting more announcements like Stack Exchange’s over the coming months. Discerning humans can probably distinguish ChatGPT output for their chosen field, for now, but it’s certain to make problems in everyday life. For example, search engine spammers taking their game to a new level with AI-produced filler far more convincing than the Markhov Chain or plagiarism techniques they use today. Firms using it for frontline support to lead customers in endless rhetorical circles even more frustrating than the precanned answers and telephone mazes of today. Phishing seniors via email with previously unimaginable sophistication.

You can’t put this genie back in the bottle. If we don’t make it, China will. Regulation’s a tempting idea, but lawmakers, barely able to sound even partially articulate in hearings over comparatively simple issues like social media, are wholly unequipped to tackle such a complex subject. The only practical answer is supremely unsatisfying: wait, and see.

Bloomberg reports how Shopify is canceling recurring meetings with more than two people to kick off 2023.

Tobi’s taking a lot of flak over it from middle managers on HN (my favorite from the high drama category: “Tobi has done this a couple times […] it comes from a place of narcissism”). I think it’s a great idea. In the early 2010s at Heroku I’d started to think that at least as far as tech was concerned, meetings were a solved problem. It was an engineering-driven culture, and most engineers hated them, so we’d schedule as few as possible, with only a few key ones per week. But then I got to Stripe and realized that I’d been living in a bubble – an uncomfortable truth of the universe is that most people love meetings, even if they think they don’t, and doubly so for well-pedigreed Stanford types most likely to work at top Silicon Valley firms. Far from being an element of change, tech was only the latest continuation of a long tradition in big American business. Every satirical pane from 90s Dilbert cartoons held perfectly true.

Based on testimony from Shopify employees, Bloomberg gets a few points wrong:

• Meeting cancellations aren’t permanent. The move wipes the slate clean, but important meetings can be put back on the docket.

• It’s not the first time Shopify’s done it. One person says that these events were originally aimed to occur every rand(600) + 300 days.

Targeting recurring meetings specifically is the right move. These get slapped onto calendars, often for large groups, and their tendency is to stay after their inertia’s built. Before long no one can even remember a time without them, and it’d be heresy to challenge their existence. Ending recurring meetings by default shifts the burden of proof to the right place – it’s now on middle management to re-justify their existence rather than falling to a maverick engineer who has to burn social capital in a risky attempt to argue for their discontinuation.

Occasionally, especially when thinking about past around the holiday season, I’ll use the Way Back Machine to take a look at sites which I used to consider the absolute pinnacle of great web design back in the day. Today’s specimen is A List Apart, a site that writes about web design. Screenshot of a 2009 capture below.

Some things that come to mind:

• We all loved small fonts back then. The body text here is ~8pt Verdana which is a style I used all over the place back in the 2000s. I’d hazard a guess that the reason a lot of us liked them is that larger fonts didn’t look anywhere near as good as they do now, before the advent of subpixel anti-aliasing and retina displays.

• A key element that makes the web today look as good as it does are fonts, which render so beautifully nowadays that it makes the web’s default look quite good, even where minimal design has been applied. Old sites should benefit from these advancements, but it never seems to translate through. Their fonts don’t look great and it makes the whole site not look great.

• The broad strokes of this old layout are excellent, and better than alistapart.com’s current site. The specifics are a bit dated and it needs niceties like responsiveness, but a little modernization would easily bring it up to date. The current era of web design is too quick to remove any and every extraneous element, and although the baseline product is decent, 2022’s internet looks uniform and somewhat bland.

• Notice all the text in images? (e.g. in the logo, “An event apart”, “A book apart”, etc.) The reason people did that is that it looked better than what the web browser could render at the time. Programs like Photoshop would yield additional anti-aliasing beyond the browser’s discrete, blocky pixels. It’s ironic that transported through time, text that was put in images to look better now looks much worse.

When all you have is a hammer (access to free Heroku apps), everything looks like a nail (dynamic apps + buildpacks for everything). On the eve of November 28’s Herokupocalypse, I pulled the static rip cord and dropped the dynamic component from apps that should’ve been static HTML pages from the get go. The totality of tooling required: Chrome and a text editor.

See also Nanoglyph 022: Time and Entropy.

Happy new year!

I always love these first few days of a new year. A fresh slate, after a week off, batteries recharged, and optimism for the year ahead. I’ve spent the last few days doing little else beyond writing, some casual coding, and running. It’s been great. Despite some of the most uncertain times in which I’ve ever lived, I’m feeling a lot better about 2023 than 2022.

Wrote a few notes on publishing iOS live photos, involving exporting a .mov from iOS and running it through ffmpeg to fix aspect ratio, scale, strip audio, and convert to web-friendly codecs. Taking live photos is a nice alternative to shooting in video mode because you can have them captured by default, so there’s a lot less mucking around with start/stop.

A major takeaway: you can make a <video> behave like a classic animated GIF by having it autoplay and loop, but only by also including the muted option, which browsers require before allowing autoplay to prevent us all from losing our minds.

<video autoplay loop muted playsinline>
    <source src="/videos/girm632/cougar-1.mp4" type="video/mp4">
    <source src="/videos/girm632/cougar-1.webm" type="video/webm">
</video>

playsinline is another magic Apple token that tells an iPhone that it’s okay to autoplay a video.

Eugyppius has a way with words:

The pre-pandemic world is gone forever. If the past year has taught us nothing else, it has taught us that much. Mass containment has permanently transformed our societies and our cultures. It has cemented the cooperative relationship between the regime and the press, and it has changed the content and the tenor of our media. Drama and panic have always sold newspapers, but our new era is characterised by an unending self-reinforcing cyclone of hyperventilation journalism, the likes of which we’ve never seen before. For the foreseeable future, I think, we will careen from one crisis to the next.

Truer words have rarely been spoken. The whole edition is well worth reading.

The Calgary Zoo yesterday, somewhere I haven’t been in years. The grounds feel smaller now, but the animals are impressive, and we got lucky with many out in the open. Amongst others:

• Extraordinary penguin exhibit – lots of them, very active, and multiple locations indoors and out. (I wish I’d gotten some photos, but was holding coffee at the time.)
• Cougars jumping, climbing, and very lively. (The polar opposite of the snow leopards.)
• A gorilla couple with a young baby that attached itself to the mother’s arm as she roamed.
• Two Komodo dragons, with one hanging out right in the open a few feet from the glass.
• A pack of wolves circling energetically within inches of the observation perch.
• A troupe of Japanese macaques, which I haven’t seen since visiting Japan. (Too bad there wasn’t an onsen.)
• An enormous Malayan tapir.
• The biggest Bactrian (two hump) camel any of us had ever seen. What a beast.

Although not as cold as last week (-30C), it was chilly out (-3C), and I’ve never been so thankful to for the butterfly conservatory’s hot, humid interior (currently minus the butterflies for the winter season) for recharging body temperature.

I played around with exporting some iOS live photos and running them through ffmpeg to make some VP9/.webm shorts. It worked reasonably well, despite the idiosyncrasies of the <video> tag.