brandur.org

The FAA’s recent trouble with NOTAM caught my eye for a couple reasons:

  • It happened 12 hours after my flight out of Calgary. I hadn’t been too happy about our 4 hour delay, but it goes to show that things can always be worse.

  • FAA traced the outage to a damaged database file. Mongo, is that you?

I’m still holding out hope for a full Silicon Valley-style outage postmorterm, but from what we can glean, it sounds less exciting than you might hope. “Database” can mean a lot of things, and from the sound of it, this is more like a flat file.

A finding from yesterday was that the file had been corrupted by manual manipulation by a pair of contractors. There were supposed to be safeguards to prevent that from happening, but they didn’t. We also found out that in line with long tradition in operations, there had been backups, but when brought online, the backups were also found to be corrupted. Absolutely classic.