On Friday, at 3:54 p.m. Eastern, the FAA halted every departure from US airports. About 3,500 planes were in the air at that moment. They all landed safely, but on the ground the stop lasted more than four hours: more than 4,000 flights canceled and over 10,000 delayed, according to FlightAware (CBS News).
The cause, according to several sources who spoke to CBS News, was an update. An update to the private communications network that links the air traffic control system: the network rebooted, then crashed, and the control centers lost communications.
The day before, half the world had been left without Outlook and Teams because of a failure in Microsoft’s servers. Two days, two huge infrastructures down because of software. And a few days ago, on a much smaller scale, it happened to me too.
What happened to US air traffic control
In order, Eastern time:
- shortly after 3:30 p.m.: radar and communications at the control centers start acting up (ABC News);
- 3:54 p.m.: the FAA orders a nationwide ground stop, meaning no plane can take off. It’s the second time since 9/11: the previous one, in January 2023, was caused by the failure of the system that sends notices to pilots;
- evening: delays in Boston, Atlanta, Miami, New York and Los Angeles, lines at the LaGuardia check-in desks and full parking lots in Houston;
- 8:15 p.m.: the FAA lifts the stop after checking the systems are stable. Flights restart slowly, and delays will go on for days.
The network involved is called FTI, the FAA Telecommunications Infrastructure, and according to CBS it is run by L3Harris, a large US defense contractor, which did not comment.
The official explanations point the same way. Transportation Secretary Sean Duffy said switches and routers were flooded with data, started giving inconsistent data and the system was shut down. Homeland Security Secretary Markwayne Mullin talked about new security patches and new equipment installed on an old system, which may have overloaded the routers. According to Bloomberg, the FBI doesn’t believe it was a cyberattack and is looking precisely at a software update (SecurityWeek).
What we still don’t know: which update, who made it and why nobody noticed beforehand that it could bring everything down. No official cause has been announced.
It wasn’t AI, and it wasn’t Microsoft
Two theories went around right away, and it’s worth clearing them up.
The first: a few days earlier the FAA had rolled out SMART, a new AI system for routing traffic, to all its centers. Duffy says it had nothing to do with it, and the company that makes it, AirSpace Intel, says the same. Mullin was more cautious: “we’re looking at that”.
The second: a link with Microsoft’s outage the day before. Mullin ruled it out.
That leaves Duffy’s words, which should be taken for what they are, politics included: the system is “incredibly old”, with telecommunications still partly analog, and “you will have more of these” unless Congress finds the money to replace it. The administration talks about roughly $50 billion. It’s not news: back in 2023 an FAA review called the FTI network obsolete and warned that a failure could cut off single centers or entire regions.
My server, a few days ago
I’m telling this because it’s the same story, in miniature.
I had to update a server before a launch. Nothing exotic: a procedure I knew, written down, tested. I followed it step by step and everything went as it should, until the reboot. On reboot the machine didn’t come back: the boot partition was broken. The launch slipped by several hours compared to the plan, the time it took to get the machine back on its feet.
I hadn’t done anything wrong, and that’s exactly the point. An update isn’t finished when the installer says “complete”: it’s finished when the system has restarted and works. In between there’s the reboot, which is the most fragile moment of any system, from my server to the US air traffic control network. That’s when you find out whether everything underneath, partitions, configurations, drivers, routers, still holds.
How much we depend on things we don’t control
Let’s line up the last few days. Email and chat for ministries, central banks and companies down because of a problem in the automated checks of Microsoft’s servers. Every plane in the United States stuck on the ground because of an update to a network run by an outside contractor. And in the weeks before, supermarket fridges tampered with remotely through old controllers left reachable from the internet.
Three different stories, but they have one thing in common: the people who paid the price had no control over what broke. The passenger in Boston doesn’t know what the FTI network is. The Japanese official doesn’t know which process got overloaded in Microsoft’s servers. The supermarket didn’t write its fridge’s firmware. And often not even whoever runs the service fully controls it: the FAA depends on L3Harris, and Microsoft’s customers depend on Microsoft.
Then there’s a paradox anyone who works on systems knows well. Updates can’t be avoided: security patches are needed, and a system that’s never updated is a system that will get breached sooner or later. But every update is also the moment when a working system can stop working. The older and more layered a system is, the riskier it is to touch it, and the riskier it is not to. The US air traffic control network sits right in the middle.
What I take home
I don’t have a solution, but a few rules I’m holding on to more tightly after this week:
- an update isn’t finished until the system has restarted. The time to plan for isn’t the install time, it’s the reboot plus the time to fix whatever doesn’t come back;
- never update right before a launch. Or, if you really have to, with a margin of hours and not minutes, as I learned the hard way;
- a plan to roll back, whether it’s a snapshot, a backup of the boot partition or a spare machine. The day you need it is not the day to invent it;
- know who you depend on. Cloud provider, network operator, outside services: you can’t control them, but you can know what happens if they stop and have an alternative for the things that matter.
Software is everywhere, and most of the time it works so well that we forget about it. Weeks like this one remind us that under every flight, every email and every fridge there’s someone who, at some point, pressed “restart”.
What about you? Has a routine update ever ruined your day? Let me know.
Sources: CBS News, ABC News, NPR, SecurityWeek, CNN.