OT: PSA Expect massive delays if you need to air travel this week

Long story short - FAA had to ground all flights nationwide starting at 6:30am this morning due to a computer systems outage of the notification system they use

The most recent update from a couple minutes ago is that some flights are resuming at Atlanta and Newark, not because systems are restored but in order to reduce congestion

And while this is good news, this overall event is not something the airline industry can easily handle. From crew staffing to needing planes to take certain flights to get planes at gates in order for future departures, any kind of ripple to the system here snowballs. And to cancel at least 3 to 4 hours of all flights for all airports and all airlines is going to be felt for a while, with the impact likely being felt for the remainder of the week, if not longer.

If you're due to fly anywhere this week, expect headaches, and may the odds be ever in your favor.

Forums: 
DISCLAIMER: Forum topics may not have been written or edited by The Key Play staff.

Comments

...hm... Hopefully it's cleared up by Friday. Wifey is flying out to Cali Friday afternoon lol...

Dear FAA,

please get all this out of your system before Monday when I am flying halfway around the globe. I've slept in an airport before and have no desire to repeat the experience.

Sincerely,
theMattBoard

I could be wrong because I haven't looked into it too much, but from what I've heard/read, it sounds like it really mostly affects domestic flights, so you might still be ok.

Given most international flights are just back and forth with the same airframe, I would thin that should be the case, as most if not all of those routes should avoid the snowball effect. Domestic flights are going to be a nightmare, however.

This is my school
This is home

Update: my domestic flight was fine. My New York to Amsterdam flight was delayed by almost an hour but thankfully, so was my Amsterdam to Bangalore flight so it worked out.

My hotel was overbooked because of a guest's medical emergency so they couldn't leave. I ended up at a different hotel for two nights. The manager not only apologized over the phone and took care of the arrangements, she came over in person to my hotel room to apologize and present me with a handwritten note and a bouquet of flowers.

The manager not only apologized over the phone and took care of the arrangements, she came over in person to my hotel room to apologize and present me with a handwritten note and a bouquet of flowers.

Don't see that very often. I'd bring my business back to them, if it happened to me.

Oh yeah, it will be my recommendation to the travel agent whenever we send people here in the future.

Bengaluru Sheraton Whitefield is hotel 1, Bengaluru Marriott Whitefield is hotel 2. (Both Marriott Bonvoy)

Bonus: The Marriott made my towels into an elephant. The service has been top notch, staff polite and helpful. Not sure how much is Indian service culture vs American service culture expectations, but I am getting seriously spoiled here.

Everything I've gone to Europe amd stayed in an American brand hotel they have been outstanding. The Best Western in Rome was really nice. The holiday ins are nice, The Hiltons and Marriotts are top notch. Why can't they be as nice here in the states.

Every time Europe tricks me in to thinking the US is so much better than it is, American Brands are luxury, you see a guy driving a Benz and I'm just like why are you driving a taxi and not a nice car like a Ford. Or look at them importing all their medications from the US, so cheap, America makes the best drugs. Look at this Big Mac, so cheap. Then I come home and wonder who thought putting a 6 person jacuzzi next to a King sized bed on the 2nd floor of a motel was a good idea? This passes for luxury?

What's funny is how mad people were at Southwest a couple of weeks ago. This could be significantly bigger.

Edit: Seems like takeoffs are getting back on track already.

Southwest's issues resulted for corporate negligence. They chose to go decade(s) without upgrading their logistics software to 'save money' and they got burned as a result. As someone who works in tech, what happened to southwest is something we're always thinking about.

Southwest's failure to deal with TechDebt will be an MBA case study for the next 30+ years.

Twitter me

As someone who also works in tech it's clear that in spite of the hammering they got in the media, the problem isn't limited to Southwest.

I had a friend who used to write code for American Airlines. Their reservation system was a mess. This is a common problem in operational environments. Yes, Southwest will be putting a lot bigger investment in software right now.

The early reports are that a file or two were corrupted in the FAA NOTAM system, and that brought the whole system down when they rebooted. So it sounds like they need to spend some infrastructure money on tech updates, too.

From what I am seeing they had a corrupted database file. It started to manifest itself Tuesday afternoon, so they went to look for the backup to restore. But the backup was corrupted too. They were able to recover something, and re-booted at 2am Wednesday, as that was the time of least impact. The reboot normally takes 90 minutes, but it didn't finish in the normal time. And kept running the reboot through 9:30am Wednesday when it finally came back up.

Either the corrupt database was corrupt for a long time before it impacted anything (maybe) or more likely, they aren't keeping backups long enough to go back before the corruption manifested itself (most likely). We go 42 days, but management has advocated for less more than once as a cost saving measure, which we have successfully thwarted.

I suspect they don't do many re-boots (90 minutes is an eternity - the hardware or OS might be ancient) and whatever they did to fix the corrupted file impacted the re-boot time.

Somebody was sweating bullets for a long time.

Nobody wants to deal with technical debt and that is what caused all of these problems. These large companies and government won't deal with technical debt until it slaps them in the face hard like it did in both of these cases. Technical debt IS a debt like financial debt. It WILL be paid sooner or later and the later it is the more it is going to cost.

This is true of all deferred maintenance.

Yet it exists in most places, as part of human nature.

An example of technical debt and a business making a horrible decision. We have a division using a piece of software that they absolutely love. Challenge is that the Vendor that published the software has been out of business for years. The architecture it runs on is no longer supported by the OS provider and the application itself has been out of support for half a decade.

Our CIO has been very clear with their VP that if it hits the fan, they will be back to paper and pencil to manage their business because the thing is being held together by baling wire and tape. And our ITSEC space has had to jump through hoops to keep this dog secure and isolated from everything else. Yet that VP refuses to consider another option. Even added scope to what is managed by the application in the last year.

I secretly hope the whole thing blows up, because if its catastrophic my people won't be able to do anything, and it will be great to watch that VP whine and squirm when it comes to light - they have been warned over and over this could happen.

The kicker is, when shit like that goes down, while you want it to all fall on the VP, and in the end it very well could, but fixing it will be a nightmare of a time for all of the lower level employees, including round the clock triaging and mandatory overtime. And, when it costs the company money, it'll be many of those same employees who get laid off while the VP gets a nice golden parachute of a buyout package to go away.

This is my school
This is home

Takeoffs are happening (just before 9AM EST I believe), but the delays from this morning are going to be problematic for the day I would imagine, maybe longer into tomorrow.

Watch this end up being a failed Microsoft patch update and the roll-back didn't go as planned. This is one of those "design for resiliency" moments that shouldn't have to be taught at all for the engineers of a system that is inherently a security system.

I'm normally pretty frugal about things, but one thing I suggest to anyone who is expecting to do substantial amounts of air travel is to splurge and get the club membership for whatever airline is the main one flying out of the airport you normally use. It is so much better spending 6 hours in the US Air or American or Star Alliance lounge than at a gate. And you can actually get a lot of work done on business trips too. And if you are travelling with a little one, you have clean bathrooms and unlimited snacks and drinks. Well worth the cost.

Recovering scientist working in business consulting

Or get the Amex Platinum card - and get trip delay protections if you buy the tickets on the card too.

Achievement unlocked: All of the Fullers

"Sam Rogers is a college football icon" SB Nation

Thanks Frank!

also tsa precheck and global entry

"Why gobble gobble chumps asks such good questions, I will never know." - TheFifthFuller

I love precheck.

Global Entry is great as well, but I do wish it was applicable to more than just air travel. Would be fantastic if we could use it when disembarking from a cruise as well.

This is my school
This is home

Precheck is GOAT. If you don't have it and you fly 4 or more times a year then literally what the heck are you doing.

"Now Miami wants to talk about it." *Cue Enter Sandman*

As much as I like global entry, I'm usually held up at the luggage carousel, and my wife, who doesn't have it sometimes beats me through the line.

That said, I'm not giving it up.

As much as I like global entry, I'm usually held up at the luggage carousel, and my wife, who doesn't have it sometimes beats me through the line.

This has routinely been my experience as well. At least everything else is a breeze

This is my school
This is home

First time i used it, my boss didn't have it. In the time it took him to go through line in Atlanta, I cleared customs, picked up and rechecked my luggage, went to the wrong gate, when to the right gate, found the lounge and got a plate of food and a beer.

If you travel international more than once a year or so I think its worth the money.

I can't tell you how unimpressed I am with sky lounges. I generally don't drink (other than a wine with meals) when traveling, and the food is generally not the kind of stuff I want to eat. If it costs anything, it's a lot to pay for a clean bathroom. For infrequent travelers, I'd recommend just going to an airport restaurant.

Still, any small advantage in air travel is appreciated if you travel a lot.

You're not going to the right lounges.

Turkish Airlines is the best at Dulles, but not even top three in Istanbul. Centurion are fantastic when they're not overcrowded. Lufthansa Senator lounges are consistently good. There are lounges in Japan, Thailand, Qatar, and Singapore that are basically spa-restaurants. And Lord help you if you have to spend hours in Athens as general population.

I won't pay for it, but i appreciate it if it is included with my ticket because it is so much less hectic than the rest of the airport.

Also, I have occasionally availed myself of a shower which is really nice after travelling for 20 hours.

edit: every time i end up at JFK on the way home, the flight back to Norfolk doesn't leave until like 11 pm and having somewhere quiet to spend the day is much better than the noise of the airport.

Honestly, when I am travelling alone, I personally value that it is quiet over anything else.

Recovering scientist working in business consulting

Agreed. And sometimes the lounges aren't the quietest place I can find.

Working within an airport restaurant can suck. Often overcrowded; no dedicated power source; table/bar can be difficult to type on; internet spotty.

My only experience is with Delta Sky lounges and they are pretty top-notch. Whether working or relaxing, the seats are comfortable with each one having a power source; everything is spaced out; the desks are generally good; lounge-specific wifi internet included; the food is generally good (not exceptional, but enjoyable enough to get seconds and worth hitting up on the way out of the arrival airport); good selection of complimentary drinks; espresso or coffee available; and a great sense of security (can use restroom without luggage). Many of the new ones have an outside terrace and decent views.

Delta sky lounge comes with certain AmEx cards, so it's an expensive yearly cost, but isn't the only benefit of the card. I used to fly all the time on standby (my ex-wife is a flight attendant for Delta) and the lounge benefits were beyond worth it. I don't have a need for lounge benefits any more, but they're awesome if you fly enough.

🦃 🦃 🦃

Had a flight from SLC to DFW yesterday and the delay was not terrible, from 4:19 to 5:45 departure. After reading the news when I woke up at my hotel yesterday I was expecting much worse.

Wednesday morning I jumped in my truck and drove from Richmond to Denver. Saw lots of the good old USA and enjoyed most of the drive. Flying is for the birds. Give me my Silverado and a Rand McNally and some podcasts. I'll drive anywhere.

You will see this game, this upset and this sign next on ESPN Sportscenter. Virginia Tech 31 Miami 7

His decision was made after a phone call with longtime Virginia Tech assistant coach Bud Foster. All Foster told him was, "We win. They don't."

Yeah, I love driving but fuck all that in the winter time. Driving across the country in the summer though? Beautiful.. Especially through Montana and Idaho, absolutely gorgeous in the summer

Edit: they're gorgeous in the winter too (done it) but it's absolutely nerve wracking as well.

Wasn't much different winter versus summer. Missouri and Kansas are flat and boring regardless. I would agree I'd rather do Montana in warmer months. I've driven to Colorado 5 times and broken down twice in Odessa MO. Super random.

You will see this game, this upset and this sign next on ESPN Sportscenter. Virginia Tech 31 Miami 7

His decision was made after a phone call with longtime Virginia Tech assistant coach Bud Foster. All Foster told him was, "We win. They don't."

Username does NOT check out.

LOL touche

You will see this game, this upset and this sign next on ESPN Sportscenter. Virginia Tech 31 Miami 7

His decision was made after a phone call with longtime Virginia Tech assistant coach Bud Foster. All Foster told him was, "We win. They don't."

Seems to be mostly sorted out.

Flew from Greensboro to Denver and back last week with nearly zero issues. Now have friends (the wife is a pilot for American, husband for Delta...lots of dorky airplane jokes but still love em) flying down from Boston next week. Hope it's all figured out before then.

Amateur superstar and idiot extraordinaire.

ALWAYS a leg for an Airplane reference! (particularly a "Johnny" one-"What do you make of this Johnny?";"I can make a hat; or a brooch, or a pterodactyl...")

From the 2018 VT-uva game-"This is when LEGENDS are made!"

This is my school
This is home

Damn... tough day at work for that guy.

It's okay, he probably got to go home early

To look for another job.