How a data system glitch turned into a holiday crisis for Sun Country

Early Tuesday morning, more than 30 hours after Sun Country Airlines lost connection to a key data center, and with no sign of the link returning, a technology manager told senior executives that an upgrade immediate software would solve everything.

But there was a catch. The mid-day software upgrade would force Sun Country to cancel more flights, just a day after the data outage forced cancellations and delays that added to the airline’s worst disruption in years .

“I had a lot of questions about it, a lot of concerns,” Sun Country COO Greg Mays said in a lengthy interview near the end of one of the toughest weeks of the season. recent history of the Minneapolis-based airline.

The holiday season ending this weekend has been troubling for airlines nationwide. The rapid spread of the latest variant of the coronavirus has sickened plane crews and other personnel, leading to flight cancellations of around 7-8% of US airline passenger capacity every day since Christmas. Prior to the COVID-19 pandemic, around 2% of flights were canceled on a typical day.

Sun Country’s woes were even greater – 16% of its passenger flights on Monday were canceled – and stemmed from a problem with pedestrian technology rather than omicron infections. Nearly 2,000 people were to be on these flights.

The airline has offered to refund tickets, provide $200 vouchers for future travel, and accept requests for reimbursement of additional expenses, such as hotels and meals.

Yet the day turned into a nightmare as cancellations, which occurred on the first set of flights on Monday morning, created a cascade of delays across its entire 90-flight schedule. Displaced passengers flooded Sun Country’s customer service website and call center. Dozens of people took to Twitter and other social media to complain.

“I will never fly with Sun Country again,” Woodbury’s Cory Williams said Friday. He first spoke to the Star Tribune on Tuesday to describe how the airline’s actions led his family to cancel a week-long trip to Arizona.

Although Sun Country has made headlines for customer service issues in the past — notorious during a late-winter storm in 2018 when it left 250 passengers stranded in Mexico — the airline rarely cancels. flights. Its completion factor, a measure of scheduled flights that are completed, is one of the highest in the airline industry at 99.9%.

“It hurts us every time we have to cancel flights,” said Mays, who joined the company in 2019 after two decades in operational roles at Delta and Alaska Airlines. “I’ve never seen anything like it here.”

With still no access to the data center Tuesday morning and no clarity about its recovery, the question facing Sun Country executives was whether to stage a repeat of Monday’s cancellations in the hopes that updated software running in another data center would bring everything back to Ordinary.

“Typically, you’d do this on a quiet early Saturday morning in a slow time of year,” said Robert Mann, an independent airline analyst in Port Washington, NY. “But you just can’t handle the airline manually. My I guess their options were slim and none.”

The problem began Sunday evening, at the end of the busiest day of Sun Country’s three-week vacation period. As the last flights of the day were in the air, the airline’s crew management system was down.

Over the past 30 years, crew management systems have evolved to do much more than their name suggests and their original purpose of scheduling pilots and flight attendants. The systems are now the operational backbone of many airlines, tracking aircraft, ensuring crew schedules comply with government restrictions and even monitoring pilot exposure to cosmic radiation.

After holding virtual meetings every two hours all night, Mays left the house at 3 a.m. to join a handful of operations managers at headquarters. The company’s top executives, chief executive Jude Bricker and chairman Dave Davis, were on a vacation trip and held meetings remotely over the next two days.

Around the same time, Scott and Mary Baumgartner of Rogers left home to catch a 5:30 a.m. Sun Country flight to Minneapolis-St. Paul International Airport in Fort Myers, Florida.

When they went through check-in and security, they noticed that none of the Sun Country flights had gate assignments.

Snow was falling, the roads were slippery and the couple thought the weather might affect their trip. “I thought maybe something bigger was going on,” Scott Baumgartner said. But when he checked his smartphone, “everything looked fine.”

Sun Country has learned from the crew management system provider – Dubai-based AIMS International – that its main data center is experiencing network difficulties. Sun Country faced a similar outage involving AIMS in July.

Without access to data from its AIMS system, the handful of people at Sun Country headquarters would not be able to verify that the half-dozen people from each crew assigned to the first round of flights were properly rested and scheduled. There was no choice but to cancel.

“Our passengers assume safety. They can only do it because we don’t,” Mays said. “If it’s not good, I tell our teams to stop the operation and I will support them.”

As more and more people arrived at the office, they directly checked the working hours of the teams. The airline worked with AIMS to access Sun Country’s database on a single computer. A camera was then placed in front of that computer screen, so that the information it contained could be viewed via video chat by others making decisions about the airline’s operations.

Outside, the snow continued to fall. At Terminal 2, passengers jostled for new flight options or simply went home. Three Sun Country charter flights carrying the University of Minnesota Marching Band to Phoenix for the Guaranteed Rate Bowl took off on time, but many other scheduled flights were delayed.

The Baumgartners used a smartphone app to find a flight on American Airlines, but in Tampa instead of Fort Myers and with a layover in Washington, D.C. They arrived at 9 p.m. Minnesota time and their son-in-law picked them up for a two-hour flight to Cape Coral.

“We wasted an entire day visiting our grandchildren,” Scott Baumgartner said.

At 8 a.m. Tuesday, Mays said he knew the airline couldn’t endure another day with such limited access to data. Sun Country’s IT staff planned to perform the software upgrade to the AIMS system at a quiet time in the first quarter and had done some preparatory work.

“Ultimately, we made the decision to make this transition,” Mays said. “We had our first push [of flights]and we’ve set a final decision for 10:30 a.m..”

Over the next two hours, Mays spoke with airline executives in Spain and Mexico, also AIMS customers, who shared how they had achieved the upgrade. Two new backup systems were prepared – one simply sheets of paper taped to a wall listing each plane by its tail number – and staff members quickly pored over training materials that showed the new look of the AIMS software.

At 10:45 a.m., Mays agreed that the update should start at 11 a.m. Cancellation notices were sent to passengers on 14 afternoon flights, seven to MSP and seven returning to MSP.

The holidays of almost 2,000 other people have been interrupted. At least this time, executives hoped, passengers would be informed before departing for airports.

“We handled this as best we could under the circumstances. We’ll see how our passengers judge us, and that will happen over time,” Mays said.

The upgrade worked, but Sun Country will eventually revise its work with AIMS, Mays said. This weekend, sub-zero weather in the Twin Cities looked more likely to be the main challenge to on-time performance. Sunday will be the second busiest day of the holiday for the carrier.

“I want to make sure our local community knows how seriously we take our commitment to our passengers,” Mays said. “We are not successful all the time. Our first concern is safety. But we always want to do our best for the passengers. We are very proud to be the local airline.”

On Friday, some of the displaced passengers said they had obtained refunds. Others, like Williams, said they had received partial refunds and were seeking more information.

Some, like Scott and Mary Baumgartner, were still waiting. The couple are due to return home on Monday from Fort Myers on Sun Country.

Comments are closed.