Failover: Failing to Fail Over Can Cost Millions

failover instead of fail illustration

When universities, banks, retailers and other businesses don’t build redundant connections and automatic link failover into their networks, they run the risk of incurring big losses. The pain of Internet outages can be avoided with network failover. The cost of failing to fail over isn’t worth the risk.

Kansas University

Early in the afternoon of March 29, construction crews inadvertently cut through a critical section of fiber on the KU campus. The fiber cut shut down Internet service to the KU campus as well as outsiders’ access to websites and applications housed on KU servers — including all of KU-based Center for Educational Testing and Evaluation (CETE) tests, administered to K-12 students in more than a dozen states. Instead of having automatic failover to an alternate circuit, connectivity failed.

As a result, Alaska cancelled its CETE contract and KU lost a stream of revenue. The Lawrence Journal-World reported KU IT estimated a “seven-digit” cost to restore the 3,000 strands of fiber and 2,000 copper pairs that were cut. Although the link is now 100% functional, it will likely need to be replaced. The outage reinforced the importance of prevention and backup plans, even with services as reliable as fiber.

According to David Day, director of external affairs for KU IT, “Building in redundancy has a cost, and you have to calculate that against the possibility of something remote happening. That’s what has always been weighed in the past and is being re-looked at now.”

As campuses adopt more Internet-based phones and faculty rely more heavily on the Internet to teach, providing automatic failover becomes essential.

“Even 10 years ago losing internet connectivity wouldn’t have been so critical,” Day said. “The need for redundancy and maintaining that connectivity has grown exponentially, as it has become such an important part of our lives.”

Amazon and Starbucks

Internet Retailer estimates that a 20-minute outage in March cost Amazon about $3.75 million. Last year a point of sale glitch at Starbucks impacting more than 60 percent of roughly 13,500 locations had the coffee giant giving away coffee and closing shops.

Santa Monica restaurant

The losses that happen with the giants happen with small businesses, too, and the impact can be devastating. A Santa Monica business hit by a widespread Internet outage in May lost sales when it was forced to close early. Mark Peters, manager at Dagwoods Pizza Parlor on Wilshire Blvd, reported his business was unable to take orders since the phone was down, unable to take internet orders since their internet was down, and unable to do business.

Peters told the Santa Monica Observer, “It’s a bad situation. We can’t take orders, and this is our big night of the week! We’re really bummed out about the whole situation. I don’t know if they (Frontier) realize that this has the potential to destroy small businesses.”

Texas bank

For banks, outages cause headaches for customers and increase manual workloads on staff. During an AT&T Internet outage due to a cut fiber line in Waco, TX, the Mart branch of The First National Bank of Central Texas found itself cut off from the home office. Brian Hand, Banking Central Manager, said, “It’s kind of taking us back to the Stone Age where we don’t have access to customer accounts, customer information, and actually running the checks and the deposit and everything else through our system.” The ATM machine was out of service, the drive-thru was closed for the day and loan processing was delayed.

Oregon manufacturer

Sunstone Circuits depends on the Internet for its printed circuit board (PCB) business. Because of its rural location in Mulino, OR, the company used to experience weekly Internet outages that hurt its reputation and employee productivity. Needing an automatic failover system that would prevent manual restores and revenue loss, they added redundant circuits to their network along with an Ecessa device to load balance traffic and provide automatic failover. Later, when they switched to fiber optic circuits, they upgraded their Ecessa device to accommodate their increased bandwidth.

“Things can happen,” said Mike Butler, Network Administrator at Sunstone. “During bridge construction in the area, contractors accidentally cut our fiber lines. Ecessa’s automatic failover worked as planned. Our operations were not interrupted.”

Knowledge is power

According to a survey by CA Technologies, small enterprises lost, on average, more than $55,000 in revenue due to IT failures each year, while midsize companies lost more than $91,000 and large companies lost more than $1,000,000. Spot Migration created a downtime calculator to illustrate revenue lost during downtime and recovery. Understanding the cost of downtime can help organizations determine how to prepare their networks for inevitable cable cuts and Internet outages.

Many small businesses don’t know redundant connections and network failover devices are available and advisable. Increasingly, telecommunications agents working with businesses of all sizes are recommending affordable, redundant services and automatic failover.

“We recommend redundant networks for our customers, because it’s the smart thing to do. Not only do clients get peace of mind that their networks will always be available, they get additional bandwidth that’s usable all the time. This has become both practical and affordable,” said Wade Acheson, Principal at Broadline Solutions, a master agency specializing in cloud, colocation and connectivity solutions.

For information about adding redundancy and automatic failover to your business network, please fill out the form below or call Ecessa at 800.669.6242