At the end of 2018, CenturyLink, one of the largest telecommunications providers in America, experienced a two-day service outage. This major outage cut internet, television and 911 services across the U.S. The outage started at 17:18 UTC on Thursday, December 27, and was reported resolved at 19:49 UTC on Saturday, December 29. CenturyLink said the outage was caused by a single network card sending bad packets.
The issue caused so much grief that the U.S. Federal Communication Commission (FCC) has launched a public investigation into this outage. FCC chairman Ajit Pai (@AjitPaiFCC) called the outage “completely unacceptable, and its breadth and duration are particularly troubling.”
But who should really be to blame for a sad lack of planning? You can’t put that all on CenturyLink. Each organization needs to take responsibility for its own business continuity.
It is possible – and prudent – to avoid damage from outages by having more than one internet connection, from more than one provider, on different backbones.
Let’s unpack that. Say you have CenturyLink. You need a second connection from a different internet service provider (ISP) offering service in your area, whether it’s fiber, DSL, satellite, 4G, microwave, or fixed wireless. Since many services physically run over the same large fiber optic cables (for example, CenturyLink handles wireless data network traffic for Verizon in some areas), you need to find one that isn’t on the same backbone as your primary ISP.
Preferably this link would come into your building from a different physical direction so that a single backhoe mishap wouldn’t be able to knock out both services.
If you’re thinking to yourself, you only have one ISP option in your area, find a wireless option. You’ll then be able to tie your services together with a load balancing and automatic failover device so when an outage does happen, your voice and data traffic instantly flow to the working connection. It costs a heckuva lot less than business downtime and losses from an outage. Being prepared saves your staff the grief of scrambling for workarounds. It makes you look smart and gives your clients and investors more confidence in your organization. In a world where we rely so heavily on the Internet, redundancy is a must-have for every business.
There is NO EXCUSE for your operation losing connectivity!
Join Ecessa on February 8 for a free, one-hour webinar that will teach you how to protect your organization against Internet outages. Register here: http://info.ecessa.com/how-to-protect-your-organization-against-internet-outages-webinar-2019
The Damage Done
CNBC reported on several organizations impacted by the CenturyLink outage. Here are a few.
Banner Health’s facilities and affiliated hospitals and clinics throughout Colorado and other states were affected by the outage. At North Colorado Medical Center in Greeley, Colorado, doctors and nurses lacked access to patient records on December 27th and clinicians had to take notes with pencil and paper, according to the Greeley Tribune.
The Idaho Statesman reported that the internet problems caused the temporary shutdown of phone services at the Idaho Department of Correction and the state’s Department of Education. Also, Idaho Central Credit Union’s network went down, according to the credit union’s Facebook page. ICCU public relations director Laura Smith told the Statesman the network problems are a result of the issues with CenturyLink. “Our call center and some ATMs are currently unavailable,” the credit union’s post said. “ACH files, checks, and wires were also affected; however, our team has begun to work on these processes.”
The outage knocked out 911 emergency call services in parts of western Washington state. According to KOMO News, some CenturyLink customers got busy signals when dialing 911. Other areas of the country that also experienced 911 outages included parts of Missouri, Idaho and Arizona.
Really – 911 service?! If anyone should have their act together, it’s our emergency responders. The Salem News said it best. “The nation’s 911 service shouldn’t fail because of one company’s troubles.” And yet it did.
We have the technology to avert future issues like this. The good news is, it’s not rocket science. It’s affordable and straightforward to install. The infrastructure just needs to get implemented.
Your business has the capacity to protect itself – and it must.
For more information or to speak with a network architect, contact Ecessa.