Yesterday (March 02, 2017), Level 3 experienced a major service issue across the United States. Because of this event, customers of Southwest Cyber Systems suffered an outage of key hosted services lasting approximately 6 hours. Examples of affected services include hosted email, IP phones, and DNS resolution. The outage was nationwide in scope, and the number of individual companies affected is still being tabulated today.
According to Level 3, the root cause came from an adjusted software process on an internal servers application that increased polling frequency, causing multiple routers to bounce or become isolated across Multiple Markets in North America. This event impacted IP and voice services. To fix the issue, Level 3 reverted the software process to stabilize services.
Today, the team at Southwest Cyber Systems analyzed several options to leverage our extended network infrastructure to create additional redundancy in our core systems, which will further reduce the impact of these types of outages in the future.