A disruption today in the networking services provided by Cloudflare is the latest in a series of outages to shine a spotlight on the fragility of the web.
An outage that affected the content delivery network (CDN) services provided by Cloudflare created a cascade of issues that caused a number of websites to become unavailable, including sites operated by the ChatGPT service hosted by OpenAI and the X social media site.
Cloudflare CTO Dane Knecht apologized for the outage that he attributed to an unspecified configuration issue rather than a cyberattack. “I won’t mince words: earlier today we failed our customers and the broader Internet when a problem in [our] network impacted large amounts of traffic that rely on us. The sites, businesses, and organizations that rely on Cloudflare depend on us being available and I apologize for the impact that we caused,” Knecht said in a post on X. “This was not an attack.”
Specifically, a routine configuration change affected a latent bug in the software underpinning the Cloudflare bot‑mitigation/challenge layer, cascading into widespread 500 errors across its CDN service.
Mark Townsend, co-founder & CTO, AcceleTrex, a website for building relationships across technology leaders interested in working for startup companies, said the outage shines a light on the need to treat CDNs as tier‑0 dependencies that create single points of failure because of concentration risks. IT teams would be well-advised to map all services gated by CDN providers to better quantify the risks they represent to the business, he added
While the Cloudflare outage lasted only roughly three hours after being resolved by 9:40am EST, the total cost of the incident is incalculable. In some cases, users of some services were merely inconvenienced. In other cases, providers of mission-critical applications and services that depend on network services provided by Cloudflare might be more deeply impacted.
The one thing that is certain is that in the wake of similar outages involving, for example, Amazon Web Services (AWS) and Microsoft, the need for higher levels of resiliency has become increasingly apparent.
The challenge is that achieving and maintaining that level of resiliency is, naturally, expensive. It essentially means having a secondary set of cloud and networking services that organizations can switch over to in the event the primary provider of those services suffers a disruption. The issue is determining what the total amount of lost revenue might be depending on how long a critical cloud or networking service is for one reason or another unavailable.
Hopefully, the providers of those services will become more adept at eliminating single-points of failure that are the root cause of most disruptions. Unfortunately, most of the providers of these services are confident they have already addressed these issues until some unanticipated event proves them wrong. The issue, of course, is that the IT leaders that decided to bet on those services being available then need to have an uncomfortable conversation with the business leaders that hired them in the expectation that web applications and services will, at least mostly, be always available at a reasonable cost.
