Major Sites and Services Back Up After Early Morning Fastly Service Disruption

Streaming Media

Internet users in the United States awoke to news that a vast swath of the world's most prominent sites—including leading OTT providers—had gone down in the early morning hours. HBO Max, Hulu, Vimeo, Amazon, Google, Twitter, Spotify, The New York Times, Reddit and The Guardian were among major sites impacted by an outage attributed to cloud computing provider Fastly.

https://www.streamingmedia.com/Articles/News/Online-Video-News/Major-Sites-and-Services-Back-Up-After-Early-Morning-Fastly-Service-Disruption-147279.aspx

It has tweeted, and also emailed to Streaming Media, that “We identified a service configuration that triggered disruptions across our POPs globally and have disabled that configuration. Our global network is coming back online.”

Earlier, Fastly said that it was “investigating potential impact to performance with our CDN services and warned that “customers may experience increased origin load as global services return.”

The outage lasted about an hour. Fastly have not yet provided comment on the precise nature of the problem.  According to Mark Hendry, director of data protection and cybersecurity at legal business DWF, some of the affected organisations sought to rectify the issue by reverting to non-CDN schemes of distribution.

“However, if this is the case, users of those websites can expect for their experience to be slower than normal until the CDN can be restored,” he says. “Whilst the outage can be considered an availability of services issue, it is not clear at this time whether any underlying data or infrastructure belonging to the affected organisations has become vulnerable as a result of the issue.”

Others have questioned the wisdom of having so much of the internet infrastructure in the hands of a few companies, causing widescale disruption when things go wrong.

Adam Smith, a software testing expert with the BCS, the Chartered Institute for IT, told the BBC that outages with CDNs “highlight the growing ecosystem of complex and coupled components that are involved in delivering internet services. Because of this, outages are increasingly hitting multiple sites and services at the same time.”

Fastly runs edge compute services including Nearline Cache, a service launched last year as the first of its commercial solutions to be built in a serverless compute environment.

The company explains that Nearline Cache allows you to automatically populate and store content in third-party cloud storage near one of Fastly’s POPs “without incurring egress costs, addressing a very real challenge for long-tail content that might get evicted from cache”. With Nearline Cache, it says, “you can populate that content back into cache, resulting in overall cost savings and improved origin offload. Plus, there’s minimal latency and no new work for customers.”

Its operations appear to be built using technology from Swedish developer Varnish Software, whose customers include Hulu, Emirates, and Tesla.

Varnish promptly put out its own Tweet denying responsibility.

Fastly’s customers include Pinterest, The New York Times, and GitHub. Its cloud partners include Google Cloud, AWS and Azure.

It made $291 million in revenue in 2020, up 45% on 2019. While its stock price initially fell on news of the outage, it was back above pre-outage pricing at the time this article was published.