Since about 2:35pm EDT today, we noticed many sites that use Amakai go down. Akamai denies any outage at their end.

Here is a brief timeline:-

2:35pm EDT @dcagle tweets, "Wow, msnbc.com is down. I can't remember that happening before.". It was then I began examining headers for the error page. The Server header was not the usual Microsoft-IIS/7.5, but rather AkamaiGHost. Some msnbc hostnames were also having DNS issues for a short while.

Then I checked facebook Akamai based image CDN profile.ak.fbcdn.net which was also down (and still is as of 4:30pm EDT). To test try accessing my profile picture. Make sure you are not loading it from your browser/ISP cache.


[user@host ~]# GET -e http://profile.ak.fbcdn.net/hprofile-ak-snc4/275590_554351037_2048277_n.jpg
Cache-Control: private, no-cache, no-store, must-revalidate
Connection: close
Date: Tue, 04 Oct 2011 20:26:03 GMT
Pragma: no-cache
Server: AkamaiGHost
Content-Length: 172
Content-Type: text/html
Expires: Sat, 01 Jan 2000 00:00:00 GMT
Client-Date: Tue, 04 Oct 2011 20:26:03 GMT
Client-Peer: 68.177.32.58:80
Client-Response-Num: 1
Mime-Version: 1.0
Title: Error

<HTML><HEAD><TITLE>Error</TITLE></HEAD><BODY>
An error occurred while processing your request.<p>
Reference #97.3620b144.1317759963.ac5f
</BODY></HTML>
[user@host ~]# 

Now most of facebook is still usable because most of my friends images would be stored in my local browser cache. Here is a test result of a page on facebook using WebPageTest.

While this was happening, there was a flood of tweets about some Apple announcement and that apple.com was down. apple.com was also down for quite a while. Some speculate that the flood of fanboys accessing (Akamai hosted) apple.com caused an overload at Akamai, I personally don't believe that to be the case. The surge from an Apple announcement is huge, but it is nothing compared to what Akamai otherwise handles.

At about 3:18pm EDT, I noticed msnbc working again.

Current status

Apple.com is back online. MSNBC came back online at about 3:18pm EDT. profile.ak.fbcdn.net is still down

The first public comment by Akamai was at 4:06 pm EDT, when they tweeted "We are aware of reported problems w/ certain Web sites, but can’t comment on specific customers. Our platform isn’t experiencing any issues."

It's still not clear as to what really happened, if we know more, we will update this post.