This article is a first in a series of three on the topic of CDN debugging. Here we will share three tips for collecting useful data, which really is the first thing you want to do when you believe something is wrong with your CDN.
curl -svo /dev/null
Curl (or cURL) is a command line tool widely used for testing and troubleshooting client-server data transfer over HTTP. The free tool comes preinstalled on Linux and Mac OS X machines and curl for Windows is available too.
When using curl to debug CDN behavior, don't ever use curl -I.
Using the I flag results in sending a HEAD request and that is often pointless and unintended.
Your users will send GET requests not HEAD requests and the CDN may treat HEAD requests very differently from GET requests.
So remember: don't use curl -I, ever.
Your curl commands should start with curl -svo /dev/null.
The -svo /dev/null is a series of flags:
-s enables silent mode; don't show progress meter or error messages.
-v enables verbose output
-o /dev/null sends output to a file, but not really because the /dev/null item does not save anything (note: on Windows, use -o NUL
So, -svo /dev/null gives you a nice and clean display of information, most importantly the request and response headers.
The -H flag can be used to send an arbitrary request header, for example a specific User-Agent string: -H 'User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25
Below you see an example curl command for fetching the (gzip) compressed version of https://edition.cnn.com/css/2.33.0/global.css.
curl -svo /dev/null --compressed 'https://edition.cnn.com/css/2.33.0/global.css'
Learn more about curl and all the flags by reading the curl manual page.
Don't test against just one POP
When debugging your CDN, make sure you get don't fall in the trap of looking into the behavior of just one POP. It's important you find out if the problem is POP specific or not and the only way to do that is to test against several/many POPs. But how do you do that?
Services like Pingdom or Catchpoint have test machines across the globe, and this is great because you can perform tests against POPs in many locations. However, the visibility you can get using such services is suboptimal. For example: in France, Level 3 CDN has POPs in Marseille and Paris, but Pingdom only has servers in a datacenter in Strasbourg. When testing from the Pingdom servers in Strasbourg, you'll likely hit the same one of two Level 3 POPs every time and consequently know nothing about the other POP.
Ideally, your CDN provides you a way to send a request to a specific POP, so you can run curl tests against each and every POP comfortably from your office computer. Unfortunately, not many CDNs provide this feature.
As far as we know, Edgio and Highwinds (part of StackPath) enable you to debug a specific POP.
Fastly enables sending requests to a specific cache node (e.g. cache-lax1427) which is nice but only useful if you know the cache node that is serving your content. In our opinion, it's more useful to be able to send a request to 'the Amsterdam POP'.
To send a request to a specific target endpoint (e.g. a CDNetworks server in Amsterdam), use the -H flag to add a Host header with your domain:
curl -svo /dev/null -H 'Host: cdn.yourdomain.com' 'https://91.194.205.21/path/to/file'
Test from real user networks, not datacenters
Even if you can run curl tests against specific CDN POPs from your office location, or from datacenters across the globe, you're still somewhat in the dark. Real users don't live in your office or in datacenters. They are at home, at work and on the go and connected to last-mile networks.
It's not uncommon for a CDN's performance to go bad only for users on one particular last-mile network. The problem may be a congested network path, suboptimal routing (e.g. users on AS9143 (Ziggo) in NL are routed to New York instead of Amsterdam) or the ISP's DNS resolvers can't reach the CDN's DNS nameservers. The only way to spot these kind of issues is by testing from devices connected to the last-mile (or: eyeball) networks.