Skip to content

CDN behavior testing with Dummy Origin

Published on Apr 5 2017

Before you sign the contract with your new CDN you want to know for a fact the CDN can do what you want and behave as you need it to behave. Right? Sure, you can read the CDN documentation and chat with support, but if you're like us, that is not good enough. You want to test rigorously, observe the CDN's behavior and match it to your functional requirements. Dummy Origin helps you do this better and with greater ease.

Dummy Origin is a simple yet special server, written in Go. It was built specifically for the purpose of evaluating CDNs in an origin pull setup. Dummy Origin is easy to install, lightweight, stable and fast. And open source!

Why use Dummy Origin and not simply put the CDN in front of your real origin with your real content?
The short answer: flexibility and ease-of-use.

  • No changes to your stack
  • Ultimate flexibility in which response headers the origin sends and seeing how the CDN then behaves
  • Easy to test a scenario like 'my origin is serving 502 responses, does the CDN serve stale?'

Features of Dummy Origin

The key feature of Dummy Origin is arbitrary response header injection. In short: whatever you send as a query string parameter in the request will be served as a header with the response.

Send ?ETag="41a0544696e19971383984fdaaeb5aed" in the request and get the ETag:"41a0544696e19971383984fdaaeb5aed" header back.

Sending a request to get back multiple headers is just as easy:
?Cache-Control=no-cache&Referer=https://www.cdnplanet.com/

The arbitrary response header injection gives you great flexibility in testing the CDN's behavior and there is no need to constantly change the server config and restart it.

The other features of Dummy Origin:

Feature Description Benefits
Gzip Compress the response with default GZip compression level. Not based on Content-Type but file extension. Currently supports .html, .css .js and .json extensions only but very easy to add more You want to know if the CDN requests files from origin compressed or uncompressed
Conditional requests Server always sends Last-Modified header with 200 responses and sends a 304 or 200 response if it gets a If-Modified-Since request (as per RFC7234 section 4.3) Enables testing if the CDN sends conditional requests after a file in cache has expired
Range requests Send the requested byte range in a 206 partial response + the appropriate Content-Length header Essential for serving large files through the CDN, for example MP4 videos
404 and 301 Serve a 404 if resource does not exist, serve a 301 if request is for /index.html Does the CDN cache 404 responses? Does the CDN forward the 301 or does it follow the redirect?
Special error generator When the server receives request in the format GET /err/<code> it returns with http status <code>. <code> must be between 400 and 599 Handy for seeing how the CDN behaves in case of variour origin error responses. E.g. does the CDN cache 404 responses?
X-Tb-Time Server always sends current time in the X-Tb-Time header Helps determine if CDN served the response from cache and how long it has been in CDN cache
Loggly support The request and response details are written to Loggly if LOGGLY_TOKEN environment variable is set (optional) Loggly gives you a near real-time view on the origin logs. This is very handy for seeing the CDN to origin requests

Missing in Dummy Origin

  • HTTPS
  • Brotli
  • Can't serve different content/headers based on the value of User-Agent or Accept-Language requests headers

Installation & configuration

Install the server: README.md on Github.

Put some files on the server. These should be similar to what you want to serve through the CDN. You may want to use dummy files, for example 15kb.jpg, 100kb.jpg, 15kb.png, 13kb.js, 160kb.js, 108mb.avi.

Do NOT put an index.html on the server so you can always see what files are on the origin by simply visiting the origin domain in your browser.

Set things up on the CDN. Configure the CDN to treat query strings as unique (file.jpg?123 is not the same as file.jpg?456). This is often the default behavior of a CDN but make sure to check. Also, make sure the CDN forwards all headers from origin to client including the X-Tb-Time header.

Configure your DNS. Use a low TTL for the origin host (e.g. 300 seconds) as this will come in handy later when you want to find out how the CDN behaves in case the origin is down (more on this later).

Testing CDN behavior: example use cases

In these examples, dummy.mydomain.com is the CDN host and dummy.origin.com is the origin host.
Our tool of choice for testing is cURL. Our cURL commands always start with curl -vo /dev/null so we send a GET request and can see the request and response headers, but not be bothered with the response body.

Do not use the -I flag to send a HEAD request to the CDN. The CDN may treat HEAD requests very differently from GET requests (and remember: your users will send GET requests not HEAD requests).

Learn more about cURL by reading Using cURL when Troubleshooting with Cloudflare or the cURL manual.

Don't cache and always fetch full file from origin

Not the typical CDN use case, but if this is what you need for your 'uncacheable' content, it needs to work well. The question here is: which Cache-Control directive(s) must be used to make the CDN behave as desired? Based on what is in RFC 7234 it seems logical to use Cache-Control:no-store. The RFC states: The "no-store" response directive indicates that a cache MUST NOT store any part of either the immediate request or response. This directive applies to both private and shared caches. Ok, let's try that

curl -vo /dev/null 'http://dummy.mydomain.com/15kb.jpg?Cache-Control=no-store'

Below you see the response headers from our tests with Fastly. Surprisingly, Fastly does not honor the no-store and serves the file from cache after a first initial cache miss.

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Cache-Control: no-store
Content-Type: image/jpeg
Last-Modified: Tue, 21 Mar 2017 10:13:25 GMT
X-Tb-Time: 2017-03-30 14:39:11.358951299 +0000 UTC
Content-Length: 15887
Accept-Ranges: bytes
Date: Thu, 30 Mar 2017 14:39:23 GMT
Via: 1.1 varnish
Age: 12
X-Served-By: cache-ams4136-AMS
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1490884763.605627,VS0,VE0
Vary: Accept-Encoding

Should we use no-cache or max-age=0? We don't want the CDN to cache the response and send conditional requests to the origin. It is important the CDN does not store the response in cache.

Testing shows no-cache does not work (again, Fastly caches the responses) but max-age=0 does work.

Cache for 300 seconds, then do origin validation

Getting a CDN to cache a file for 300 seconds is easy: Cache-Control=max-age=300. The key question here is: after the file expires in cache, will the CDN send a conditional or unconditional request to origin? One can argue the conditional request is optimal, because if the file on the origin has not changed, the origin can send a lightweight 304 response ("No, the file has not changed, it is fine to serve the cached object!") and the CDN can quickly respond to the client. This is faster than when the origin sends the full file to CDN.

RFC 7234 does not state a cache must send a conditional request for cache validation. Therefore it should be not a big surprise if the CDN sends unconditional requests (although not great for performance).

curl -vo /dev/null 'http://dummy.mydomain.com/15kb.jpg?Cache-Control=max-age=300'

Fastly and CDNetworks play nice and send conditional requests. StackPath however sends unconditional requests to origin, meaning their caches always fetch the full file. We could see this in the origin logs and by inpecting the value of the X-Tb-Time header in the CDN to client response. We also tried with Cache-Control=max-age=300,must-revalidate but no luck.

HTTP/1.1 200 OK
Date: Fri, 31 Mar 2017 11:20:16 GMT
Content-Type: image/jpeg
Content-Length: 33956
Access-Control-Allow-Origin: *
Cache-Control: max-age=20,must-revalidate
Last-Modified: Fri, 31 Mar 2017 09:49:42 GMT
X-Tb-Time: 2017-03-31 11:20:16.091275242 +0000 UTC
Server: NetDNA-cache/2.2
Accept-Ranges: bytes
X-Cache: EXPIRED
Connection: keep-alive

We contacted StackPath about this behavior and they quickly responded to say the CDN can be configured to send conditional requests if the customer wants that.

Cache for 5 minutes, may serve stale for 15 minutes

curl -vo /dev/null 'http://dummy.mydomain.com/15kb.jpg?Cache-Control=max-age=300&stale-while-revalidate=900&stale-if-error=900'

As far as we know, Fastly and CDN77 are the only CDNs that support the stale-while-revalidate and stale-if-error Cache-Control extensions. We tested the behavior of Fastly in three situations:

1. Origin host does not resolve in DNS (NXDOMAIN)
In summary: long after the DNS TTL of our origin must have expired in Fastly caches/systems, Fastly still served stale content. That's nice.

More interestingly, Fastly also kept fetching fresh content from our origin that had not yet made it to their caches. Does Fastly remember the last known IP address of the origin and use that in case of origin resolution failure? Or was this behavior due to Fastly keeping a long-lived connection with the origin? We're not sure and need to dig in deeper.

2. Origin serves 400 response
Something goes wrong at the origin and it starts serving 400 Bad request responses. We expected the stale-if-error to kick in here but Fastly simply forwarded the 400 response to the client. As it turns out, our expectation was wrong because RFC 5861: HTTP Cache-Control Extensions for Stale Content states an error is any situation that would result in a 500, 502, 503, or 504 HTTP response status code being returned

3. Origin drops all incoming traffic
In our third scenario we changed our origin DNS record to point to 72.66.115.13. This is the IP address behind blackhole.webpagetest.org. Pat Meenan, creator of WebPageTest.org set this up back in 2011 to enable testing for frontend SPOFs in WPT. The WPT blackhole will not return any response to any request.

Did Fastly serve stale? No. Fastly served 503 responses after ~ 1 second and these 503 responses were served with the Retry-After: 0 header. This is actually to be expected with Fastly out of the box. Their Serving stale content document describes the default behavior: If Varnish cannot contact the origin for any reason, a 503 error will be generated The doc then shows and explains the custom VCL code to have Fastly serve stale in this case (and two other cases).

FYI: StackPath can serve stale content based on customer configured status codes (404, 502, 503 etc). This works as expected but StackPath does not serve stale in case of NXDOMAIN or blackhole.
Read more about serve stale in our Serve stale CDN Guide.

It's open source!

Dummy Origin is on Github with MIT license. Fork away!


Comments