Before you sign the contract with your new CDN you want to know for a fact the CDN can do what you want and behave as you need it to behave. Right? Sure, you can read the CDN documentation and chat with support, but if you're like us, that is not good enough. You want to test rigorously, observe the CDN's behavior and match it to your functional requirements. Dummy Origin helps you do this better and with greater ease.
Dummy Origin is a simple yet special server, written in Go. It was built specifically for the purpose of evaluating CDNs in an origin pull setup. Dummy Origin is easy to install, lightweight, stable and fast. And open source!
Why use Dummy Origin and not simply put the CDN in front of your real origin with your real content?
The short answer: flexibility and ease-of-use.
- No changes to your stack
- Ultimate flexibility in which response headers the origin sends and seeing how the CDN then behaves
- Easy to test a scenario like 'my origin is serving 502 responses, does the CDN serve stale?'
Features of Dummy Origin
The key feature of Dummy Origin is arbitrary response header injection. In short: whatever you send as a query string parameter in the request will be served as a header with the response.
?ETag="41a0544696e19971383984fdaaeb5aed" in the request and get the
ETag:"41a0544696e19971383984fdaaeb5aed" header back.
Sending a request to get back multiple headers is just as easy:
The arbitrary response header injection gives you great flexibility in testing the CDN's behavior and there is no need to constantly change the server config and restart it.
The other features of Dummy Origin:
|Gzip||Compress the response with default GZip compression level. Not based on
||You want to know if the CDN requests files from origin compressed or uncompressed|
|Conditional requests||Server always sends
||Enables testing if the CDN sends conditional requests after a file in cache has expired|
|Range requests||Send the requested byte range in a 206 partial response + the appropriate
||Essential for serving large files through the CDN, for example MP4 videos|
|404 and 301||Serve a 404 if resource does not exist, serve a 301 if request is for
||Does the CDN cache 404 responses? Does the CDN forward the 301 or does it follow the redirect?|
|Special error generator||When the server receives request in the format
||Handy for seeing how the CDN behaves in case of variour origin error responses. E.g. does the CDN cache 404 responses?|
|X-Tb-Time||Server always sends current time in the
||Helps determine if CDN served the response from cache and how long it has been in CDN cache|
|Loggly support||The request and response details are written to Loggly if
||Loggly gives you a near real-time view on the origin logs. This is very handy for seeing the CDN to origin requests|
Missing in Dummy Origin
- Can't serve different content/headers based on the value of
Installation & configuration
Install the server: README.md on Github.
Put some files on the server. These should be similar to what you want to serve through the CDN. You may want to use dummy files, for example 15kb.jpg, 100kb.jpg, 15kb.png, 13kb.js, 160kb.js, 108mb.avi.
index.htmlon the server so you can always see what files are on the origin by simply visiting the origin domain in your browser.
Set things up on the CDN.
Configure the CDN to treat query strings as unique (
file.jpg?123 is not the same as
file.jpg?456). This is often the default behavior of a CDN but make sure to check.
Also, make sure the CDN forwards all headers from origin to client including the
Configure your DNS. Use a low TTL for the origin host (e.g. 300 seconds) as this will come in handy later when you want to find out how the CDN behaves in case the origin is down (more on this later).
Testing CDN behavior: example use cases
In these examples,
dummy.mydomain.com is the CDN host and
dummy.origin.com is the origin host.
Our tool of choice for testing is cURL. Our cURL commands always start with
curl -vo /dev/null so we send a GET request and can see the request and response headers, but not be bothered with the response body.
Do not use the
-I flag to send a HEAD request to the CDN. The CDN may treat HEAD requests very differently from GET requests (and remember: your users will send GET requests not HEAD requests).
Don't cache and always fetch full file from origin
Not the typical CDN use case, but if this is what you need for your 'uncacheable' content, it needs to work well.
The question here is: which Cache-Control directive(s) must be used to make the CDN behave as desired?
Based on what is in RFC 7234 it seems logical to use
The RFC states:
The "no-store" response directive indicates that a cache MUST NOT store any part of either the immediate request or response. This directive applies to both private and shared caches.
Ok, let's try that
curl -vo /dev/null 'http://dummy.mydomain.com/15kb.jpg?Cache-Control=no-store'
Below you see the response headers from our tests with Fastly. Surprisingly, Fastly does not honor the
no-store and serves the file from cache after a first initial cache miss.
HTTP/1.1 200 OK Access-Control-Allow-Origin: * Cache-Control: no-store Content-Type: image/jpeg Last-Modified: Tue, 21 Mar 2017 10:13:25 GMT X-Tb-Time: 2017-03-30 14:39:11.358951299 +0000 UTC Content-Length: 15887 Accept-Ranges: bytes Date: Thu, 30 Mar 2017 14:39:23 GMT Via: 1.1 varnish Age: 12 X-Served-By: cache-ams4136-AMS X-Cache: HIT X-Cache-Hits: 1 X-Timer: S1490884763.605627,VS0,VE0 Vary: Accept-Encoding
Should we use
max-age=0? We don't want the CDN to cache the response and send conditional requests to the origin. It is important the CDN does not store the response in cache.
no-cache does not work (again, Fastly caches the responses) but
max-age=0 does work.
Cache for 300 seconds, then do origin validation
Getting a CDN to cache a file for 300 seconds is easy:
The key question here is: after the file expires in cache, will the CDN send a conditional or unconditional request to origin?
One can argue the conditional request is optimal, because if the file on the origin has not changed, the origin can send a lightweight 304 response ("No, the file has not changed, it is fine to serve the cached object!") and the CDN can quickly respond to the client. This is faster than when the origin sends the full file to CDN.
RFC 7234 does not state a cache must send a conditional request for cache validation. Therefore it should be not a big surprise if the CDN sends unconditional requests (although not great for performance).
curl -vo /dev/null 'http://dummy.mydomain.com/15kb.jpg?Cache-Control=max-age=300'
Fastly and CDNetworks play nice and send conditional requests. StackPath however sends unconditional requests to origin, meaning their caches always fetch the full file. We could see this in the origin logs and by inpecting the value of the
X-Tb-Time header in the CDN to client response. We also tried with
Cache-Control=max-age=300,must-revalidate but no luck.
HTTP/1.1 200 OK Date: Fri, 31 Mar 2017 11:20:16 GMT Content-Type: image/jpeg Content-Length: 33956 Access-Control-Allow-Origin: * Cache-Control: max-age=20,must-revalidate Last-Modified: Fri, 31 Mar 2017 09:49:42 GMT X-Tb-Time: 2017-03-31 11:20:16.091275242 +0000 UTC Server: NetDNA-cache/2.2 Accept-Ranges: bytes X-Cache: EXPIRED Connection: keep-alive
We contacted StackPath about this behavior and they quickly responded to say the CDN can be configured to send conditional requests if the customer wants that.
Cache for 5 minutes, may serve stale for 15 minutes
curl -vo /dev/null 'http://dummy.mydomain.com/15kb.jpg?Cache-Control=max-age=300&stale-while-revalidate=900&stale-if-error=900'
1. Origin host does not resolve in DNS (NXDOMAIN)
In summary: long after the DNS TTL of our origin must have expired in Fastly caches/systems, Fastly still served stale content. That's nice.
More interestingly, Fastly also kept fetching fresh content from our origin that had not yet made it to their caches. Does Fastly remember the last known IP address of the origin and use that in case of origin resolution failure? Or was this behavior due to Fastly keeping a long-lived connection with the origin? We're not sure and need to dig in deeper.
2. Origin serves 400 response
Something goes wrong at the origin and it starts serving
400 Bad request responses.
We expected the
stale-if-error to kick in here but Fastly simply forwarded the 400 response to the client.
As it turns out, our expectation was wrong because RFC 5861: HTTP Cache-Control Extensions for Stale Content states
an error is any situation that would result in a 500, 502, 503, or 504 HTTP response status code being returned
3. Origin drops all incoming traffic
In our third scenario we changed our origin DNS record to point to 220.127.116.11. This is the IP address behind blackhole.webpagetest.org. Pat Meenan, creator of WebPageTest.org set this up back in 2011 to enable testing for frontend SPOFs in WPT. The WPT blackhole will not return any response to any request.
Did Fastly serve stale? No. Fastly served 503 responses after ~ 1 second and these 503 responses were served with the
Retry-After: 0 header. This is actually to be expected with Fastly out of the box. Their Serving stale content document describes the default behavior:
If Varnish cannot contact the origin for any reason, a 503 error will be generated
The doc then shows and explains the custom VCL code to have Fastly serve stale in this case (and two other cases).
FYI: StackPath can serve stale content based on customer configured status codes (404, 502, 503 etc). This works as expected but StackPath does not serve stale in case of NXDOMAIN or blackhole.
Read more about serve stale in our Serve stale CDN Guide.
It's open source!
Dummy Origin is on Github with MIT license. Fork away!