This is article two in a series of three blog posts.
Yesterday we published Google DNS and OpenDNS usage stats that we generated from the data of millions of end-user tests. Interesting as this is, the real goals of those tests was to a) find out which CDNs support edns-client-subnet and b) gain insight in the impact of edns-client-subnet on CDN performance. In this article, we explain what edns-client-subnet is, its relevance to CDN performance and show which CDNs currently support it. In article 3, to be published tomorrow, we will show real-world performance data and make clear how edns-client-subnet (significantly) impacts CDN performance.
Some CDNs use DNS to determine the geographical location of the user. They can't use the IP address of the client for this, because it is masked by the DNS resolver, and so the CDNs use the IP address of the DNS resolver instead. In case of the Google DNS or OpenDNS servers, for many end users those servers are not close to them, simply because these providers don't have servers in every country and every ISP's network. For example, OpenDNS does not have DNS servers in South-America (network map). Someone in Brazil using OpenDNS will likely hit their resolver in Florida. The CDN will then think the user is in Florida and as a result it will serve content to the user from a server far away (Florida, not Brazil) resulting in a slow experience.
The two diagrams below illustrate how things work for a user in Thailand who wants to connect to a hostname on Akamai, using either the DNS resolver of his ISP or the resolver of OpenDNS.
The user ends up hitting a low-latency Akamai edge server in Thailand because Akamai's DNS server can accurately detect the user's geolocation. This is good.
Now Akamai cannot accurately detect the user's geolocation and believes the user is in Singapore. The user will connect to a high-latency Akamai edge server in Singapore. This is bad.
There are two solutions a CDN can implement to counter this problem. They can either use anycast for HTTP or support edns-client-subnet. We'll take a quick look at anycast for HTTP but focus on edns-client-subnet.
Anycast for HTTP
CDNs that use anycast for HTTP are not affected by the problems with geo targetting using DNS since they resolve to the same IP for all users (or all users in one region; EdgeCast). They rely on BGP to direct users to the closest server based on preferred path. Using anycast for HTTP may come with its own set of problems, but that is beyond the scope of this post.
The CDN's authoritative name server (ADNS) will respond with an IP address regardless of the user's location and that will lead the user to a nearby edge server wherever the user is.
To mitigate the problem of DNS based geo-targetting, Google proposed a technical solution to the issue in an IETF draft Client subnet information in DNS requests. This is an experimental DNS extension that allows DNS resolvers to pass the client's IP address (or part of) to compatible authoritative DNS servers. The CDN's DNS server can then use this information to better determine where the end user is. Google DNS and OpenDNS implemented this solution as part of the Global Internet Speedup initiative in August 2011.
The drawback is the experimental nature of the spec and limited support in existing DNS server software. Only OpenDNS and Google Public DNS seems to support it on the resolver side. With both these providers, one must apply to be "whitelisted" in order to receive client's subnet with the query. The whitelisting procedure is pretty straightforward. Contact OpenDNS or Google, tell them your hostnames, nameservers IPs and they will probably whitelist you within a couple of days without any fuss.
Our test DNS server supported edns-client-subnet since the start of this month, and no resolver apart from Google and OpenDNS sent us queries with client subnet information.
The following illustration shows how supporting edns-client-subnet should result in getting a low-latency connection to the CDN.
Which CDNs support edns-client-subnet
|Bitgravity||No||Anycast, so no problem|
|CacheFly||No||Anycast, so no problem|
|CloudFlare||No||Anycast, so no problem|
|EdgeCast||Yes||Anycast, so no problem|
|Fastly||No||Not big problem now|
|Highwinds||No||Anycast, so No problem|
|Level 3||No||High impact|
|NetDNA||No||Anycast, so no problem|
CDNs that need to take action
Quite a few CDN providers don't do anycast for HTTP and don't support edns-client-subnet: Akamai, CDNetworks, CloudFront, Fastly, Internap, Level3 and Limelight. These are the CDNs that provide worse performance to some Google DNS and OpenDNS users than possible. But the size of the problem is not equal for all these CDNs. Fastly is the newcomer in the CDN market and currently has a limited number of POPs (4 in the US and 3 in Europe). Supporting edns-client-subnet will not improve performance all that much for them. Performance of the bigger CDNs however, like Akamai and Limelight, could improve significantly for many end-users if they would support edns-client-subnet. They have thousands of servers in many, many locations and quite a few Google DNS and OpenDNS users currently don't connect to an edge server inside the network of their ISP, but rather to a server in some other country or even continent.
A few more things to say
CDNetworks is listed as a participant in the Global Internet Speedup initiative, but they actually don't support edns-client-subnet. That is odd. The same goes for Bitgravity and CloudFlare, but since these CDNs do anycast, it is a none-issue.
EdgeCast is the only CDN with anycast for HTTP that supports edns-client-subnet, but it shouldn't really matter much because they do anycast (per region actually: North-America, Europe, APAC, ...). Perhaps EdgeCast can improve the effectiveness of their regional anycast architecture just a little bit with edns-client-subnet.
This is what I did to test support for edns-client-subnet by CDNs:
I downloaded the patch to dig from http://wilmer.gaa.st/edns-client-subnet/ and installed it. Actually, I copied the patched dig as dig-client into my path for easy access.
sajal@sajal-laptop:~$ dig-client gp1.wac.v2cdn.net @ns1.edgecastcdn.net +client=220.127.116.11 ; <<>> DiG 9.9.1-P3 <<>> gp1.wac.v2cdn.net @ns1.edgecastcdn.net +client=18.104.22.168 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8715 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ; CLIENT-SUBNET: 22.214.171.124/32/14 ;; QUESTION SECTION: ;gp1.wac.v2cdn.net. IN A ;; ANSWER SECTION: gp1.wac.v2cdn.net. 3600 IN A 126.96.36.199 ;; Query time: 56 msec ;; SERVER: 188.8.131.52#53(184.108.40.206) ;; WHEN: Tue Oct 16 17:08:39 2012 ;; MSG SIZE rcvd: 91 sajal@sajal-laptop:~$
The response from EdgeCast shown above means that EdgeCast is saying that the response is only valid for 220.127.116.11/14 (i.e. 18.104.22.168 - 22.214.171.124). It may be the case that the CDN has only whitelisted Google/OpenDNS to receive client-subnet. In that case we can query Google. Google passes along subnet information to everyone.
sajal@sajal-laptop:~$ dig-client gp1.wac.v2cdn.net +client=126.96.36.199 @188.8.131.52 ; <<>> DiG 9.9.1-P3 <<>> gp1.wac.v2cdn.net +client=184.108.40.206 @220.127.116.11 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5615 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ; CLIENT-SUBNET: 18.104.22.168/32/14 ;; QUESTION SECTION: ;gp1.wac.v2cdn.net. IN A ;; ANSWER SECTION: gp1.wac.v2cdn.net. 2192 IN A 22.214.171.124 ;; Query time: 60 msec ;; SERVER: 126.96.36.199#53(188.8.131.52) ;; WHEN: Tue Oct 16 17:13:56 2012 ;; MSG SIZE rcvd: 74 sajal@sajal-laptop:~$
My bash script is here