This is article 3 in a series of three blog posts.

Two days ago we published Google DNS and OpenDNS usage stats and in yesterday's article we explained edns-client-subnet, gave an overview of which CDNs support edns-client-subnet and stated that the impact of not supporting edns-client-subnet on CDN performance is high for several CDNs, including Akamai. Today we present real-world performance data that backs this up and gives good insight in how CDN performance for Google DNS and OpenDNS users compares against all users.

Performance impact for small file static object

As mentioned before, we collect performance data at Turbobytes for many CDNs using a custom RUM (Real User Monitoring) solution. In a nutshell: JavaScript code in webpages, executes after page finishes loading and then fetches a 15 KB file from one of the CDNs. The timing data gets beaconed back to our server. Our DNS tests are initiated from the same JavaScript code and that enables us to gain insight in CDN performance for users on Google DNS or OpenDNS, versus all users.

Below you see four charts, one for each of these countries: United States, Italy, Indonesia and Brazil. Each chart shows, for several CDNs, by how much median total time to download that 15 KB file improved or got worse for users on Google DNS, OpenDNS or either one of these public DNS services, versus all users.
The charts can help answer questions like:

Note: a positive percentage means performance is worse, negative means performance is better.

United States

Delta in median total loadtime for 15 kb object - Public vs other DNS resolvers - US - Oct 2012

Akamai performance degrades by 20% - 40% on both public DNS services and by that is the CDN that is most negatively impacted. Performance of Internap, EdgeCast and CloudFront for Google DNS users is much worse than for all users, while it actually improves a bit for OpenDNS users. ChinaCache needs to look into how they route OpenDNS users. Something is going horribly wrong there. With the exception of the EdgeCast/Google DNS combination, for all CDNs that do anycast for HTTP (and Fastly), performance is slightly better for the users of the two public DNS services versus all users.
We analyzed ~280,000 measurements per CDN, except for Akamai (~80,000).The lower count for Akamai does not impact the data shown here. We validated that by looking at different 80,000 sized samples for other CDNs.

Italy

Delta in median total loadtime for 15 kb object - Public vs other DNS resolvers - Italy - Oct 2012

For most CDNs, performance improves a bit, on average circa 10%. But there are three CDNs who give Google DNS and OpenDNS users a relatively bad performance: Internap, Level3 and ChinaCache. Internap shows the same picture as for US: performance is much worse for Google DNS users, while slightly better for OpenDNS users. Level3 has no issues in the US, but in Italy OpenDNS users get almost a 100% worse performance compared to all users. ChinaCache seems to have the same issue(s) as in the US: fine for Google DNS users but a whopping ~170% worse performance for OpenDNS users.
Measurements per CDN is ~30,000 (Akamai ~20,000).

Indonesia

Delta in median total loadtime for 15 kb object - Public vs other DNS resolvers - Indonesia - Oct 2012

Google DNS and OpenDNS are very popular in Indonesia, so CDNs that want to deliver good performance there should make sure all is good for users of those public DNS services. The chart clearly shows that is far from true. Let's start positive: CacheFly and Bitgravity are the only CDNs that delivers better performance to users of both public DNS services. Performance of Highwinds, CloudFront, NetDNA and Fastly is only impacted a little bit and EdgeCast and CDN77 are not so bad either. Internap, CDNetworks, Level3, ChinaCache and Akamai are the ones that need to take action, and especially Akamai and Internap. They have the biggest, negative performance delta for Google DNS users (30%+) and Google DNS is far more popular than OpenDNS in Indonesia.
Measurements per CDN is ~80,000 (Akamai ~45,000).

Brazil

Delta in median total loadtime for 15 kb object - Public vs other DNS resolvers - Brazil - Oct 2012

The situation in Brazil is ... different. There is really no CDN that delivers significantly better performance to Google DNS or OpenDNS users. Only Highwinds delivers a mere 4% better performance to Google DNS users. Highwinds has a local POP in Brazil and Google DNS has resolvers in Brazil and apparently this works out. CloudFront, Akamai, CDNetworks and CDN77 also have POP(s) in Brazil, but they deliver relatively bad performance to Google DNS users, with Akamai performance being impacted most (+57% !). EdgeCast stands out too with a ~40% worse performance for Google DNS users. That is odd, because EdgeCast does not have a POP in Brazil or in any other country in South-America. They always serve into Brazil from the US. So why is performance 40% worse for Google DNS users from Brazil? Are they serving from a different POP in the US? We don't know. CloudFront is also likely serving from US instead of from their POP in Brazil.
Measurements per CDN is ~80,000 (Akamai ~45,000).

The big table

Select a CDN from the dropdown menu to view a table with the total time median delta for Google DNS, OpenDNS or the aggregate of these public DNS services, per country. The column 'Pct' shows what % of users in that country used that DNS service and the column 'Delta' shows by how much CDN performance differs from performance all users get. By default the table is sorted on percentage of users on a public DNS service, from high to low, but you can sort on any column by clicking one of the small arrows.

Akamai

A positive percentage means performance is worse, negative means performance is better.

RegionCountOpenDNSGoogle DNSAll Public DNS
PctDeltaPctDeltaPctDelta
Turkey11,44511.94 %4.67 %26.10 %13.27 %38.03 %10.65 %
Viet Nam7,8335.34 %85.70 %27.75 %6.01 %33.09 %16.35 %
Indonesia46,0515.28 %27.91 %16.41 %23.84 %21.69 %24.97 %
Malaysia38,3822.06 %144.82 %10.07 %-32.96 %12.13 %-6.59 %
Italy20,7152.92 %7.35 %7.21 %-2.13 %10.13 %0.24 %
Thailand9,9631.89 %218.66 %7.50 %16.13 %9.38 %32.49 %
Brazil46,2121.67 %61.25 %7.65 %57.23 %9.32 %58.12 %
India31,9792.65 %96.11 %6.19 %76.42 %8.84 %81.87 %
Russian Federation12,0030.90 %32.83 %7.76 %-14.13 %8.66 %-12.46 %
Poland28,5951.06 %-7.44 %7.39 %9.93 %8.46 %7.44 %
Czech Republic6,4980.54 %-11.68 %7.88 %0.34 %8.42 %0.34 %
Spain20,2860.88 %28.87 %6.66 %21.78 %7.54 %22.31 %
Mexico45,6150.67 %40.18 %6.42 %83.71 %7.10 %77.01 %
Netherlands14,3471.98 %-3.21 %5.04 %20.18 %7.02 %13.30 %
United States (Illinois)9,9692.73 %56.39 %4.21 %129.07 %6.94 %105.29 %
United States (Michigan)6,0403.56 %41.94 %3.33 %104.84 %6.89 %80.65 %
Canada (Quebec)11,5853.79 %90.91 %3.05 %42.15 %6.84 %75.62 %
Argentina11,7081.39 %56.50 %5.36 %48.23 %6.75 %49.41 %
Canada (Ontario)29,5663.73 %114.95 %2.95 %97.20 %6.68 %107.48 %
Morocco8,0480.48 %-1.78 %6.10 %46.96 %6.59 %44.44 %

In summary

Many people use the Google DNS or OpenDNS services. From our data the combined 'market share' is ~8% globally, and in some (big) countries much higher. Both public DNS services allow CDNs to have the edns-client-subnet be passed on to them, so they can determine with more accuracy what the geolocation of the end-user is. In consequence, the CDNs can provide a better user experience by serving content from a POP closeby over a low-latency connection. We listed 9 CDNs that would benefit from supporting/using edns-client-subnet, and only two actually support edns-client-subnet: CDN77 and ChinaCache. Others, including Akamai, Internap and CDNetworks, do not currently. This really is too bad, because from the performance data we collected, it is clear these CDNs deliver (much) worse performance currently in many countries to Google DNS and OpenDNS users. We believe performance can greatly improve if they would support edns-client-subnet. But edns-client-subnet is not a magic bullet. The performance data clearly shows that some non-anycast CDNs (ChinaCache) need to do more and in some countries the anycast CDNs are not servicing the public DNS users as well as they should (EdgeCast). Our final message is to Google DNS and OpenDNS: please make an effort to publish an uptodate map/list of the resolver locations, so CDNs can use that to keep their systems uptodate. Even better would be to proactively inform them of location changes. Maybe you do this already, but ... please crank it up a notch. Txs.