President Obama is now on the record stating that internet providers have “a legal obligation not to block or limit your access to a website.” We agree and want to thank the President for following in our footsteps by telling the world that throttling is bad. Backblaze itself does not actively throttle our customers. The President went on to say, “In plain English, I’m asking [the FCC] to recognize that for most Americans, the internet has become an essential part of everyday communication and everyday life.” We couldn’t agree more, and wholeheartedly wish for increased bandwidth to all our users.
No Net Neutrality Bites Backblaze
Backblaze and by proxy, our customers, were victimized by the absence of net neutrality during the Comcast-Netflix war from November 2013 through February 2014. Here’s what went down. Beginning in late November 2013, a small number of our customers started reporting slow data restores and downloads from a Backblaze data center to their computer. Upon investing their claims, we examined our internal network, reconfigured our restore servers, and changed the caching code on our servers to “fix” the problem.
None of the changes seemed to work as the number of support calls continued to rise.
The First Smoking Gun
We continued to investigate, but the trouble was that nothing was consistent. People were using different versions of the product, different platforms, different versions of their operating systems, and they were located in different parts of the country and around the world. Digging in further, we found that there was a slow down in downloaded network traffic on our Comcast/Cogent and Comcast/XO connections. They were being severely throttled. All of our other data center network connections were fine, except those two.
The Second Smoking Gun
In early January, Backblaze analyses compared the download speeds of our Oakland data center versus our Sacramento data center over comparable 24 hour periods. The Sacramento data center is serviced by the Comcast/Cogent and Comcast/XO connections. The Oakland data center uses Abovenet and other providers.
Observe that download network traffic from the Oakland data center is nearly flat (the thick black line). In the Sacramento data center, download network traffic decreased starting at 14:00 (2 p.m. Pacific Standard Time) and continued decreasing until about 22:00 (10 p.m. PST), then recovered. This pattern continued each day.
Rerouting Around the Throttle
In mid-January we began rerouting download traffic from our Sacramento data center to our Oakland data center, then from Oakland to the outside using Abovenet. Customer download speeds from the Sacramento data center to the customer increased 10 times versus the throttled speeds we were getting previously. The reroutes were working, but we couldn’t get a good answer as to why the Comcast/Cogent and Comcast/XO connections in the Sacramento data center were “broken.”
The Final Smoking Gun
On February 23, 2014, Comcast and Netflix reach a deal on routing Netflix network traffic. On March 1st, Backblaze removed our reroutes and returned to our original network configuration prior to the war. Downloads from our Sacramento data center were now running once again over the Comcast/Cogent and Comcast/XO network connections.
The result: All downloads now occur at normal speeds, including downloads during the 2 p.m. to 8 p.m. timeframe. Once the Comcast/Netflix deal was done, it took less than a week for the problem to disappear.
TL;DR
To summarize, Comcast had a problem with the amount of traffic Netflix was using. Yet, services completely unrelated to Netflix were punished, meaning consumers and businesses—innocent bystanders—unrelated to Netflix were punished, and all this occurred without notice or explanation from Comcast. That should not be the future of the internet.
We have prepared a document titled “Findings and Analysis of Slowdowns on the Comcast Network,” which is a chronological review used to support the assertions in this post.
