Recently there has been a lot of coverage about a disagreement between Comcast and Level(3) related to carriage of streaming video from Netflix. On November 29 Level(3) issued a press release that claimed “Comcast is effectively putting up a toll booth at the borders of its broadband Internet access network, enabling it to unilaterally decide how much to charge for content which competes with its own cable TV and Xfinity delivered content.” These claims confuse peering with network neutrality. This is also an interesting clash of two competitors’ distinct strategies.
Netflix Picks Level(3) for their Content Delivery Network
In November Level(3) won a contract to become the primary Content Delivery Network (CDN) provider for Netflix (Reuters Canada Report). Previously Akamai was Netflix’s primary CDN. The switch to Level(3) is a BIG deal because Netflix streaming content can represent up to 20% of all downstream Internet traffic during primetime television watching hours. That estimation comes from this report from network policy solutions provider Sandvine issued in October.
The FCC and pundits took much interest in Level(3)’s claims. This attention comes at a sensitive time for broadband providers in general and Comcast specifically:
- The Comcast – NBC Universal merger is still under review by the FCC and the Department of Justice.
- FCC Chairman Julius Jenachowski has called a vote on a network neutrality policy for December 21.
Comcast sent a letter to the FCC (pdf) explaining their perspective on the disagreement. Comcast counterclaims “Level 3 is trying to game the process of peering – one that has worked well and consensually, without government interference, for over a decade – in order to gain a unique and unfair advantage for its own expanding CDN service.”
Everything You Could Ever Possibly Want to Know about CDNs
CDNs like Akamai, Level(3), Amazon, AT&T and Limewire deliver high-volume, delay intolerant Internet traffic faster than can be provided over a general purpose Internet backbone. Your Netflix streaming movie, your YouTube video, your friends’ photos on Facebook and your iTunes download all rely on CDNs. Generally CDNs consist of the following 3 components:
- CDN-owned Internetworking Protocol (IP)-based backbones that employ traffic prioritization technologies such as MPLS and DiffServ. These technologies are used in corporate Internet backbones to differentiate the performance and priority of various types of traffic.
- Distributed servers that duplicate content with high-volumes of downloads by end users. When you click on today’s most-read article from the New York Times or “The History of Dance” on YouTube the content is being served up by a CDN server instead of the New York Times or YouTube servers.
- Interconnection with broadband providers. The closer these interconnects are to the user requesting the content the faster and more efficiently the content can be served. These interconnects are commercial arrangements between the CDNs and broadband providers. The more interconnects a CDN has the more reliable the content delivery. These interconnects are based on standard peering agreements (see more about peering below).
The purpose behind CDNs is to distribute infrastructure to achieve efficiency and improve end users’ experience. Building a CDN is a capital expense, and there is a direct relationship between the capital deployed, network efficiency and the end user experience. The better CDNs charge more to cover the up-front capital costs of building their superior CDN and can demand a premium because of the end user experience they are able to deliver.
Being Judged by a Jury of Your Internet Backbone Peers
Most people know that the Internet is a network of networks that use the Internetworking Protocol. These include public nets, private nets, long-haul nets, metropolitan nets, local nets, fat nets, skinny nets, nets that climb on rocks…
Two nodes on the Internet do not have to be connected to the same network to exchange traffic. As users access Google, Netflix, Facebook or any other sources of content available via the Internet the traffic to and from those sources traverses multiple networks. Each “hop” comes with a (usually very minor) delay.
Networks are constantly trading traffic back and forth with one another. Networks that exchange a lot of traffic with establish private peering interconnections to facilitate those exchanges. The idea is that peers Network Provider A and Network Provider B will move roughly equivalent amounts of traffic over one another’s network. If Network Provider B puts a lot of traffic onto Network Provider A’s network and carries a much smaller amount of traffic for Network Provider A, then Network Provider B will compensate Network Provider A. Network Provider A incurs real and opportunity costs to deploy and manage additional Internet backbone capacity to carry Network Provider B’s traffic.
Peering is an industry-led balance of payments regime formulated to ensure fairness in an interconnected network infrastructure. A balance of payments for terminating international telephone calls serves as a decades-old precedent and model for this approach.
Level(3) Needs More Interconnects to Serve Netflix Content to Comcast’s Customers
Prior to their winning the Comcast traffic, Level(3) was not a leading CDN, having only acquired the relatively small CDN business from Savvis in 2006 to augment their own wholesale Internet offerings. Level(3) have not made major capital investments to their CDN by building the large number of interconnects with other Internet providers to provide a high-quality CDN. Level(3)’s Internet backbone business was so much larger than their CDN business that peer Internet providers, including Comcast, usually compensated Level(3) because of the deficit in traffic they carried for Level(3) compared to the Internet traffic Level(3) carried for them. Level(3)’s CDN was not putting anywhere near as much traffic onto Comcast’s networks as Comcast’s customers were putting on Level(3)’s backbone.
Akamai is far and away the leading CDN with network assets deployed very close to end users to minimize the number of network hops any of its customers’ content must make to reach end users. CDN is Akamai’s core business. In strategic terms, Akamai differentiate themselves by focus and differentiation based quality, while Level(3) pursue a cost leadership strategy.
When Netflix sought bids for a new CDN, Level(3) offered a low-cost alternative to Akamai. No doubt the Netflix RFP specified service level agreement (SLA) including metrics of end users’ experience. I am a Netflix customer, and often after I watch streaming content I receive an e-mail from Netflix asking me about the quality of the experience. I can only imagine that one of the metrics to which Netflix CDN partners must abide is a percentage of customer respondents confirming an excellent or acceptable viewing experience.
I do not know whether or not the Level(3) team realized the capital investment that would be required to deliver those SLAs. According to Ars Technica, Level(3) asked Comcast to eat the cost of nearly 30 additional CDN interconnects. The Comcast team agreed that 6 new interconnects would address their peering traffic deficit with Level(3). Additional interconnects and traffic would turn Comcast’s traffic deficit with Level(3) into a surplus and thus require Level(3) to compensate Comcast accordingly. It was in response to this point that Level(3) issued their press release and sought relief from the FCC by claiming that Comcast was seeking to block its customers from accessing Netflix’s streaming media services. Netflix is the lead in cutting edge services that are challenging traditional television program models such as Comcast’s cable television business. Level(3) were suggesting that Comcast was using its market power in their broadband business to protect their cable television business.
Conclusion and What’s Next
What did Level(3) hope to achieve with their complaint? I cannot say for certain. Based on the long-standing peering regime I cannot imagine that the leadership at Level(3) assumed that there was any legitimacy to their claims. There is no FCC action warranted in this circumstance, and no consideration of Level(3)’s complaint should be considered in the FCC and DoJ’s review of the Comcast merger with NBC Universal.
As I was considering this blog entry I wrote a series of tweets that outlined my arguments. Fellow Titterer Pranav Desai commented that it’s perhaps time to re-examine the peering regime and that we will see more cases like the Level(3) – Comcast conflict as “over the top” television services compete with the television services offered by cable and telecommunications providers. These are great points, and I really appreciate the discussion.
On the point of re-examining peering I can offer a definitive “maybe.” It is not for Level(3) to unilaterally re-define a decade-old industry self-governing regime. Level(3) undercut a competitor and then sought to offload the cost of delivering on their Netflix contract onto Comcast. That’s not right.
What dimensions of competition will come to play as over-the-top media services such as Netflix, Hulu, iTunes and others eat into traditional television programming offered by Comcast, Verizon (my employer) and others who provide both television programming and broadband Internet connectivity? The Netflix-Level(3)-Comcast dust-up is an example of the conflicts we might see in the coming years.