Netflix Picks Level(3) for their Content Delivery Network

In  November Level(3) won a contract to become the primary Content Delivery Network (CDN) provider for Netflix (Reuters Canada Report).  Previously Akamai was Netflix’s primary CDN.  The switch to Level(3) is a BIG deal because Netflix streaming content can represent up to 20% of all downstream Internet traffic during primetime television watching hours.  That estimation comes from this report from network policy solutions provider Sandvine issued in October.

The FCC and pundits took much interest in Level(3)’s claims.  This attention comes at a sensitive time for broadband providers in general and Comcast specifically:

• The Comcast – NBC Universal merger is still under review by the FCC and the Department of Justice.
• FCC Chairman Julius Jenachowski has called a vote on a network neutrality policy for December 21.

Comcast sent a letter to the FCC (pdf) explaining their perspective on the disagreement.  Comcast counterclaims “Level 3 is trying to game the process of peering – one that has worked well and consensually, without government interference, for over a decade – in order to gain a unique and unfair advantage for its own expanding CDN service.”

Everything You Could Ever Possibly Want to Know about CDNs

CDNs like Akamai, Level(3), Amazon, AT&T and Limewire deliver high-volume, delay intolerant Internet traffic faster than can be provided over a general purpose Internet backbone.  Your Netflix streaming movie, your YouTube video, your friends’ photos on Facebook and your iTunes download all rely on CDNs.  Generally CDNs consist of the following 3 components:

1. CDN-owned Internetworking Protocol (IP)-based backbones that employ traffic prioritization technologies such as MPLS and DiffServ.  These technologies are used in corporate Internet backbones to differentiate the performance and priority of various types of traffic.
2. Distributed servers that duplicate content with high-volumes of downloads  by end users.  When you click on today’s most-read article from the New York Times or “The History of Dance” on YouTube the content is being served up by a CDN server instead of the New York Times or YouTube servers.
3. Interconnection with broadband providers.  The closer these interconnects are to the user requesting the content the faster and more efficiently the content can be served.  These interconnects are commercial arrangements between the CDNs and broadband providers.  The more interconnects a CDN has the more reliable the content delivery.  These interconnects are based on standard peering agreements (see more about peering below).

The purpose behind CDNs is to distribute infrastructure to achieve efficiency and improve end users’ experience.  Building a CDN is a capital expense, and there is a direct relationship between the capital deployed, network efficiency and the end user experience.  The better CDNs charge more to cover the up-front capital costs of building their superior CDN and can demand a premium because of the end user experience they are able to deliver.

Being Judged by a Jury of Your Internet Backbone Peers

Most people know that the Internet is a network of networks that use the Internetworking Protocol.  These include public nets, private nets, long-haul nets, metropolitan nets, local nets, fat nets, skinny nets, nets that climb on rocks…

Two nodes on the Internet do not have to be connected to the same network to exchange traffic.  As users access Google, Netflix, Facebook or any other sources of content available via the Internet the traffic to and from those sources traverses multiple networks.  Each “hop” comes with a (usually very minor) delay.

Networks are constantly trading traffic back and forth with one another.  Networks that exchange a lot of traffic with establish private peering interconnections to facilitate those exchanges.  The idea is that peers Network Provider A and Network Provider B will move roughly equivalent amounts of traffic over one another’s network.  If Network Provider B puts a lot of traffic onto Network Provider A’s network and carries a much smaller amount of traffic for Network Provider A, then Network Provider B will compensate Network Provider A.  Network Provider A incurs real and opportunity costs to deploy and manage additional Internet backbone capacity to carry Network Provider B’s traffic.

Peering is an industry-led balance of payments regime formulated to ensure fairness in an interconnected network infrastructure.  A balance of payments for terminating international telephone calls serves as a decades-old precedent and model for this approach.

Level(3) Needs More Interconnects to Serve Netflix Content to Comcast’s Customers

Prior to their winning the Comcast traffic, Level(3) was not a leading CDN, having only acquired the relatively small CDN business from Savvis in 2006 to augment their own wholesale Internet offerings.  Level(3) have not made major capital investments to their CDN by building the large number of interconnects with other Internet providers to provide a high-quality CDN.  Level(3)’s Internet backbone business was so much larger than their CDN business that peer Internet providers, including Comcast, usually compensated Level(3) because of the deficit in traffic they carried for Level(3) compared to the Internet traffic Level(3) carried for them.  Level(3)’s CDN was not putting anywhere near as much traffic onto Comcast’s networks as Comcast’s customers were putting on Level(3)’s backbone.

Akamai is far and away the leading CDN with network assets deployed very close to end users to minimize the number of network hops any of its customers’ content must make to reach end users.  CDN is Akamai’s core business.  In strategic terms, Akamai differentiate themselves by focus and differentiation based quality, while Level(3) pursue a cost leadership strategy.

When Netflix sought bids for a new CDN, Level(3) offered a low-cost alternative to Akamai.  No doubt the Netflix RFP specified service level agreement (SLA) including metrics of end users’ experience.  I am a Netflix customer, and often after I watch streaming content I receive an e-mail from Netflix asking me about the quality of the experience.  I can only imagine that one of the metrics to which Netflix CDN partners must abide is a percentage of customer respondents confirming an excellent or acceptable viewing experience.

I do not know whether or not the Level(3) team realized the capital investment that would be required to deliver those SLAs.  According to Ars Technica, Level(3) asked Comcast to eat the cost of nearly 30 additional CDN interconnects.  The Comcast team agreed that 6 new interconnects would address their peering traffic deficit with Level(3).  Additional interconnects and traffic would turn Comcast’s traffic deficit with Level(3) into a surplus and thus require Level(3) to compensate Comcast accordingly.  It was in response to this point that Level(3) issued their press release and sought relief from the FCC by claiming that Comcast was seeking to block its customers from accessing Netflix’s streaming media services.  Netflix is the lead in cutting edge services that are challenging traditional television program models such as Comcast’s cable television business.  Level(3) were suggesting that Comcast was using its market power in their broadband business to protect their cable television business.

Conclusion and What’s Next

What did Level(3) hope to achieve with their complaint?  I cannot say for certain.  Based on the long-standing peering regime I cannot imagine that the leadership at Level(3) assumed that there was any legitimacy to their claims.  There is no FCC action warranted in this circumstance, and no consideration of Level(3)’s complaint should be considered in the FCC and DoJ’s review of the Comcast merger with NBC Universal.

As I was considering this blog entry I wrote a series of tweets that outlined my arguments.  Fellow Titterer Pranav Desai commented that it’s perhaps time to re-examine the peering regime and that we will see more cases like the Level(3) – Comcast conflict as “over the top” television services compete with the television services offered by cable and telecommunications providers.  These are great points, and I really appreciate the discussion.

On the point of re-examining peering I can offer a definitive “maybe.”  It is not for Level(3) to unilaterally re-define a decade-old industry self-governing regime.  Level(3) undercut a competitor and then sought to offload the cost of delivering on their Netflix contract onto Comcast.  That’s not right.

What dimensions of competition will come to play as over-the-top media services such as Netflix, Hulu, iTunes and others eat into traditional television programming offered by Comcast, Verizon (my employer) and others who provide both television programming and broadband Internet connectivity?  The Netflix-Level(3)-Comcast dust-up is an example of the conflicts we might see in the coming years.

My Verizon Wireless MiFi, next to standard business card for size comparison.

If you’ve been reading this blog for any period of time you have no doubt concluded that I am a gadget hound and tech nerd.  Would it surprise you to learn that when I travel for business I usually travel with two laptops?  It’s a sickness, I know.  I’m a firm believer in keeping my work work on my work PC (an HP) and my personal material and projects on my MacBook Pro.  In most hotels this has resulted in a life-or-death decision of which computer will be registered for the (usually expensive and slow) hotel broadband connection.  In a few instances in the past I was able to use my Apple Airport Express to connect multiple devices to the hotel broadband.  In those instances the connection is generally very slow, and often I find myself unable to use my VPN to connect to my work e-mail and other intranet resources.  Bummer.

Even when I bite the bullet and decide that my work PC will be the sole digital link to the outside world performance is inconsistent.  In my years of travel I have found it amazing how many times my VPN did not work.  It still also amazes me how many hotels still treat broadband as an optional amenity.  It’s not– a hotel room might as well not have electricity or running water.  The net is central to how I and many others live our lives today and is non-negotiable if I am going to remain a productive employee while on the road.  It also still amazed me the rates hotels charge for Internet connectivity: $12.95/day seems to be the standard.  You don’t learn the VPN won’t function until after you’ve connected the work laptop and tried to connect.  Want to cut bait and just use the personal machine?  That’s another $12.50, please.  Don’t even THINK of connecting your iPhone or other wifi-enabled smart phone (you can generally fall back on your usable if somewhat slow 3G connection there).  Coffee shops and other hotspots also have spotty, inconsistent support for VPN connectivity.  A lot of productivity has been lost struggling to get a VPN connection only to give up and just resign myself to days of catching up on e-mail and other tasks when next I am at home or in the office.  To put it mildly, connectivity when traveling sucks.

I don’t have to worry about that anymore.  Since I bought my MiFi I’ve had one day of meetings outside of the office and one business trip.  On my day running around the Washington metro area I was able to use my down time to great productive effect   The VPN works flawlessly every time.  I can connect my PC, Macintosh and iPhone all to a blazing fast (in wireless terms) network with great coverage.  On my recent business trip I did a speed check to find that I was getting 1097 Kbps down and 652 Kbps up.  While it’s not as fast as my FiOS connection at home (I will limit myself to pimping one Verizon product in this post) it’s faster than most hotel broadband connections.

I’m not the only one who loves this devices and have made productive use of the MiFi.  Andy Abramson of VoIP Watch and Bob Gourley of CTOVision have both sung the praises of their MiFis.  Guy Kawasaki made a great post to the American Express OPEN for Small Business blog highlighting some valuable use cases for his Sprint MiFi, and some relevant to people who are not afflicted with my tech nerdery:

• In your hotel room
• Traveling with kids
• MacBook Air, iPhone and iPod Touch owners
• Smartphone users using VoIP such as Skype
• Making a sales pitch when you need a reliable and fast Internet connection
• Conference attendance (often wifi at a conference is either completely unavailable or an additional daily expense.  Now you can even use the MiFi”s support for multiple connections to make friends and influence people).
• Speaking or presenting when you need an Internet connection (a requirement I can say with experience many venues are challenged to provide reliably)
• Alternative to tethering your computer with a mobile phone

One challenge I have had with the MiFi is maintaining a charge on the device.  On my recent business trip I learned that my MiFi as well as a few other devices that charge via USB do not like my Belkin travel surge suppressor.  This is a handy three-outlet surge suppressor that also has two USB ports to charge devices without the need for additional power adapters.  This I think is a problem more of the Belkin than the MiFi, because my iPhone also would not take a charge from this device.  So the one cautionary advice I would offer is that travelers should take the MiFi’s power adapter on the road with them just to be safe.  The MiFi can be charged via USB from your computer, and I found this to be somewhat idiosyncratic and felt that the MiFI didn’t get the full charge it does when plugged in directly to an outlet.

Overall this is a device I strongly recommend.  The retail price is competitive with standard 3G adapters for laptops (that only support connectivity for that single device).  You barely have to travel more than once a month to make the $60 monthly price (5 Gigabytes cap) more cost-effective than paying for daily connectivity at hotels, in airports or coffee shops.  The MiFi has already paid for itself this month and kept me happier and more productive in the bargain.

With Apple, Amazon.com, NetFlix and Microsoft pushing downloadable movies and cable and phone companies peddling a plethora of on-demand, high-definition content, the day is coming when the stacks of plain vanilla DVDs that clutter many home entertainment centers will go the way of the CD collection.

The increase in bandwidth and hard drive capacity in particular are brining us closer to a reality where all of our content comes down the pipe.  I haven’t bought a CD in ages, and the only CDs I’ve bought in the last few years are for music that I was unable to find on any legitimate (or otherwise) digital store.

What trends could make me be wrong about this?  For starters I could be drastically over-estimating the rate at which high-bandwidth penetration will reach the necessary tipping point.  There’s also always the possibility that a “legacy” technology will continue to have takers within specific populations well into the future.  People still use dial-up where there is no broadband or for redundancy.  Portable CD players are still in use among groups that cannot afford the infrastructure one needs to use MP3 players (you need a computer and an MP3 player and sufficient computer literacy to use them together at the bare minimum).

Within the mainstream, though, I really do think the HD-DVD vs, Blu-Ray standards war is much less relevant than the video content standards wars that went before it.

AT&T Inc. announced Monday that subscribers to its higher-speed broadband services will have free access to its Wi-Fi hotspots throughout the country.

Here’s an article with more detail from CNet.

Kudos to AT&T for finding a service differentiator to cable-based broadband in their service territory. Broadband competitors in AT&T-served markets will find it difficult to counter this. AT&T DSL customers who have iPhones should be particularly happy because now they can use their Wi-Fi functionality of their iPhones in locations where AT&T offers Wi-Fi hotspot service.

Technorati Tags: Competitive Intelligence, Telecom.

“Imagine accessing YouTube.com and MySpace.com literally on the fly,” Sprint Chief Executive Gary Forsee told a news conference, referring to the popular online video and social networking sites.

Imagine that– a telecommunications executive not bitching about how people aren’t paying enough for his pipes but rather being delighted at the notion of delivering connectivity and value to customers.

Some commentators have questions the ability for Sprint to make a profit following such a capital investment, but I have to contrast this effort with some efforts to bypass the local exchange carriers in the past. Specifically I’m contrasting this $4 billion in capital investment to the around $100 billion pre-SBC AT&T spent to buy cable companies TCI and MediaOne in the late 1990s. At the time I actually thought AT&T’s strategy would pay off, and I think Wall Street unduly beat up on then AT&T Chairman C. Michael Armstrong, forcing him to tear down this strategy. Thanks to the progressive march of technology Sprint will get to bypass the ILECs for around 4 percent of what AT&T once spent. If Sprint can market the service well I think they’ve got a good shot at making this profitable in short order.

Also, keep in neutral, Sprint. Stay classy, San Diego.

Tags: Sprint, WiMax

I came across this piece from a website called the Loudoun Connection, a source Id never even known existed before now.

I was surprised to read the article and learn that one of the options the county government was considering was the possibility of purchasing the franchise from Adelphia. This is evidently an option that becomes an option for the government as a result of a county ordinance should the ownership of the company holding the franchise change hands.

Adelphia are claiming that the ordinance is not applicable as a result of their bankruptcy filing. This sounds like an interpretive stretch to me, and I fail to see the basis for Adelphias claim here. I know Im not a lawyer, but this smacks to me as another potential example of the abuse of bankruptcy protection as a management tool.

Even though the Adelphia claim has no merit I can see, the consideration that the county would take over ownership of the cable system as a public utility sounds like an awful option to me. Ever since moving to Loudoun Ive had the feeling that Adelphia is one step above a fly-by-night operation in terms of their infrastructure, customer service and product offerings. It took them forever to roll out broadband, their customer service (or lack thereof) makes it clear they have no real idea what theyre doing and the poor response to technical problems with broadband makes me wonder what their level of technical expertise must be. Since the news of the pending sale to Comcast and Time Warner Ive been counting the days until a company that can only improve our situation in Loudoun County takes ownership of the system.

I dont think the county government would improve the situation. Ive got a strong suspicion that the county doesnt know how to manage an Internet service, and I cant imagine the county being willing to make the investments which very clearly need to be made. The article I found goes back and forth as to whether or not the threat to buy back the system from Adelphia is a simply a tactic to raise the franchise fees or if the county is actually serious about taking over the system and turning cable television and broadband into a public utility. I sincerely hope its simply a negotiation tactic.

While I agreed with most of what Richard had to say, I didn’t necessarily follow or agree with this one comment:

There was only one displeasing issue pointed out for me by Brough Turner in his presentation: the telcos are squeezed also from the lower layer (layer 0 is he named it). The access is also not a safe heaven in future, because there is (will be) competition here too by independent FTTH providers. They will lease, rent or sell the fiber to the customer. And if the customer has finally a fiber to the home, he is set. The currently existing bandwidth gap is solved. And the customer now may choose his ISP at the co-location room.

I would really like to believe this is the model that will prevail. I still have significant concern, however, of the difficulty of a market for true facility-based competition for last mile services. This is not that I think the market cannot be there. Quite the opposite. My supposition is still that local authorities are going to be reluctant to offer up rights of way to competitive providers who are not incumbent telcos, cable companies or maybe (if they’re generous) utility companies. My read of the new telecom bill reflects this worry.

My own telecom experience didn’t lead me to think too, too much of this at the time. The original job posting looks pretty standard for a company with heavy data hosting requirements that is going to need lots of hosting space and bandwidth capacity to support a data-intensive platform such as Google. Considering the volumes of information they’re dealing with, dark fiber seems like a perfectly cost-effective path to take. The per-unit cost of dark fiber is significantly less than leased capacity, but only the highest-volume customers can make it cost effective. Likewise, Google put so much information onto the Internet backbone, that they must pay a lot in interconnect charges. As Google begin to offer more and more high-bandwidth content such as audio and video, it would become a significant expense to their bottom line. Using dark fiber and effectively building their own Internet backbone would help them better control these costs.

Om speculates that one possibility is that Goolgle could build out a nationwide Wi-Fi network. This is the bit of his article which is likely to get the most attention. The business model would be a mix of subscription access and location-based advertising. I’m not sure dark fiber purchases lend themselves to this end, because there would be an awful lot of local access connectivity to individual Wi-Fi locations which would be required. However, Om may have better insider information than I: Business 2.0 has learned from telecom insiders that Google is already building such a network, though ostensibly for many reasons.

The Wi-Fi thing doesnt make sense to me. Wi-Max maybe. But I doubt Google would want to deal with all of the local access issues involved in a major Wi-Fi build out (dark fiber wont help you here) nor the negotiations with owners of the venues in which Wi-Fi would be made available.

A reason which makes much more sense to me, and that Om outlays in his article as well, is that taking a dark fiber approach helps Google eliminate the expense of putting large amounts of information onto ISP networks. Om writes:

An even more compelling reason for Google to build its own network is that it could save the company millions of dollars a month. Here’s why: Every time a user performs a search on Google, the data is transmitted over a network owned by an ISP — say, Comcast (CMCSK) — which links up with Google’s servers via a wholesaler like AboveNet. When AboveNet bridges that gap between Google and Comcast, Google has to pay as much as $60 per megabit in IP transit fees. As Google adds bandwidth-intensive services, those costs will increase. Big networks owned by the likes of AT&T (T) get around transit fees by striking “peering” arrangements, in which the networks swap traffic and no money is exchanged. By cutting out middlemen like AboveNet, Google could share traffic directly with ISPs to avoid fees.

This makes much more sense to me, and fits in better with Google’s current content-oriented strategy.

From the latter bill:
No State statute, regulation, or other State legal requirement may prohibit or have the effect of prohibiting any public provider from providing, to any person or any public or private entity, advanced telecommunications capability or any service that utilizes the advanced telecommunications capability provided by such provider.

This bit seems to run counter to a section of the Broadband Investment and Consumer Choice Act, as quoted here from the summary of the bill as linked:

A state or local government seeking to provide a communications service must give notice of and permit non-government entities to bid to provide such service preference is to be given to non-government entities. Existing government owned network offerings are grandfathered.

While the two clauses do not directly contradict one another, they seem to be in a somewhat different spirit. The former makes it clear that governments (most likely at the state level) will not prohibit municipalities from building their own communications infrastructure. The latter, while fine in concept, would likely open up an opportunity for commercial organizations to tie up community efforts in court battles. Hopefully the full text of the bill (I have not been able to find it as of yet) will address what constitutes a community having done their due diligence in giving commercial firms a right of first refusal.

In principal I do believe commercial organizations are better suited to provide communications services, and using tax dollars to fund the build-out of infrastructure is a gray area. However, with commercial companies such as the incumbent telcos and cable television providers picking and choosing the neighborhoods in which to build out advanced services, you cant help but encourage cities to build out their own offerings such as municipal Wi-Fi. However, if cities were to open up their rights of way, franchises and zoning in order to facilitate a more competitive last mile infrastructure build, they would make it easier for competing low-cost commercial firms to offer services in underserved areas. There is a certain degree to which the municipalities themselves create the problem they then must then try to solve.

Theres always the hope that Wi-Max will make this entire discussion irrelevant.

We should all probably expect to see more of this as FTTP buildouts progress. Ugly, ugly trenches dug along the front of each of our yards and residential streets torn assunder. Ah well… such is the price to pay for wicked fast broadband.