What can we do to actually fix Verizon's Netflix throttling?

[bbinnard]

Obviously there are lots of folks (including me, located in the Los Angeles area) who are furious about Verizon's unacceptably poor delivery of Netflix content.  There are either numerous pauses waiting for buffering or reduced resolution due to Netflix increasing compression ratios to reduce network traffic.  Neither of these are what we are paying for, and both are unacceptable.

This chart

http://ispspeedindex.netflix.com/results/usa/graph

clearly shows Verizon's steadily decreasing performance delivering Netflix content.  At this rate the delivery speed will drop to zero in a few more months.  Abd we are expected to continue paying for this?

So far, despite countless postings here about this issue, nothing has been done or appears to be planned to be done.  So what can we, the customers and consumers, do to cause some positive action?  

How about some sort of petition sent to the FCC ( but addressed to whom or what department?)  Or perhaps we flood the head of Verizon (who is that?) with emails or actual letters.  Or how about letters to the editors of major newspapers and/or wire services?  How about emails to all the popular PC & Internet oriented magazines & blogs?

My sense is that the only way to actually make something happen to fix this situation is to raise enough attention that Verizon management realizes it is in the company's best interest to stop stonewalling and instead actually do something.

To borrow a quote from Paddy Chayefsky: "I'm mad as hell and I'm not going to take it anymore!"

[thetick1234]

You can get your local cable company if they are higher on the Netflix buddies list. 

[bbinnard]

thetick1234:

Yeah, I'm fortinate that in my area there is an alternative to FIOS - Cox Cable, which, surprisingly enough, is near the top of the Netflix speed chart.  However, switching to Cox does not do anything to address the basic problem and therefore seems like admitting defeat right out of the box.

chomper87:

I appreciate all the info you provided, but I think you are missing the point.  Granted, I should have said my FIOS plan is 35/35, more than enough bandwidth to handle anything Netflix throws out, and when watching Netflix I have only one device online - a hardwired PS3 which also has more than enough throughput for Netflix traffic.

The point I think you have missed is that although it is possible to list a whole host of reasons why Netflix performance at the TV set is poor, these are just excuses for providing poor service.  Netflix and FIOS promise full quality HD video with 5.1 sound.  That's what they sell, and that's what we pay for.  Providing less than that means we are not getting what we pay for, or what we expect.  

The Netflix problem is so widespread, and has been mentioned in so many places (other than FIOS), that it is clearly a ubiquitous one and must have some sort of common denominator.  Encoding is not the problem - the reason Netflix creates different encoders is to accommodate different screen sizes.  The encoding is done once, the results are stored on a Netflix server, and the stored file is transmitted to the end user when requested.  

Transmitted is the key word - the end user's experience is totally dependent on whether or not the transmitted data stream can keep up with the viewing device's data requirements.  Buffering delays happen when this is not possible, and this has nothing to do with how the source material is encoded.

OpenConnect is all about throughput, not anything else.  Granted, the problem of asymmetrical bandwidth traffic is not simple, but these problems are basically related to money and not technology.  Google's gigabit network shows what you can do if you really want to.

Verizon management simply has to decide whether or not it wants to provide the throughput it promised us.  My intent is to find a way to get them to do this.

[chomper87]

No doubt, there's a widespread issue. As for the part about licensing and encoding - I was simply tring to state the obvious that if it's a 480p 1.25Mb encode for a Movie on a certain device, then there's realy no expectation that the streaming bitrate will be much higher. No, it won't impact buffering. But if I measure that network stream, I might only see a max of 2Mb.

Likewise if my device only supports 2 channel audio, well I'm not going to get 5.1 audio.

In you're setup you certainly should get the HD with 5.1.

Netflix and FiOS haven't promised anything together. You pay Verizon - they promise 35/35. You separately pay Netflix they promise HD with 5.1 (for you're PS3). There's no agreement between Verizon and Netflix. It might be unfair to you and the other consumers, but that's how it is.

My FiOS seems to be just fine to companies which actually have agreements with Verizon.

Including Level3:

http://i.imgur.com/H7x3jsi.png

Including CenturyLink:

http://i.imgur.com/iENOIjp.png

Once again, Verizon isn't being evil here or anything. Just like if I tried to download something from you're FiOS 35/35 connection. I will ONLY get a Max of 35 download even though I'm on the 75/35 package. Verizon can't control whether you want to pay for something more like the 150/65. That's up to you. Verizon isn't just going to give you a free upgrade so that I can download 75Mb from you're server. Verizon isn't going to give away free services for things which it normally charges for.

Google gigabit is not a large ISP. They are not a Network Service provider or a backbone provider or a Tier 1. Google is a Customer of companies like Level3 and Verizon.

traceroute to www.google.com (173.194.115.113), 30 hops max, 40 byte packets
 1  L100.TAMPFL-VFTTP-135.verizon-gni.net (96.254.188.1)  1.377 ms  1.118 ms  3.265 ms
 2  G0-9-2-4.TAMPFL-LCR-22.verizon-gni.net (130.81.140.62)  4.481 ms  4.901 ms  4.100 ms
 3  ae2-0.TPA01-BB-RTR2.verizon-gni.net (130.81.199.82)  7.699 ms  31.255 ms  72.756 ms
 4  0.ae4.XL2.MIA19.ALTER.NET (152.63.8.117)  10.548 ms  10.831 ms  9.965 ms
 5  0.xe-9-1-1.GW1.MIA19.ALTER.NET (152.63.84.218)  10.254 ms  10.413 ms  10.086 ms
 6  google-gw.customer.alter.net (63.65.188.38)  10.007 ms  11.720 ms *

 google-gw.customer.alter.net <-- Alter.net is Verizon.

So OpenConnect makes sense for Google. They can save money. CUT COSTS to Level3 and CUT COSTS to Verizon by doing the free OpenConnect Appliance and OpenConnect Peering.

There's no technical reason why Verizon isn't on the OpenConnect. Once again, have you also complained to Netflix?

As for DNS changs and VPN.

Using a VPN takes your traffic away from the shortest path by distance, but may be faster in cases when the path would otherwise be congested. Networks generally aren't intelligent enough to automatically route around congestion.

http://arstechnica.com/information-technology/2014/02/netflix-slow-on-verizon-or-comcast-a-vpn-might...

Okay, you do some manual rerouting. Doesn't necessarily prove or disprove anything other than this other route perfoms better right now.

Arstechnica has some info here:
http://arstechnica.com/information-technology/2013/07/why-youtube-buffers-the-secret-deals-that-make...

It's can be difficult to really pinpoint a reason.

[optivity]

"Netflix and FiOS haven't promised anything together. You pay Verizon - they promise 35/35. You separately pay Netflix they promise HD with 5.1 (for you're PS3). There's no agreement between Verizon and Netflix. It might be unfair to you and the other consumers, but that's how it is."

I am paying Verizon for 35 Mbps Internet service.

Netflix streams Super HD @ 7 Mbps.

So why can't I receive HD video from Netflix with Verizon?

[chomper87]

So why can't I receive HD video from Netflix with Verizon

The simplest explanation is Verizon and Netflix have no agreements between each other. You're part is done. You have presumably done everything you can. You aren't satisfied with Verizon / Netfix. You can try switch to another ISP and/or switch to another video streaming service like Amazon.

[VUser50]

---

@chomper87 wrote:

So why can't I receive HD video from Netflix with Verizon

The simplest explanation is Verizon and Netflix have no agreements between each other. You're part is done. You have presumably done everything you can. You aren't satisfied with Verizon / Netfix. You can try switch to another ISP and/or switch to another video streaming service like Amazon.

---

So Netflix is already paying for their bandwidth needs. But the person who they pay runs into a problem when it connects to Verizon. The actual roads are big enough, the bridge between them, however, is too narrow and there is debate over who pays. Additionally, because there is so much data coming through, Verizon wants to charge a toll.

The issue is that most customers don't have much choice in ISPs so Verizon has a unfair advantage in price disputes (especially when they partner with competing services), hence why the FCC should be involved.  

If they are currently abusing that power is a bit more debatable. The prices being charged aren't available or easy to interpret. It also depends on your views on how peering should work. Historical, yes, the person sending data pays but I think the person requesting the traffic makes more sense (even though implementation of that would be hard.) That somewhat biases me to be supportive of Netflix.

[chomper87]

---

@VUser50 wrote:

So Netflix is already paying for their bandwidth needs. But the person who they pays runs into a problem when it connects to Verizon. The actual roads are big enough, the bridge between them, however, is too narrow and there is debate over who pays. Additionally, because there is so much data coming through, Verizon wants to charge a toll.

The issue is that most customers don't have much choice in ISPs so Verizon has a unfair advantage in price disputes (especially when they partner with competing services), hence why the FCC should be involved.

---

I think as time goes on hopefully data exchange between carriers will evolve. Were moving past the historical days. Level3 has a great modern twist - they are moving to "Bit Miles"

http://www.level3.com/en/legal/ip-traffic-exchange-policy/

Verizon certainly does evil things. They lobby for monopoly control of areas. They force municipalities into agreements that prevent competition. They have a lot of power, no debate about that.

I love the idea of small ISPs or a muni ISP. Competition would be great.

[chomper87]

I love how people tend to somehow avoid any blame on Netflix OR any other number of things that could be affecting performance. This is not a Net Neutrality, FCC, or a throttling issue. You haven't listed any of you're specs or you're FiOS package. How do we know that YOU aren't on the FiOS 3 Mb down / 1 Mb Up package? Then that would be YOUR problem. Do we know how many FiOS 3 / 1 subscribers Netflix includes in their stats? NO. How about how many that are on the FiOS 5 / 2 package. NO.

OTHER FACTORS besides BLAME VERIZON include:

Wired vs wireless

Cat5e vs MoCA

You're equipment including computer, OS, software, router.

Other traffic on you're equipment while attempting to stream Netflix

Squirrels. Yes the tiny animal that apparently likes to chew on fiber cables.

http://blog.level3.com/culture/what-we-learned-from-you-on-squirrels-and-fiber-cuts/

http://blog.level3.com/level-3-network/the-10-most-bizarre-and-annoying-causes-of-fiber-cuts/

Other than Squirrels, the other 9 items listed by Level 3.

The NETFLIX device you use - how many are there? A quick overview here: http://en.wikipedia.org/wiki/List_of_Netflix_compatible_devices

I sure bet they all perform exactly the same and perform at a super high performance. Oh whats that, no they don't perform the same. Some only do 720p. Some might not even do 720p. Some are only stereo instead of 5.1 audio. Some are HTML5, some are Silverlight, some have a native app.

What about if you have an XBOX. You have to pay you're ISP, Netflix and XBOX Live Gold. What a bunch of nonsense.

NETFLIX encoding and source material. How many ways can they encode a single video?

http://gigaom.com/2012/12/18/netflix-encoding/

I'm sure glad that sounds simple. It wouldn't have anything to do with the 900 fragmented Netflix devices out there, right?

Xboxes, iPads, connected TVs: Netflix streams to a lot of different devices. More than 900, to be precise. And many of them have different screen sizes, bitrate requirements and codec support. That’s why Netflix is doing a whole lot of encoding

What license to they have for to stream a video? 480p, 720p?

BTW there's 1000 Netflix Android variations alone.

To put device diversity in context, we see almost around 1000 different devices streaming Netflix on Android every day.

http://techblog.netflix.com/2012/03/testing-netflix-on-android.html

Netflix data exchange.

Netflix has Open Connect Peering and a Cache server (Open Connect Appliance). They pay an ISP ABC. They probably still pay for a another CDN XYZ. Super. Now what about that agreement between ISP ABC, CDN XYZ with Verizon.

Here's what their Open Connect letter looks like to Verizon: Dear Verizon, we offer for free, these services. Information on our Open Connect Peering and Open Connect Appliance.

Carbon Copy: ATT, CenturyLink

Verizon's responce: What. LOL? Huh? We aren't really in the BUSINESS of providing FREE services. You can pay us to put YOURE appliance in OUR datacenters, central offices, for colocation etc. You can also pay us for transit services or other data services.

Which ISPs benefit from Netflix Open Connect? Well certainly the smaller ones. Rather than Small ISP LLC pay for a 10 Gigabit connection to a Large ISP LIKE Verizon, they will end up saving money by using Netflix's Open Connect Appliance and potentially the Peering. So instead paying Verizon for that 10 Gigabit connection, they now only have to pay Verizon for a 5 Gigabit connection.

Perhaps you should send some messages to Netflix and point out some of Verizon's services that Netflix could PAY for

Verizon offers a number of readily available solutions for interconnecting providers who send significantly more traffic than they receive from Verizon’s networks. Solutions such as cloud, hosting, Partner Ports and others are designed specifically to provide a cost-effective means of delivering very large volumes of out-of-balance traffic. Other large streaming video providers (and/or network service providers carrying such one-way traffic) are already taking advantage of these solutions and seeing immediate benefits. These solutions are available today to Cogent, Netflix and any other content or network service provider with similar traffic profiles.

http://publicpolicy.verizon.com/blog/entry/unbalanced-peering-and-the-real-story-behind-the-verizon-...

http://www.terremark.com/services/it-infrastructure/

Having said all of that, it would be great for - us - the consumer of FiOS if Verizon used Open Connected, but as shown above, it's not going to happen because.

As for some of the news media on this topic really hit the fan when Netflix and Comcast had their agreement. Some of them are so full of nonsense. They have no idea who Verizon is outside of DSL - FiOS - Wireless, or who Level3 is.

[SoNi67]

@chomper87

We are not stupid, stop serving us the Verizon coolaide.

Easy test:

Every evening, my Netflix slowes down to 350-500kbps (I can measure that in IE using Silverlight). Quality is execrable, below SD.

Switching to a VPN provider (like unlocator.com, requires just a change in my DNS server address) brings the speed to 3000kbps INSTANTLY!

So "hiding" the endpoint from Verizon solves the "congestion". How is that not the Verizon's premeditated fault?

[santosSoCal]

First of all, thank you for the abundant information you have provided.

Yes, I agree, the networking environment can be very complicated.

You seemed rather well versed on Verizon's position and service capabilities

but the issue is what kind of service is Verizon able to deliver to me.

I spoke with Version tech support and they verified that the speeds to/from my residence are

in compliance with the service level I pay for.

So, the conclusion the support technician and I reached was that, yes,  Netflix is being throttled.

The poor guy knows that this is a political/business hot potato, but couldn't offer any help.

Although you listed the variety of services that Verizon offers.  Do you happen to know which services

Netflix already subscribes to?

I, as a consumer, do not want to be pawn with the two heavyweights.

[Donnnie]

I'm by no means an expert on this particular issue, but I can say this. Months ago I checked out Netflix's diagnostic video ("Example Short 23.976") which shows in real time your Bitrate, Resolution, and par , and it took less than a minute to reach 1920x1080. Now, the same test rarely gets beyond 720x480, why?

[VUser50]

This may soon be a moot point. Verizon has indicated that they are talking with Netflix about a deal similar to Comcast's where Netflix will pay for a direct pipe to the network. (and while easily unrelated, I've personally seen some improvements.) Yay ISP victory!

While Verizon does have an incentive to charge Netflix high (what is high is debatable) peering/connection prices, at the end of the day unless they are super ridiculous Netflix can pay. It is more of an issue with new smaller players. In fact, Netflix may actually see a benefit to high prices since they set a barrier to entry for new streaming services.

If you still are concerned enough to write, you should write the chairmen whoose emails are on this page. Instead of writing that they should stop throttling, which is a kind of basic phrasing/ a simplistic reduction of the issue, you might want to say they should look into ways of regulating the peering and connection agreements set by the ISPs. Even people who may be sympathetic to Verizon in the Netflix dispute should see the fundamental issue with a monopoly (or often part of a duopoloy) being free to set the prices for streaming competitors. 

[bbinnard]

VUser50:

I saw that announcement about Verizon joining OpenConnect, but so far it is just an annouincement and the skeptic in me wonders if that's all it will ever be.  I guess time will tell.

Remember, Verizon was the company that got the Net Neutrality decision nullified.

In the meantime, here is an article that describes the problem quite nicely:

http://arstechnica.com/information-technology/2014/02/netflix-packets-being-dropped-every-day-becaus...

And here is what seems to be a simple/cheap/free way to work around the problem:

http://thevpn.guru/vpn-vs-smart-dns-proxy-unblock-location/

I think I might try the latter approach of specifying an alternate DNS server pair on my PS3.

But I wonder - what if there were a petition with 2,000 (or some other number) of names sent to the Verizon CEO, and what if that same petition were published in national newspapers and news websites?  Might that not create enough social pressure to force a change that actually worked?

[SoNi67]

Change the DNS setting in your router or in the computer/TV that you use for Netflix. Instead of Automatic (which brings Verizon DNS servers), use a service that "goes around" Verizon obvious blockage:

https://unlocator.com/

It is a free (for now, beta stage) service.

[optivity]

Nothing much other than to express our dissatisfaction with the only means Verizon cares about: cut service options or switch providers.

The Internet is rapidly devolving to an environment that is ruled by the political oligarchy that is bought and paid for by a few large corporations (e.g. Verizon, Comcast/Time Warner, etc.). With the demise of  "Net Neutrality," Verizon now dictates what services they will provide by determining throughput speeds across their network.

Verizon has intentionally harmed its subscriber base by:

1) Litigating and successfully having the courts to overturn the FCC's Net Neutrality rule.

2) Allowing their peer networks between CDNs (e.g. Cogent) to become so congested we can no longer receive HD video streams from Netflix.

Netflix provides a service that directly competes with Verizon's Red Box Instant and PPV offerings.

Verizon CEO Lowell McAdam pays its customers lip service by saying he will reach an agreement with Netflix "soon" (sic) however, given Verizon comprises about 6% of the CATV and 20% of the broadband markets, they are in no position to compel Netflix to adhere to Verizon's terms to restore HD video streaming w/FiOS.

Netflix has offered Verizon a "free" solution to mitigate the congestion between CDN's that Comcast recently agreed to implement while Verizon has pa$$ed on. What does this tell us regarding Verizon's desire to resolve the Netflix HD video streaming debacle?

"Netflix Open Connect Content Delivery Network

ISPs can directly connect their networks to Open Connect for free. ISPs can do this either by free peering with us at common Internet exchanges, or can save even more transit costs by putting our free storage appliances in or near their network."

[chomper87]

Netflix has offered Verizon a "free" solution to mitigate the congestion between CDN's that Comcast recently agreed to implement while Verizon has pa$$ed on. What does this tell us regarding Verizon's desire to resolve the Netflix HD video streaming debacle?

It tells us that Verizon is not in the BUSINESS of providing FREE services. Comcast and Netflix didn't agree to for it to be free. OpenConnect is a fine idea. I'm not arguing that. From Verizon's point of view it doesn't make sense. Same as ATT, CenturyLink.

[SoNi67]

What FREE service {please keep your posts courteous}?

We are PAYING for our services. We are customers. Verizon promised us 50MBPS connection to the INTERNET, not just to a speedtest site!

They don't deliver 3MBPS with netflix, but when I hide the endpoint suddenly that's not an issue anymore!

Obiously is not Netflix fault, since I can get that speed using a VPN, is obviously Verizon's fault.

I don't care why they cannot provide what they promise, but this is false advertising.

[chomper87]

What FREE service

 That's in reference to Netflix OpenConnect.

Verizon promised us 50MBPS connection to the INTERNET

I agree ISP marketing should be more explicit. But i'm sure ISPs for simplicity market packages as Internet service. You don't get 50Mbps to the Internet. You get 50Mbps to Verizon. Verizon is just but one of many networks.

not just to a speedtest site

I see, well the Internet makes no sense without companies like Level3 and CenturyLink. These aren't just "some random speed tests".

They don't deliver 3MBPS with netflix

 Who doesn't deliver? Verizon? Netflix?

since I can get that speed using a VPN, is obviously Verizon's fault

Not quite. Okay so you have a good VPN. Well perhaps that VPN is on Netflix OpenConnect. You've rerouted around some congestion

[SoNi67]

---

@chomper87 wrote:

Not quite. Okay so you have a good VPN. Well perhaps that VPN is on Netflix OpenConnect. You've rerouted around some congestion

---

I don't have a "good VPN". That specific  VPN does not provide internet service, it relies on my existing connection.

If I can reach the destination with faster speeds  when they "hide" my destinaton from Verizon, it is obviously a foul play from Verizon.

Otherwise you claim that a free beta service has a better pipe than Verizon. This is just not true.

They provide VPN service for everyone in the world. Do you think that they have such an internet connection that can handle ALL the traffic? No, they just change the headers of the packets for me, so my provider cannot "snoop" my destination and apply their traffic shaping.

[chomper87]

---

@SoNi67 wrote:

it relies on my existing connection.

If I can reach the destination with faster speeds  when they "hide" my destinaton from Verizon, it is obviously a foul play from Verizon.

---

I think you mentioned Unlocator somewhere? Unloacter is not VPN, it is a DNS based.

Some users have Unlocator confused with a VPN service, which we are not. We are a DNS service and you don’t get a new IP address when using Unlocator (like you would with VPN)

Your traffic remains unchanged. Encrypted traffic remains encrypted and unencrypted remain unencrypted

https://support.unlocator.com/customer/portal/articles/1463062-difference-between-unlocator-and-vpn

Unless I've missed something, its just a DNS gizmo. Instead of the default directing Netflix traffic to Miami or whatever, it somewhat fakes you're location so that you are directed to a server in Atlanta or a server in Dallas. Etc. Verizon could still potentially detect that you are connecting to a Netflix server.