What can we do to actually fix Verizon's Netflix throttling?
bbinnard
Contributor - Level 3

Obviously there are lots of folks (including me, located in the Los Angeles area) who are furious about Verizon's unacceptably poor delivery of Netflix content.  There are either numerous pauses waiting for buffering or reduced resolution due to Netflix increasing compression ratios to reduce network traffic.  Neither of these are what we are paying for, and both are unacceptable.

This chart

http://ispspeedindex.netflix.com/results/usa/graph

clearly shows Verizon's steadily decreasing performance delivering Netflix content.  At this rate the delivery speed will drop to zero in a few more months.  Abd we are expected to continue paying for this?

So far, despite countless postings here about this issue, nothing has been done or appears to be planned to be done.  So what can we, the customers and consumers, do to cause some positive action?  

How about some sort of petition sent to the FCC ( but addressed to whom or what department?)  Or perhaps we flood the head of Verizon (who is that?) with emails or actual letters.  Or how about letters to the editors of major newspapers and/or wire services?  How about emails to all the popular PC & Internet oriented magazines & blogs?

My sense is that the only way to actually make something happen to fix this situation is to raise enough attention that Verizon management realizes it is in the company's best interest to stop stonewalling and instead actually do something.

To borrow a quote from Paddy Chayefsky: "I'm mad as hell and I'm not going to take it anymore!"

0 Likes
63 Replies
sac3528
Newbie
sorry but you're just wrong. On FiOS I can't use netflix at all. On FiOS however, I am NOT paying by the Gigabyte. I am paying by the gb on my lte plan with Verizon and there I see no bandwidth throttling. Same device, same content, different result.
0 Likes
nick_d
Enthusiast - Level 3
I don't understand the part about level 3 sending more traffic to Verizon than Verizon sends to them. Umm, isn't that the case for any streaming service? The user doesn't need to upload near the amount of data that they download.
I think if Verizon didn't **bleep** off every customer with abysmal customer service and nickel and dime tactics (see being able to watch shows off one dvr in the house to other STBs in the house at the rate of $20 per month. On top
of the cable box rental fee. They charge more than they would lose when looking at the cost of a standard box vs dvr) they may have more of the public on their side. Verizon is worse than congress, they just do everything they can to block anything they don't like, whether it's what their customers want/need or not.
0 Likes
PJL
Master - Level 3

Yest it is the case for any content ISP (that they send more than they receive), but that imbalance should be accounted for in the peering agreement contact between them and the consumer ISP (like Verizon).  The contract allows certain transit levels to and from the parties.  Either party is obligated to only support the receipt of the level specified -- no more.  That's what we're seeing with the Netflix traffic coming to the Verizon consumer ISP network from NTT, L3, and Telia (to name three of which I am aware).

This problem is slowly going away.  Users at east coast locations have reported that Netflix traffic now comes directly from Netflix without going through a third-party like NTT, etc.  Dallas-Fort Worth uses this week reported the same direct routing.  This is a result of the announcement awhile back that Netflix was going to connect directly to the Verizon network rather than routing through third parties.  (For those who care this is not "Netflix Open Connect" but rather a direct Netflix connection.  There is a big difference between the two.)

0 Likes
Hubrisnxs
Legend
http://publicpolicy.verizon.com/blog/entry/level-3s-selective-amnesia-on-peering


Apparently Level 3 is being intellectually dishonest with that latest post. They had the same complaint and were very public about it in the past. Now they have changed their story.
0 Likes
monsterlab
Enthusiast - Level 2

So, how do those of you who are still blaming Netflix explain this post from Level3?:

http://blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/

nick_d
Enthusiast - Level 3

Haha Monstrlab, I was just coming to post that blog link myself... here are the contents for those who do not want to open a new tab (source: http://blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/)


Verizon’s Accidental Mea Culpa

David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix’s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon’s own paying broadband consumers.

His explanation for Netflix’s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.

The diagram then shows this one little bar, suggestively color-coded in red so you know it’s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.

But here’s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizon’s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because that’s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network – IN BOTH DIRECTIONS.

So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?

Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3’s network interconnects with Verizon’s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.

So let’s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion exists.

lvltvzw

Verizon has confirmed that everything between that router in their network and their subscribers is uncongested – in fact has plenty of capacity sitting there waiting to be used. Above, I confirmed exactly the same thing for the Level 3 network. So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers. Simple. Something we’ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused. So Verizon, not Level 3 or Netflix, causes the congestion. Why is that? Maybe they can’t afford a new port card because they’ve run out – even though these cards are very cheap, just a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that’s the case, we’ll buy one for them. Maybe they can’t afford the small piece of cable between our two ports. If that’s the case, we’ll provide it. Heck, we’ll even install it.

But, here’s the other interesting thing also shown in the Verizon diagram. This congestion only takes place between Verizon and network providers chosen by Netflix. The providers that Netflix does not use do not experience the same problem. Why is that? Could it be that Verizon does not want its customers to actually use the higher-speed services it sells to them? Could it be that Verizon wants to extract a pound of flesh from its competitors, using the monopoly it has over the only connection to its end-users to raise its competitors’ costs?

To summarize: All of the networks have ample capacity and congestion only occurs in a small number of locations, locations where networks interconnect with some last mile ISPs like Verizon. The cost of removing that congestion is absolutely trivial. It takes two parties to remove congestion at an interconnect point. I can confirm that Level 3 is not the party refusing to add that capacity. In fact, Level 3 has asked Verizon for a long time to add interconnection capacity and to deliver the traffic its customers are requesting from our customers, but Verizon refuses.

Why might that be? Maybe we should ask David Young.


0 Likes
Kestrel3
Contributor - Level 2

Could this problem also explain why many users have been seeing such poor actual upload speeds, but the speed tests to the closest Verizon servers look fine?  Uplolad speeds used to be great, but have dropped to about 10% of what they used to be.

see this thread:

http://forums.verizon.com/t5/FiOS-Internet/Slow-real-world-upload-speed/td-p/654849

0 Likes
DuneSleeper
Newbie

Mine just started getting bad a couple of weeks ago. So far, it is averaging 1 reload per streamed show. It is still very annoying. It just keeps getting more expensive while the service gets worse.

0 Likes
TennisFreak
Enthusiast - Level 3

So I am going to start a youtube campain with proof Verizon is throttling Netflix specifically.

Last night I tried a "proof of concept" test to see what would happen and it went exactly as I planned.  This week I will produce and post a full video of this to try and prove Verizon is throttling Netflix intentionally.


Here is what I did last night.  Also I performed this test twice once over my in house wi-fi and then again over my hard wired gigabit network with identical results.

1. Open netflix on xbox one and try to watch How I met your mother and then World War Z.

2. Experience approiximately 5 minutes of buffering for video to load.  Video begins to play for approximately 20 seconds then stops and buffers again.  

3. After 5 mintues of attempted buffering I give up and shut down the xbox.

4. Load Netflix on my Lumia 1520 phone connected to wifi.  Experience same issues as xbox one.

5. Open coputer internet browser and test speed on speedtest.net.  Recieve full 15 down 5 up speeds.

6. Watch youtube on my internet browser in full HD with no buffering.

7. Turn xbox one on and load youtube. 

8. Open youtube app and load video, experience about 15 secods and then commence to watch full HD stream of video.

9. Re-open Netflix on xbox one and try to play How I met your mother.

10. Experience about 3 minutes of buffering then stream loads in extremely low quality and plays.  Picture is pixelated and blocky.

11. After about 3 minutes of watching How I met your mother stream improves to what I would consider standard definition quality.

12. Disconnect my Lumia 1520 from wifi and have it running exclusively on LTE.

13. Open netflix and play exact same episode of How I met your mother simultaneously as the xbox one.

14. Experience approximately 10 seconds of buffering and stream commences to play in HD quality with no further buffering while the xbox one is barely maintaining a standard definition quality stream.

15. Re-connect my Lumia 1520 to local wifi

16. Open speedtest.net app and run test.  Confirm getting full 15 down 5 upload speeds.

17. Open netflix and try to resume How I met your mother episode. 

18. Experience almost 1 mintue of buffering before stream starts in a very low quality (almost below standard definition).

I know my test is no silver bullet pinning Verizon to the wall but I feel that it is more than adequate in showing that something is going on WITH VERIZON and not Netflix.  When you can watch Netflix perfectly on a mobile phone LTE connection but barely get it to play on a local gigabit hard wired network SOMETHING IS WRONG.

0 Likes
Hubrisnxs
Legend

I am sorry, I just wanted to point out that your test shows the opposite.  It shows that Youtube and Google have an appropriate amount of bandwidth where netflix doesn't?

it doesn't show that verizon is doing anything to netflix's traffic.

That same test you just performed, if you removed netflix from the picture, and substituted ME instead, would prove that I might have a DSL line or a really cheapy slow FiOS connection, not that Verizon was throttling ME.

0 Likes
TennisFreak
Enthusiast - Level 3

Hubrisnxs

Maybe you didnt read all of my post then.


Because I clearly stated that while Netlix was not playing worth a dang on my FIOS connection I could simultaneously load Netflix via my LTE connection on my Lumia 1520 and watch the same show in full HD with no buffering.

0 Likes
Hubrisnxs
Legend

no I Actually read that.  That means that it took a different path.  That's all. 

a REAL GOOD test is if you had APPLE TV at your house, Netflix signed an agreement with apple to route those customers across a premium link.    you wouldn't have any problems if you did that.    


@TennisFreak wrote:

@Hubrisnxs

Maybe you didnt read all of my post then.


Because I clearly stated that while Netlix was not playing worth a dang on my FIOS connection I could simultaneously load Netflix via my LTE connection on my Lumia 1520 and watch the same show in full HD with no buffering.



In a little known, but public fact, anyone who is on Comcast  or Verizon and using Apple TV to stream Netflix wasn’t having quality problems. The reason for this is that Netflix is using Level 3 and Limelight to stream their content specifically to the Apple TV device. What this shows is that Netflix is the one that decides and controls how they get their content to each device and whether they do it via their own servers or a third party. Netflix decides which third party CDNs to use and when Netflix uses their own CDN, they decide whom to buy transit from, with what capacity, in what locations and how many connections they buy, from the transit provider. Netflix is the one in control of this, not Comcast or any ISP.


0 Likes
monsterlab
Enthusiast - Level 2

This sounds like a good test, and I've thought about doing something similar by tethering my laptop to my LTE connection and show how much better the connection is. The key is to do a traceroute while on both connections to prove that the route is exactly the same. I haven't done this yet, so I can't confirm. If the routes to Netflix's CDN are the same over both FiOS and your cellular carrier's LTE connection, then the argument that it's the Netflix's transit providers' fault doesn't hold water. If they are indeed using different routes, then it will be harder to place blame.

0 Likes
monsterlab
Enthusiast - Level 2

Let me clarify. The route will definitely not be the same, as you will be dealing with two different ISPs (FiOS and your celluar carrier), but if the END POINT on Netflix's end is the same, then it is hard to blame Netflix. I was in the mountains last week on a tiny little ISP's (Comporioum) connection getting a perfect 1080p stream from Netfilx. I get home last night, and I can't even get a decent SD picture over my 75/35 FiOS connection. Based on this, I'm finding it hard to blame Netflix.

0 Likes
bbinnard
Contributor - Level 3

image

If you look at the graph version of this chart at http://ispspeedindex.netflix.com/results/usa/graph starting in Feb 2013 you'll see a steady decrease in Netflix/FIOS performance. And they still want us to pat $10/mo more for Quantum?
image
0 Likes
Hubrisnxs
Legend

@bbinnard wrote:
If you look at the graph version of this chart at http://ispspeedindex.netflix.com/results/usa/graph starting in Feb 2013 you'll see a steady decrease in Netflix/FIOS performance. And they still want us to pat $10/mo more for Quantum?


funny how if you go to google / Youtube ISP rankings, you'll find FiOS Fully HD Qualified.

Wonder why google's and netflix's report differ.

Wonder why netflix's CDN Partners are so high in ranking where those who didn't play ball are rated low?

Things that make you go hmmmmmmmmmmmmmmmmmmmmmmmm

0 Likes
Normx
Enthusiast - Level 2

April 6 2014

Greetings,

I just received my proxyvote for Verizon Com inc.

What a shocker that Verizon has asked the shareholders to vote againts network neutrality, or curbing lobbying activity, or anything that might be benificial for the customer. As soon as Netflix cuts a deal with Verizon, speeds will change. Of couse, Verizon will blame Netflix, Netflix will blame Verizon,  for the cost increase you, the customer,  will have to pay.

What ever you do, don't cut the wire. Just keep paying, and paying, and paying.....

0 Likes
DuneSleeper
Newbie

It won't let me give kudos to your comment, but I totally kudo you!

0 Likes
MusicalPhysics

I tried to file a complaint with the FCC but despite their previous net neutrality rules they state they don't directly regulate ISPs.

 So, I filed complaints with my state attorney general and my local better business bureau. If anything actually happens as a result of these complaints I'll post here.

0 Likes
MusicalPhysics

*signed*

 Most of the time I can only stream shows at 240p even though I pay for 50 Mbit internet. This is ridiculous.

0 Likes