What can we do to actually fix Verizon's Netflix throttling?
bbinnard
Contributor - Level 3

Obviously there are lots of folks (including me, located in the Los Angeles area) who are furious about Verizon's unacceptably poor delivery of Netflix content.  There are either numerous pauses waiting for buffering or reduced resolution due to Netflix increasing compression ratios to reduce network traffic.  Neither of these are what we are paying for, and both are unacceptable.

This chart

http://ispspeedindex.netflix.com/results/usa/graph

clearly shows Verizon's steadily decreasing performance delivering Netflix content.  At this rate the delivery speed will drop to zero in a few more months.  Abd we are expected to continue paying for this?

So far, despite countless postings here about this issue, nothing has been done or appears to be planned to be done.  So what can we, the customers and consumers, do to cause some positive action?  

How about some sort of petition sent to the FCC ( but addressed to whom or what department?)  Or perhaps we flood the head of Verizon (who is that?) with emails or actual letters.  Or how about letters to the editors of major newspapers and/or wire services?  How about emails to all the popular PC & Internet oriented magazines & blogs?

My sense is that the only way to actually make something happen to fix this situation is to raise enough attention that Verizon management realizes it is in the company's best interest to stop stonewalling and instead actually do something.

To borrow a quote from Paddy Chayefsky: "I'm mad as hell and I'm not going to take it anymore!"

0 Likes
63 Replies
SoNi67
Contributor - Level 1

Well, true, it might still detect/conter it at some point. I know is not a true VPN, it just "hides" the destination.

But exactly that's why it proves my point that Verizon ACTIVELLY criples our service for their interest. Is not a Netflix issue, is purelly a Verizon choice. Nothing changes in the data path, except the fact that Verizon cannot sniff it like any regular DNS request.

Verizon advertise their speed as being "internet access". If it does a traffic-shaping then is just limited Internet access service.

Their TOS doesn't say nothing about that and frankly I think they are in breach of contract for doing that.

0 Likes
optivity
Contributor - Level 3

Currently, I pay Verizon $204 for a Triple Play package that includes a 35/35 Mbps Internet connection.

 

Nowhere in Verizon's TOS do they say I can attain (nearly) wire speed to site A but not from site B.

 

Verizon's blazing fast FiOS Internet speed advertisements and their customer service rep's incessant push to upgrade to Quantum FiOS is disingenuous at best.

 

The best part of being a month-to-month subscriber is once they milk this cow dry, it will be sayonara Verizon hello Time Warner.

0 Likes
chomper87
Enthusiast - Level 3
Currently, I pay Verizon $204 for a Triple Play package that includes a 35/35 Mbps Internet connection.

Seems high. What's you're TV package? My total is $116 - Including all taxes - for FiOS Prime TV and 75/35 FiOS Quantum

 

Nowhere in Verizon's TOS do they say I can attain (nearly) wire speed to site A but not from site B.

I'm not aware of any ISP that explicitly mentions this. Usually ISP's now market their service as "Up To".  Verizon does say: Actual speeds will vary.

Verizon's blazing fast FiOS Internet speed advertisements and their customer service rep's incessant push to upgrade to Quantum FiOS is disingenuous at best.

 Certainly, I'm sure it happens. The rep says: "Well you can upgrade you're service to help with Netflix." And of course they are incorrect. You already have 35/35.

0 Likes
Erasmus354
Enthusiast - Level 2

@SoNi67 wrote:

Well, true, it might still detect/conter it at some point. I know is not a true VPN, it just "hides" the destination.

But exactly that's why it proves my point that Verizon ACTIVELLY criples our service for their interest. Is not a Netflix issue, is purelly a Verizon choice. Nothing changes in the data path, except the fact that Verizon cannot sniff it like any regular DNS request.

Verizon advertise their speed as being "internet access". If it does a traffic-shaping then is just limited Internet access service.

Their TOS doesn't say nothing about that and frankly I think they are in breach of contract for doing that.


I think you are a bit confused about what is actually happening. I think the road/bridge analogy is the best example that might help you. Verizon is not (at least there is no evidence) actively crippling connections to Netflix. They aren't detecting you connecting to Netflix and then slowing down that connection.

Instead Verizon is simply allowing the connections to Netflix to become overloaded, which in essence slows down the connection for everyone.

Think of the road and bridges. Verizon has an 8 lane highway on their end, and Netflix has an 8 lane highway on their end. The bridge between Netflix and Verizon is only 2 lanes. Normally during low traffic it doesn't matter and everything moves just fine. However when you reach a certain point it causes a traffic jam and everyone slows down. Now there isn't just one bridge between Verizon and Netflix, there are many all spaced out in different locations. What you did with your VPN wasn't hide your destination, you changed it. You used a different bridge to cross from Verizon to Netflix and happened to use a bridge that wasn't having a traffic jam.


Now this isn't to say that Verizon isn't at fault here. They are deliberately allowing the connections to get clogged. Historically when that happens both ends do a no cost peer (Netflix pays for their half, Verizon pays for their half) and the bridge would be made wider to accomodate more traffic. Verizon is changing that paradigm and is telling Netflix that if they want to make the connection wider to alleviate the congestion that Netflix will have to pay Verizon to do that.

There is nothing you can do to fix Netflix speed aside from using a VPN to get around the congested connections or switching to another ISP.

SoNi67
Contributor - Level 1

@Erasmus354 wrote:

@SoNi67 wrote:

Well, true, it might still detect/conter it at some point. I know is not a true VPN, it just "hides" the destination.

But exactly that's why it proves my point that Verizon ACTIVELLY criples our service for their interest. Is not a Netflix issue, is purelly a Verizon choice. Nothing changes in the data path, except the fact that Verizon cannot sniff it like any regular DNS request.

Verizon advertise their speed as being "internet access". If it does a traffic-shaping then is just limited Internet access service.

Their TOS doesn't say nothing about that and frankly I think they are in breach of contract for doing that.


I think you are a bit confused about what is actually happening. I think the road/bridge analogy is the best example that might help you. Verizon is not (at least there is no evidence) actively crippling connections to Netflix. They aren't detecting you connecting to Netflix and then slowing down that connection.

Instead Verizon is simply allowing the connections to Netflix to become overloaded, which in essence slows down the connection for everyone. 


OK, so explain how by just changing the DNS the route sudenly becomes less congested???

Like I said, unlocator.com is not a VPN, it just "hides" my request from my ISP provider. My packets still gets from Netflix to me trough Verizon network, including the "bottleneck" that you keep assuming that exist. But, since Verizon cannot **bleep** where they are coming from, it cannot traffic shape them.

The proof is that I get 10x the speed that I get when I use Verizon DNS.

Other DNS providers don't work because they are "transparent", Verizon still can "see" the endpoint.

This is not just "not taking care of upgrading the system", it is activelly throttling my connection.

If you like the examples is the difference between "not repairing holes in the Netflix road" and "sending crews to make the holes in the Netflix road".

0 Likes
chomper87
Enthusiast - Level 3
OK, so explain how by just changing the DNS the route sudenly becomes less congested???

I'm not sure how else to explain it for you. The Destination IP adress changes. The Destination Server changes. The Destination server locaion is changed. The route to that new server has potentially changed. Verizon CAN STILL SEE you're IP address (the source) and the destination IP address.

Go play around on some Looking Glass traceroutes for a while. Do traceroutes to "amazon.com" and change the location (source server)

https://www.sprint.net/lg/lg_start.php

http://lookingglass.level3.net/traceroute/lg_tr_main.php

Here's some examples on DNS for "amazon.com"

Provider~~~~~Src Location~~Dest IP

----------------------------------------------------------

Sprint~~~~~~~Wash, DC~~~72.21.194.212

Sprint~~~~~~~Miami, FL~~~72.21.215.232

Level3~~~~~~Miami, FL~~~~72.21.215.232

XO~~~~~~~~~Miami, FL~~~~205.251.242.54

Level3~~~~~~Tampa, FL~~~72.21.215.232

VZ FiOS~~~~~Tampa, FL~~~176.32.98.166

VZ Business~~Dallas, TX~~~205.251.242.54

VZ FiOS~~~~~Dallas, TX~~~~72.21.194.212

 

And so forth. Verizon still sees you're IP and the Dest IP. You just are using another DNS to give you some other IP for a Server which is somewhere else. Unlocator might differentiate it's DNS by pursuing the Video streaming market such as Netflix and Amazon and spend time scouring for IP addresses and destination servers for these services. Verizon could blacklist all these IPs.

Verizon could do something as simple and USE ARIN list to block, throttle, limit, or shape entire ranges of systems:

http://whois.arin.net/rest/org/AMAZON-4/nets

0 Likes
Erasmus354
Enthusiast - Level 2

@SoNi67 wrote:

@Erasmus354 wrote:

@SoNi67 wrote:

Well, true, it might still detect/conter it at some point. I know is not a true VPN, it just "hides" the destination.

But exactly that's why it proves my point that Verizon ACTIVELLY criples our service for their interest. Is not a Netflix issue, is purelly a Verizon choice. Nothing changes in the data path, except the fact that Verizon cannot sniff it like any regular DNS request.

Verizon advertise their speed as being "internet access". If it does a traffic-shaping then is just limited Internet access service.

Their TOS doesn't say nothing about that and frankly I think they are in breach of contract for doing that.


I think you are a bit confused about what is actually happening. I think the road/bridge analogy is the best example that might help you. Verizon is not (at least there is no evidence) actively crippling connections to Netflix. They aren't detecting you connecting to Netflix and then slowing down that connection.

Instead Verizon is simply allowing the connections to Netflix to become overloaded, which in essence slows down the connection for everyone. 


OK, so explain how by just changing the DNS the route sudenly becomes less congested???

Like I said, unlocator.com is not a VPN, it just "hides" my request from my ISP provider. My packets still gets from Netflix to me trough Verizon network, including the "bottleneck" that you keep assuming that exist. But, since Verizon cannot **bleep** where they are coming from, it cannot traffic shape them.

The proof is that I get 10x the speed that I get when I use Verizon DNS.

Other DNS providers don't work because they are "transparent", Verizon still can "see" the endpoint.

This is not just "not taking care of upgrading the system", it is activelly throttling my connection.

If you like the examples is the difference between "not repairing holes in the Netflix road" and "sending crews to make the holes in the Netflix road".


You have to understand a little bit about what DNS is. So Netflix.com doesn't always go to the same place. When you try to go to Netflix.com your DNS service needs to tell you what that address actually is and how to get there. So there might be Netflix servers in New York, Chicago, Atlanta, Dallas, and Los Angeles. The Verizon DNS servers will try to route you to the closest connection, for arguments sake we will say New York. So if lots of people around New York are all trying to watch Netflix at the same time the New York connection will get bogged down.

If you change your DNS you change how your connection gets routed. Now instead of being told to go to New York to connect to Netflix, the Unlocator DNS might tell you to go to Los Angeles. Well over on the west coast people are still at work so the connection isn't clogged. Sure the signal might take an extra 20 milliseconds to get there and cross a couple more hops, but the connection to Netflix isn't clogged so you end up with an overall faster connection.

You can get the same effect yourself without changing your DNS if you want to. 

1) Find out the IP address where the connection is slowing down.

2) Block that IP address

This will force the Verizon DNS to route you to a different server and will do the same thing as changing your DNS. It is just forcing your route to change to a different destination. That is all.

0 Likes
bbinnard
Contributor - Level 3

There is another factor at play here that I haven't seen mentioned yet, and that is the inherent characteristic of Ethernet networks to experience rapidly degraded performance/throughput once they get past about 30% busy.  The reason for this is inherent in the basic architecture of Ethernet - on a single Ethernet cable (like Cat5) only one device can be transmitting at a time.

Before a device transmits it tests the cable to see if it is busy transmitting someone else's signal.  If it (the cable) is busy, the Ethernet device (actually it's the network card in the device) waits a random amount of time and then tries to transmit again.  This process continues until the the device finds that the cable is free and it then transmits the current data packet.  Then the process starts all over again.

This architecture is called "indeterminant" because it is impossible to predict how long it will take any specific amount of data to get transmitted - again becuase there is no telling how busy the cable will be or how many times a particular Ethernet adapter will have to wait/retry it's transmissions.  In general, if the cable is more than abotu 30% busy there are a lot of waits/retries that happen, and this significantly degrades overall network performance.

Interestingly enough, there is (or was) an alternate network architecture that does not have this problem.  It is called a "deterministic" network becuase it is always possible to predict exactly how long it will take to transmit a known amount of data.  This kind of network was invented by IBM and is called Token RIng.  It has several major advantages over Ethernet, but was never widely adopted because IBM wanted to charge money for every Token RIng device, whereas Ethernet was given away for free.

0 Likes
optivity
Contributor - Level 3

Level 3 blames Internet slowdowns on ISPsโ€™ refusal to upgrade networks
Cogent and Netflix aren't the only ones haggling with Verizon and AT&T.

http://arstechnica.com/information-technology/2014/03/level-3-blames-internet-slowdowns-on-isps-refu...

U.S. Pay-TV Subscriptions Fall for First Time as Streaming Gains

http://www.bloomberg.com/news/2014-03-19/u-s-pay-tv-subscriptions-fall-for-first-time-as-streaming-g...

0 Likes
chomper87
Enthusiast - Level 3

Certainly there's all sorts of quirks including Ethernet collisions. There's variations in switching methodologies, eg who out there is using the new OpenFlow. Also, not all connections are 100% Ethernet either. ATM still has a strong hold. On the Verizon FiOS side, this is one of the differences in FiOS BPON vs FiOS GPON. BPON is ATM based. This applies not just to services like DSL and FiOS BPON, but for interconnects between service providers.

As for Level3's recent statement, no doubt there's a lot of validity. Especially with the statement of:

monopoly rent-seeking conduct Level 3 has observed

Looking at more towards Level3 / Comcast thing - I don't see how in any normal universe Level3 would pay Comcast. - Except in ours of course because that's what ended up happening. How many service providers can Netflix get? Many. How many can the normal consumer get? 1? 2 perhaps. Comcast undoubtably used it's monopolistic powers. If Comcast wants to charge Netflix for a direct connection, that's fine I suppose - Netflix can decline.

The real solution - I don't think is to necessarily regulate Interconnect agreements. It's to ban laws that create monopolies in the first place.

http://arstechnica.com/tech-policy/2014/01/who-wants-competition-big-cable-tries-outlawing-municipal...

http://www.muninetworks.org/content/big-bucks-why-north-carolina-outlawed-community-networks

http://www.theatlanticcities.com/technology/2013/03/why-are-there-no-big-cities-municipal-broadband-...

There are cities that would potentially like to get a Small private ISP and / or a Muni network which can't.

0 Likes
bbinnard
Contributor - Level 3

Here's the petition - if you are reading this please follow the link and sign the petition.

http://www.thepetitionsite.com/869/336/946/verizon-intentionally-crippling-netflix-streaming-over-it...

0 Likes
SteveVi
Enthusiast - Level 3

You can also join the Hey Verizon, your FiOS customers want their Netflix back! discussion on  reddit.

The only way Verizon is going to fix this is if it gets enough negative press.

0 Likes
SoNi67
Contributor - Level 1

Signed. It won't matter, only money talks. I will switch service as soon as I can.

0 Likes
SoNi67
Contributor - Level 1

@chomper87 wrote:

On the Verizon FiOS side, this is one of the differences in FiOS BPON vs FiOS GPON. BPON is ATM based. This applies not just to services like DSL and FiOS BPON, but for interconnects between service providers.


It makes no difference in this issue. I have the I-211M-K ONT. Thats's GPON. Still sucks with Netflix. Un-sucks if I use a different DNS provider that hides the endpoint. I just test it again last night.

You keep making escuses for Verizion but fail to address this relavant information about DNS change.

Are you working for Verizon?

0 Likes
SoNi67
Contributor - Level 1

PS: For everybody information BPON is not ATM based, is broadband PON. APON is ATM (hence the name).

Chopme87 just creates whitenoise on the threads, to cover Verizon deficiencies in this very specific matter.

0 Likes
glugglug
Enthusiast - Level 3

It's not just Netflix being slowed down by this sillyness.  The other day I was browsing Verizon's own online streaming to try to find a show my wife wanted to watch.  It was slow as hell.  I checked, and Verizon's own site was using an Akamai CDN which was being routed through cogent like Netflix at the time.

0 Likes
bbinnard
Contributor - Level 3

According to this posting 

http://forums.verizon.com/t5/FiOS-Internet/VUDU-HDX-streaming-problems/td-p/684569

it isn't only Netflix that is being throttled.  Makes me wonder if Verizon is going to start limiting everything except their own transmissions.

I also wonder why Verizon has been completely silent on this matter, why no one who has any real leverage with Verizon has at least commented on the situation, and what it will take for some real action on this matter to occur.

0 Likes
optivity
Contributor - Level 3

"Makes me wonder if Verizon is going to start limiting everything except their own transmissions."

Give that man a cigar! You have hit the proverbial nail on the head.

0 Likes
chomper87
Enthusiast - Level 3
Verizon is changing that paradigm and is telling Netflix that if they want to make the connection wider to alleviate the congestion that Netflix will have to pay Verizon to do that.

Verizon isn't chaning anything. Netflix has a catalog of videos. Netflix wants to stream those videos. Netflix needs to PAY for an ISP. Netflix needs to PAY for a CDN. What's so difficult to understand? If I start my own Video catalog and streaming service, should I get a free services from Verizon? Am I going to get a Free ISP? A free CDN? NO.

Okay so now you say, well Netflix does pay for an ISP, Netflix does pay for a CDN. Not not to Verizon. Netflix pays company XYZ. Okay, well now company XYZ and Verizon now have to come up with an agreement. Go read about "Depeering"

http://en.wikipedia.org/wiki/Peering#Depeering

I see Level3 and XO had disagreements.

I see Level3 and Cogent had disagreements.

I see Sprint and Cogent had disagreements.

THIS IS NOTHING NEW!!!!!!!!!!!!!!!!

Level3 and Comcast had disagreements.

http://arstechnica.com/tech-policy/2011/02/peers-or-not-comcast-and-level-3-slug-it-out-at-fccs-door...

Cogent and Verizon have disagreements.

http://arstechnica.com/information-technology/2014/02/netflix-packets-being-dropped-every-day-becaus...

Once again Netflix can directly pay Verizon for some services including:

Dedicated network connection

http://www.verizonenterprise.com/resources/factsheets/fs_informatio-%20that-drives-your-business-for...

http://www.verizonenterprise.com/products/networking/

http://www.terremark.com/services/it-infrastructure/

And just because NETFLIX Pays for one ISP doesn't mean they get a second one free. If Netflix want a direct connection to Verizon, they will pay. Just like I would, and just like you will. And just like I pay for Verizon FiOS, that doesn't get me a Free Bright House connection even if most of my traffic ends up on Bright House or originates from Bright House. I'd have to PAY Bright House for a direct connection.

Unless you are perhaps a Company such as Level3, Sprint, ATT, CenturyLink then you can say to Verizon:

We'll carry you're traffic, if you carry ours, but even then we'll still might have some disaagreements
nick_d
Enthusiast - Level 3

@chomper87 wrote:
Verizon isn't chaning anything. Netflix has a catalog of videos. Netflix wants to stream those videos. Netflix needs to PAY for an ISP. Netflix needs to PAY for a CDN. What's so difficult to understand? If I start my own Video catalog and streaming service, should I get a free services from Verizon? Am I going to get a Free ISP? A free CDN? NO.

I'm not sure you understand how enterprise networks interconnect. Netflix IS ALREADY paying for their bandwidth from a top tier provider (at speeds greater than 1Tbps in many connection points and a fee of tens of thousands if not more). They ALREADY HAVE a CDN system using open peering. They're not trying to access the internet using bandwidth from Verizon like consumers do.  

Essentially you should think of Netflix as a really great burger place drawing tons of customers from all over. Those customers are using a highway to get to the burger place -- that highway is Verizon's network, Comcasts network, AT&Ts network, etc... a network for which the customers have already paid to access when they pay their monthy internet bill. So what verizon has done is what Chris Christie's office did in NJ -- they took their wide highway down to a lane or two and just let it back up. This is not Netflix demanding bandwidth from Verizon this is Verizon's customers demanding access to Netflix.  Netflix doesn't just send traffic without it being requested!! I don't know how that is so difficult to understand.  Verizon sees that a lot of it's customers are watching netflix and they feel like they should get more money for Netflix's success -- absolutely ridiculous.  It's like the mall seeing that a lot more people want to shop at a particular store they they try to charge that store more because they are popular - explain how that is fair.  If your HOA or apartment complex started charging you a fee on top of your rent because you got more fedex packages than others in the neighborhood, would you be okay with that?  I sure wouldn't.

0 Likes
PJL
Master - Level 3

@nick_d wrote:

@chomper87 wrote:
Verizon isn't chaning anything. Netflix has a catalog of videos. Netflix wants to stream those videos. Netflix needs to PAY for an ISP. Netflix needs to PAY for a CDN. What's so difficult to understand? If I start my own Video catalog and streaming service, should I get a free services from Verizon? Am I going to get a Free ISP? A free CDN? NO.

I'm not sure you understand how enterprise networks interconnect. Netflix IS ALREADY paying for their bandwidth from a top tier provider (at speeds greater than 1Tbps in many connection points and a fee of tens of thousands if not more). They ALREADY HAVE a CDN system using open peering. They're not trying to access the internet using bandwidth from Verizon like consumers do.  

Essentially you should think of Netflix as a really great burger place drawing tons of customers from all over. Those customers are using a highway to get to the burger place -- that highway is Verizon's network, Comcasts network, AT&Ts network, etc... a network for which the customers have already paid to access when they pay their monthy internet bill. So what verizon has done is what Chris Christie's office did in NJ -- they took their wide highway down to a lane or two and just let it back up. This is not Netflix demanding bandwidth from Verizon this is Verizon's customers demanding access to Netflix.  Netflix doesn't just send traffic without it being requested!! I don't know how that is so difficult to understand.  Verizon sees that a lot of it's customers are watching netflix and they feel like they should get more money for Netflix's success -- absolutely ridiculous.  It's like the mall seeing that a lot more people want to shop at a particular store they they try to charge that store more because they are popular - explain how that is fair.  If your HOA or apartment complex started charging you a fee on top of your rent because you got more fedex packages than others in the neighborhood, would you be okay with that?  I sure wouldn't.


I don't believe you take into account peering agreement contracts between the CDNs (aka content provider ISPs) and the consumer ISPs like Verizon.  That peering agreement contract may not be settlement-free.  I think it's a private peering agreement.

0 Likes