google / youtube slow over fios. definitely verizon's fault (research inside), please fix

[sbrudenell]

i have fios in pennsylvania. for several months, i've been observing a problem where many google / youtube servers are slow / stuttering in response. it seems to be getting worse.

the problem is easiest to notice with youtube videos. from the user's perspective, sometimes i can't even play a 240p video without it buffering. sometimes reloading the page makes it load fully almost instantly; sometimes changing (even *increasing*) the resolution makes it load instantly. the problem doesn't seem to depend on the time of day.

i have found many other threads on this and other forums (youtube support, dslreports) complaining about consistent youtube trouble over fios, so i think this problem might affect a lot of users.

from my perspective over the network, it's pretty obvious to me that the problem is a certain set of routers under verizon's control which border google's network.

here's my research:

at the network level, there seems to be severe (north of 40%!) packetloss to all hosts behind this router.

here's a youtube content host for a video i just watched:

steve-local@server:~$ host v16.lscache2.c.youtube.com

v16.lscache2.c.youtube.com has address 173.194.8.38

steve-local@server:~$ sudo ping -f -c 100 173.194.8.38

PING 173.194.8.38 (173.194.8.38) 56(84) bytes of data.

...............................................

--- 173.194.8.38 ping statistics ---

100 packets transmitted, 53 received, 47% packet loss, time 1398ms

rtt min/avg/max/mdev = 17.373/20.978/23.746/1.382 ms, pipe 3, ipg/ewma 14.127/20.853 ms

much of google seems afflicted:

steve-local@server:~$ sudo ping -f -c 100 www.google.com

PING www.l.google.com (72.14.204.103) 56(84) bytes of data.

............................................

--- www.l.google.com ping statistics ---

100 packets transmitted, 56 received, 44% packet loss, time 1436ms

rtt min/avg/max/mdev = 17.329/20.117/22.279/1.221 ms, pipe 2, ipg/ewma 14.507/19.689 ms

this amount of packetloss is totally unacceptable.

here's my traceroute to that content host:

steve-local@server:~$ traceroute 173.194.8.38

traceroute to 173.194.8.38 (173.194.8.38), 30 hops max, 60 byte packets

 1  10.92.1.1 (10.92.1.1)  0.295 ms  0.286 ms  0.307 ms

 2  L100.PITBPA-VFTTP-11.verizon-gni.net (72.95.222.1)  7.894 ms  7.948 ms  7.923 ms

 3  G12-0-0-447.PITBPA-LCR-04.verizon-gni.net (130.81.180.36)  8.034 ms  8.020 ms  8.011 ms

 4  P5-0-0.PITBPA-LCR-03.verizon-gni.net (130.81.27.250)  15.034 ms  17.704 ms  17.679 ms

 5  so-15-3-0-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net (130.81.28.204)  17.537 ms  17.650 ms  17.665 ms

 6  0.so-2-3-0.XL3.IAD8.ALTER.NET (152.63.30.17)  17.899 ms 0.so-2-1-0.XL3.IAD8.ALTER.NET (152.63.30.2)  14.760 ms 0.so-1-2-0.XL3.IAD8.ALTER.NET (152.63.37.117)  14.821 ms

 7  TenGigE0-6-1-0.GW7.IAD8.ALTER.NET (152.63.32.194)  19.366 ms  20.904 ms TenGigE0-4-0-1.GW7.IAD8.ALTER.NET (152.63.37.154)  19.827 ms

 8  google-gw.customer.alter.net (152.179.50.106)  37.220 ms  37.202 ms  37.190 ms

 9  216.239.46.250 (216.239.46.250)  19.648 ms  91.456 ms 216.239.46.248 (216.239.46.248)  14.902 ms

10  216.239.48.113 (216.239.48.113)  21.163 ms  21.098 ms 216.239.48.109 (216.239.48.109)  75.934 ms

11  66.249.95.143 (66.249.95.143)  15.683 ms  18.064 ms  18.093 ms

12  173.194.8.38 (173.194.8.38)  20.528 ms  21.418 ms  22.429 ms

hop #8 (152.179.50.106) is where the problems start for me. all the earlier alter.net hops return pings with no loss, but i experience the same 40%+ loss to 152.179.50.106 and all hops afterward.

the bogus name "google-gw.customer.alter.net" does not resolve.

some other problematic google hosts show 152.179.50.62 (also with bogus name google-gw.customer.alter.net) as the problematic hop to google, but is otherwise similar. 152.179.50.106 and 152.179.50.62 are the only two i have seen so far.

there are some google hosts that also go through alter.net that have no problems at all (note that www.youtube.com just serves youtube's ui, not the videos themselves):

steve-local@server:~$ host www.youtube.com

www.youtube.com is an alias for youtube-ui.l.google.com.

youtube-ui.l.google.com has address 64.233.169.93

youtube-ui.l.google.com has address 64.233.169.136

youtube-ui.l.google.com has address 64.233.169.190

youtube-ui.l.google.com has address 64.233.169.91
steve-local@server:~$ sudo ping -f -c 100 64.233.169.91

PING 64.233.169.91 (64.233.169.91) 56(84) bytes of data.

--- 64.233.169.91 ping statistics ---

100 packets transmitted, 100 received, 0% packet loss, time 1307ms

rtt min/avg/max/mdev = 19.909/22.841/28.183/1.482 ms, pipe 3, ipg/ewma 13.204/23.052 ms
steve-local@server:~$ traceroute 64.233.169.91

traceroute to 64.233.169.91 (64.233.169.91), 30 hops max, 60 byte packets

 1  10.92.1.1 (10.92.1.1)  0.286 ms  0.282 ms  0.302 ms

 2  L100.PITBPA-VFTTP-11.verizon-gni.net (72.95.222.1)  8.147 ms  8.144 ms  8.135 ms

 3  G12-0-0-347.PITBPA-LCR-03.verizon-gni.net (130.81.180.34)  10.521 ms  10.571 ms  10.546 ms

 4  so-15-3-0-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net (130.81.28.204)  17.800 ms  17.855 ms  17.830 ms

 5  0.so-2-3-0.XL3.IAD8.ALTER.NET (152.63.30.17)  20.314 ms 0.xe-4-1-0.XL3.IAD8.ALTER.NET (152.63.2.249)  17.850 ms 0.so-2-3-0.XL3.IAD8.ALTER.NET (152.63.30.17)  20.334 ms

 6  0.so-3-2-0.XL3.NYC4.ALTER.NET (152.63.0.182)  25.099 ms  27.268 ms 0.xe-1-1-1.XL3.NYC4.ALTER.NET (152.63.1.146)  24.744 ms

 7  TenGigE0-4-2-0.GW8.NYC5.ALTER.NET (152.63.16.73)  27.278 ms TenGigE0-6-4-0.GW8.NYC4.ALTER.NET (152.63.21.121)  63.215 ms TenGigE0-6-0-0.GW8.NYC4.ALTER.NET (152.63.22.41)  60.751 ms

 8  Internet-gw.customer.alter.net (152.179.72.66)  23.130 ms  21.442 ms  21.323 ms

 9  216.239.43.114 (216.239.43.114)  28.817 ms 209.85.252.215 (209.85.252.215)  23.902 ms  23.769 ms

10  72.14.239.93 (72.14.239.93)  38.671 ms  43.725 ms  40.372 ms

11  64.233.175.109 (64.233.175.109)  24.275 ms  21.282 ms  22.422 ms

12  72.14.232.21 (72.14.232.21)  33.984 ms 216.239.49.145 (216.239.49.145)  36.295 ms  36.274 ms

13  yo-in-f91.1e100.net (64.233.169.91)  21.187 ms  21.325 ms  21.154 ms

this host goes through alter.net, but hits a different border gateway (152.179.72.66, Internet-gw.customer.alter.net) on the way to google.

all google-owned sites aggressively use dns-based localization, so www.google.com doesn't always resolve to a host on the other side of the problem routers. this explains why refreshing the page sometimes helps, as a slightly different address can make all the difference.

users can sort of work around the problem by using a different public dns server (8.8.8.8 and 8.8.4.4; 4.2.2.1 through 4.2.2.6; or 74.82.42.42). if you're lucky, you'll get a favorable address for the host you're trying to get to.

the addresses of the problem routers, 152.179.50.106 and 152.179.50.62, are owned by verizon business (http://bgp.he.net/ip/152.179.50.106 , http://bgp.he.net/AS701, http://en.wikipedia.org/wiki/UUNET). i'm only a networking hobbyist, but i don't see any way that that this can be anyone's responsibility other than verizon's.

so please fix it.

[sbrudenell]

i never said financial loss.

i'm paying for a service and not receiving it. some packetloss is unavoidable, but 50% loss during daytime hours is totally unacceptable. this is in no way the service i'm paying for.

pinging one host from a single connection doesn't tell you much, true. that's why i tested the reachability of

 - the problem router

 - the hop before the problem router

 - google hosts behind the problem router

 - other google hosts not behind the problem router

 - other non-google hosts

all from

 - my fios connection

 - 3 other sites whene i have servers

 - 3 other sites using dslreports' smokeping

to try to isolate the problem.

the problem i see is that 99% of forum posts consist of "halp my conn iz bad its not my fault". so few of them have a real problem, and it's so hard to get them to provide useful information, that they're easy to ignore. my hope is to provide enough information to get the attention of the right people.

perhaps i provided too much information, since you don't seem to have read anything i wrote?

i counter your proof-by-authority argument with a challenge: tell me what you've seen in your 20 years of experience can explain my observations, other than an overloaded router verizon needs to fix. put your money where your mouth is.

the problem still persists, by the way.

[rupok93]

I have been getting the same exact problem! Usually its not as bad as it has been for the last couple of weeks!

Before changing resolution or refressing the page worked but now it doesn't work at all. I thought it was just me but I asked my friend who also has fios and he has the same problem. We are in Jersey City, NJ,

Verizon and google PLEASE FIX THIS PROBLEM!!!

To the people saying its another problem, its not. When many people complain about the same problems with same situations its not an individuals fault.

[netmastr32]

Sorry to tell you.

This is the DNS Verizon assigns to the IP address on the customers side of the link between Verizon and the customer router.

google-gw.customer.alter.net

That is why it has Customer in the name.

Here is another customer router off Verizon

limelight.customer.alter.net

Verizon managed devices have this as there DNS Name

IAD8.ALTER.NET  - All CAPS

The issue is on the Google Router.

[Hubrisnxs]

google's getting chincy.

[weedeater1]

IAD = Dulles, VA. Which is where Verizon Business (the old MCI/Worldcom) has its headquarters.

Someone needs to get on to Google about this.

[fake_usa]

FIOS in NJ has issues streaming youtube. 

I can watch videos flawlessly via lightpath and comcast.

Same videos on FIOS are unwatchable. Constant buffering.

FOIS support said I would have to pay FIOS $89 to troubleshot my issue. LOL

[Namronorman]

Chat rep also suggested I pay $89.99 for a solution. What is this, some sort of scheme to nickel and dime your customers? I just use a proxy through a VPS I rent and it works flawlessly. This whole ordeal has been an issue for me now for about a week.

[Hubrisnxs]

did you try the alternate dns trick that was posted? 

[hahmed1983]

Hey guys, 

Anyone found a fix yet. It appears this issue has been going on from 2011, I am also from northern virginia just 3 min away from Verizon HQ. The youtube issue is terrible. I can't watch even 240p videos and it buffers to death. I have freaking quantom 75/35 MB connection. I test the videos even on wired connection with no luck. Speedtest on wired give me 70MB and on wireless 60 MB. 

All other videos work fine such as netflix and other sources than youtube. 

I am seriously now thinking to disconnect FIOS as it make no sense to pay for having 75MB connection.

let me know if some one has find anything interesting,

Thanks

[Kardinal]

I'd been having this issue for a long time and had worked around it by blocking the CDN (blocking that range of IP's some have mentioned), but today, miraculously, YouTube is fast without that.  A fellow FIOS user on Google Plus mentioned that his YouTube issues were fixed this morning, and so I checked mine, and mine are as well.

I'm desperately curious what Verizon did to fix it, but it seems to work perfectly now.  None of the hesitation, no spining wheels, no queuing...just smoooothe 1080P.

(50/50 FIOS in Fairfax, VA)

[jcoxiii]

I wish i was seeing the same, 1 or 2 videos from Epic Meal Time loaded fast and played at 1080p, but that was short lived. Now not even playing, I have 50/15, tired over wired and wireless, and I'm the only one on the network. Located in the southern Balitmore MD Suburbs.

[toozie21]

I am having the EXACT same issues.  Evenings are alway worse, but it is basically not usable at that point.  I can do a speedtest and get 25/5Mbps, yet I have the same bottleneck!

[wonslung]

This problem is very real.   It's been happening for months in richmond va and effects everyone i know with fios.

When i call customer support they either act like i'm crazy or put me through a bunch of hoops that don't work and never COULD work.   Finally today i was able to get one of them to admit that it is a real issue, but they wouldn't take any of the blame, saying that it's google's fault, and i should contact google.   I just spent 30 minutes trying to find the proper channel to contact google, but ultimately couldn't find the proper contact info.  They do have a phone number but they aren't taking any support calls on a sunday.

One other thing, as a work around i've been using free proxies or a vpn.   This proves that the problem is between verizon and google somewhere...Someone is dropping the ball.  

[Taffer]

I am in Midlothian VA, near Richmond.

This crap is driving me crazy. Youtube is my second most visited site after Google search itself. Which is a shame, because I can barely watch Youtube on FIOS. It's pure luck whether or not it's going to let me watch. I don't mean it takes a long time to load either; it just doesn't load videos at all, a great deal of the time.

I've already gone through support and had them waste my time with fixes that did absolutely nothing. I first got fed up with it back around April 2013. At some point after that the problem seemed to go away so I forgot about it. It is now September 2013 and I can't watch Youtube again.

If this keeps happening, I'm afraid I'm going to have to switch back to Comcast. I don't care for Comcast, but at least I could watch Youtube.

This really sucks. I have absolutely no problem with ANYTHING else. All websites are perfect, gaming, etc is just perfect, BUT YOUTUBE is IMPOSSIBLE with FIOS.

We are talking about FIOS! Why must it be this way? Verizon, step up your game.

[NetBias]

Wow. I am also in MIdlothian. There is another thread about Youtube as well, and I can confirm it is purely a Verizon issue today. Comcast works fine. However, I have seen Comcast fail in the past as well.

The bottom line is that this is a net neutrality issue. Verizon does not keep up with Cogent's bandwith requirements because then they only get paid once instead of twice, if you stream RedBox.

http://arstechnica.com/information-technology/2013/07/why-youtube-buffers-the-secret-deals-that-make...

And no, buying more bandwidth from Verizon won't help.

[toozie21]

I had the same problem, it resolved itself, and now it is back again.  It is a pretty embarrassing problem in my mind, but it doesn't seem to bother Verizon much...

[specialp]

The reason why it does not bother Verizon is that they want Youtube and Netflix to be slow.  You see Verizon could CDN with them but they refuse to.  CDN means Google and Netflix put in caching servers to make it so you do not have to leave the Verizon network to get content.  Cablevison users have no problem with YouTube and Netflix being practically unwatchable during peak times due to the fact that Cablevision CDNs. Many ISPs CDN.  

Google and Netflix have offered to put in these CDN servers for free.  Verizon refuses because they view them as a threat to their TV business and they want them to pay transit to their network.  Make no mistake this could be solved very easily but they are being greedy.  What is the point in advertising super fast connections when the connection to the 2 biggest bandwidth sources is slow?!  Most people do not know the game of ISP peering so they presume it is the content provider's fault.  It is partially but Verizon is being uncooperative.

[sdcjn]

SAME ISSUE HERE in PA.  Been happening for months on and off.  It's been really bad the past week - been in touch with support 3 times and no resolution yet.