google / youtube slow over fios. definitely verizon's fault (research inside), please fix

[sbrudenell]

i have fios in pennsylvania. for several months, i've been observing a problem where many google / youtube servers are slow / stuttering in response. it seems to be getting worse.

the problem is easiest to notice with youtube videos. from the user's perspective, sometimes i can't even play a 240p video without it buffering. sometimes reloading the page makes it load fully almost instantly; sometimes changing (even *increasing*) the resolution makes it load instantly. the problem doesn't seem to depend on the time of day.

i have found many other threads on this and other forums (youtube support, dslreports) complaining about consistent youtube trouble over fios, so i think this problem might affect a lot of users.

from my perspective over the network, it's pretty obvious to me that the problem is a certain set of routers under verizon's control which border google's network.

here's my research:

at the network level, there seems to be severe (north of 40%!) packetloss to all hosts behind this router.

here's a youtube content host for a video i just watched:

steve-local@server:~$ host v16.lscache2.c.youtube.com

v16.lscache2.c.youtube.com has address 173.194.8.38

steve-local@server:~$ sudo ping -f -c 100 173.194.8.38

PING 173.194.8.38 (173.194.8.38) 56(84) bytes of data.

...............................................

--- 173.194.8.38 ping statistics ---

100 packets transmitted, 53 received, 47% packet loss, time 1398ms

rtt min/avg/max/mdev = 17.373/20.978/23.746/1.382 ms, pipe 3, ipg/ewma 14.127/20.853 ms

much of google seems afflicted:

steve-local@server:~$ sudo ping -f -c 100 www.google.com

PING www.l.google.com (72.14.204.103) 56(84) bytes of data.

............................................

--- www.l.google.com ping statistics ---

100 packets transmitted, 56 received, 44% packet loss, time 1436ms

rtt min/avg/max/mdev = 17.329/20.117/22.279/1.221 ms, pipe 2, ipg/ewma 14.507/19.689 ms

this amount of packetloss is totally unacceptable.

here's my traceroute to that content host:

steve-local@server:~$ traceroute 173.194.8.38

traceroute to 173.194.8.38 (173.194.8.38), 30 hops max, 60 byte packets

 1  10.92.1.1 (10.92.1.1)  0.295 ms  0.286 ms  0.307 ms

 2  L100.PITBPA-VFTTP-11.verizon-gni.net (72.95.222.1)  7.894 ms  7.948 ms  7.923 ms

 3  G12-0-0-447.PITBPA-LCR-04.verizon-gni.net (130.81.180.36)  8.034 ms  8.020 ms  8.011 ms

 4  P5-0-0.PITBPA-LCR-03.verizon-gni.net (130.81.27.250)  15.034 ms  17.704 ms  17.679 ms

 5  so-15-3-0-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net (130.81.28.204)  17.537 ms  17.650 ms  17.665 ms

 6  0.so-2-3-0.XL3.IAD8.ALTER.NET (152.63.30.17)  17.899 ms 0.so-2-1-0.XL3.IAD8.ALTER.NET (152.63.30.2)  14.760 ms 0.so-1-2-0.XL3.IAD8.ALTER.NET (152.63.37.117)  14.821 ms

 7  TenGigE0-6-1-0.GW7.IAD8.ALTER.NET (152.63.32.194)  19.366 ms  20.904 ms TenGigE0-4-0-1.GW7.IAD8.ALTER.NET (152.63.37.154)  19.827 ms

 8  google-gw.customer.alter.net (152.179.50.106)  37.220 ms  37.202 ms  37.190 ms

 9  216.239.46.250 (216.239.46.250)  19.648 ms  91.456 ms 216.239.46.248 (216.239.46.248)  14.902 ms

10  216.239.48.113 (216.239.48.113)  21.163 ms  21.098 ms 216.239.48.109 (216.239.48.109)  75.934 ms

11  66.249.95.143 (66.249.95.143)  15.683 ms  18.064 ms  18.093 ms

12  173.194.8.38 (173.194.8.38)  20.528 ms  21.418 ms  22.429 ms

hop #8 (152.179.50.106) is where the problems start for me. all the earlier alter.net hops return pings with no loss, but i experience the same 40%+ loss to 152.179.50.106 and all hops afterward.

the bogus name "google-gw.customer.alter.net" does not resolve.

some other problematic google hosts show 152.179.50.62 (also with bogus name google-gw.customer.alter.net) as the problematic hop to google, but is otherwise similar. 152.179.50.106 and 152.179.50.62 are the only two i have seen so far.

there are some google hosts that also go through alter.net that have no problems at all (note that www.youtube.com just serves youtube's ui, not the videos themselves):

steve-local@server:~$ host www.youtube.com

www.youtube.com is an alias for youtube-ui.l.google.com.

youtube-ui.l.google.com has address 64.233.169.93

youtube-ui.l.google.com has address 64.233.169.136

youtube-ui.l.google.com has address 64.233.169.190

youtube-ui.l.google.com has address 64.233.169.91
steve-local@server:~$ sudo ping -f -c 100 64.233.169.91

PING 64.233.169.91 (64.233.169.91) 56(84) bytes of data.

--- 64.233.169.91 ping statistics ---

100 packets transmitted, 100 received, 0% packet loss, time 1307ms

rtt min/avg/max/mdev = 19.909/22.841/28.183/1.482 ms, pipe 3, ipg/ewma 13.204/23.052 ms
steve-local@server:~$ traceroute 64.233.169.91

traceroute to 64.233.169.91 (64.233.169.91), 30 hops max, 60 byte packets

 1  10.92.1.1 (10.92.1.1)  0.286 ms  0.282 ms  0.302 ms

 2  L100.PITBPA-VFTTP-11.verizon-gni.net (72.95.222.1)  8.147 ms  8.144 ms  8.135 ms

 3  G12-0-0-347.PITBPA-LCR-03.verizon-gni.net (130.81.180.34)  10.521 ms  10.571 ms  10.546 ms

 4  so-15-3-0-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net (130.81.28.204)  17.800 ms  17.855 ms  17.830 ms

 5  0.so-2-3-0.XL3.IAD8.ALTER.NET (152.63.30.17)  20.314 ms 0.xe-4-1-0.XL3.IAD8.ALTER.NET (152.63.2.249)  17.850 ms 0.so-2-3-0.XL3.IAD8.ALTER.NET (152.63.30.17)  20.334 ms

 6  0.so-3-2-0.XL3.NYC4.ALTER.NET (152.63.0.182)  25.099 ms  27.268 ms 0.xe-1-1-1.XL3.NYC4.ALTER.NET (152.63.1.146)  24.744 ms

 7  TenGigE0-4-2-0.GW8.NYC5.ALTER.NET (152.63.16.73)  27.278 ms TenGigE0-6-4-0.GW8.NYC4.ALTER.NET (152.63.21.121)  63.215 ms TenGigE0-6-0-0.GW8.NYC4.ALTER.NET (152.63.22.41)  60.751 ms

 8  Internet-gw.customer.alter.net (152.179.72.66)  23.130 ms  21.442 ms  21.323 ms

 9  216.239.43.114 (216.239.43.114)  28.817 ms 209.85.252.215 (209.85.252.215)  23.902 ms  23.769 ms

10  72.14.239.93 (72.14.239.93)  38.671 ms  43.725 ms  40.372 ms

11  64.233.175.109 (64.233.175.109)  24.275 ms  21.282 ms  22.422 ms

12  72.14.232.21 (72.14.232.21)  33.984 ms 216.239.49.145 (216.239.49.145)  36.295 ms  36.274 ms

13  yo-in-f91.1e100.net (64.233.169.91)  21.187 ms  21.325 ms  21.154 ms

this host goes through alter.net, but hits a different border gateway (152.179.72.66, Internet-gw.customer.alter.net) on the way to google.

all google-owned sites aggressively use dns-based localization, so www.google.com doesn't always resolve to a host on the other side of the problem routers. this explains why refreshing the page sometimes helps, as a slightly different address can make all the difference.

users can sort of work around the problem by using a different public dns server (8.8.8.8 and 8.8.4.4; 4.2.2.1 through 4.2.2.6; or 74.82.42.42). if you're lucky, you'll get a favorable address for the host you're trying to get to.

the addresses of the problem routers, 152.179.50.106 and 152.179.50.62, are owned by verizon business (http://bgp.he.net/ip/152.179.50.106 , http://bgp.he.net/AS701, http://en.wikipedia.org/wiki/UUNET). i'm only a networking hobbyist, but i don't see any way that that this can be anyone's responsibility other than verizon's.

so please fix it.

[sbrudenell]

mtr shows a concise, visual view of the problem:

root@router:~# mtr -4 --report -w -c 100 www.google.com  
HOST: router                                           Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- L100.PITBPA-VFTTP-11.verizon-gni.net              0.0%   100    8.8   6.2   3.8   9.7   1.3
  2.|-- G12-0-0-347.PITBPA-LCR-03.verizon-gni.net         0.0%   100    8.5   7.3   4.0  10.9   1.3
  3.|-- so-15-3-0-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net  1.0%   100   11.8  16.4  11.0 104.4  14.2
  4.|-- 0.so-2-1-0.XL3.IAD8.ALTER.NET                     0.0%   100   13.1  20.2  12.3 134.9  17.5
    |  `|-- 152.63.30.17
    |   |-- 152.63.36.233
    |   |-- 152.63.32.133
    |   |-- 152.63.2.249
    |   |-- 152.63.37.117
    |   |-- 152.63.34.201
  5.|-- TenGigE0-4-0-1.GW7.IAD8.ALTER.NET                 0.0%   100   14.5  17.1  12.4  31.8   2.6
    |  `|-- 152.63.37.78
    |   |-- 152.63.37.82
    |   |-- 152.63.32.190
    |   |-- 152.63.32.194
    |   |-- 152.63.37.150
  6.|-- google-gw.customer.alter.net                     47.0%   100   53.2  51.5  43.5 101.8   8.5
  7.|-- 216.239.46.248                                   54.0%   100   16.9  20.9  13.5 139.6  22.1
  8.|-- 66.249.94.54                                     53.0%   100   20.7  22.6  14.0  32.6   5.0
    |  `|-- 66.249.94.46
  9.|-- iad04s01-in-f103.1e100.net                       45.0%   100   24.3  20.5  17.7  25.6   1.7

 what should i conclude here? is fios' network failing? is fios / verizon doomed? why can't i watch a youtube video without verizon losing half of it?

perhaps i should report the imminent collapse of the fios network to my favorite tech blog.

[spacedebris]

I have to go back and try to find the article, but I read that Google is aware of the problem. It actually affects more than just Verizon customers. Many ISP's are having the same problem. I was reading that the problem is that many of the Google gateways were overloaded with too many connections. So they were working with Verizon and other ISP's to add additional gateway connections and bandwidth. To be honest, I just skimmed over it as it really has never affected my connnection. Youtube always plays smoothly to me. But if I remember right it, the article was saying that it had a timeline saying about 3rd Qtr of 2011. If I can find it, I'll post a link.

[Icarus1]

I was doing some troubleshooting on my connection tonight and found this exact same problem:  specifically to 152.179.50.62 google-gw.customer.alter.net, in my case.  

Alter.net used to be UUnet, MCI/Worldcom, and is now owned by Verizon.  This is solidly a Verizon issue. 

I wish we knew how to reach out to the right people from this. 

[sbrudenell]

unfortunately, the very fact that you don't have any trouble underscores verizon's need to respond here. traffic is getting from verizon to google with no problem in some circumstances / for some customers, but not all.

in any case, i hope you are wrong about the 3q 2011 date. sustained 50% loss to a major content network is unacceptable to begin with; a multiple-month turnaround time for a fix makes me think about demanding reimbursement.

[Anti-Phish1]

---

@sbrudenell wrote:

makes me think about demanding reimbursement.

---

From whom?  youtube is a free service.  The MTR you posted shows there is nothing wrong with your FIOS connection.  All the packet loss is occurring at the boundary to google's network (node 6) and within google's network (nodes 7 - 9). 

[sbrudenell]

the problem router (or at least its address) is owned by verizon, not google. the hops after the problem router all return pings just fine from any connection other than my fios. I find no evidence that there is anything wrong within google's network.

[smith6612]

152.179.50.106 and it's related google-gw.customer.alter.net (this name has many different IPs) hops have been problematic for two and a half years for me. I've got a monitor set up both from home and from my web server. The packet loss you've noticed doesn't seem to really occur so much coming from my web server, however it does seem the trend for loss coming from my connection occurs mostly during the Winter and Summer months, mainly during the evening. At the same time I also experience slow speeds to many YouTube video servers. Google websites, excluding Google Maps, not so much. Other ISPs in my area do see slow-downs to YouTube, but maybe at different times of the year. They do tend to fair better than Verizon during certain times of the day though, often achieving higher speeds and better latency.

I will say though that the DSL certainly does feel faster during the very early morning hours. Just seems to have better response times to sites despite no variation in latency. Speeds and jitter remain the same the whole time though. Right now YouTube and the Google sites are working fine for me, despite the packet loss coming from that hop and sequential hops.

[sbrudenell]

ugh. I didn't actually mean to make this post the accepted solution. I fat fingered the reply button, and now I can't find a way to undo this. can a mod help me here?

to your post: I do find different isps have different amounts of loss to the same google servers. but in all my testing so far, fios has been consistently the worst.

I have been running a smokeping test from dslreports to the problem router all day. so far it seems to have the same 50% loss as I see from over the fios connection itself. I'll continue to monitor it.

[ElizabethS]

There you go. Feel free to mark the post you feel best solves your issue.

[sbrudenell]

thanks for that.

http://www.dslreports.com/r3/smokeping.cgi?target=network.d8737730e60a9d7171215e82ba9b1b99.CA1

http://www.dslreports.com/r3/smokeping.cgi?target=network.d8737730e60a9d7171215e82ba9b1b99.NY

http://www.dslreports.com/r3/smokeping.cgi?target=network.d8737730e60a9d7171215e82ba9b1b99.KS

the above links show independent ping tests from 3 different sites to 152.179.50.106 over the last 24 hours. it does seem i was wrong that this router is continuously overloaded -- it seems to have 0 loss (green line) from 12am to 8am this morning.

however, you can still clearly see the packetloss is atrocious (purple / red line) during all other times of the day.

this loss is directly correlated with (though not a direct measurement of) performance of google / youtube from my fios connection -- i.e., unacceptable at most times of the day.

does verizon have any official response to this?

[Hubrisnxs]

---

@sbrudenell wrote:

thanks for that.

http://www.dslreports.com/r3/smokeping.cgi?target=network.d8737730e60a9d7171215e82ba9b1b99.CA1

http://www.dslreports.com/r3/smokeping.cgi?target=network.d8737730e60a9d7171215e82ba9b1b99.NY

http://www.dslreports.com/r3/smokeping.cgi?target=network.d8737730e60a9d7171215e82ba9b1b99.KS

the above links show independent ping tests from 3 different sites to 152.179.50.106 over the last 24 hours. it does seem i was wrong that this router is continuously overloaded -- it seems to have 0 loss (green line) from 12am to 8am this morning.

however, you can still clearly see the packetloss is atrocious (purple / red line) during all other times of the day.

this loss is directly correlated with (though not a direct measurement of) performance of google / youtube from my fios connection -- i.e., unacceptable at most times of the day.

does verizon have any official response to this?

---

The Verizon Community Forums are where customers help other customers. If you need assistance for specific Verizon account issues, please contact us.

@Remeber, these are community forums,  to get an official response - you may have to DM the Fios Twitter Supprt @ www.twitter.com/verizonsupport or the Direct Forums over at DSL reports.  or simply contact Verizon.  I would try the twitter and dslreports first, it feels like those are manned by higher level technicians.    Both are secure, and your privacy maintained.

[houkou_onchi]

Definitely verizon's fault? Unfortunately I don't think you understand some of the fundamentals how transit and the internet works (Even I don't claim to be an expert on this). This is likely googls over-saturating its transit to Verizon which Verizon could do nothing about or faulty hardware somewhere which google should either shut down the links with the problem and/or contact google to fix.

Just because the IP is owned by Verizon doesn't necesarily actually mean much that much; especially when its the IP before it hits google's network and doesn't for sure mean is VZ and not google. The fact is when you buy transit from a provider (like cogent, verizon, internap, AT&T, comcast, above.net, NTT, etc.. ) you will get one of their IP addresses which will be assigned to the border router so other people can route traffic to you and for setting up BGP/static routes, etc...

Here is an example when I traceroute to my work from Verizon:

traceroute to dreamhost.com (208.97.187.204), 30 hops max, 46 byte packets
 1  router.houkouonchi.jp (1.1.1.1)  0.193 ms  0.142 ms  0.506 ms
 2  L103.LSANCA-DSL-29.verizon-gni.net (71.110.63.1)  1.146 ms  0.960 ms  0.863 ms
 3  G0-3-1-7.LSANCA-LCR-22.verizon-gni.net (130.81.146.60)  1.941 ms  1.517 ms  2.401 ms
 4  so-6-0-0-0.LAX01-BB-RTR2.verizon-gni.net (130.81.29.126)  2.023 ms  2.015 ms  2.165 ms
 5  0.so-1-3-0.XL4.LAX15.ALTER.NET (152.63.114.149)  2.440 ms  2.368 ms  3.784 ms
 6  0.xe-11-0-0.BR1.LAX15.ALTER.NET (152.63.115.94)  2.164 ms  4.076 ms  2.421 ms
 7  te7-2.ccr02.lax05.atlas.cogentco.com (154.54.13.85)  2.316 ms  2.460 ms  2.388 ms
 8  te0-2-0-2.ccr21.lax01.atlas.cogentco.com (154.54.30.197)  2.846 ms  4.633 ms  3.957 ms
 9  te7-1.mpd03.lax01.atlas.cogentco.com (154.54.28.142)  3.226 ms  4.963 ms  3.517 ms
10  38.122.20.218 (38.122.20.218)  2.717 ms  5.225 ms  2.633 ms
11  ip-66-33-201-114.dreamhost.com (66.33.201.114)  3.568 ms  3.586 ms  5.171 ms
12  dreamhost.com (208.97.187.204)  3.298 ms  3.482 ms  3.631 ms

Ok so that 10th hop is still cogentco right? Its cogent's IP adress, If I saw a big latency spike and/or packetloss at that hop its gotta be cogentco's fault right?.. Nope! That is just the IP address cogent supplied us with. When I access it I just get our border router:

$ ssh 38.122.20.218
Password:

Alc-Border01#

My point is that its impossible to know for sure whos side fault lies but its most likely on google's end from the hop name. It is very possible that Google is simply pushing too much traffic on some of their routers and that is why there is an excessive amount of packetloss. That being said this is likely something that *both* google and Verizon will need to get involved with (likely goole needs to deal with VZ directly on this) as there is obviously a problem here between their two networks. Either way this certainly can't just be written off as VZ's problem simply because its VZ's IP address. That is like me saturating my connection causing 500ms latency and you guys complaining to VZ for the high latency to my server just because I am saturating it and running my link at capacity but it must be VZ's fault because they own the IP address.

Trust me if our cogent connection was at capacity you would be seeing packetloss on that 10th hop (38.122.20.218) just like you are right now to google on VZ.

[Thunker1]

"Just because the IP is owned by Verizon doesn't necesarily actually mean much that much; especially when its the IP before it hits google's network and doesn't for sure mean is VZ and not google."

This is just wrong. The IP is registered to verizon as a business class router.

It is their responsibilty and I as a customer of theirs cannot go anywhere on the Internet without passing through the overloaded piece of crap.

A network ahead....yea right!

[houkou_onchi]

---

@Thunker wrote:

"Just because the IP is owned by Verizon doesn't necesarily actually mean much that much; especially when its the IP before it hits google's network and doesn't for sure mean is VZ and not google."

This is just wrong. The IP is registered to verizon as a business class router.

It is their responsibilty and I as a customer of theirs cannot go anywhere on the Internet without passing through the overloaded piece of crap.

A network ahead....yea right!

---

Um.  Please show me where that IP is registerd as a 'verizon business class router.' Atleast provide some evidence with your claim like I did.

[ttist25]

This is definitely a Verizon issue.  I'm a network engineer and have clients all over the country.  I had a DSL user who was experiencing slow connection speeds and while I was troubleshooting, I had them pinging google.com with -t switch.  Thinking there was an intermittent connectivity issue because of the amount of packet loss and it took FOREVER for me to figure out that it was only google.com that had the dropped packets.  yahoo.com had steady replies as did any other domain I tried pinging.  

I thought perhaps it was a localized issue and Verizon just didn't give a hoot because of the small user base in the area.   Then I got FIOS at my home.  I noticed that, although my Internet speeds were really fast, certain pages took FOREVER to load.  When I looked at the status bar for those pages, the URL it was requesting was some kind of Google image server (I can't remember the exact URL but when I looked it up it was owned by google.  

So then I did some testing and hopped around the country from client to client via VPN tunnel connections and guess what - where the remote location was a provider other than Verizon, steady pings 0% packet loss to Google.  EVERYWHERE there was Verizon service DSL or otherwise, SERIOUS packet loss.  

Hmmmmmm?  

It doesn't take a rocket scientist to figure out where the problem lies here.  Verizon - get your routers straight.  Where the heck are these packets going?  

[Hubrisnxs]

have you guys done a google search for "packet loss to youtube"

Do you see how many different providers are talked about?

What's kinda funny is if you put the same search request in Bing! it gives you slightly more authoritative information about the reasons why, and that google and youtube acknowledged and were trying to fix etc.....    

just thought that was interesting

[mike19691]

Hi I live in PA as well and have the exact same problems with youtube you are describing and I  have FIOS 20/5. I was wondering is you also live in an MDU (multi-dwelling unit)/apartment building as I do, where we get our FIOS internet over the telephone line.

[sbrudenell]

no, i'm in a single-family unit (i.e., a house). i have a verizon-installed NID on the side of my house: fiber goes in one side, and ethernet comes out the other.

this problem is unrelated to physical connections. my physical connection and my routing to the rest of the internet remain crystal clear and fantastically stable. my routing to google was similar before the last few months.

it's likely that this problem affects some percentage of all fios users in the northeastern united states -- there seem to be enough anecdotes around the internet to support this. dsl users may be affected too -- i expect this problem to be under-reported by dsl users, because it's easier for them (and forum contributors and techs) to blame trouble on the physical connection.

[Dwooten76]

I have also noticed this problem with many Streaming sites and with Games that I play.  I have also googled and found this post plus many more complaints not listed here.  It sounds like to me that this is a major problem with many Verizon users across the US.  I wonder how long it will take to fix this problem and what type of reimbursement they will give.  I am loosing money out here because of this problem.  I am unable to buy the products and use them effectively because of these problems.

[boom4u]

they arent responsible for your financial loss so you will not be compensated, no ISP has ever been responsible for financial loss, if it is so important to you financially, you should have a backup PC, backup montiors, and a backup internet connection, this is your responsibility, not verizon's

pingtests barely count as research, they are examples of a problem occuring to you, valuable data in the right hands to be reviewed, probably not beneficial in a forum.

Ive worked at ISP's and cable plants, and various other internet related entities, for over 20 years...