Child Item

Site Totals

61368 Questions Answered 267 Categories

google / youtube slow over fios. definitely verizon's fault (research inside), please fix

i have fios in pennsylvania. for several months, i've been observing a problem where many google / youtube servers are slow / stuttering in response. it seems to be getting worse.

the problem is easiest to notice with youtube videos. from the user's perspective, sometimes i can't even play a 240p video without it buffering. sometimes reloading the page makes it load fully almost instantly; sometimes changing (even *increasing*) the resolution makes it load instantly. the problem doesn't seem to depend on the time of day.

i have found many other threads on this and other forums (youtube support, dslreports) complaining about consistent youtube trouble over fios, so i think this problem might affect a lot of users.

from my perspective over the network, it's pretty obvious to me that the problem is a certain set of routers under verizon's control which border google's network.

here's my research:

at the network level, there seems to be severe (north of 40%!) packetloss to all hosts behind this router.

here's a youtube content host for a video i just watched:

steve-local@server:~$ host v16.lscache2.c.youtube.com

v16.lscache2.c.youtube.com has address 173.194.8.38

steve-local@server:~$ sudo ping -f -c 100 173.194.8.38

PING 173.194.8.38 (173.194.8.38) 56(84) bytes of data.

...............................................

--- 173.194.8.38 ping statistics ---

100 packets transmitted, 53 received, 47% packet loss, time 1398ms

rtt min/avg/max/mdev = 17.373/20.978/23.746/1.382 ms, pipe 3, ipg/ewma 14.127/20.853 ms

much of google seems afflicted:

steve-local@server:~$ sudo ping -f -c 100 www.google.com

PING www.l.google.com (72.14.204.103) 56(84) bytes of data.

............................................

--- www.l.google.com ping statistics ---

100 packets transmitted, 56 received, 44% packet loss, time 1436ms

rtt min/avg/max/mdev = 17.329/20.117/22.279/1.221 ms, pipe 2, ipg/ewma 14.507/19.689 ms

this amount of packetloss is totally unacceptable.

here's my traceroute to that content host:

steve-local@server:~$ traceroute 173.194.8.38

traceroute to 173.194.8.38 (173.194.8.38), 30 hops max, 60 byte packets

1 10.92.1.1 (10.92.1.1) 0.295 ms 0.286 ms 0.307 ms

2 L100.PITBPA-VFTTP-11.verizon-gni.net (72.95.222.1) 7.894 ms 7.948 ms 7.923 ms

3 G12-0-0-447.PITBPA-LCR-04.verizon-gni.net (130.81.180.36) 8.034 ms 8.020 ms 8.011 ms

4 P5-0-0.PITBPA-LCR-03.verizon-gni.net (130.81.27.250) 15.034 ms 17.704 ms 17.679 ms

5 so-15-3-0-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net (130.81.28.204) 17.537 ms 17.650 ms 17.665 ms

6 0.so-2-3-0.XL3.IAD8.ALTER.NET (152.63.30.17) 17.899 ms 0.so-2-1-0.XL3.IAD8.ALTER.NET (152.63.30.2) 14.760 ms 0.so-1-2-0.XL3.IAD8.ALTER.NET (152.63.37.117) 14.821 ms

7 TenGigE0-6-1-0.GW7.IAD8.ALTER.NET (152.63.32.194) 19.366 ms 20.904 ms TenGigE0-4-0-1.GW7.IAD8.ALTER.NET (152.63.37.154) 19.827 ms

8 google-gw.customer.alter.net (152.179.50.106) 37.220 ms 37.202 ms 37.190 ms

9 216.239.46.250 (216.239.46.250) 19.648 ms 91.456 ms 216.239.46.248 (216.239.46.248) 14.902 ms

10 216.239.48.113 (216.239.48.113) 21.163 ms 21.098 ms 216.239.48.109 (216.239.48.109) 75.934 ms

11 66.249.95.143 (66.249.95.143) 15.683 ms 18.064 ms 18.093 ms

12 173.194.8.38 (173.194.8.38) 20.528 ms 21.418 ms 22.429 ms

hop #8 (152.179.50.106) is where the problems start for me. all the earlier alter.net hops return pings with no loss, but i experience the same 40%+ loss to 152.179.50.106 and all hops afterward.

the bogus name "google-gw.customer.alter.net" does not resolve.

some other problematic google hosts show 152.179.50.62 (also with bogus name google-gw.customer.alter.net) as the problematic hop to google, but is otherwise similar. 152.179.50.106 and 152.179.50.62 are the only two i have seen so far.

there are some google hosts that also go through alter.net that have no problems at all (note that www.youtube.com just serves youtube's ui, not the videos themselves):

steve-local@server:~$ host www.youtube.com

www.youtube.com is an alias for youtube-ui.l.google.com.

youtube-ui.l.google.com has address 64.233.169.93

youtube-ui.l.google.com has address 64.233.169.136

youtube-ui.l.google.com has address 64.233.169.190

youtube-ui.l.google.com has address 64.233.169.91
steve-local@server:~$ sudo ping -f -c 100 64.233.169.91

PING 64.233.169.91 (64.233.169.91) 56(84) bytes of data.

--- 64.233.169.91 ping statistics ---

100 packets transmitted, 100 received, 0% packet loss, time 1307ms

rtt min/avg/max/mdev = 19.909/22.841/28.183/1.482 ms, pipe 3, ipg/ewma 13.204/23.052 ms
steve-local@server:~$ traceroute 64.233.169.91

traceroute to 64.233.169.91 (64.233.169.91), 30 hops max, 60 byte packets

1 10.92.1.1 (10.92.1.1) 0.286 ms 0.282 ms 0.302 ms

2 L100.PITBPA-VFTTP-11.verizon-gni.net (72.95.222.1) 8.147 ms 8.144 ms 8.135 ms

3 G12-0-0-347.PITBPA-LCR-03.verizon-gni.net (130.81.180.34) 10.521 ms 10.571 ms 10.546 ms

4 so-15-3-0-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net (130.81.28.204) 17.800 ms 17.855 ms 17.830 ms

5 0.so-2-3-0.XL3.IAD8.ALTER.NET (152.63.30.17) 20.314 ms 0.xe-4-1-0.XL3.IAD8.ALTER.NET (152.63.2.249) 17.850 ms 0.so-2-3-0.XL3.IAD8.ALTER.NET (152.63.30.17) 20.334 ms

6 0.so-3-2-0.XL3.NYC4.ALTER.NET (152.63.0.182) 25.099 ms 27.268 ms 0.xe-1-1-1.XL3.NYC4.ALTER.NET (152.63.1.146) 24.744 ms

7 TenGigE0-4-2-0.GW8.NYC5.ALTER.NET (152.63.16.73) 27.278 ms TenGigE0-6-4-0.GW8.NYC4.ALTER.NET (152.63.21.121) 63.215 ms TenGigE0-6-0-0.GW8.NYC4.ALTER.NET (152.63.22.41) 60.751 ms

8 Internet-gw.customer.alter.net (152.179.72.66) 23.130 ms 21.442 ms 21.323 ms

9 216.239.43.114 (216.239.43.114) 28.817 ms 209.85.252.215 (209.85.252.215) 23.902 ms 23.769 ms

10 72.14.239.93 (72.14.239.93) 38.671 ms 43.725 ms 40.372 ms

11 64.233.175.109 (64.233.175.109) 24.275 ms 21.282 ms 22.422 ms

12 72.14.232.21 (72.14.232.21) 33.984 ms 216.239.49.145 (216.239.49.145) 36.295 ms 36.274 ms

13 yo-in-f91.1e100.net (64.233.169.91) 21.187 ms 21.325 ms 21.154 ms

this host goes through alter.net, but hits a different border gateway (152.179.72.66, Internet-gw.customer.alter.net) on the way to google.

all google-owned sites aggressively use dns-based localization, so www.google.com doesn't always resolve to a host on the other side of the problem routers. this explains why refreshing the page sometimes helps, as a slightly different address can make all the difference.

users can sort of work around the problem by using a different public dns server (8.8.8.8 and 8.8.4.4; 4.2.2.1 through 4.2.2.6; or 74.82.42.42). if you're lucky, you'll get a favorable address for the host you're trying to get to.

the addresses of the problem routers, 152.179.50.106 and 152.179.50.62, are owned by verizon business (http://bgp.he.net/ip/152.179.50.106 , http://bgp.he.net/AS701, http://en.wikipedia.org/wiki/UUNET). i'm only a networking hobbyist, but i don't see any way that that this can be anyone's responsibility other than verizon's.

so please fix it.

Comments

sbrudenell

mtr shows a concise, visual view of the problem:

root@router:~# mtr -4 --report -w -c 100 www.google.com  
HOST: router                                           Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- L100.PITBPA-VFTTP-11.verizon-gni.net              0.0%   100    8.8   6.2   3.8   9.7   1.3
  2.|-- G12-0-0-347.PITBPA-LCR-03.verizon-gni.net         0.0%   100    8.5   7.3   4.0  10.9   1.3
  3.|-- so-15-3-0-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net  1.0%   100   11.8  16.4  11.0 104.4  14.2
  4.|-- 0.so-2-1-0.XL3.IAD8.ALTER.NET                     0.0%   100   13.1  20.2  12.3 134.9  17.5
    |  `|-- 152.63.30.17
    |   |-- 152.63.36.233
    |   |-- 152.63.32.133
    |   |-- 152.63.2.249
    |   |-- 152.63.37.117
    |   |-- 152.63.34.201
  5.|-- TenGigE0-4-0-1.GW7.IAD8.ALTER.NET                 0.0%   100   14.5  17.1  12.4  31.8   2.6
    |  `|-- 152.63.37.78
    |   |-- 152.63.37.82
    |   |-- 152.63.32.190
    |   |-- 152.63.32.194
    |   |-- 152.63.37.150
  6.|-- google-gw.customer.alter.net                     47.0%   100   53.2  51.5  43.5 101.8   8.5
  7.|-- 216.239.46.248                                   54.0%   100   16.9  20.9  13.5 139.6  22.1
  8.|-- 66.249.94.54                                     53.0%   100   20.7  22.6  14.0  32.6   5.0
    |  `|-- 66.249.94.46
  9.|-- iad04s01-in-f103.1e100.net                       45.0%   100   24.3  20.5  17.7  25.6   1.7

what should i conclude here? is fios' network failing? is fios / verizon doomed? why can't i watch a youtube video without verizon losing half of it?

perhaps i should report the imminent collapse of the fios network to my favorite tech blog.

spacedebris

I have to go back and try to find the article, but I read that Google is aware of the problem. It actually affects more than just Verizon customers. Many ISP's are having the same problem. I was reading that the problem is that many of the Google gateways were overloaded with too many connections. So they were working with Verizon and other ISP's to add additional gateway connections and bandwidth. To be honest, I just skimmed over it as it really has never affected my connnection. Youtube always plays smoothly to me. But if I remember right it, the article was saying that it had a timeline saying about 3rd Qtr of 2011. If I can find it, I'll post a link.

Icarus1

I was doing some troubleshooting on my connection tonight and found this exact same problem: specifically to 152.179.50.62 google-gw.customer.alter.net, in my case.

Alter.net used to be UUnet, MCI/Worldcom, and is now owned by Verizon. This is solidly a Verizon issue.

I wish we knew how to reach out to the right people from this.

sbrudenell

unfortunately, the very fact that you don't have any trouble underscores verizon's need to respond here. traffic is getting from verizon to google with no problem in some circumstances / for some customers, but not all.

in any case, i hope you are wrong about the 3q 2011 date. sustained 50% loss to a major content network is unacceptable to begin with; a multiple-month turnaround time for a fix makes me think about demanding reimbursement.

Anti-Phish1

@sbrudenell wrote:
makes me think about demanding reimbursement.
From whom? youtube is a free service. The MTR you posted shows there is nothing wrong with your FIOS connection. All the packet loss is occurring at the boundary to google's network (node 6) and within google's network (nodes 7 - 9).

sbrudenell

the problem router (or at least its address) is owned by verizon, not google. the hops after the problem router all return pings just fine from any connection other than my fios. I find no evidence that there is anything wrong within google's network.

smith6612

152.179.50.106 and it's related google-gw.customer.alter.net (this name has many different IPs) hops have been problematic for two and a half years for me. I've got a monitor set up both from home and from my web server. The packet loss you've noticed doesn't seem to really occur so much coming from my web server, however it does seem the trend for loss coming from my connection occurs mostly during the Winter and Summer months, mainly during the evening. At the same time I also experience slow speeds to many YouTube video servers. Google websites, excluding Google Maps, not so much. Other ISPs in my area do see slow-downs to YouTube, but maybe at different times of the year. They do tend to fair better than Verizon during certain times of the day though, often achieving higher speeds and better latency.

I will say though that the DSL certainly does feel faster during the very early morning hours. Just seems to have better response times to sites despite no variation in latency. Speeds and jitter remain the same the whole time though. Right now YouTube and the Google sites are working fine for me, despite the packet loss coming from that hop and sequential hops.

sbrudenell

ugh. I didn't actually mean to make this post the accepted solution. I fat fingered the reply button, and now I can't find a way to undo this. can a mod help me here?

to your post: I do find different isps have different amounts of loss to the same google servers. but in all my testing so far, fios has been consistently the worst.

I have been running a smokeping test from dslreports to the problem router all day. so far it seems to have the same 50% loss as I see from over the fios connection itself. I'll continue to monitor it.

ElizabethS

There you go. Feel free to mark the post you feel best solves your issue.

mike19691

Hi I live in PA as well and have the exact same problems with youtube you are describing and I have FIOS 20/5. I was wondering is you also live in an MDU (multi-dwelling unit)/apartment building as I do, where we get our FIOS internet over the telephone line.

sbrudenell

thanks for that.

http://www.dslreports.com/r3/smokeping.cgi?target=network.d8737730e60a9d7171215e82ba9b1b99.CA1

http://www.dslreports.com/r3/smokeping.cgi?target=network.d8737730e60a9d7171215e82ba9b1b99.NY

http://www.dslreports.com/r3/smokeping.cgi?target=network.d8737730e60a9d7171215e82ba9b1b99.KS

the above links show independent ping tests from 3 different sites to 152.179.50.106 over the last 24 hours. it does seem i was wrong that this router is continuously overloaded -- it seems to have 0 loss (green line) from 12am to 8am this morning.

however, you can still clearly see the packetloss is atrocious (purple / red line) during all other times of the day.

this loss is directly correlated with (though not a direct measurement of) performance of google / youtube from my fios connection -- i.e., unacceptable at most times of the day.

does verizon have any official response to this?

sbrudenell

no, i'm in a single-family unit (i.e., a house). i have a verizon-installed NID on the side of my house: fiber goes in one side, and ethernet comes out the other.

this problem is unrelated to physical connections. my physical connection and my routing to the rest of the internet remain crystal clear and fantastically stable. my routing to google was similar before the last few months.

it's likely that this problem affects some percentage of all fios users in the northeastern united states -- there seem to be enough anecdotes around the internet to support this. dsl users may be affected too -- i expect this problem to be under-reported by dsl users, because it's easier for them (and forum contributors and techs) to blame trouble on the physical connection.

Dwooten76

I have also noticed this problem with many Streaming sites and with Games that I play. I have also googled and found this post plus many more complaints not listed here. It sounds like to me that this is a major problem with many Verizon users across the US. I wonder how long it will take to fix this problem and what type of reimbursement they will give. I am loosing money out here because of this problem. I am unable to buy the products and use them effectively because of these problems.

boom4u

they arent responsible for your financial loss so you will not be compensated, no ISP has ever been responsible for financial loss, if it is so important to you financially, you should have a backup PC, backup montiors, and a backup internet connection, this is your responsibility, not verizon's

pingtests barely count as research, they are examples of a problem occuring to you, valuable data in the right hands to be reviewed, probably not beneficial in a forum.

Ive worked at ISP's and cable plants, and various other internet related entities, for over 20 years...

sbrudenell

i never said financial loss.

i'm paying for a service and not receiving it. some packetloss is unavoidable, but 50% loss during daytime hours is totally unacceptable. this is in no way the service i'm paying for.

pinging one host from a single connection doesn't tell you much, true. that's why i tested the reachability of

- the problem router

- the hop before the problem router

- google hosts behind the problem router

- other google hosts not behind the problem router

- other non-google hosts

all from

- my fios connection

- 3 other sites whene i have servers

- 3 other sites using dslreports' smokeping

to try to isolate the problem.

the problem i see is that 99% of forum posts consist of "halp my conn iz bad its not my fault". so few of them have a real problem, and it's so hard to get them to provide useful information, that they're easy to ignore. my hope is to provide enough information to get the attention of the right people.

perhaps i provided too much information, since you don't seem to have read anything i wrote?

i counter your proof-by-authority argument with a challenge: tell me what you've seen in your 20 years of experience can explain my observations, other than an overloaded router verizon needs to fix. put your money where your mouth is.

the problem still persists, by the way.

Hubrisnxs

@sbrudenell wrote:
thanks for that.
http://www.dslreports.com/r3/smokeping.cgi?target=network.d8737730e60a9d7171215e82ba9b1b99.CA1
http://www.dslreports.com/r3/smokeping.cgi?target=network.d8737730e60a9d7171215e82ba9b1b99.NY
http://www.dslreports.com/r3/smokeping.cgi?target=network.d8737730e60a9d7171215e82ba9b1b99.KS
the above links show independent ping tests from 3 different sites to 152.179.50.106 over the last 24 hours. it does seem i was wrong that this router is continuously overloaded -- it seems to have 0 loss (green line) from 12am to 8am this morning.
however, you can still clearly see the packetloss is atrocious (purple / red line) during all other times of the day.
this loss is directly correlated with (though not a direct measurement of) performance of google / youtube from my fios connection -- i.e., unacceptable at most times of the day.
does verizon have any official response to this?

The Verizon Community Forums are where customers help other customers. If you need assistance for specific Verizon account issues, please contact us.

@Remeber, these are community forums, to get an official response - you may have to DM the Fios Twitter Supprt @ www.twitter.com/verizonsupport or the Direct Forums over at DSL reports. or simply contact Verizon. I would try the twitter and dslreports first, it feels like those are manned by higher level technicians. Both are secure, and your privacy maintained.

rupok93

I have been getting the same exact problem! Usually its not as bad as it has been for the last couple of weeks!

Before changing resolution or refressing the page worked but now it doesn't work at all. I thought it was just me but I asked my friend who also has fios and he has the same problem. We are in Jersey City, NJ,

Verizon and google PLEASE FIX THIS PROBLEM!!!

To the people saying its another problem, its not. When many people complain about the same problems with same situations its not an individuals fault.