High latency and poor speed tests lately

[MisterJefferson]

For the past 4-5 days, I've been working with various Verizon techs to resolve a high latency/slow Internet problem.  Their ultimate solution was that they have no idea why it's doing this, and since I can reach a few websites at full speed and their line tests show I'm operating at full provisioning, it's not their problem.  "We can't control traffic outside of our networks."  Well, my results are also showing problems with the Verizon servers, so I would think they should be more concerned.  Most websites, however, exhibit the problem.  So I'm stuck, looking for a better answer than "go away."  Maybe some of you are experiencing a similar problem with latency or sporadic speeds?

My provisioning is 20M/5M.  I'm in the L.A. area. (In the traceroutes below, I masked Hop 2 for privacy.)

I've experienced this problem with 2 computers (1 desktop, 1 laptop), both using Ethernet ports; one test using wireless on the laptop.  I've plugged the router directly into the ONT coax to eliminate my house wiring as the culprit.  Westell 9100EM router.

A few websites seem okay, and speedtest.net results always show provisioning at full speed to the L.A. server, but the problem persists when using the non-L.A. server.  With speakeasy.net, even the L.A. server exhibits the problem.  The latency gets worse as the night goes on, sometimes over 1000ms for some traceroute hops.  In the morning, it seems to be back to normal.

Thanks for any help....

Jeff

Some examples...

In the past 2 mornings, I've had good results:

Speed Test (speakeasy.net, Los Angeles): ~ 19798K/4581K

Speed Test (speedtest.net, Los Angeles): always shows full provisioning ~19.74M, 4.56M, 19ms ping

Speed Test (speedtest.net, Dallas): ~ 19.81M/4.37M, 49ms ping

Tracing route to www.l.google.com [74.125.19.104]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  myrouter.home [192.168.1.1]
  2     8 ms     7 ms     7 ms  xxxx.xxxxx-xxxxx-xxxxx.verizon-gni.net [173.55.xxx.xxx]
  3     8 ms    10 ms     9 ms  G4-0-6-1142.LSANCA-LCR-11.verizon-gni.net [130.81.140.122]
  4    11 ms    53 ms    11 ms  so-4-3-0-0.LAX01-BB-RTR1.verizon-gni.net [130.81.29.244]
  5    13 ms    12 ms    12 ms  0.so-1-2-0.XL3.LAX15.ALTER.NET [152.63.10.133]
  6    12 ms    11 ms    13 ms  0.ge-7-0-0.BR2.LAX15.ALTER.NET [152.63.116.157]
  7     9 ms    12 ms    42 ms  xe-11-0-0.edge1.SanJose3.level3.net [4.68.111.249]
  8    14 ms    12 ms    17 ms  ae-63-60.ebr3.LosAngeles1.Level3.net [4.69.144.52]
  9    21 ms    34 ms    35 ms  ae-2.ebr3.SanJose1.Level3.net [4.69.132.9]
 10    26 ms    19 ms    19 ms  ae-83-83.csw3.SanJose1.Level3.net [4.69.134.234]
 11    21 ms    22 ms    22 ms  ae-3-89.edge1.SanJose1.Level3.net [4.68.18.142]
 12    23 ms    22 ms    23 ms  GOOGLE-INC.edge1.SanJose1.Level3.net [4.79.43.146]
 13    29 ms    34 ms    22 ms  209.85.251.98
 14    20 ms    23 ms    19 ms  nuq04s01-in-f104.1e100.net [74.125.19.104]

In the past 4-5 evenings, I've had poor results:

Speed Test (speakeasy.net, Los Angeles): ~ 2441K/194K

Speed Test (speedtest.net, Los Angeles): always shows full provisioning ~19.74M, 4.56M, 19ms ping

Speed Test (speedtest.net, Dallas): ~ 8.28M/1.22M, 49ms ping

Tracing route to www.l.google.com [74.125.19.105]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  myrouter.home [192.168.1.1]
  2     7 ms     7 ms     7 ms  xxxx.xxxxx-xxxxx-xxx.verizon-gni.net [173.55.xxx.xxx]
  3   531 ms   502 ms   514 ms  G4-0-6-1242.LSANCA-LCR-12.verizon-gni.net [130.81.140.162]
  4   519 ms   506 ms   530 ms  so-4-3-0-0.LAX01-BB-RTR2.verizon-gni.net [130.81.29.246]
  5   563 ms   564 ms   553 ms  0.so-1-2-0.XL4.LAX15.ALTER.NET [152.63.10.125]
  6   577 ms   544 ms   524 ms  0.ge-6-1-0.BR2.LAX15.ALTER.NET [152.63.116.153]
  7   477 ms   469 ms   459 ms  xe-10-1-0.edge1.LosAngeles9.Level3.net [4.68.63.129]
  8    11 ms    17 ms    19 ms  ae-93-90.ebr3.LosAngeles1.Level3.net [4.69.144.244]
  9   515 ms   559 ms   536 ms  ae-2.ebr3.SanJose1.Level3.net [4.69.132.9]
 10    20 ms    19 ms    19 ms  ae-83-83.csw3.SanJose1.Level3.net [4.69.134.234]
 11   524 ms   499 ms   544 ms  ae-3-89.edge1.SanJose1.Level3.net [4.68.18.142]
 12   490 ms   479 ms   464 ms  GOOGLE-INC.edge1.SanJose1.Level3.net [4.79.43.146]
 13    24 ms    22 ms    32 ms  209.85.251.98
 14    23 ms    22 ms   489 ms  nuq04s01-in-f105.1e100.net [74.125.19.105]

[chernabog]

Similar problem for me too. I just had FIOS installed and I'm getting excellend download/upload but terrible ping. My results are as follows:

Tracing route to google.com [74.125.67.100]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  myrouter.home [192.168.1.1]
  2     4 ms     4 ms     5 ms  L100.LSANCA-VFTTP-142.verizon-gni.net [173.55.53.1]
  3     9 ms     9 ms     8 ms  G4-0-6-1142.LSANCA-LCR-11.verizon-gni.net [130.81.140.122]
  4     9 ms     9 ms     8 ms  so-4-3-0-0.LAX01-BB-RTR1.verizon-gni.net [130.81.29.244]
  5    11 ms    10 ms    10 ms  0.so-1-2-0.XL3.LAX15.ALTER.NET [152.63.10.133]
  6    11 ms    10 ms    11 ms  0.ge-6-0-0.BR2.LAX15.ALTER.NET [152.63.116.149]
  7    10 ms     9 ms    10 ms  192.205.35.161
  8    66 ms    65 ms    66 ms  cr2.la2ca.ip.att.net [12.123.30.134]
  9    66 ms    65 ms    65 ms  cr2.dlstx.ip.att.net [12.122.28.177]
 10   104 ms    97 ms   131 ms  cr1.attga.ip.att.net [12.122.28.173]
 11    65 ms    64 ms    64 ms  12.123.22.5
 12   135 ms   124 ms   135 ms  12.88.97.6
 13   110 ms   199 ms    92 ms  72.14.233.54
 14    66 ms   119 ms   119 ms  72.14.239.127
 15   152 ms   167 ms   162 ms  209.85.255.194
 16   120 ms   107 ms   111 ms  gw-in-f100.1e100.net [74.125.67.100]

Trace complete.

Granted these results are also 4x better than they were 10 min ago. I was getting most pings in the 400 range but a couple were over 600! What's going on Verizon?

[spacedebris]

MisterJefferson, your problem looks to be with hop #3. That is a Verizon router just after the gateway router for you. This will need to be addressed by a Verizon Network technician. Your hops to your router and your hop to the gateway look fine. Verizon will need to reboot that second router or re-route your hops.

Chernabog, yours is more difficult, you dont have any obvious big issue. However you do have a moderate jump at hop number 8. this is off the verizon network and is out on the internet. on one of the ATT routers. Unlikely you are going to be able to do anything about this. And as it is off the Verizon network, there would be nothing verizon could do about it either. Usually though, if it is a router needing rebooting, it will generally self correct after a few days. Or you could be unlucky enough to be going through a system that is overloaded or just plain old. This can be frustraiting. I myself play a lot of online games and if the server is in Washington DC (I'm in California) one of the hops I have to go through is in kentucky with sprintlink and their routers constantly give me a 500ms ping and I havent been able to get it corrected in over 4 years. Something we sometimes have to live with when it is out of our control.

[spacedebris]

MisterJefferson I also noticed that the Hop number 3 changed between your good results and your bad results.

On the good it was

3     8 ms    10 ms     9 ms  G4-0-6-1142.LSANCA-LCR-11.verizon-gni.net [130.81.140.122]

On the bad it was

3   531 ms   502 ms   514 ms  G4-0-6-1242.LSANCA-LCR-12.verizon-gni.net [130.81.140.162]

So it looks like when they switch you to that LCR-12 you have an issue. I'd contact Verizon Tech support, tell them that you are having a network issue and have them contact a network technition and have them look at that IP 130.81.140.162. It likely needs to be rebooted, reset, or replaced.

[nestacio]

Well, I'm glad it's not just me -- also getting terrible ping the last week or so. I used to see pings in the ~20ms range throughout the entire day, but now I am experiencing the same behavior: good ping in the morning, and successively worse and worse values throughout the day. Sometimes going higher than ~900ms in the evening. I'm also in Los Angeles.

Here are the results of my traceroute:

traceroute: Warning: www.l.google.com has multiple addresses; using 74.125.19.105

traceroute to www.l.google.com (74.125.19.105), 64 hops max, 52 byte packets

 1  myrouter (192.168.1.1)  0.634 ms  0.331 ms  0.478 ms

 2  l100.lsanca-vfttp-142.verizon-gni.net (173.55.53.1)  7.985 ms  6.892 ms  7.483 ms

 3  g4-0-6-1242.lsanca-lcr-12.verizon-gni.net (130.81.140.162)  210.618 ms  219.254 ms  202.147 ms

 4  so-4-3-0-0.lax01-bb-rtr2.verizon-gni.net (130.81.29.246)  205.109 ms  194.284 ms  210.358 ms

 5  0.so-1-2-0.xl4.lax15.alter.net (152.63.10.125)  259.831 ms  201.562 ms  207.601 ms

 6  0.ge-6-1-0.br2.lax15.alter.net (152.63.116.153)  15.467 ms  24.865 ms  27.213 ms

 7  xe-10-1-0.edge1.losangeles9.level3.net (4.68.63.129)  227.344 ms  214.266 ms  198.670 ms

 8  ae-93-90.ebr3.losangeles1.level3.net (4.69.144.244)  11.946 ms

    ae-73-70.ebr3.losangeles1.level3.net (4.69.144.116)  214.004 ms

    ae-83-80.ebr3.losangeles1.level3.net (4.69.144.180)  207.058 ms

 9  ae-2.ebr3.sanjose1.level3.net (4.69.132.9)  217.253 ms  228.709 ms  245.033 ms

10  ae-73-73.csw2.sanjose1.level3.net (4.69.134.230)  27.427 ms  34.813 ms  36.956 ms

11  ae-2-79.edge1.sanjose1.level3.net (4.68.18.78)  250.562 ms  239.283 ms  232.337 ms

12  google-inc.edge1.sanjose1.level3.net (4.79.43.146)  232.339 ms  211.787 ms  235.336 ms

13  209.85.251.98 (209.85.251.98)  252.116 ms  271.735 ms  229.820 ms

14  nuq04s01-in-f105.1e100.net (74.125.19.105)  19.967 ms  21.635 ms  25.225 ms

So hop #3 is the start of my woes; it is the EXACT switch/router location as MisterJefferson: 130.81.140.162.

What can we do about this? I'm assuming (perhaps wrongly) that Verizon is not equipped to troubleshoot this issue over the phone; I don't have a lot of confidence, based on other support calls to various services, that the customer support person has a script to deal with this type of repair request -- since it's not a repair that needs to be done to customer-owned equipment.

It's reassuring that the issue can be duplicated by someone other than myself. I'm hoping to resolve this since we have otherwise been really happy with the quality of the FIOS connection, which is amazing.

N

[spacedebris]

Yup, you have the same problem with the same router. Once one of you get to Verizon tech support and speak to a network technician, then once the techs are able to get that router either rebooted replaced then both your issues will likely be corrected.  Your right that one of the front line agents that you get when you call, will not be able to help you. They will need to get this to one of the network technicians. Hopefully, you'll get an agent that knows to send network related trouble up the line.

 3  g4-0-6-1242.lsanca-lcr-12.verizon-gni.net (130.81.140.162) 21 0.618 ms  219.254 ms  202.147 ms

maybe one of the employees here on the board can take it to one of their network techs for a look.

[VZ_Brett]

MisterJefferson .. If your still having a latency issue please direct message me your info so I can get this issue resolved thanks Brett

[nestacio]

Great to see you here Brett - MisterJefferson and I have the same issue; I took a look at the new traceroute info (as of this morning 10.23.09) and there is an improvement in the routing:

traceroute: Warning: www.l.google.com has multiple addresses; using 74.125.19.104

traceroute to www.l.google.com (74.125.19.104), 64 hops max, 52 byte packets

 1  myrouter (192.168.1.1)  0.564 ms  0.350 ms  0.488 ms

 2  l100.lsanca-vfttp-142.verizon-gni.net (173.55.53.1)  7.978 ms  6.874 ms  19.982 ms

 3  g4-0-6-1142.lsanca-lcr-11.verizon-gni.net (130.81.140.122)  10.477 ms  8.849 ms  10.232 ms

 4  so-4-3-0-0.lax01-bb-rtr1.verizon-gni.net (130.81.29.244)  12.486 ms  12.092 ms  12.217 ms

 5  0.so-1-2-0.xl3.lax15.alter.net (152.63.10.133)  12.218 ms  11.845 ms  12.466 ms

 6  0.ge-7-0-0.br2.lax15.alter.net (152.63.116.157)  99.945 ms

    0.ge-6-0-0.br2.lax15.alter.net (152.63.116.149)  107.061 ms  122.270 ms

 7  xe-11-0-0.edge1.sanjose3.level3.net (4.68.111.249)  12.437 ms

    xe-10-1-0.edge1.losangeles9.level3.net (4.68.63.129)  116.309 ms

    xe-11-0-0.edge1.sanjose3.level3.net (4.68.111.249)  11.872 ms

 8  ae-63-60.ebr3.losangeles1.level3.net (4.69.144.52)  14.479 ms  17.096 ms

    ae-73-70.ebr3.losangeles1.level3.net (4.69.144.116)  94.683 ms

 9  ae-2.ebr3.sanjose1.level3.net (4.69.132.9)  109.825 ms  111.811 ms  112.170 ms

10  ae-83-83.csw3.sanjose1.level3.net (4.69.134.234)  112.451 ms  106.826 ms  107.922 ms

11  ae-3-89.edge1.sanjose1.level3.net (4.68.18.142)  22.468 ms  21.347 ms  22.722 ms

12  google-inc.edge1.sanjose1.level3.net (4.79.43.146)  122.187 ms  117.049 ms  137.157 ms

13  209.85.251.98 (209.85.251.98)  127.426 ms  122.161 ms  149.677 ms

14  nuq04s01-in-f104.1e100.net (74.125.19.104)  145.379 ms  144.021 ms  127.655 ms

The ping is still hovering around ~130ms so the problematic equipment appears to be at hop #6 and #7 which do not seem to belong to Verizon; can the route be tweaked further?

Thanks for working on this,

nestacio

[spacedebris]

Yeah, your hop 3 is back to router #11 again. same as MisterJefferson does in the morning. In the morning you two go through #11 but in the afternoon, you go through router #12 and #12 is the problem.

unfortunatly each hop determines where the next one will go. so hop 5 determins hop 6 and so forth. Now your good on the verizon system and good on the first hop out of the verizon network. Verizon will not be able to do anything to avoid hop 6 now. Now it is a problem with the alter.net systems. This is the problem I run into, bad hops beyond Verizon. Unfortunatly there is likely nothing that can be done, short of contacting alter.net that is and praying that they give a **bleep** about someone other than their own customers. like I said in an earlier post. I've been having the same kind of problem with some routers run by sprintlink and it took me nearly 6 months to get ahold of them and they basically told me to take a flying leap since I was not their customer.

[nestacio]

Ok, Spacedebris is correct, the problem is NOT resolved, LCR-12 at 130.81.140.162 is NOTbeing re-routed, both MisterJefferson and I are being switched over from LCR-11 to LCR-12 in the evenings here in Los Angeles.

This is a problem.

Here is the traceroute as of 5:30pm Friday 10.23.09:

.......

Tracing route to www.l.google.com [74.125.19.103]

over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  myrouter.home [192.168.1.1]

  2     8 ms     6 ms     7 ms  L100.LSANCA-VFTTP-142.verizon-gni.net [173.55.53.1]

  3   458 ms   444 ms   459 ms  G4-0-6-1242.LSANCA-LCR-12.verizon-gni.net [130.81.140.162]

  4   448 ms   409 ms   430 ms  so-4-3-0-0.LAX01-BB-RTR2.verizon-gni.net [130.81.29.246]

  5   416 ms   412 ms   421 ms  0.so-1-2-0.XL4.LAX15.ALTER.NET [152.63.10.125]

  6    10 ms     9 ms    10 ms  0.ge-7-1-0.BR2.LAX15.ALTER.NET [152.63.116.161]

  7   414 ms   406 ms   419 ms  xe-10-1-0.edge1.LosAngeles9.Level3.net [4.68.63.129]

  8   471 ms   494 ms   509 ms  ae-73-70.ebr3.LosAngeles1.Level3.net [4.69.144.116]

  9   513 ms   502 ms   525 ms  ae-2.ebr3.SanJose1.Level3.net [4.69.132.9]

 10   531 ms   534 ms   531 ms  ae-93-93.csw4.SanJose1.Level3.net [4.69.134.238]

 11   514 ms   454 ms   449 ms  ae-4-99.edge1.SanJose1.Level3.net [4.68.18.206]

 12   453 ms   437 ms   454 ms  GOOGLE-INC.edge1.SanJose1.Level3.net [4.79.43.146]

 13   440 ms   447 ms   454 ms  209.85.251.98

 14   492 ms   474 ms   446 ms  nuq04s01-in-f103.1e100.net [74.125.19.103]

Trace complete.

.......

Brett / Verizon: can you fix this equipment? A 458ms ping is unacceptable. I will look around the customer service pages to see if I can open a ticket to get this resolved. If not, I will try and escalate to a manager.

In addition, alter.net is owned by Verizon Business; see prior posts. LCR-11 works in the morning until it hits alter.net. There is a breakdown in the equipment when it hits that equipment as well. Verizon, wth?

Neil