Verizon Fios Routing Issue (Server-Side)
Ifrahim
Enthusiast - Level 2

I ran a tracert and noticed that around the same time everyday. My internet latency would spike, and I have found the issue. I ran a tracert to one of the servers which I often use, and noticed that one of the hops spikes to 20+ latency. This is very unusual because I live in PA, and the routing to NY should be less than 10ms. This particular issues only happens from 8:00pm to 9:00pm everyday. I kinda got sick of it because my neighborhood pole supports up to 6 people. And half of them don't have Fios.  I know for a fact it isn't network congestion either. If someone can bring up this issue to Verizon's head technicians to fix this issue. And other than that my Fios internet is great, and I have not issues with it. 

 


1)

<1 ms <1 ms <1 ms 192.168.1.1

2)

<1 ms 3 ms 3 ms lo0-100.PHLAPA-VFTTP-322.verizon-gni.net [72.94.44.1]

3)

9 ms 7 ms 8 ms B3322.PHLAPA-LCR-21.verizon-gni.net [130.81.25.32]

4 * * * Request timed out.


5)

22 ms 17 ms 16 ms ae-14.bar4.Philadelphia1.Level3.net [4.68.38.217]

6)

8 ms 8 ms 6 ms ae2.3611.edge2.NewYork6.level3.net [4.69.209.82]


7 * * * Request timed out.


8 * * * Request timed out.

Labels (1)
0 Likes
Reply
1 Solution
smith6612
Community Leader
Community Leader

An important note to keep in mind when understanding a traceroute, is whether or not the latency you see at one hop cascades to the rest of the hops in a traceroute. Level3 / Lumen has their routers set to treat ICMP traffic used by traceroute as low priority. This is also true with other providers like Verizon. I don't see anything wrong with Hop 5 other than perhaps the management backplane on the router is a little busy at that time.

Here is similar behavior but with my ISP.

 

 

Tracing route to www.google.com [142.251.35.164]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.1.1
  2     9 ms     9 ms    10 ms  142-254-xxx-xxx.inf.spectrum.com [142.254.xxx.xxx]
  3   151 ms    33 ms    22 ms  lag-63.<>.netops.charter.com [24.58.xxx.xxx]
  4    13 ms    16 ms     9 ms  lag-27.<>.netops.charter.com [24.58.xxx.xxx]
  5    18 ms    24 ms    26 ms  lag-29.rcr01albynyyf.netops.charter.com [24.58.32.58]
  6    26 ms    27 ms    29 ms  lag-26.nycmny837aw-bcr00.netops.charter.com [24.30.201.130]
  7    29 ms    25 ms    30 ms  209.85.172.46
  8    30 ms    28 ms    25 ms  142.251.78.59
  9    28 ms    31 ms    28 ms  142.251.64.5
 10    29 ms    31 ms    28 ms  lga25s78-in-f4.1e100.net [142.251.35.164]

 

 

 

The latency spikes don't cascade from Hop 3 to the rest of the trace, and given it is almost 1AM, there isn't congestion in the mix, either. Housemates are playing their games without latency spikes or packet loss.

Now, it IS possible you're hitting congestion somewhere that is leading to small amounts of packet loss or very brief latency spikes not being seen reliably in a ping or traceroute. That could be anywhere, and without running a tool like pathping, mtr, or ping plotter, it can be hard to figure out where that may be. It might also be a problem on the return path, which you won't be able to see with your traceroute - the host on the other side would need to traceroute to your IP address to see the other side. The return path may also not be via Level3/Lumen as well, and can take a different path back on Verizon's network. It's also very possible for switching to be asymmetric (traffic exits on one link in a bundle, returns on another link in the same bundle).

A good question to ask (and apologies if I glossed over this detail) is whether or not the performance problems you're observing occur to all services / on different endpoints, and not exclusive to something routing over Level3 / Lumen. If that's the case then the problem is likely something within Verizon's network. If that's not the case, the problem could be out on the public Internet or even  at Verizon's network edge between them and another provider.

 

 

View solution in original post

5 Replies
Cang_Household
Community Leader
Community Leader

Level3.net belongs to Level 3 Communications (Lumen Technologies), so it is not on the VZ network anymore.

VZ follows the hot-potato routing philosophy to hand over the traffic to downstream ISPs as soon as possible.

Ifrahim
Enthusiast - Level 2

If it is being passed down to another ISP, isn’t VZ the one doing the routing in the first place. And is there a solution or do I have to contact Lumen, about this issue. The problem still persists. I should say. 

0 Likes
Reply
Ifrahim
Enthusiast - Level 2

I need a solution, cause this is driving me crazy. I usually come home from work, and find that my internet isn’t even routing correctly.

0 Likes
Reply
Cang_Household
Community Leader
Community Leader

By your logic, does VZ need to guarantee connection and speeds to every corner of the Internet?

This is not what the ToS stipulated. The connection you bought is the last mile delivery, meaning the section of the VZ network from a central office to your house. The speeds and link are guaranteed between the central office to your ONT.

Other experts please chime in. @gs0b @smith6612 

smith6612
Community Leader
Community Leader

An important note to keep in mind when understanding a traceroute, is whether or not the latency you see at one hop cascades to the rest of the hops in a traceroute. Level3 / Lumen has their routers set to treat ICMP traffic used by traceroute as low priority. This is also true with other providers like Verizon. I don't see anything wrong with Hop 5 other than perhaps the management backplane on the router is a little busy at that time.

Here is similar behavior but with my ISP.

 

 

Tracing route to www.google.com [142.251.35.164]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.1.1
  2     9 ms     9 ms    10 ms  142-254-xxx-xxx.inf.spectrum.com [142.254.xxx.xxx]
  3   151 ms    33 ms    22 ms  lag-63.<>.netops.charter.com [24.58.xxx.xxx]
  4    13 ms    16 ms     9 ms  lag-27.<>.netops.charter.com [24.58.xxx.xxx]
  5    18 ms    24 ms    26 ms  lag-29.rcr01albynyyf.netops.charter.com [24.58.32.58]
  6    26 ms    27 ms    29 ms  lag-26.nycmny837aw-bcr00.netops.charter.com [24.30.201.130]
  7    29 ms    25 ms    30 ms  209.85.172.46
  8    30 ms    28 ms    25 ms  142.251.78.59
  9    28 ms    31 ms    28 ms  142.251.64.5
 10    29 ms    31 ms    28 ms  lga25s78-in-f4.1e100.net [142.251.35.164]

 

 

 

The latency spikes don't cascade from Hop 3 to the rest of the trace, and given it is almost 1AM, there isn't congestion in the mix, either. Housemates are playing their games without latency spikes or packet loss.

Now, it IS possible you're hitting congestion somewhere that is leading to small amounts of packet loss or very brief latency spikes not being seen reliably in a ping or traceroute. That could be anywhere, and without running a tool like pathping, mtr, or ping plotter, it can be hard to figure out where that may be. It might also be a problem on the return path, which you won't be able to see with your traceroute - the host on the other side would need to traceroute to your IP address to see the other side. The return path may also not be via Level3/Lumen as well, and can take a different path back on Verizon's network. It's also very possible for switching to be asymmetric (traffic exits on one link in a bundle, returns on another link in the same bundle).

A good question to ask (and apologies if I glossed over this detail) is whether or not the performance problems you're observing occur to all services / on different endpoints, and not exclusive to something routing over Level3 / Lumen. If that's the case then the problem is likely something within Verizon's network. If that's not the case, the problem could be out on the public Internet or even  at Verizon's network edge between them and another provider.