Verizon Fios Routing Issue (Server-Side)

I ran a tracert and noticed that around the same time everyday. My internet latency would spike, and I have found the issue. I ran a tracert to one of the servers which I often use, and noticed that one of the hops spikes to 20+ latency. This is very unusual because I live in PA, and the routing to NY should be less than 10ms. This particular issues only happens from 8:00pm to 9:00pm everyday. I kinda got sick of it because my neighborhood pole supports up to 6 people. And half of them don't have Fios. I know for a fact it isn't network congestion either. If someone can bring up this issue to Verizon's head technicians to fix this issue. And other than that my Fios internet is great, and I have not issues with it.

<1 ms <1 ms <1 ms 192.168.1.1

<1 ms 3 ms 3 ms lo0-100.PHLAPA-VFTTP-322.verizon-gni.net [72.94.44.1]

9 ms 7 ms 8 ms B3322.PHLAPA-LCR-21.verizon-gni.net [130.81.25.32]

4 * * * Request timed out.

22 ms 17 ms 16 ms ae-14.bar4.Philadelphia1.Level3.net [4.68.38.217]

8 ms 8 ms 6 ms ae2.3611.edge2.NewYork6.level3.net [4.69.209.82]

7 * * * Request timed out.

8 * * * Request timed out.

Accepted answers

smith6612

An important note to keep in mind when understanding a traceroute, is whether or not the latency you see at one hop cascades to the rest of the hops in a traceroute. Level3 / Lumen has their routers set to treat ICMP traffic used by traceroute as low priority. This is also true with other providers like Verizon. I don't see anything wrong with Hop 5 other than perhaps the management backplane on the router is a little busy at that time.

Here is similar behavior but with my ISP.

Tracing route to www.google.com [142.251.35.164]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 9 ms 9 ms 10 ms 142-254-xxx-xxx.inf.spectrum.com [142.254.xxx.xxx]
3 151 ms 33 ms 22 ms lag-63.<>.netops.charter.com [24.58.xxx.xxx]
4 13 ms 16 ms 9 ms lag-27.<>.netops.charter.com [24.58.xxx.xxx]
5 18 ms 24 ms 26 ms lag-29.rcr01albynyyf.netops.charter.com [24.58.32.58]
6 26 ms 27 ms 29 ms lag-26.nycmny837aw-bcr00.netops.charter.com [24.30.201.130]
7 29 ms 25 ms 30 ms 209.85.172.46
8 30 ms 28 ms 25 ms 142.251.78.59
9 28 ms 31 ms 28 ms 142.251.64.5
10 29 ms 31 ms 28 ms lga25s78-in-f4.1e100.net [142.251.35.164]

The latency spikes don't cascade from Hop 3 to the rest of the trace, and given it is almost 1AM, there isn't congestion in the mix, either. Housemates are playing their games without latency spikes or packet loss.

Now, it IS possible you're hitting congestion somewhere that is leading to small amounts of packet loss or very brief latency spikes not being seen reliably in a ping or traceroute. That could be anywhere, and without running a tool like pathping, mtr, or ping plotter, it can be hard to figure out where that may be. It might also be a problem on the return path, which you won't be able to see with your traceroute - the host on the other side would need to traceroute to your IP address to see the other side. The return path may also not be via Level3/Lumen as well, and can take a different path back on Verizon's network. It's also very possible for switching to be asymmetric (traffic exits on one link in a bundle, returns on another link in the same bundle).

A good question to ask (and apologies if I glossed over this detail) is whether or not the performance problems you're observing occur to all services / on different endpoints, and not exclusive to something routing over Level3 / Lumen. If that's the case then the problem is likely something within Verizon's network. If that's not the case, the problem could be out on the public Internet or even at Verizon's network edge between them and another provider.

All comments

Cang_Household

Level3.net belongs to Level 3 Communications (Lumen Technologies), so it is not on the VZ network anymore.

VZ follows the hot-potato routing philosophy to hand over the traffic to downstream ISPs as soon as possible.

GoingHam

If it is being passed down to another ISP, isn’t VZ the one doing the routing in the first place. And is there a solution or do I have to contact Lumen, about this issue. The problem still persists. I should say.

GoingHam

I need a solution, cause this is driving me crazy. I usually come home from work, and find that my internet isn’t even routing correctly.

Cang_Household

By your logic, does VZ need to guarantee connection and speeds to every corner of the Internet?

This is not what the ToS stipulated. The connection you bought is the last mile delivery, meaning the section of the VZ network from a central office to your house. The speeds and link are guaranteed between the central office to your ONT.

Other experts please chime in. @gs0b
@smith6612

smith6612

Here is similar behavior but with my ISP.