TRACE ROUTE Odd Results FiOS Network

[alwayscurious]

This is a security related post for the experienced users.

My first phone call to Verizon Tech Support yielded an answer of:

Change your wi-fi password.

I did that but of course, no difference, so thought I would ask here.

BACKGROUND:

Am a longtime FiOS Quantum customer in the mid-atlantic area, with Windows 10 systems. My connectivity is working just fine. I have multiple security products running on my home network. I am using Verizon's DHCP recommended settings for my router. The DNS IP addresses are verified as belonging to Verizon.

When I open a command window and run a TRACERT command to check an IP address or domain name, I expect the display of results to  show every hop, or timeout, on routers from my location to the destination IP address.

THE ODD RESULTS:

As of yesterday, Nov 29, 2018, TRACERT only shows 2 HOPS, no matter what IP address or domain name I type in. The first hop is my own router, the second hop is the destination IP. The milliseconds time will vary, but the response is very quick.

This is especially odd because some of the IP addresses that I have checked would take serveral hops (in the past) and confirmed on https://dawhois.com. Multiple hops do not send results as fast as the 2 hop responses are occuring.

I am not receiving any errors that ICMP echo requests are being suppressed.

My firewall(s) are not showing notifications of any compromise or network intrusion.

QUESTIONS:

1. Is anyone aware of a compromise that would produce the 2 hop results?

2. Could this be a man-in-the-middle (MITM) situation, where cached ARP or DNS results are being returned instantly?

3. Could Verizon is suppressing ICMP echo requests for security reasons?

4. Could Verizon be using a technology to display cached results for speed?

5. Is anyone else getting only 2 hops on their TRACERT results.  Try an opposite coast query. Example, I am on east coast, so a west coast for LATIMES.COM has multiple hops.

6. Could this be related to Eternal Blue vulnerability at VERIZON network level that is currently in the news?

https://www.digit.in/security-software/nsa-hacking-tools-being-used-to-attack-thousands-of-computers...

https://www.theregister.co.uk/2018/11/30/akamai_routerwreckers_active/

Your thoughtful comments and guidance would be greatly appreciated.

Many thanks in advance!

Always Curious

[kilrathiace]

i get the same thing except i get only 1 hop but i am on direct ethernet to ont.

[ingenium13]

I thought there was something wrong with my equipment (pfsense). I noticed the exact same thing and have been trying to get it resolved. On Linux and FreeBSD (pfsense), traceroute works, but mtr (more advanced traceroute) fails, always showing my upstream gateway as the final hop. To my knowledge, mtr and traceroute both use the same technique, but they must be doing something differently for one to work and one to fail.

mtr -rn 8.8.8.8
Start: Thu Dec  6 18:32:26 2018
HOST: server                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.42.1               0.0%    10    0.2   0.3   0.2   0.3   0.0
  2.|-- 8.8.8.8                    0.0%    10    1.3   1.5   0.9   3.4   0.7


traceroute -n 8.8.8.8                                   
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  192.168.42.1  0.187 ms  0.150 ms  0.143 ms
 2  * * *
 3  100.41.195.180  2.888 ms 100.41.195.182  1.664 ms 100.41.195.180  2.823 ms
 4  * * *
 5  * * *
 6  140.222.2.199  11.667 ms  11.576 ms  11.580 ms
 7  204.148.79.46  11.514 ms  10.497 ms  16.379 ms
 8  108.170.246.65  12.063 ms 108.170.246.1  11.468 ms 108.170.246.33  13.188 ms
 9  209.85.252.23  9.609 ms 72.14.239.85  11.341 ms 108.170.229.65  12.840 ms
10  8.8.8.8  10.440 ms  9.811 ms  10.049 ms

Inbound mtr works correctly:

mtr -rn MYIP
Start: Thu Dec  6 15:34:08 2018
HOST: MYVM.xen.prgmr.com          Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 71.19.144.2                0.0%    10    0.2   0.3   0.2   0.5   0.0
  2.|-- 216.218.133.65             0.0%    10    0.8   1.0   0.7   2.4   0.5
  3.|-- 184.105.65.114             0.0%    10    2.1   1.2   0.9   2.1   0.3
  4.|-- 204.255.168.25             0.0%    10    1.2   1.0   0.9   1.2   0.0
  5.|-- 140.222.231.33             0.0%    10   64.0  64.3  63.1  65.8   0.9
  6.|-- 100.41.195.181             0.0%    10   62.9  63.1  62.5  65.9   0.9
  7.|-- MYIP                       0.0%    10   62.6  62.5  62.4  62.7   0.0

[Capricorn1]

What I got from my Windows 10 desktop through my router/firewall is:

tracert www.bandhphotovideo.com

Tracing route to bhphotovideo.com [74.113.188.100]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms lan-if.inside.local [192.168.3.1]
2 4 ms 4 ms 4 ms lo0-100.CLPPVA-VFTTP-304.verizon-gni.net [72.86.37.1]
3 7 ms 7 ms 7 ms B3304.CLPPVA-LCR-21.verizon-gni.net [130.81.221.132]
4 * * * Request timed out.
5 9 ms 9 ms 9 ms 0.et-5-3-0.BR1.IAD8.ALTER.NET [140.222.239.79]
6 * 10 ms 9 ms lag-78.ear2.WashingtonDC12.Level3.net [4.68.75.97]
7 * * * Request timed out.
8 12 ms 12 ms 12 ms B-H-PHOTO-V.ear1.Newark1.Level3.net [4.34.123.46]
9 14 ms 12 ms 11 ms 74.113.191.201
10 20 ms 19 ms 19 ms 74.113.191.206
11 18 ms 19 ms 19 ms 74.113.188.100

Trace complete.

[smith6612 · Accepted solution]

The difference may have to do with ICMP Echo vs UDP Echo. Windows uses ICMP whereas a number of *nix based tools will prefer to use UDP unless you specify the -I (ICMP) Flag during command launch, or otherwise have the defaults configured differently. This is for the same reason that a WIndows machine may be able to complete a traceroute whereas Linux / Mac hosts will "star out" at the end of the trace. Not all hosts will treat a UDP Echo / Trace the same as an ICMP Echo / Trace

A number of users over at DSLReports have also noticed similar behavior. A quick summary of the thread seems to suggests that Verizon may have accidentially disabled TTL propagation in areas where network changes are being made to support IPv6 on FiOS. Check this out: https://www.dslreports.com/forum/r32136909-Has-Vz-disabled-TTL-propagation

[Capricorn1]

The must not be making any changes like that in my area (yet). From my Ubuntu firewall, I get the following:

traceroute www.bandhphotovideo.com
traceroute to www.bandhphotovideo.com (74.113.188.100), 30 hops max, 60 byte packets
 1  lo0-100.CLPPVA-VFTTP-304.verizon-gni.net (72.86.37.1)  7.149 ms  7.152 ms  7.140 ms
 2  B3304.CLPPVA-LCR-22.verizon-gni.net (130.81.221.134)  4.557 ms B3304.CLPPVA-LCR-21.verizon-gni.net (130.81.221.132)  7.118 ms  7.108 ms
 3  * * *
 4  0.et-7-3-0.BR1.IAD8.ALTER.NET (140.222.239.83)  4.368 ms  4.387 ms  4.377 ms
 5  * * *
 6  * * *
 7  B-H-PHOTO-V.ear1.Newark1.Level3.net (4.34.123.46)  9.861 ms  9.831 ms  9.211 ms
 8  74.113.191.201 (74.113.191.201)  11.753 ms  11.754 ms  11.740 ms
 9  74.113.191.206 (74.113.191.206)  16.602 ms  19.164 ms  19.115 ms
10  74.113.188.100 (74.113.188.100)  17.405 ms  17.317 ms  17.301 ms

traceroute -I www.bandhphotovideo.com
traceroute to www.bandhphotovideo.com (74.113.188.100), 30 hops max, 60 byte packets
 1  lo0-100.CLPPVA-VFTTP-304.verizon-gni.net (72.86.37.1)  6.288 ms  6.290 ms  6.290 ms
 2  B3304.CLPPVA-LCR-21.verizon-gni.net (130.81.221.132)  6.353 ms  6.362 ms  6.361 ms
 3  * * *
 4  0.et-5-3-0.BR1.IAD8.ALTER.NET (140.222.239.79)  8.841 ms  8.860 ms  8.862 ms
 5  * * *
 6  * * *
 7  B-H-PHOTO-V.ear1.Newark1.Level3.net (4.34.123.46)  11.914 ms  11.540 ms  11.508 ms
 8  74.113.191.201 (74.113.191.201)  11.498 ms  12.397 ms  12.389 ms
 9  74.113.191.206 (74.113.191.206)  19.720 ms  19.904 ms  19.899 ms
10  74.113.188.100 (74.113.188.100)  19.904 ms  19.881 ms  19.902 ms

traceroute -T www.bandhphotovideo.com
traceroute to www.bandhphotovideo.com (74.113.188.100), 30 hops max, 60 byte packets
 1  lo0-100.CLPPVA-VFTTP-304.verizon-gni.net (72.86.37.1)  4.507 ms  4.497 ms  4.485 ms
 2  B3304.CLPPVA-LCR-22.verizon-gni.net (130.81.221.134)  7.078 ms  7.068 ms  7.058 ms
 3  * * *
 4  0.et-5-3-0.BR1.IAD8.ALTER.NET (140.222.239.79)  6.931 ms 0.et-7-3-0.BR1.IAD8.ALTER.NET (140.222.239.83)  4.406 ms  4.405 ms
 5  * * *
 6  * * *
 7  B-H-PHOTO-V.ear1.Newark1.Level3.net (4.34.123.46)  9.338 ms  9.332 ms  11.858 ms
 8  74.113.191.201 (74.113.191.201)  12.360 ms  11.841 ms  11.802 ms
 9  74.113.191.206 (74.113.191.206)  16.678 ms  19.967 ms  17.376 ms
10  74.113.188.100 (74.113.188.100)  19.897 ms  19.960 ms  19.924 ms

[jonjones1]

As with you it is not reduced to two hops as original poster was saying.

has to be a glitch in certain areas.

[B_P2]

Yes, I started experiencing the same around the same time. I do not believe it's a security issue of any kind, rather a functional thing which Vz has created either on purpose, or accidentially. Inbound works fine.

C:\>tracert 8.8.8.8

Tracing route to google-public-dns-a.google.com [8.8.8.8]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms 192.168.0.1
2 9 ms 6 ms 6 ms google-public-dns-a.google.com [8.8.8.8]

Trace complete.

C:\>ping -i 3 8.8.8.8

Pinging 8.8.8.8 with 32 bytes of data:
Reply from 130.81.223.134: TTL expired in transit.
Reply from 130.81.223.134: TTL expired in transit.
Reply from 130.81.223.134: TTL expired in transit.
Reply from 130.81.223.134: TTL expired in transit.

Ping statistics for 8.8.8.8:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

[user912835]

There is no such thing as a glitch or bug that can reliably produce such a consistent result. The replies are always there in 2 ms and spoof the intended target. This has to be deliberate (or at best, a misconfiguration of technology written for places with strict internet controls like financial business call centers and China).

Technically the equipment is configured to "seamlessly" provide disinformation, but for what reason? I'm sure they'll say "security" but security of what... my guess is their ability to hide net neutrality violations and network operations problems. You can't make a convincing argument that the traffic is being routed unfairly or that their peers are overloaded if you can't get any visibility into it whatsoever.

[CRobGauth]

If you read back in the thread, it is just ICMP that is disabled.

there are still ways that you can track it.

someone made true comment it happened during network changes to implement ipv6

sometimes network upgrades offer new defaults that aren't expected.

especially if they use Linux or Unix based test tools, this issue would not be seen.