TRACE ROUTE Odd Results FiOS Network

[alwayscurious]

This is a security related post for the experienced users.

My first phone call to Verizon Tech Support yielded an answer of:

Change your wi-fi password.

I did that but of course, no difference, so thought I would ask here.

BACKGROUND:

Am a longtime FiOS Quantum customer in the mid-atlantic area, with Windows 10 systems. My connectivity is working just fine. I have multiple security products running on my home network. I am using Verizon's DHCP recommended settings for my router. The DNS IP addresses are verified as belonging to Verizon.

When I open a command window and run a TRACERT command to check an IP address or domain name, I expect the display of results to  show every hop, or timeout, on routers from my location to the destination IP address.

THE ODD RESULTS:

As of yesterday, Nov 29, 2018, TRACERT only shows 2 HOPS, no matter what IP address or domain name I type in. The first hop is my own router, the second hop is the destination IP. The milliseconds time will vary, but the response is very quick.

This is especially odd because some of the IP addresses that I have checked would take serveral hops (in the past) and confirmed on https://dawhois.com. Multiple hops do not send results as fast as the 2 hop responses are occuring.

I am not receiving any errors that ICMP echo requests are being suppressed.

My firewall(s) are not showing notifications of any compromise or network intrusion.

QUESTIONS:

1. Is anyone aware of a compromise that would produce the 2 hop results?

2. Could this be a man-in-the-middle (MITM) situation, where cached ARP or DNS results are being returned instantly?

3. Could Verizon is suppressing ICMP echo requests for security reasons?

4. Could Verizon be using a technology to display cached results for speed?

5. Is anyone else getting only 2 hops on their TRACERT results.  Try an opposite coast query. Example, I am on east coast, so a west coast for LATIMES.COM has multiple hops.

6. Could this be related to Eternal Blue vulnerability at VERIZON network level that is currently in the news?

https://www.digit.in/security-software/nsa-hacking-tools-being-used-to-attack-thousands-of-computers...

https://www.theregister.co.uk/2018/11/30/akamai_routerwreckers_active/

Your thoughtful comments and guidance would be greatly appreciated.

Many thanks in advance!

Always Curious

[kilrathiace]

i get the same thing except i get only 1 hop but i am on direct ethernet to ont.

[ingenium13]

I thought there was something wrong with my equipment (pfsense). I noticed the exact same thing and have been trying to get it resolved. On Linux and FreeBSD (pfsense), traceroute works, but mtr (more advanced traceroute) fails, always showing my upstream gateway as the final hop. To my knowledge, mtr and traceroute both use the same technique, but they must be doing something differently for one to work and one to fail.

mtr -rn 8.8.8.8
Start: Thu Dec  6 18:32:26 2018
HOST: server                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.42.1               0.0%    10    0.2   0.3   0.2   0.3   0.0
  2.|-- 8.8.8.8                    0.0%    10    1.3   1.5   0.9   3.4   0.7


traceroute -n 8.8.8.8                                   
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  192.168.42.1  0.187 ms  0.150 ms  0.143 ms
 2  * * *
 3  100.41.195.180  2.888 ms 100.41.195.182  1.664 ms 100.41.195.180  2.823 ms
 4  * * *
 5  * * *
 6  140.222.2.199  11.667 ms  11.576 ms  11.580 ms
 7  204.148.79.46  11.514 ms  10.497 ms  16.379 ms
 8  108.170.246.65  12.063 ms 108.170.246.1  11.468 ms 108.170.246.33  13.188 ms
 9  209.85.252.23  9.609 ms 72.14.239.85  11.341 ms 108.170.229.65  12.840 ms
10  8.8.8.8  10.440 ms  9.811 ms  10.049 ms

Inbound mtr works correctly:

mtr -rn MYIP
Start: Thu Dec  6 15:34:08 2018
HOST: MYVM.xen.prgmr.com          Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 71.19.144.2                0.0%    10    0.2   0.3   0.2   0.5   0.0
  2.|-- 216.218.133.65             0.0%    10    0.8   1.0   0.7   2.4   0.5
  3.|-- 184.105.65.114             0.0%    10    2.1   1.2   0.9   2.1   0.3
  4.|-- 204.255.168.25             0.0%    10    1.2   1.0   0.9   1.2   0.0
  5.|-- 140.222.231.33             0.0%    10   64.0  64.3  63.1  65.8   0.9
  6.|-- 100.41.195.181             0.0%    10   62.9  63.1  62.5  65.9   0.9
  7.|-- MYIP                       0.0%    10   62.6  62.5  62.4  62.7   0.0

[smith6612 · Accepted solution]

The difference may have to do with ICMP Echo vs UDP Echo. Windows uses ICMP whereas a number of *nix based tools will prefer to use UDP unless you specify the -I (ICMP) Flag during command launch, or otherwise have the defaults configured differently. This is for the same reason that a WIndows machine may be able to complete a traceroute whereas Linux / Mac hosts will "star out" at the end of the trace. Not all hosts will treat a UDP Echo / Trace the same as an ICMP Echo / Trace

A number of users over at DSLReports have also noticed similar behavior. A quick summary of the thread seems to suggests that Verizon may have accidentially disabled TTL propagation in areas where network changes are being made to support IPv6 on FiOS. Check this out: https://www.dslreports.com/forum/r32136909-Has-Vz-disabled-TTL-propagation

[Capricorn1]

The must not be making any changes like that in my area (yet). From my Ubuntu firewall, I get the following:

traceroute www.bandhphotovideo.com
traceroute to www.bandhphotovideo.com (74.113.188.100), 30 hops max, 60 byte packets
 1  lo0-100.CLPPVA-VFTTP-304.verizon-gni.net (72.86.37.1)  7.149 ms  7.152 ms  7.140 ms
 2  B3304.CLPPVA-LCR-22.verizon-gni.net (130.81.221.134)  4.557 ms B3304.CLPPVA-LCR-21.verizon-gni.net (130.81.221.132)  7.118 ms  7.108 ms
 3  * * *
 4  0.et-7-3-0.BR1.IAD8.ALTER.NET (140.222.239.83)  4.368 ms  4.387 ms  4.377 ms
 5  * * *
 6  * * *
 7  B-H-PHOTO-V.ear1.Newark1.Level3.net (4.34.123.46)  9.861 ms  9.831 ms  9.211 ms
 8  74.113.191.201 (74.113.191.201)  11.753 ms  11.754 ms  11.740 ms
 9  74.113.191.206 (74.113.191.206)  16.602 ms  19.164 ms  19.115 ms
10  74.113.188.100 (74.113.188.100)  17.405 ms  17.317 ms  17.301 ms

traceroute -I www.bandhphotovideo.com
traceroute to www.bandhphotovideo.com (74.113.188.100), 30 hops max, 60 byte packets
 1  lo0-100.CLPPVA-VFTTP-304.verizon-gni.net (72.86.37.1)  6.288 ms  6.290 ms  6.290 ms
 2  B3304.CLPPVA-LCR-21.verizon-gni.net (130.81.221.132)  6.353 ms  6.362 ms  6.361 ms
 3  * * *
 4  0.et-5-3-0.BR1.IAD8.ALTER.NET (140.222.239.79)  8.841 ms  8.860 ms  8.862 ms
 5  * * *
 6  * * *
 7  B-H-PHOTO-V.ear1.Newark1.Level3.net (4.34.123.46)  11.914 ms  11.540 ms  11.508 ms
 8  74.113.191.201 (74.113.191.201)  11.498 ms  12.397 ms  12.389 ms
 9  74.113.191.206 (74.113.191.206)  19.720 ms  19.904 ms  19.899 ms
10  74.113.188.100 (74.113.188.100)  19.904 ms  19.881 ms  19.902 ms

traceroute -T www.bandhphotovideo.com
traceroute to www.bandhphotovideo.com (74.113.188.100), 30 hops max, 60 byte packets
 1  lo0-100.CLPPVA-VFTTP-304.verizon-gni.net (72.86.37.1)  4.507 ms  4.497 ms  4.485 ms
 2  B3304.CLPPVA-LCR-22.verizon-gni.net (130.81.221.134)  7.078 ms  7.068 ms  7.058 ms
 3  * * *
 4  0.et-5-3-0.BR1.IAD8.ALTER.NET (140.222.239.79)  6.931 ms 0.et-7-3-0.BR1.IAD8.ALTER.NET (140.222.239.83)  4.406 ms  4.405 ms
 5  * * *
 6  * * *
 7  B-H-PHOTO-V.ear1.Newark1.Level3.net (4.34.123.46)  9.338 ms  9.332 ms  11.858 ms
 8  74.113.191.201 (74.113.191.201)  12.360 ms  11.841 ms  11.802 ms
 9  74.113.191.206 (74.113.191.206)  16.678 ms  19.967 ms  17.376 ms
10  74.113.188.100 (74.113.188.100)  19.897 ms  19.960 ms  19.924 ms

[jonjones1]

As with you it is not reduced to two hops as original poster was saying.

has to be a glitch in certain areas.

[glnzglnz]

Smith, alwayscurious and others - Same here, in NYC in zip code 10023, and started about the same time.

Smith - can you give us step-by-step for getting the full tracert info?  Where exactly do we put what flag in a traceroute query?

And what do the super-fast two-line results mean?  Is it possible that I got to the distant end point in only 2ms??

[lanboyo]

Short answer: UDP and TCP works as expected, ICMP is odd.

MTR, all OS:  select the UDP option - mtr -u 8.8.8.8 , for example. _ mtr -buz 8.8.8.8 _ is my goto.

Windows: The windows tracert has no options to switch from icmp. Download a version of MTR,  install cygwin or the windows 10 bash shell environment, and use the unix traceroute, or some other solution. Not your dad.

Linux, Mac : See MTR above, but to observe the issue that others see and not feel left out, _ traceroute -P icmp 8.8.8.8 _

TMI -

It looks like they are rewriting the TTL on the outbound ICMP and the TTL on the icmp response.  Actually it looks like a TTL 1 packet hitting their network is responded by a spoof.

traceroute -P icmp na1.salesforce.com
traceroute: Warning: na1.salesforce.com has multiple addresses; using 136.147.103.71
traceroute to viv.l2.salesforce.com (136.147.103.71), 64 hops max, 72 byte packets
1 openwrt (192.168.1.1) 1.991 ms 1.097 ms 1.020 ms
2 dcl7-phx.viv-phx.salesforce.com (136.147.103.71) 4.577 ms 3.887 ms 5.135 ms

Baltimore to Phoenix and back in  ms is faster than lightspeed...

traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 72 byte packets
1 openwrt (192.168.1.1) 2.125 ms 1.042 ms 1.101 ms
2 google-public-dns-a.google.com (8.8.8.8) 5.915 ms 4.238 ms 4.865 ms

This server is an anycast server, so it is actually more like 7 MS, but still.... These packets never reach the end hosts.

ping na1.salesforce.com
PING viv.l2.salesforce.com (136.147.100.71): 56 data bytes
64 bytes from 136.147.100.71: icmp_seq=0 ttl=246 time=69.084 ms
64 bytes from 136.147.100.71: icmp_seq=1 ttl=246 time=71.081 ms
64 bytes from 136.147.100.71: icmp_seq=2 ttl=246 time=71.865 ms
64 bytes from 136.147.100.71: icmp_seq=3 ttl=246 time=70.611 ms

You see my point. 

70 ms makes more sense with lightspeed and all.

But UDP and tcp aren't munged, and pinging normally isn't messed with, so I assume there is some kind of screweyness with low TTL ICMP.

sudo tcpdump -nnnntttt -vvv -s2000 icmp or host 8.8.8.8
tcpdump: data link type PKTAP
tcpdump: listening on pktap, link-type PKTAP (Apple DLT_PKTAP), capture size 2000 bytes
2019-01-05 17:42:23.311669 IP (tos 0x0, ttl 1, id 37968, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 1, length 52
2019-01-05 17:42:23.312541 IP (tos 0xc0, ttl 64, id 9080, offset 0, flags [none], proto ICMP (1), length 100)
192.168.1.1 > 192.168.1.232: ICMP time exceeded in-transit, length 80
IP (tos 0x0, ttl 1, id 37968, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 1, length 52
2019-01-05 17:42:23.313559 IP (tos 0x0, ttl 1, id 37969, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 2, length 52
2019-01-05 17:42:23.314394 IP (tos 0xc0, ttl 64, id 9081, offset 0, flags [none], proto ICMP (1), length 100)
192.168.1.1 > 192.168.1.232: ICMP time exceeded in-transit, length 80
IP (tos 0x0, ttl 1, id 37969, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 2, length 52
2019-01-05 17:42:23.314476 IP (tos 0x0, ttl 1, id 37970, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 3, length 52
2019-01-05 17:42:23.315248 IP (tos 0xc0, ttl 64, id 9082, offset 0, flags [none], proto ICMP (1), length 100)
192.168.1.1 > 192.168.1.232: ICMP time exceeded in-transit, length 80
IP (tos 0x0, ttl 1, id 37970, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 3, length 52
2019-01-05 17:42:23.315298 IP (tos 0x0, ttl 2, id 37971, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 4, length 52
2019-01-05 17:42:23.320313 IP (tos 0xc0, ttl 254, id 0, offset 0, flags [none], proto ICMP (1), length 72)
8.8.8.8 > 192.168.1.232: ICMP echo reply, id 37967, seq 4, length 52
2019-01-05 17:42:23.320958 IP (tos 0x0, ttl 2, id 37972, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 5, length 52
2019-01-05 17:42:23.325093 IP (tos 0xc0, ttl 254, id 0, offset 0, flags [none], proto ICMP (1), length 72)
8.8.8.8 > 192.168.1.232: ICMP echo reply, id 37967, seq 5, length 52
2019-01-05 17:42:23.325165 IP (tos 0x0, ttl 2, id 37973, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 6, length 52
2019-01-05 17:42:23.330340 IP (tos 0xc0, ttl 254, id 0, offset 0, flags [none], proto ICMP (1), length 72)
8.8.8.8 > 192.168.1.232: ICMP echo reply, id 37967, seq 6, length 52

[zielkj1]

That's an explanation, not a solution.  Should not be marked as such.

Still not working as of Oct 29 2019 (over a year now).

[CRobGauth]

The solution told you how to get what you wanted.

Verizon made a change to their network.

You have options to get the same data you had before.

Highly unlikely they will change their network back.

[zielkj1]

The solution told you how to get what you wanted.  -  IS IT HIDDEN?  WHERE?

Verizon made a change to their network. - OBVIOUSLY

You have options to get the same data you had before.  NO I DON'T.  ENLIGHTEN ME

Highly unlikely they will change their network back.  YOU KNOW THAT HOW?

[dslr595148]

---

@zielkj wrote:

The solution told you how to get what you wanted.  -  IS IT HIDDEN?  WHERE?

You have options to get the same data you had before.  NO I DON'T.  ENLIGHTEN ME

---

REF the post that is marked as solved, which is post 5 of this thread.

Unix/Linux OSes use, by default, UDP in their tracerouting, which seems to be fine. Windows uses ICMP by default, which is broken.

[ee16]

This is most certainly not solved, and appears to be an explicit implementation by Verizon to obscure distributed latency measurements in their network by intentionally interfering with a basic internet protocol commonly used for this purpose (ICMP).

Verizon spoofs (replies with a fake source address) at their first hop. They do this to ICMP packet where TTL=1.  The spoofed address incorrectly signals to your traceroute client that it has reached its destination in only a couple of hops (based on most residential network setups) at an impossibly low latency.

This is easy to see and work around if you have access to a linux machine with a more sophisticated traceroute that allows specification of initial TTL using the -f flag set to 3.  This results in Verizon's first hop seeing TTL=2 (one decremented by your home router).  The traceroute then proceeds but lack the first two hops (your home router, Verizon's first hop router).

ICMP traceroute with initial TTL=3:

traceroute -I -f 3 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
3 b3365.nycmny-lcr-22.verizon-gni.net (130.81.30.150) 5.895 ms 5.884 ms 5.885 ms
4 * * *
5 0.et-10-1-2.gw15.nyc1.alter.net (140.222.227.145) 4.103 ms 4.110 ms 4.109 ms
6 72.14.214.38 (72.14.214.38) 4.488 ms 4.491 ms 4.522 ms
7 108.170.225.6 (108.170.225.6) 4.826 ms 4.827 ms 4.874 ms
8 172.253.70.5 (172.253.70.5) 4.697 ms 2.678 ms 2.669 ms
9 dns.google (8.8.8.8) 2.195 ms 2.090 ms 2.083 ms

Traceroute with an initial TTL=2 showing that Verizon instantly spoofs Google's address:

traceroute -I -f 2 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
2 dns.google (8.8.8.8) 3.670 ms 3.704 ms 3.705 ms

What a normal Windows user sees and many here are reporting:

traceroute -I -f 1 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 gw (192.168.1.1) 0.247 ms 0.279 ms 0.280 ms
2 dns.google (8.8.8.8) 2.120 ms 2.162 ms 2.212 ms

This is not solved, and UDP traceroute is not a valid replacement.  Please open support tickets to Verizon about this and follow up with a complaint to the FCC here: https://consumercomplaints.fcc.gov/hc/en-us

This will not get solved without voices.

[gs0b]

You might want to visit this thread:

https://www.dslreports.com/forum/r32764318-FiOS-and-ICMP-traceroute

[Capricorn1]

What I got from my Windows 10 desktop through my router/firewall is:

tracert www.bandhphotovideo.com

Tracing route to bhphotovideo.com [74.113.188.100]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms lan-if.inside.local [192.168.3.1]
2 4 ms 4 ms 4 ms lo0-100.CLPPVA-VFTTP-304.verizon-gni.net [72.86.37.1]
3 7 ms 7 ms 7 ms B3304.CLPPVA-LCR-21.verizon-gni.net [130.81.221.132]
4 * * * Request timed out.
5 9 ms 9 ms 9 ms 0.et-5-3-0.BR1.IAD8.ALTER.NET [140.222.239.79]
6 * 10 ms 9 ms lag-78.ear2.WashingtonDC12.Level3.net [4.68.75.97]
7 * * * Request timed out.
8 12 ms 12 ms 12 ms B-H-PHOTO-V.ear1.Newark1.Level3.net [4.34.123.46]
9 14 ms 12 ms 11 ms 74.113.191.201
10 20 ms 19 ms 19 ms 74.113.191.206
11 18 ms 19 ms 19 ms 74.113.188.100

Trace complete.

[B_P2]

Yes, I started experiencing the same around the same time. I do not believe it's a security issue of any kind, rather a functional thing which Vz has created either on purpose, or accidentially. Inbound works fine.

C:\>tracert 8.8.8.8

Tracing route to google-public-dns-a.google.com [8.8.8.8]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms 192.168.0.1
2 9 ms 6 ms 6 ms google-public-dns-a.google.com [8.8.8.8]

Trace complete.

C:\>ping -i 3 8.8.8.8

Pinging 8.8.8.8 with 32 bytes of data:
Reply from 130.81.223.134: TTL expired in transit.
Reply from 130.81.223.134: TTL expired in transit.
Reply from 130.81.223.134: TTL expired in transit.
Reply from 130.81.223.134: TTL expired in transit.

Ping statistics for 8.8.8.8:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

[user912835]

There is no such thing as a glitch or bug that can reliably produce such a consistent result. The replies are always there in 2 ms and spoof the intended target. This has to be deliberate (or at best, a misconfiguration of technology written for places with strict internet controls like financial business call centers and China).

Technically the equipment is configured to "seamlessly" provide disinformation, but for what reason? I'm sure they'll say "security" but security of what... my guess is their ability to hide net neutrality violations and network operations problems. You can't make a convincing argument that the traffic is being routed unfairly or that their peers are overloaded if you can't get any visibility into it whatsoever.

[CRobGauth]

If you read back in the thread, it is just ICMP that is disabled.

there are still ways that you can track it.

someone made true comment it happened during network changes to implement ipv6

sometimes network upgrades offer new defaults that aren't expected.

especially if they use Linux or Unix based test tools, this issue would not be seen.