Internet routing issue: Unable to ping and traceroute 128.0.0.1

The idea on the Juniper aspect could be pretty much spot on. Verizon loves Juniper. Their gear jush pushes data like nothing else. Problem is, I have absolutely no idea as to what version of JUNOS Verizon has in use, or if their backbone consists of a mix of gear rather than pure Juniper or for that matter how their gear is configured. My DSL line runs through something older than Verizon's ERX routers used on the DSL network, which would be a Redback/Erricson (legacy!) unit that Verizon's been trying to retire for ages now in my area. That may be a deterministic factor if it hasn't been configured to route to 128.0.0.1 correctly...

For comparison, I VPN'd into the VPN Gateway we have at the Datacenter from home (hence latency that appears). My company has their own global network (they are their own ISP) and in order to even reach the server it is going through a boatload of Juniper AND Cisco (not to mention others) equipment. This trace does go through a Juniper, a properly configured one at that.

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:\Users\>tracert 128.0.0.1

Tracing route to 128.0.0.1 over a maximum of 30 hops

1 159 ms * 221 ms *.com [....]
2 239 ms 202 ms 203 ms *.com [....]
3 125 ms 198 ms 124 ms *.com [....]
4 180 ms 200 ms 202 ms *.com [....]
5 222 ms 203 ms 101 ms *.com [....]
6 127 ms 202 ms 199 ms 64...
7 222 ms 203 ms 204 ms 64....
8 334 ms 246 ms 367 ms xe-8-2-0.ams10.ip4.tinet.net [213.200.81.145]
9 388 ms 407 ms 402 ms surfnet-gw.ip4.tinet.net [77.67.72.110]
10 241 ms 193 ms 292 ms 128.0.0.1

Trace complete.

C:\Users\>

Notice, it does complete, but that's because this is a datacenter and we need to be reachable everywhere without killing security or violating RFCs. Not a good example, but right now anything's up in the air. I've asked a Verizon employee to stop by and see if they can get the ball rolling, but until then I guess we'll wait and see.

All comments

Pretty tough to get to the right level of support for Verizon. The employees at this forum can help but it does take some time. Essentially you're looking to talk to network operations. I'm able to reach that address at work (Datacenter) on my Windows and Linux machines. Trace falls flat on it's face at home. Makes it to the DSL edge router and then stops. Sounds like a routing issue that was happening with SoftLayer just a few months ago that was resolved when an upstream provider was taken out of the equation.

C:\Users\>tracert 128.0.0.30

Tracing route to 128.0.0.30 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 192.168.3.1
2 9 ms 9 ms 8 ms 10.15.3.1
3 * * * Request timed out.
4 * * * Request timed out.
5 * ^C
C:\Users\>

http://forums.verizon.com/t5/FiOS-Internet/Intermittent-timeouts-pinging-external-FIOS-gateway-VZ-says/m-p/438883/message-uid/438883#U438883

FWIW, I am at a friends house on at&t uverse and I can't reach that site either.

on a trace,

I time out after 77.67.72.110 and then it fully dies from there on out. surfnet-gw.ip4.tinet.net

prisaz

OK This is what I see based on your other post here.

If someone were running a network monitoring utility or tool that constantly blasted my network, I would block all their traffic, or at least ICMP traffic. So if you were doing that, you may have caused Verizon's network ICMP traffic to be disabled or blacklisted on the RIPE edge routers for the address block mentioned. Now that your other timeout problem is resolved, I would suggest contacting one of the people at the links provided at the end of this very long post. I am happy your timeout issue is resolved, and perhaps I am wrong in the fact that a flood of constant ICMP traffic caused this issue. But the NOC where I work would do this in a heart beat if it was disrupting other traffic, or assumed to be a threat to the network.

Looks like someone is blocking ICMP traffic from your IP. I don't think it is something Verizon can resolve unless they contact the people on the other end. Are you having trouble reaching sites at those addresses, or just can't ping or tracert to IPs?

I can hit this network web site. http://www.tng.de/web/guest/home

I can tracert to the IP just before the one you are trying to hit. So it looks like an issue over there, where traffic is being blocked for your IP. See if you can hit the IP just before the one you are trying to reach. If you can, it is a routing issue over there.

My trace

Tracing route to gi5-1-400.ams1.tng.de [213.178.75.99]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms ipcop.localdomain [x.x.x.x]
2 4 ms 4 ms 5 ms x.x.x.x
3 5 ms 4 ms 5 ms G0-12-3-4.WASHDC-LCR-22.verizon-gni.net [130.81.184.74]
4 3 ms 4 ms 5 ms so-14-0-0-0.RES-BB-RTR2.verizon-gni.net [130.81.22.56]
5 6 ms 7 ms 7 ms 0.ae2.BR1.IAD8.ALTER.NET [152.63.32.158]
6 6 ms 9 ms 9 ms ae17.edge1.washingtondc12.level3.net [4.68.62.137]
7 6 ms 7 ms 14 ms vl-3602-ve-226.ebr2.Washington12.Level3.net [4.69.158.38]
8 6 ms 7 ms 14 ms ae-5-5.ebr2.Washington1.Level3.net [4.69.143.221]
9 86 ms 87 ms 87 ms ae-43-43.ebr2.Paris1.Level3.net [4.69.137.57]
10 94 ms 102 ms 94 ms ae-46-46.ebr1.Frankfurt1.Level3.net [4.69.143.137]
11 93 ms 94 ms 99 ms ae-61-61.csw1.Frankfurt1.Level3.net [4.69.140.2]
12 93 ms 94 ms 94 ms ae-62-62.ebr2.Frankfurt1.Level3.net [4.69.140.17]
13 103 ms 99 ms 99 ms ae-48-48.ebr1.Dusseldorf1.Level3.net [4.69.143.177]
14 104 ms 104 ms 104 ms ae-4-4.car1.Hamburg1.Level3.net [4.69.133.181]
15 103 ms 104 ms 104 ms 62.140.29.67
16 103 ms 104 ms 104 ms po1.ham1.tng.de [213.178.75.10]
17 113 ms 124 ms 129 ms gi5-1-400.ams1.tng.de [213.178.75.99]

Trace complete.

Your trace

The IP in blue, may be a valid network starting address, but may not have anything sitting on that IP, or that responds to ICMP traffic.

Is there a specific site you are trying to reach? If it can be provided. Or another site you know is active on that subnet?

Based on the NMAP ping scan for that subnet. all ICMP traffic is blocked, because none of the 256 IPs responded to ping requests.

Starting Nmap 5.51 ( http://nmap.org ) at 2012-05-12 07:33 Eastern Daylight Time

Nmap done: 256 IP addresses (0 hosts up) scanned in 4.55 seconds

OK I went one further. I may have made someone upset, but only sent pings to 65536 IP addresses, and only a 10 minute list of ICMP ping requests at port 8. And yes ICMP traffic is disabled for that entire address block. By who? I would say Verizon or RIPE.

Starting Nmap 5.51 ( http://nmap.org ) at 2012-05-12 07:43 Eastern Daylight Time

Nmap done: 65536 IP addresses (0 hosts up) scanned in 580.12 seconds.

http://www.ripe.net/contact

http://www22.verizon.com/customersupport/contactus/

https://www.verizonbusiness.com/us/Support/contact/

One would have to ask Ripe about any ICMP blocks established for 128.*;
though, I would find it a bit strange putting an ICMP block in place,
because specifically 128.0.0.1 is supposed to pinged by RIPE Atlas probes
from all over the world, so RIPE would have to expect significant incoming
ICMP traffic to that, and one other, also unreachable address within the
same network. I'll check with them, see what they say.

Anyway, good point with tracerouting the last system in front of the target:

ckeck@radio $ traceroute 213.178.75.99
traceroute to 213.178.75.99 (213.178.75.99), 64 hops max, 40 byte packets
1 192.168.100.2 (192.168.100.2) 0.750 ms 0.932 ms 0.389 ms
2 L300.DLLSTX-VFTTP-63.verizon-gni.net (71.252.134.1) 5.590 ms 4.517 ms 4.908 ms
3 G0-5-2-2.DLLSTX-LCR-22.verizon-gni.net (130.81.107.138) 4.900 ms 4.742 ms 4.922 ms
4 so-5-0-0-0.DFW9-BB-RTR2.verizon-gni.net (130.81.199.36) 4.866 ms 4.753 ms 4.946 ms
5 0.xe-8-0-0.BR1.DFW13.ALTER.NET (152.63.4.17) 7.387 ms 67.265 ms 7.493 ms
6 ae6.edge2.dallas3.level3.net (4.68.62.165) 7.348 ms 7.268 ms 7.413 ms
[...]
17 ae-48-48.ebr1.Dusseldorf1.Level3.net (4.69.143.177) 137.209 ms
ae-46-46.ebr1.Dusseldorf1.Level3.net (4.69.143.169) 134.815 ms
ae-47-47.ebr1.Dusseldorf1.Level3.net (4.69.143.173) 132.222 ms
18 ae-4-4.car1.Hamburg1.Level3.net (4.69.133.181) 137.321 ms 137.134 ms 137.410 ms
19 62.140.29.67 (62.140.29.67) 142.353 ms 139.749 ms 152.430 ms
20 po1.ham1.tng.de (213.178.75.10) 139.856 ms 139.672 ms 139.879 ms
21 gi5-1-400.ams1.tng.de (213.178.75.99) 149.994 ms * 151.839 ms
ckeck@radio $

Works. Cool. Verizon routes through alter.net, then level3.net to tng.de.
So that routing across the big pond works. Devices along the way respond
to ping. Let's try another one, this time from work:

ckeck@xxxxxx $ traceroute 128.0.0.1
traceroute to 128.0.0.1 (128.0.0.1), 30 hops max, 40 byte packets
[...]
9 cr1.dlstx.ip.att.net (12.123.18.110) 6.272 ms 8.165 ms 8.125 ms
10 12.122.212.9 (12.122.212.9) 3.563 ms 5.237 ms 3.454 ms
11 192.205.32.26 (192.205.32.26) 3.844 ms 4.104 ms 3.734 ms
12 xe-8-3-0.ams10.ip4.tinet.net (89.149.186.133) 121.603 ms xe-8-2-0.ams10.ip4.tinet.net (213.200.81.145) 121.837 ms 120.957 ms
13 surfnet-gw.ip4.tinet.net (77.67.72.110) 131.626 ms 130.516 ms 128.696 ms
14 128.0.0.1 (128.0.0.1) 122.272 ms 121.487 ms 122.427 ms
ckeck@xxxxxx $

77.67.72.110 doesn't respond to traceroute, but can be pinged, means
it can be reached:

ckeck@radio $ traceroute 77.67.72.110
traceroute to 77.67.72.110 (77.67.72.110), 64 hops max, 40 byte packets
1 192.168.100.2 (192.168.100.2) 0.885 ms 0.778 ms 0.642 ms
2 L300.DLLSTX-VFTTP-63.verizon-gni.net (71.252.134.1) 5.464 ms 4.697 ms 4.964 ms
3 G0-5-2-2.DLLSTX-LCR-21.verizon-gni.net (130.81.107.46) 4.881 ms 4.812 ms 4.894 ms
4 so-5-0-0-0.DFW9-BB-RTR1.verizon-gni.net (130.81.199.34) 7.688 ms 7.269 ms 7.420 ms
5 0.ge-6-3-0.XT3.DFW9.ALTER.NET (152.63.96.193) 7.404 ms 7.265 ms 7.421 ms
6 0.so-7-1-0.XL3.CHI13.ALTER.NET (152.63.64.109) 32.424 ms 32.219 ms 32.431 ms
7 TenGigE0-4-1-0.GW2.CHI13.ALTER.NET (152.63.65.209) 34.895 ms
TenGigE0-4-2-0.GW2.CHI13.ALTER.NET (152.63.67.102) 32.277 ms 32.225 ms
8 tinet-gw.customer.alter.net (152.179.92.6) 34.934 ms 34.712 ms 34.886 ms
9 xe-11-0-1.ams10.ip4.tinet.net (89.149.185.210) 132.159 ms 132.190 ms
xe-8-3-0.ams10.ip4.tinet.net (89.149.186.133) 149.942 ms
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
^Cckeck@radio $

ckeck@radio $ ping 77.67.72.110
PING 77.67.72.110 (77.67.72.110): 56 data bytes
64 bytes from 77.67.72.110: icmp_seq=0 ttl=250 time=130.493 ms
64 bytes from 77.67.72.110: icmp_seq=1 ttl=250 time=129.746 ms
64 bytes from 77.67.72.110: icmp_seq=2 ttl=250 time=135.399 ms
64 bytes from 77.67.72.110: icmp_seq=3 ttl=250 time=129.598 ms
^C
--- 77.67.72.110 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/stddev = 129.598/131.309/135.399/2.386 ms
ckeck@radio
Ok, new game, new luck, change to wireless broadband. TMobile says:

C:\Documents and Settings\keck>tracert 128.0.0.1

Tracing route to 128.0.0.1 over a maximum of 30 hops

1 * * * Request timed out.
2 875 ms 463 ms 383 ms 10.170.204.48
[...]
10 774 ms 359 ms 392 ms 10.177.7.246
11 881 ms 432 ms 406 ms ge-8-13.car2.Dallas1.Level3.net [4.71.171.101]
12 1041 ms 47 ms 39 ms ae-2-70.edge4.Dallas3.Level3.net [4.69.145.77]
13 45 ms 40 ms 39 ms xe-8-1-2.dal33.ip4.tinet.net [77.67.71.221]
14 164 ms 159 ms 183 ms xe-11-1-2.ams10.ip4.tinet.net [89.149.185.242]
15 * 3642 ms 175 ms surfnet-gw.ip4.tinet.net [77.67.72.110]
16 153 ms 173 ms 175 ms 128.0.0.1

Trace complete.

C:\Documents and Settings\keck>

We already know about 77.67.72.110. Good enough.

What bugs me about this is that traceroute to 128.0.0.1 falls silent beyond
L300.DLLSTX-VFTTP-63.verizon-gni.net (71.252.134.1). I would expect reponses
from other Verizon equipment, especially because L300 does not appear to talk
directly to the outside world. Could it be that L300 has a configuration issue?

Forgot to mention. I'm running a RIPE Atlas probe, check

https://atlas.ripe.net/

for details. My probe was, and is preconfigured (means I didn't mess with
it) to ping 128.0.0.1 and 128.0.24.1. Neither one has been reachable since
middle of March. Same, btw, goes for labs.ripe.net at 193.0.6.153. In other
words, I don't think that this recent nmap-scan caused RIPE to blacklist
Verizon, because the reachability issue started weeks ago.

Is this all academic or do you have an actual functional problem?

This definitely isn't academic. Something that is supposed to work, and works for others, doesn't work here.

OK.

If RIPE was actively blocking his particular IP address or was blocking ICMP/UDP datagrams, why is it from work I can reach the server and actually make use of it? I can't even touch the server from my home in a different area, on different IP blocks (DSL vs. FiOS, they use different pools) and why do UDP datagrams also not go through? It fails at the edge router which from previous knowledge of Verizon's network, always means there's a routing failure occurring.

Yeah I know the latency is rediculous, but that's my DSL line for you 🙂 . I doubt this is RIPE doing something as I'm familiar enough with the IP in question.

@CKeck wrote:
This definitely isn't academic. Something that is supposed to work, and works for others, doesn't work here.

well I understand that much. What I don't understand is how this affects you. are you not able to log into a server or a site for a use?

or is it part of monitoring?

@Smith6612 wrote:
Pretty tough to get to the right level of support for Verizon. The employees at this forum can help but it does take some time. Essentially you're looking to talk to network operations

wouldn't you just do a who is on the VZ server and send an email to their NOC?

it is listed...