Verizon (or someone) dropping packets from Uni of Pitt

I work for Pitt, and we have a Verizon FiOS residential customer who is unable to access any websites in a particular subnet. We actually had the same issue with someone who is geographically in the same area, and gets an IP from the same range (108.32.0.0/12) a few months ago. That person actually ended up switching to Comcast to fix the problem.

I personally have FiOS and get an IP in the 74.96.0.0/12 range and have no issues, as does several other employees that we had test...just none in the 108.32.0.0/12 range.

I had the user run a packet capture, and I ran one on our load balancer, and we can see her packets leaving correctly, hitting the load balancer, and the load balancer doing its job, and returning packets, but the user never gets any return packets. All we see from their side is retransmissions.

We have a ticket opened with Verizon, but I don't want to run into the same issue where the user gets fed up with no one being able to help on Verizon's side that they leave.

I'm not sure if any network engineers from Verizon read this board, or if problems can be escalated. It is a problem when a student is forced to use their cell phone tethering to access their school's website.

Thanks!

Accepted answers

Kepler2

It was the DSCP tagging. I set up a web server in our development environment and added DSCP tagging and the user was not able to pull a website, despite me seeing traffic from her address coming in via tcpdump, and responding correctly.

Once I removed the DSCP tagging from the zone, the user was able to get to the website without issue. So I tested it twice more -- adding/removing and it is the problem.

I then performed traceroutes to the user, with and without DSCP tagging, and it always follows the same path, ending on a Verizon node, so that node, or another after it that isn't responding to my ICMP is either re-routing, or dropping traffic tagged with DSCP 43. That is 100% a misconfiguration on Verizon's part, as they should not trust DSCP tags coming from outside networks.

All comments

Hubrisnxs

Hi there. They don't normally, - the site is a peer to peer forum, with some standard forum administraton, but we've seen a lot of users get out of a range like that fairly easily by following the steps outlined below.

Pls feel free to report back and let us know if it helped.

They have to log into the Verizon router. 192.168.1.1 user name is admin and the password is printed on the sticker as "password" or "login password" People like to put the wpa2 or WEP key here for some reason, so try to avoid that, this is different then that. Use these instructions to log in

How to find Router Default Username and Password | FiOS ...

Then follow these steps

Actiontec MI424-WR - RELEASE DHCP
Click on MY NETWORK icon at the top.
Select NETWORK CONNECTIONS from the menu on the left.
Select BROADBAND CONNECTION (coax or ethernet) depending on your connection to the ONT.
Click SETTINGS
Click RELEASE
Click APPLY
Disconnect the router immediately to prevent it from re-requesting a DHCP lease.

After you have disconnected it, if you reconnect it too fast, it will simply get the same IP and or the same subnet. So you want to leave it disconnected for at least 5 full minutes, and preferably 10 -15. After that wait period, reconnect it, check the IP again and it should be at the least a different subnet.

Kepler2

Thanks for the look!

The user was sent a new router by Verizon prior to calling us -- So the different MAC requesting an IP should have accomplished the same thing. Would it be worth having the user unplug the cable/battery from the box installed in their basement, where the fiber terminates? I'm iffy on recommending this since they have to unscrew that bolt that says that only technicians should do that.

I should mention that she went through three calls with Verizon, so they did the power cycling, hardware replacement, and normal troubleshooting. The bigger issue is that I remembered that a few months ago, someone had the same problem from the same IP block.

Hubrisnxs

@Kepler wrote:
Thanks for the look!

The user was sent a new router by Verizon prior to calling us -- So the different MAC requesting an IP should have accomplished the same thing. Would it be worth having the user unplug the cable/battery from the box installed in their basement, where the fiber terminates? I'm iffy on recommending this since they have to unscrew that bolt that says that only technicians should do that.

I should mention that she went through three calls with Verizon, so they did the power cycling, hardware replacement, and normal troubleshooting. The bigger issue is that I remembered that a few months ago, someone had the same problem from the same IP block.

The normal troubleshooting including getting a new router, won't accomplish any of what you need. The process outlined will in fact get you a new ip on different subnets.

The router, and you can test this yourself, when you unplug it and plug it back in, you in fact get literally the same IP address back.

That defeats the whole purpose.

The release of the IP and Mac Binding is Key, but the more important thing is the length of time you leave it disconnected.

In fact even if you had a Verizon rep on the line and asked him to clear that arp/mac, and you rebooted it with a Verizon tech on the phone, you would still get the same ip.

The ONT Doesn't cary the ARP table, so rebooting it won't be of any use to you.

Kepler2

It would accomplish what I need -- you said it yourself:

"The release of the IP and Mac Binding is Key, but the more important thing is the length of time you leave it disconnected".

Getting a whole new router, where the WAN port has a whole new MAC address makes the DHCP lease for the old MAC irrelevant.

"The process outlined will in fact get you a new ip on different subnets."

I don't see how this gets a new IP on a different subnet -- if the DHCP request from the WAN interface to ff:ff:ff:ff:ff:ff is being responded to by whatever DHCP server is serving the 108.32.x.x addr, why would that server decide to pick a new range? Unless there was a configuration issue that allows DHCP requests to that 108.32.x.x DHCP server (bad DHCP relay address on the layer 3 maybe), which has been fixed, this may not work.

I know clearing ARP wouldn't fix it, as the DHCP server doesn't care about the router's ARP table.

What you described was trying to get the DHCP lease gone -- which a new MAC address on a new modem would do.

Hubrisnxs

Most people, yourself included, don't don't a router and then wait 5-15 minutes to connect a new one. So I'm not quite sure what you mean by that, but verizon areas are served by several gateway routers handing out ip's to its customers from various subnets. The proof is simply in the pudding the method mentioned works flawlessly.

Hubrisnxs

Also a new Mac on a new modem would cause the connection to flat out stop working. Verizon does very strict Mac authentication and binding. The second a new Mac attempts to connect, the gateway refuses the connection. Calling in, they clear that binding and a new home router gets connected in seconds (again defeating the purpose) Rather than the time described. The time described only works if the lease is broken and the connection is sitting vacant and not occupying an address block. This process allows enough time for the ip to go back into the pool of ips and get cycled to a new customer. These routers are employed in a tier I network and serving an impressively large customer base. That's how or why it's typically going to throw you on a different subnet vs the same, largely because that subnet is likely close to a threshold at various parts of the day.

Kepler2

I was able to contact the user, the steps in post one did not work. User obtained an IP in the same range. She left it disconnected for nearly 2.5 hours while she went to class.

Hubrisnxs

2 and a half hours should have worked, the Verizon lease on the IP address is at 2 hours. Some area's might be set to their old 4 hour lease, BUT if you did the release dhcp and immediately turned it off (if it wasn't it would have grabbed that ip within seconds) then that might have been an issue.

I can tell you that 90% of people here don't need to leave it off for any more than 5 minutes if they did it right. The other ten percent are a mixture, of simply not doing it right, or having to leave it off overnight.

You may want to have her try again, or simply leave it off over night when she goes to bed.

If it keeps grabbing the same subnet, then either you're doing something that no one else is doing, or there's a very very very weird problem in her area, that no one else has seen before, and if that's the case, you'll want to contact Verizon direct.

I wouldn't recommend front line support for that, I'd say reach out to their twitter support or their direct support at the following link.

Those are advanced technicians that would be able to help you better than going through the front line.

Kepler2

Once I removed the DSCP tagging from the zone, the user was able to get to the website without issue. So I tested it twice more -- adding/removing and it is the problem.

Hubrisnxs

Does that conform to an rfc? I noticed verizon is pretty strict about that in general. You might want to send that to their NOC

Kepler2

If you can tell me how to contact their NOC, I will. 😛

DSCP values are always present in an IP packet, it is just normally 0, unless you change it. For instance, I did the same test with a co-worker who has Comcast, and they actually rewrote the value to 8. It isn't an RFC thing, but a policy thing. We use DSCP to re-route traffic for a particular subnet (set it to 43). I have no idea what Verizon is doing with it.

As far as the actual problem, it seems local to one router, whatever is servicing this user's particular area (we had 2 others who had the same issue, both from the same geographical area). On our side, we're going to strip DSCP at our borders or something to resolve this for the user, but it is still something Verizon should fix.

Hubrisnxs

most places are just NOC@

I've gotten responses from ip-noc@verizon.com NOC@verizon.com and also DNS@verizon.com

If noc doesn't answer me, I would normally email the dns address, and ask them if they can pass along, or point me to the correct email address.

Verizon's actual noc telephone number is unpublished from what I remember. There's a lot of providers who publish that info, I haven't seen Verizon's published in a LoOoOng time, before they bought UUNET/MCI Business