Fix Westernport MD

[BCS23]

I am not a Verizon customer; I work for a local PC repair shop.  For several years now some of our customers in Westernport Maryland have complained about 'slow computers'.

What they are really experiencing is terrible Internet service.  Many of them have over 40% packet loss at night, latency over 1000ms all day, flaky connections, and other errors.  Every single one of them has 10.23.16.1 as their modem's gateway, and running traceroute to anything shows the latency spike there.

I've informed many of them that the blame lies, not on their computers, but on Verizon DSL.  Many of them have called Verizon over and over, only to be blamed for the poor service they receive.  One of my customers has a stack of modems in their closet, as Verizon support simply throws new modems at the problem any time they themselves detect an issue with the connection.

There does not seem to be an avenue of communication for me to convey this to Verizon, but I'm getting fed up of not being able to help my customers while Verizon turns a blind eye to an entire town. 

I'm hoping someone can point out a way I can talk to a Verizon network technician as not their customer, because when I try to call on my customer;s behalf I am talked down to and ignored.  Frequently the phone support will not even forward me to anyone who would even know what I'm talking about.

[smith6612]

Have you tried talking to the folks in Verizon Direct over at DSLReports?

https://www.dslreports.com/forum/vzdirect

The endpoint you're describing is usually a PPP session terminating on a GWR. These GWRs for DSL are typically Juniper ERX1440, or routers of similar calibur. Old routers, 80Gbps maximum of routing capacity (less due to ATM), and many due to load are seeing congestion, or otherwise just showing their age. But I have a good suspicion they've been overworked for quite a while, and the spawn of Netflix and YouTube has made the problem apparent.

My central office has four ERX1440s that I know of. Of the four, there's three which are peaking at >100% congested according to Verizon. The one I'm on has been peaking around 85% at night on a 30 day average. Prior to being on the Juniper I'm on right now, I was on one of the congested ones, and would see what your customers are describing from the hours of 5PM to 1AM. Congestion (losing 70+% of the bandwidth at night, packet loss of 4% to 30%). The strange part is, the VZ techs I spoke with didn't see congestion between my DSLAM or any intermediate ATM Circuits in beteen, which leads to me the Juniper is hitting a processing limit. Unfortunately VZ didn't let me dig into the problem far enough with them to actually help offer a solution (I help keep many major Internet services working at as a means of living). I do know that my area has two problems which would impact the Junipers: Buffer over-run conditions due to bad provisioning at the GWR towards certain DSLAMs (Litespan 2000 specifically), and the per-line QoS profile they apply to limit a customer's rate at the GWR (not just at the DSLAM), causing them to run quite inefficiently with the number of customers being serviced out of a single router.

These Junipers are doing a lot more than just routing. They're doing Egress and Ingress filtering (port-based). Encapsulating/Depcasulating Ethernet from PPPoE sessions. Rate limiting all circuits on a per-line basis. Handling PPPoE sessions (probably with lots of DSL circuits flapping because many lines are in poor shape), doing bandwidth logging so VZ knows when you hit the 1TB cap DSL has, just to name a few. There have also been CGNAT trials done on some of these Junipers, which really doesn't help them out at all being they're not made to do NAT quickly.

Hopefully you can get someone to address whatever the problem is via VZDirect. My neighbors who are on the overloaded Juniper routers all complain about slowdowns almost every night. I seem to be the only one in the neighborhood with working DSL. It's been over two years since I had my congestion issue, spent 6 months working with VZ to fix it, and apparently no effort has been made to resolve them for my neighbors, who call in weekly. After I suggested putting in some upgraded equipment (Or better yet FiOS). So, best of luck.

If people can get FiOS, point them to that. Much newer network. Rock solid. Put the DSL out of it's misery.

[smith6612]

Just to show how consistent the service is *supposed* to be. This is what a few months of working with Verizon accomplished. Note the "dips" in the graphs are usually due to other traffic on the network (upload) or hardware accelerated NAT in the router causing traffic to bypass the CPU.

The key part about this graph is the times at which certain speeds are occurring. Notice, consistency during peak hours. That is what DSL has been touted to provide. Problem is, VZ hasn't been doing much upgrading or maintenance to the DSL network. So traffic is growing but upgrades at the last mile are not occurring as often. No excuse for those in FiOS areas to still have DSL though 😄

30 second view24 hour view

[BCS23]

"No excuse for those in FiOS areas to still have DSL though"

It's three times as expensive.  Almost all of my customers who are experiencing problems would just switch to Comcast if they could aford it.  It doesn't matter anyway, I don't expect FiOS to be available within 50 miles of Westernport.

[BCS23]

Also, thank you for the information.  I'll make a post over there and see if I can get some of my customers to do the same.