Routing Issues
jpstone1022
Enthusiast - Level 1

Hello,

I live in northern VA. I connect to services via TCP to two servers: one in Ashburn, VA (literally a 5 minute drive to the server), and one in DC (abou ta 30 minute drive to the server). About 30% of the time, Verizon FiOS decides to route me to California in order to resolve one or both of the servers. This creates extremely high latency (80-120ms) and makes services unusable. Releasing/renewing DHCP on my home router does not help.

As an example, I took a Windows tracert to the server in DC that I'm currently pinging 81ms to. Here is the output (I highlighted the evidence that I'm going all the way to CA):

Tracing route to 50.97.59.34-static.reverse.softlayer.com [50.97.59.34]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 4 ms 4 ms 4 ms L100.WASHDC-VFTTP-153.verizon-gni.net [108.28.24
.1]
3 26 ms 10 ms 11 ms G0-3-4-1.WASHDC-LCR-21.verizon-gni.net [130.81.1
05.178]
4 16 ms 15 ms 24 ms so-12-1-0-0.RES-BB-RTR1.verizon-gni.net [130.81.
51.230]
6 99 ms 72 ms 71 ms 0.ae4.XL3.SJC7.ALTER.NET [140.222.226.203]
7 68 ms 73 ms 74 ms TenGigE0-6-1-0.GW3.SJC7.ALTER.NET [152.63.50.26]

8 88 ms 75 ms 87 ms be-209-pe03.11greatoaks.ca.ibone.comcast.net [23
.30.206.93]
9 70 ms 70 ms 71 ms 75.149.228.214
10 74 ms 75 ms 73 ms ae0.bbr02.cs01.den01.networklayer.com [173.192.1
8.149]
11 79 ms 76 ms 75 ms ae0.bbr02.eq01.chi01.networklayer.com [173.192.1
8.130]
12 74 ms 74 ms 73 ms ae0.bbr02.eq01.wdc02.networklayer.com [173.192.1
8.154]
13 75 ms 74 ms 74 ms ae1.dar01.sr01.wdc01.networklayer.com [173.192.1
8.193]
14 75 ms 73 ms 74 ms po1.fcr03.sr02.wdc01.networklayer.com [208.43.11
8.157]
15 75 ms 74 ms 73 ms 50.97.59.34-static.reverse.softlayer.com [50.97.
59.34]

Trace complete.

How do I get this fixed? Can Verizon create static routes for me, since I use these services all the time? I never got these routing issues when I was with Comcast. It only started happening when I switched to Verizon FiOS. Latency of this magnitude--due to poor routing decisions by Verizon's routers--is unacceptable and makes no sense.

Thanks,

Justin

0 Likes
13 Replies
eljefe2
Master - Level 1

Yours is a fairly common problem reported here on the forum.   There hasn't been an official solution offered by Verizon, but what you might try is selecting a different DNS server to see if that improves your routing any.

You can use a free utility, DNSBench, to identify the best performing server(s) at your location:

https://www.grc.com/dns/benchmark.htm

Some users have reported good luck using Google's DNS...8.8.8.8 and/or 8.8.4.4

Good luck.

0 Likes
jpstone1022
Enthusiast - Level 1

Hi eljefe,

Unless I'm missing something, DNS is irrelevant to this issue since I'm connecting directly to an IP address.

The issue is with Verizon's routers. Either a) they've configured their routing protocols poorly or b) they're doing some kind of load balancing that sends my packets off to another router dedicated to routing traffic as far away as possible before getting to its destination. 😄

All it would take to fix this is for Verizon to dedicate some resources to get their routing in order. It appears Comcast has done this, as I never experienced it with them.

In all seriousness, if this continues, I will cut my Verizon service and go back to Comcast. Routing me to CA to get to a server 5 miles away is unacceptable in any context. It's just extremely poor design.

0 Likes
eljefe2
Master - Level 1

What you say makes sense, jpstone.  I was thinking of connecting to a domain name.  However, I just ran a couple of experiements tracing the route to a specific IP address with different DNS servers in place.  I got a different routing with the different DNS servers.

That may just be a conincidence but it might be worth your giving it a try.

As for giving up on Verizon and going back to Comcast, I can't blame you if you get better service from the cable provider.  Here, for whatever reasons, my experience with FiOS has been much better than it was prviously with our local cable company.

Good luck.  I hope you find a solution.

0 Likes
jpstone1022
Enthusiast - Level 1

I tried different DNS servers just for giggles. Same issue.

0 Likes
eljefe2
Master - Level 1

Yeah, I was going to post an edit that I tried the IP address you need with different DNS servers and I also had the same results. Smiley Frustrated

Maybe a VPN would help?   Or maybe Comcast is indeed the answer.

Sorry.  I'll hope that another user here chimes in with a better solution that works.

0 Likes
jpstone1022
Enthusiast - Level 1

In my humble opinion, the answer would be for Verizon to fix these routing issues. Especially if this is a common problem. Why are they content to unintelligently route their users' traffic and create high latency? This should be a high priority for them. Especially if their competitor has it right.

0 Likes
CRobGauth
Community Leader
Community Leader

Your initial complaint is high latency.

What level of latency are you expecting?

And Verizon doesn't have direct control over routing for specific address within their network.

It is possible that a closer gateway was down or is congested.

Or that the server you are hitting has changed providers and they are advertising that network out of Ca to Verizon.

0 Likes
jpstone1022
Enthusiast - Level 1

@CRobGauth wrote:

Your initial complaint is high latency.


Yes, high latency is the problem. In theory, you could route me to CA to get to a server on the east coast that's 5 minutes away from where I live if you can somehow manage to keep the latency to below 20ms. But this is reality, and that is impossible. So the real issue is that Verizon is sending me to California to get to a server 20 minutes away in DC.


@CRobGauth wrote:

What level of latency are you expecting?


I said it in my original post: 80-120ms.


@CRobGauth wrote:

And Verizon doesn't have direct control over routing for specific address within their network.


This is absolutely false. For example, Verizon could institute static routes if they wanted. At the very least, they could identify what it is in their routing protocols that causes traffic destined for a server 10 miles away to be sent on a 6,000 mile journey. If this issue happened in a closed network at a fortune 500 company, it would be considered unacceptable and the routing would be fixed.


@CRobGauth wrote:

It is possible that a closer gateway was down or is congested.


Then they need to fix their topology. If it was down, they should have backups that keep my packet journey within a reasonable milage, rather than turning 10 miles into 6,000. If it's conjested, perhaps I'd get less latency going through the conjested router than traversing 6,000 miles?!


@CRobGauth wrote:

Or that the server you are hitting has changed providers and they are advertising that network out of Ca to Verizon.


As I mentioned in my original post, the issue is interminttent and happens only 30% of the time. If it was a changed provdier/advertising issue, then it would happen 100% of the time.

Also, they have had the same provider since 2011. Just for kicks, I went to someone's house that has Comcast and the latency was 17ms, and the traceroute was much more sane. After I returned home, the issue remained with Verizon. 

0 Likes
CRobGauth
Community Leader
Community Leader

Couple of comments:

I asked what you expected for latency.

You said you stated it in your post. But your post said you measured 80-120. Then said Comcast was 17. So you are really expecting in the 20 range I would assume.

Changing routing is not as easy as you report. Putting statics in a network the size of Verizon's is impossible. The number of nodes that would need to be updated would be huge. I work on a private corporate network. We would never correct a problem like this with statics. The problem would be found with the routing and corrected appropriately. Problem is, Verizon probably has little interest in the correcting a routing issue for just one network. Espeically for a residential user (unfortunately that is the reality. Those of us that are residential, get no SLA and minimal support when it comes to correcting this type of issue).

0 Likes
jpstone1022
Enthusiast - Level 1

@CRobGauth wrote:

I asked what you expected for latency.

You said you stated it in your post. But your post said you measured 80-120. Then said Comcast was 17. So you are really expecting in the 20 range I would assume.


My apologies. I misread your comment. But yes, for a server just a few miles away, I would expect no more than 25ms.


@CRobGauth wrote:

Changing routing is not as easy as you report. Putting statics in a network the size of Verizon's is impossible. The number of nodes that would need to be updated would be huge.


From my understanding, this is a pretty common complaint by customers to Verizon--frequent, incredulous routes to nearby servers. This isn't something that should take fancy network statistic monitoring. Routing protocols--if configured properly--should handle this with ease. So either a) Verizon has bad network design, or b) they are intentionally sending this traffic to distant routers to allow other traffic through that they feel is more important. If the former, shame on them. If the latter, shame on them.

But to address your point oabout the number of nodes that would need to be updated--again, this isn't an issue. Routing protocols propogate themselves...no need to manually update each router. Just configure a neighbor zone appropriately and let it propegate. Issue resolved.


CRobGauth wrote:Problem is, Verizon probably has little interest in the correcting a routing issue for just one network.  

This isn't "just one network". I frequently use 2 servers: one in Ashburn, VA and one in DC. Both on different networks. Both exhibit this issue with Verizon. Couple that with all the other complaints users are voicing, and I would say it's not an issue with any one network--except, say, Verizon's network, and their routers. So if they have "little interest" in resolving an issue like this, I will certainly take my business elsewhwere.

Can someone from Verizon confirm this is the case? Do you have "little interest" in fixing these issues? If no response from Verizon directly, then I would assume CRobGauth's viewpoint represents Verizon's.


CRobGauth wrote:Espeically for a residential user (unfortunately that is the reality. Those of us that are residential, get no SLA and minimal support when it comes to correcting this type of issue).

Again, is this true, Verizon? Let me know now and I will take my business elsewhere. I have no interest in being in a one-sided relationship.

0 Likes
CRobGauth
Community Leader
Community Leader

Keep in mind that this is mainly a peer to peer support forum

Don't expect a formal response from Verizon here.

And my experience is that all providers treat residential and business class services different.
That's why they pay more. Pay more, get better service.

0 Likes
jpstone1022
Enthusiast - Level 1

CRobGauth wrote:

Keep in mind that this is mainly a peer to peer support forum

Don't expect a formal response from Verizon here.


I don't expect one, but as a customer, I deserve one.


And my experience is that all providers treat residential and business class services different.
That's why they pay more. Pay more, get better service.


If they want to lose customers, then they should definitely adopt this sort of attitude. Not saying business class shouldn't take priority, but customers are customers, and Verizon should aspire to excellence. If they want to sweep an on-going, widely-affecting, multiple-customer impacting issue like this under the rug, then that says a lot.

0 Likes
seitzbg
Enthusiast - Level 3

It is not unusual for Verizon's backbone routing to be complete crap.  Not to mention peering is horrible with a lot of major players like Amazon/AWS. I get ~1Mbps downloads from dropbox since they host on AWS.  I can upload quite fast to AWS.   Some of their other peering has been getting better since all of the net neutrality stuff hit the law books and has been on the news but clearly some of it is still horrible.

During peek hours I get 70ms to 8.8.8.8 from one of my five IPs, however if I try another I get 20ms. Either they dont care, don't want to spend the money, or have monkeys working there.  The bad part about this is you can't even call and get support because they will just run you through a script or send a technician out to fix a problem on THEIR network.

I too am in the VA area.