- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@frnkblk wrote:...
On the outside chance that that routing is actually symmetric: PJL, can you provide traceroutes to 65.55.251.214 and 65.55.53.235, as well as 65.204.48.118 and 161.221.89.118? Perhaps we can identify which path is the problem one.
Here are the traces:
Tracing route to 65.55.251.214 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms 192.168.1.1
2 5 ms 5 ms 5 ms L100.LSANCA-VFTTP-128.verizon-gni.net [96.229.59.1
3 6 ms 6 ms 6 ms P9-1.PITBPA-LCR-03.verizon-gni.net [130.81.55.176]
4 7 ms 7 ms 58 ms 130.81.199.38
5 6 ms 6 ms 6 ms 0.so-7-2-0.XL3.LAX15.ALTER.NET [152.63.113.241]
6 7 ms 8 ms 7 ms TenGigE0-4-1-0.GW4.LAX15.ALTER.NET [152.63.115.178
7 7 ms 7 ms 7 ms microsoft.customer.alter.net [152.179.21.26]
8 41 ms 41 ms 42 ms xe-9-0-0-0.sn1-96cb-1b.ntwk.msn.net [207.46.42.178
9 78 ms 78 ms 78 ms xe-0-1-0-0.blu-96c-1b.ntwk.msn.net [207.46.43.23]
10 77 ms 77 ms 77 ms ten9-2.blu-76c-1a.ntwk.msn.net [207.46.47.155]
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.
Trace complete.
Tracing route to msnbc.com [65.55.53.235] over a maximum of 30 hops:
1 <1 ms <1 ms 1 ms 192.168.1.1
2 5 ms 4 ms 5 ms L100.LSANCA-VFTTP-128.verizon-gni.net [96.229.59.1]
3 8 ms 8 ms 8 ms G0-5-2-4.LSANCA-LCR-22.verizon-gni.net [130.81.183.176]
4 9 ms 8 ms 10 ms so-4-1-0-0.LAX01-BB-RTR2.verizon-gni.net [130.81.151.248]
5 10 ms 10 ms 11 ms 0.so-6-0-0.XT2.LAX9.ALTER.NET [152.63.10.157]
6 60 ms 10 ms 12 ms 0.so-7-1-0.XL4.LAX15.ALTER.NET [152.63.1.242]
7 11 ms 10 ms 12 ms TenGigE0-5-1-0.GW4.LAX15.ALTER.NET [152.63.115.182]
8 10 ms 9 ms 10 ms microsoft.customer.alter.net [152.179.21.26]
9 10 ms 10 ms 10 ms xe-3-0-0-0.lax-96cbe-1a.ntwk.msn.net [207.46.47.10]
10 20 ms 19 ms 19 ms xe-7-0-2-0.by2-96c-1a.ntwk.msn.net [207.46.42.176]
11 43 ms 41 ms 41 ms ge-5-2-0-0.co2-64c-1a.ntwk.msn.net [207.46.40.84]
12 42 ms 40 ms 41 ms xe-0-2-0-0.co2-96c-2b.ntwk.msn.net [207.46.42.203]
13 50 ms 41 ms 41 ms xe-7-3-0-0.co1-96c-1b.ntwk.msn.net [207.46.40.246]
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.
Trace complete.
Tracing route to 65.204.48.118 over a maximum of 30 hops
1 <1 ms <1 ms 1 ms 192.168.1.1
2 4 ms 5 ms 21 ms L100.LSANCA-VFTTP-128.verizon-gni.net [96.229.59.1]
3 6 ms 6 ms 7 ms P9-1.PITBPA-LCR-03.verizon-gni.net [130.81.55.176]
4 6 ms 7 ms 6 ms so-4-1-0-0.LAX01-BB-RTR1.verizon-gni.net [130.81.151.246]
5 6 ms 7 ms 6 ms 0.ae1.BR3.LAX15.ALTER.NET [152.63.2.129]
6 7 ms 7 ms 7 ms 204.255.168.134
7 91 ms 90 ms 92 ms cr2.la2ca.ip.att.net [12.123.30.150]
8 89 ms 91 ms 92 ms cr1.slkut.ip.att.net [12.122.30.29]
9 89 ms 91 ms 91 ms cr2.dvmco.ip.att.net [12.122.30.26]
10 91 ms 91 ms 91 ms cr1.cgcil.ip.att.net [12.122.31.86]
11 111 ms 91 ms 91 ms cr2.cgcil.ip.att.net [12.122.2.54]
12 94 ms 96 ms 94 ms cr1.n54ny.ip.att.net [12.122.1.1]
13 87 ms 87 ms 87 ms cr81.nw2nj.ip.att.net [12.122.105.30]
14 92 ms 92 ms 92 ms 12.122.115.105
15 81 ms 82 ms 82 ms 12.94.85.66
16 * * * Request timed out.
17 81 ms 81 ms 82 ms 65.204.48.118
Trace complete.
Tracing route to 161.221.89.118 over a maximum of 30 hops
1 <1 ms 1 ms <1 ms 192.168.1.1
2 5 ms 4 ms 4 ms L100.LSANCA-VFTTP-128.verizon-gni.net [96.229.59.1]
3 6 ms 6 ms 7 ms P9-1.PITBPA-LCR-03.verizon-gni.net [130.81.55.176]
4 27 ms 7 ms 6 ms 130.81.199.38
5 6 ms 6 ms 7 ms 0.ae1.BR3.LAX15.ALTER.NET [152.63.2.129]
6 7 ms 6 ms 7 ms 192.205.35.161
7 92 ms 91 ms 89 ms cr2.la2ca.ip.att.net [12.123.30.134]
8 89 ms 91 ms 92 ms cr1.slkut.ip.att.net [12.122.30.29]
9 88 ms 87 ms 87 ms cr2.dvmco.ip.att.net [12.122.30.26]
10 89 ms 89 ms 91 ms cr1.cgcil.ip.att.net [12.122.31.86]
11 88 ms 88 ms 91 ms cr2.cgcil.ip.att.net [12.122.2.54]
12 94 ms 95 ms 95 ms cr1.n54ny.ip.att.net [12.122.1.1]
13 85 ms 85 ms 85 ms cr83.n54ny.ip.att.net [12.122.105.50]
14 87 ms 86 ms 86 ms gar1.n54ny.ip.att.net [12.122.105.9]
15 84 ms 82 ms 81 ms 12.94.184.186
16 81 ms 82 ms 82 ms 161.221.88.6
17 81 ms 81 ms 82 ms 161.221.89.118
Trace complete.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Will the lease break/renew IP fix work on a Fios business account that's using a static IP address?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Probably not, as the "new IP" fix requires getting a new/different IP address so that hopefully the return traffic goes over a different (L1/L2) path than the existing path.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Had a feeling. Fabulous. Now I need to see if I can get them to assign me a new static IP address. Thanks for the reply.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Mordeen wrote:Had a feeling. Fabulous. Now I need to see if I can get them to assign me a new static IP address. Thanks for the reply.
Are you assigned a single static IP address? Or are you assigned a range of static IP addresses? Does your router actually use dhcp to get that address? Or is it set to a single IP address?
If you're assigned a range of IP addresses, then you are in the perfect position to change it yourself and test how it works for each individual IP address. It also arms you/us with more information about characteristics of the issue. Things like "every odd IP works" or "every even IP works" or "every 4th IP works" would be most illuminating. Some of the participants in this thread have voices that I believe can reach engineering in Verizon, but only if the details and facts are repeatable. Your physical location might also be part of that equation. So far, we've seen mostly San Diego and Inland Empire reporting this, but one of the participants was also in Long Beach.
If not assigned a range, then yeah, you'll have to get someone on the phone to assign you a new IP. What is it now? During the change to a new IP, do any of them not work? And what IP starts working when you get it changed?
For reference, this IP is working for me: 173.58.47.108 (has been since a tech support agent broke the lease 2 weeks ago and I got this IP)
this is the IP that was not working for me: 173.58.43.42
Here is the conversation when I was troubleshooting with a CDN provider when we figured out that I couldn't reach their west coast node but I could reach their Dallas and Virginia nodes:
<cannonball> Working from office in Los Angeles: Savvis -> Cogent (te0-3-0-7.ccr21) -> 38.99.68.170 -> 38.127.199.74
<cannonball> Not working: Verizon -> AlterNet -> Cogent (te0-3-0-5.ccr2) -> 38.99.68.170 -> 38.127.199.74
<crucially> cannonball: huh, you get all the way to 38.127.199.74?
<cannonball> The tracepath does, yes.
<crucially> ok
<crucially> fun
<cannonball> There are some "no reply" sprinkled in the two paths I pasted into the pastebin. Didn't figure they were that important.
<crucially> mtr gets around those usually
<cannonball> openssl s_client -connect flassets.a.ssl.fastly.net:443 hangs for me, no connection made.
<cannonball> Yeah, you're right, mtr does handle displaying those a lot nicer.
<crucially> mtr will also show you multiple paths
<crucially> better
<crucially> cannonball: can you msg me your end point so i can mtr back/tcpdump
<cannonball> Doing the openssl command from work connects and does the cert exchange.
<cannonball> Sure, do you want work (working) or home (non-working) ip?
<crucially> the non working one
<crucially> cannonball: you are west coast right?
<cannonball> 173.58.43.42 (home, Verizon FIOS) Yes, Los Angeles area (Murrieta, CA)
<cannonball> $ host flassets.a.ssl.fastly.net
<cannonball> flassets.a.ssl.fastly.net has address 38.99.68.180
<cannonball> I assume I'm the only person complaining about this, both for James and for Fastly. So I wouldn't rule out a local issue.
<crucially> I get to 0.ae3.SJC01-BB-RTR1.verizon-gni.NET (tracing from one of the west coast nodes)
<crucially> then it stops
<DocWilco> I can mtr all the wai to 173.58.43.42 from home
<crucially> oh great
<cannonball> does anybody know if enabling ipv6 (miredo, the linux implementation of teredo) would have any adverse effects?
<DocWilco> although, with some heave loss
<crucially> cannonball: if you put in your hsots file
<crucially> 38.99.68.181
<crucially> instead of 180
<crucially> and see if it works
<cannonball> doing
<crucially> cannonball: no, ipv6 should not matter
<DocWilco> a15 can reach him
<crucially> DocWilco: cache-s22 can reach him
<crucially> s1 can not
<DocWilco> yeah
<DocWilco> and 0.ae3.SJC01-BB-RTR1.verizon-gni.NET is where it ends
<DocWilco> I'm going to go with Verizon routing problem
<crucially> from 181 I get 0.ge-2-0-0.SJC01-BB-RTR2.verizon-gni.net
<crucially> from 180 I get 7. 0.ae3.SJC01-BB-RTR1.verizon-gni.NET
<cannonball> Testing with curl from commandline, neither 180 or 181 work. Got a couple other IP's I could try forcing?
<crucially> cannonball: cache-d20.hosts.fastly.net
<cannonball> Yeah, I tend to think it's Verizon too. I'm stumped why the tracepath gets all the wya.
<crucially> cannonball: asymmetric routing
<cannonball> cache-d20 works with curl, testing int he browser.
<crucially> the return path is different
<crucially> cannonball: try varnish-v11
<crucially> and varnish-v12
<cannonball> browser worked, trying varnish-v11 and varnish-v12 next.
<cannonball> ...browser worked with cache-d20
<cannonball> ...curl works with varnish-v11
<cannonball> ..and works with varnish-v12
<crucially> ok
<crucially> so it isn't cogent/verizon
<cannonball> So it's a Verizon issue, not you guys. My apologies for bugging you, I'll call them (though I'm just a FIOS customer, I don't have a NOC contact). If I can figure out who the peer is, that would at least help them.
<crucially> cannonball: d20 is in dallas, v\d is in virginia
<crucially> I will give you the reverse tracepaths
<cannonball> I'm sure I'll go through voicemail hell just to get told I need to reboot my wifi gateway 😕 but I gotta pester them.
<cannonball> thank you for your help guys.
<crucially> hang on
<cannonball> am holding, just being polite 🙂
<cannonball> Not enough people say thank you in irc, wanted to make sure I wasn't one of them.
<crucially> cannonball: i want to try a few more things
<cannonball> My typo of "fastfly" also explains why I couldn't find jack about you guys in google. I was beginning to lose faith in the Goo.
<crucially> heheh
<crucially> can you ping 180 and 181
<cannonball> I'm ready to try anything you want.
<crucially> i want to verify we see the requests
<cannonball> pinging 180 right now
<crucially> ok, i see them
<crucially> now one more check
<cannonball> I get responses back from 181.
<cannonball> But I get nothing back from 180.
<crucially> yeah, makes sense
<crucially> though unclear why curl doesn't work to 181
<crucially> ok, both icmp replies leave our netowkr
<DocWilco> friend of mine also on Verizon FIOS can't reach s1 either
<crucially> it is probably a line card that has an error in it for this path
<cannonball> Is .180 the s1 node?
<crucially> yeah
<crucially> 181 is s22
<crucially> you should be able to reach s22
<crucially> if icmp works
<cannonball> I can ping it, mtr shows it being reached, testing with tcptraceroute now.
<DocWilco> nevermind, he could reach them, but cogent routers are very heavily loaded
<cannonball> curl does something weird. It must think it's connected because it displays the download status line, it just never moves.
<crucially> cannonball: what is the curl line you use?
<DocWilco> crucially: see the email I just sent, check out hops 12 and 13, those times are pretty high
<cannonball> curl --verbose https://flassets.a.ssl.fastly.net/stylesheets/base_packaged.css?323c1e4e6e864
<crucially> hmm
<crucially> DocWilco: shouldn't matter, thats just control plane
<crucially> has s1 had a traffic drop
<DocWilco> crucially: the fact that it's responsing slowly means it's heavily loaded though
<DocWilco> not to the point where it's dropping stuff going through, but still
<crucially> DocWilco: on the control plane, not switching plane
<crucially> routing/switching
<crucially> there is no way you can tell if the forward plane is loaded or not
<cannonball> icmp is usually dropped first, so if it's not dropping it, the cpu is probably not near its limits.
<cannonball> but yeah, tells you very little about forwarding.
<cannonball> Nobody has said anything in the Outages mailing list nor NANOG, so I think it's isolated to Verizon, and maybe only my area.
<cannonball> If you're not seeing traffic dips, then I think you guys probably can move on to other, more important things.
<DocWilco> he's getting loss on those routers now
<crucially> DocWilco: using mtr?
<DocWilco> yeah
<crucially> and using tcptrace?
<DocWilco> he said UDP
<crucially> yeah
<crucially> unless we trace back, it is impossible to say where the loss is
<cannonball> As a kick it in the shin test, I can go release and renew the IP on my wifi gateway, trying to get a new IP, and see if the problem persists. But that really doesn't _fix_ anything.
<crucially> cannonball: let us know if it continues and we can start to kick our isps
<crucially> cannonball: if it is the src+dst ip flow that has a error on a line card it might help
I called tech support, got someone who was really clueful and was NOT reading from a script, but was unable to escalate. He said the only thing he could do was break the lease, then I renewed, getting a new IP, and it worked fine. He also sent me a new router (was an old revision Actiontec, Rev A IIRC, and now I have a Rev F IIRC). My IP lease has not changed since I got the new router and replaced the old one.
I posted all of this in the hopes that some of this would be useful to someone in narrowing down the scope of this issue.
...Todd
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We are able to access the sites mentioned in Baltimore area. Hope this helps. It is a server problem.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@topdog: Glad you can access the sites mentioned from Baltitmore. The issue appears to affect Verizon FiOS customers in SoCal, likely due to some networking equipment in that region. There's no indication that the issue is a server problem.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@topdog wrote:We are able to access the sites mentioned in Baltimore area. Hope this helps. It is a server problem.
There has been a lot of discussion about various things here, so I want to provide some clarification:
All of the affected parties are SoCal Verizon FIOS users, both consumer and business grade, who are accessing other sites in SoCal. San Diego Credit Union (and various other banks), an MLS (real estate) website, Verizon Forums itself, an adult oriented site that uses a CDN that has a node in L.A. serving images and css, etc. This is a regional issue. And the issue as far as I was able to determine based on *end-to-end* testing is that the packets left my house and got to the server(s), the server sent it's reply, the reply left the edge of the provider's network, but the packet never made it back to me.
Sometimes the issue can be somewhat negated by using (for example) Google's DNS, because the IP that Google's DNS returns for a geo-located host might be different than the (local to SoCal) Verizon DNS servers. If that IP is local to, for example, Northern California, then the routing to and from that site works just fine. But depending on the site and its infrastructure, if the IP that it ultimately routes to is still the same host in SoCal, no connection can be initiated. It just hangs.
I hope this helps to better define the scope of what we're seeing and fighting.
...Todd
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@mrballcb wrote:
@topdog wrote:We are able to access the sites mentioned in Baltimore area. Hope this helps. It is a server problem.
There has been a lot of discussion about various things here, so I want to provide some clarification:
All of the affected parties are SoCal Verizon FIOS users, both consumer and business grade, who are accessing other sites in SoCal. San Diego Credit Union (and various other banks), an MLS (real estate) website, Verizon Forums itself, an adult oriented site that uses a CDN that has a node in L.A. serving images and css, etc. This is a regional issue. And the issue as far as I was able to determine based on *end-to-end* testing is that the packets left my house and got to the server(s), the server sent it's reply, the reply left the edge of the provider's network, but the packet never made it back to me.
Sometimes the issue can be somewhat negated by using (for example) Google's DNS, because the IP that Google's DNS returns for a geo-located host might be different than the (local to SoCal) Verizon DNS servers. If that IP is local to, for example, Northern California, then the routing to and from that site works just fine. But depending on the site and its infrastructure, if the IP that it ultimately routes to is still the same host in SoCal, no connection can be initiated. It just hangs.
I hope this helps to better define the scope of what we're seeing and fighting.
...Todd
I think you have correctly stated things. I still have an open ticket with Verizon. Their level 2 tech contacted my sunday and said that they are aware of the problem and that it is likely a routing table issue, and that he was going to have the router cache cleared fo my routes. He said it might be at the CO, but when I pointed out to him that there were many others in Southern California experiencing the same issues, he acknowledged it as a bigger problem. They were supposed to resolve this within 24 hours. It's not going on 48 since he contacted me and I don't believe it's fixed, but I will monitor and report back.
I told him I didn't want a new IP address for my router node -- when they did that for my neighbor, they broke his FiOS TV caller ID and remote DVR. He said that was no longer their fix (for the things it broke) and that clearing router caches is now the fix.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I live in Redlands and have had the problem not being able to access craigslist. Had done all the troubleshooting that Microsoft had available. Yesterday, I was pretty much on the phone all day off and on with Verizon Tech. Last resort, they transferred me to a pay tech, which was ridiculous. Not going to pay for tech. Supposed to be included in the DSL. I have the slowest one, which workds fine for me. This morining, I was prepared to cancel the service, but to my surprise, I was able to access craigslist. I'm sure it must have been a Verizon problem. A few weeks ago, I also had the KB2585542 problem. Luckily, I was able to fix by restoring system, and not allowing automatic updates. By elimination, I four the KB2585542 problem. Now, I am afraid to download any updates.
