- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Since about Monday I'm unable to access various sites. I have confirmed they are up because I can access them from a nearby Starbucks and a neighbor's connection, neither of which are on FIOS.
This is one site I can't access that I confirmed is online:
http://store.steampowered.com/
When I tried to access thiis site using IE 9 (from my Windows 7 PC), I get this message:
Internet Explorer cannot display the webpage
When I click on the Diagnose Connection Problems button, I get this:
website is online but isn't responding to connection attempts
This is happening on multiple sites.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I contacted tech support via chat, had him release/renew my IP address lease, and afterward I was able to get to the sites that I previously could not. So that fix definitely can work on an individual basis, though it's really just steering around the problem rather than properly repairing it... but still worth doing.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The higher netblock is part of a seperate, smaller allocation (Handle NET-71-181-128-0-1)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I stand corrrected, thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Anyone still having problems with this? Someone posted similar issues on dslreports. I wonder if this has to do with how VZ does its weird load balancing depending on Odd/even IP. If anyone is still having issues with this where the traceroute shows packets being dropped it might be interesting to try some other ips in the same /24 and see if it is specific to odd/even ips (You might notice that your 2nd (real) hop is a different router/IP depending on even/odd IP address. In my case this also causes a significant latency increase as well. Example:
traceroute to vnsc-bak.sys.gtei.net (4.2.2.2), 30 hops max, 46 byte packets
1 router.houkouonchi.jp (1.1.1.1) 0.164 ms 0.133 ms 0.523 ms
2 L103.LSANCA-DSL-29.verizon-gni.net (71.110.63.1) 1.026 ms 0.756 ms 0.724 ms
3 G0-3-1-7.LSANCA-LCR-22.verizon-gni.net (130.81.146.60) 52.357 ms 39.216 ms 39.918 ms
4 so-4-1-0-0.LAX01-BB-RTR2.verizon-gni.net (130.81.151.248) 127.499 ms 36.677 ms 1.965 ms
5 0.ae2.BR3.LAX15.ALTER.NET (152.63.2.133) 2.655 ms 2.285 ms 2.105 ms
6 4.68.63.245 (4.68.63.245) 3.378 ms 2.930 ms 3.413 ms
7 ae-11-60.car1.LosAngeles1.Level3.net (4.69.144.3) 6.407 ms 2.594 ms *
8 vnsc-bak.sys.gtei.net (4.2.2.2) 2.616 ms 2.485 ms 2.517 ms
root@dekabutsu: 07:50 AM :~# traceroute -I 4.2.2.3
traceroute to vnsc-lc.sys.gtei.net (4.2.2.3), 30 hops max, 46 byte packets
1 router.houkouonchi.jp (1.1.1.1) 1.231 ms 0.133 ms 0.106 ms
2 L103.LSANCA-DSL-29.verizon-gni.net (71.110.63.1) 1.419 ms 0.753 ms 0.728 ms
3 G0-3-1-7.LSANCA-LCR-21.verizon-gni.net (130.81.140.224) 5.940 ms 5.479 ms 5.345 ms
4 so-4-1-0-0.LAX01-BB-RTR1.verizon-gni.net (130.81.151.246) 5.749 ms 5.525 ms 5.656 ms
5 0.ae1.BR3.LAX15.ALTER.NET (152.63.2.129) 5.867 ms 6.135 ms 5.869 ms
6 4.68.63.245 (4.68.63.245) 6.303 ms 5.828 ms 5.841 ms
7 ae-21-70.car1.LosAngeles1.Level3.net (4.69.144.67) 6.048 ms 6.154 ms *
8 vnsc-lc.sys.gtei.net (4.2.2.3) 5.309 ms 5.796 ms 5.482 ms
root@dekabutsu: 07:50 AM :~# traceroute -I 4.2.2.4
traceroute to 4.2.2.4 (4.2.2.4), 30 hops max, 46 byte packets
1 router.houkouonchi.jp (1.1.1.1) 0.237 ms 0.433 ms 0.140 ms
2 L103.LSANCA-DSL-29.verizon-gni.net (71.110.63.1) 1.149 ms 1.274 ms 1.022 ms
3 G0-3-1-7.LSANCA-LCR-22.verizon-gni.net (130.81.146.60) 1.891 ms 1.518 ms 1.252 ms
4 so-6-0-0-0.LAX01-BB-RTR2.verizon-gni.net (130.81.29.126) 2.013 ms 1.923 ms 2.208 ms
5 0.xe-11-1-0.BR1.LAX15.ALTER.NET (152.63.112.5) 2.270 ms 3.088 ms 2.369 ms
6 ae6.edge1.LosAngeles9.level3.net (4.68.62.169) 3.105 ms 3.515 ms 3.042 ms
7 ae-31-80.car1.LosAngeles1.Level3.net (4.69.144.131) 105.685 ms 228.988 ms 18.215 ms
8 4.2.2.4 (4.2.2.4) 2.736 ms 2.515 ms 2.485 ms
root@dekabutsu: 07:50 AM :~# traceroute -I 4.2.2.5
traceroute to 4.2.2.5 (4.2.2.5), 30 hops max, 46 byte packets
1 router.houkouonchi.jp (1.1.1.1) 0.247 ms 0.166 ms 0.145 ms
2 L103.LSANCA-DSL-29.verizon-gni.net (71.110.63.1) 1.008 ms 0.814 ms 0.862 ms
3 G0-3-1-7.LSANCA-LCR-21.verizon-gni.net (130.81.140.224) 5.608 ms 5.413 ms 5.338 ms
4 so-4-1-0-0.LAX01-BB-RTR1.verizon-gni.net (130.81.151.246) 5.535 ms 5.540 ms 5.493 ms
5 0.ae1.BR3.LAX15.ALTER.NET (152.63.2.129) 5.950 ms 5.933 ms 5.857 ms
6 4.68.63.245 (4.68.63.245) 36.847 ms 8.010 ms 5.879 ms
7 ae-11-60.car1.LosAngeles1.Level3.net (4.69.144.3) 5.346 ms * *
8 4.2.2.5 (4.2.2.5) 5.478 ms 5.342 ms 5.478 ms
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Even though this thread is labeled "DNS issues", there's been nothing shared that actually documents a DNS issue. Everything has pointed to something in Verizon's network that's mishandling packets, packets on the return to the customer.
@houkou_onchi: Yes, while your traceroutes show differnet paths depending on the destination IP address, that doesn't matter so much because they all complete. So you haven't actually demonstrated an issue with DNS at all. ๐
If you can share a traceroute with us of sites that fail, that may be helpful, but the solution seems to be, in pretty well every case, getting another IP address that's not affected by whatever issue that Verizon has with their network gear.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am seeing issues getting web responses only from certain sites (e.g. msnbc.com, barnsandnoble.com, abcnews.go.com, foxnews.com, etc.) when I use the Verizon's default DNS settings in Windows (obtain DNS server address automatically). (Other sites like dslreports.com and cnn.com are fine.) When I switch from the obtain DNS server address automatically setting (which gave me the 192.168.1.1 primary and 68.238.64.12) to the another DNS server -- Level 3 (4.2.2.2 and 4.2.2.3) - all sites are near-instantenous, including those that I could not get to before. Then I manually tried 68.238.64.12 (the Verizon server added for automatic settings) with no secondary and it failed -- and the Windows diagnostics said I had a DNS issue. I then tried 192.168.1.1 and got the same results. I then went back to the Level 3 DNS and all works perfectly. My router has been factory reset (twice).
My neighbor had the same issue. Verizon released and renewed his IP address to give him a new WAN address -- and they broke his caller ID and remote DVR functions on FiOS TV!. His Internet seems "better" though, but this band-aid fix introduces other problems. I've put in a request on DSL Reports for this issue and I will try to get Verizon's attention Monday.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you 'dig' the fauly domain names using the bad DNS servers and share the results? If the DNS servers truly don't work for you, we'll see bad/no results.
i.e.
dig msnbc.com @192.168.1.1
dig msnbc.com @68.238.64.12
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I don't have dig, so I just used nslookup, which produced an interesting result. When I used L3 4.2.2.2 DNS (which gives me great results browsing to msnbc.com) I got 65.55.251.214 for msnbc.com, a "problem site." When I used 68.238.64.12 (the Verizon default DNS secondary) the result was 65.55.53.235 and when I used 192.168.1.1 (the Verizon default DNS primary) I also got 65.55.53.235 for msnbc.com.
So I tried barnsandnoble.com, also a "problem site." With 4.2.2.2 I get 65.204.48.118 161.221.89.118. With 68.238.64.12 I get 161.221.89.118 65.204.48.118 (backwards from the first), and the same for 192.168.1.1.
And for DSL report, a "non-problem site" With 4.2.2.2 I get 209.123.109.175. With 68.238.64.12 I get 209.123.109.175 (the same), and the same for 192.168.1.1.
But somehow I think this is still a routing issue?!?
Your thoughts please?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks, PJL. What this shows is that querying a different recursive DNS server returns a different IP address(es) . This is
standard behavior for those websites that want to direct users to a different server (perhaps to a certain geographical region), which is done based on the "known" location of the recursive DNS server making the query. I'm going to guess that's why, for some Verizon FiOS SoCal users and websites, changing the DNS server helps regain access.
What PJL's testing showed was that differnet IP addresses for the same website are accessible over Verizon's FiOS network in SoCal, which further confirms that something with the path (we believe return path) is affecting end-to-end connectivity.
On the outside chance that that routing is actually symmetric: PJL, can you provide traceroutes to 65.55.251.214 and 65.55.53.235, as well as 65.204.48.118 and 161.221.89.118? Perhaps we can identify which path is the problem one.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@frnkblk wrote:...
On the outside chance that that routing is actually symmetric: PJL, can you provide traceroutes to 65.55.251.214 and 65.55.53.235, as well as 65.204.48.118 and 161.221.89.118? Perhaps we can identify which path is the problem one.
Here are the traces:
Tracing route to 65.55.251.214 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms 192.168.1.1
2 5 ms 5 ms 5 ms L100.LSANCA-VFTTP-128.verizon-gni.net [96.229.59.1
3 6 ms 6 ms 6 ms P9-1.PITBPA-LCR-03.verizon-gni.net [130.81.55.176]
4 7 ms 7 ms 58 ms 130.81.199.38
5 6 ms 6 ms 6 ms 0.so-7-2-0.XL3.LAX15.ALTER.NET [152.63.113.241]
6 7 ms 8 ms 7 ms TenGigE0-4-1-0.GW4.LAX15.ALTER.NET [152.63.115.178
7 7 ms 7 ms 7 ms microsoft.customer.alter.net [152.179.21.26]
8 41 ms 41 ms 42 ms xe-9-0-0-0.sn1-96cb-1b.ntwk.msn.net [207.46.42.178
9 78 ms 78 ms 78 ms xe-0-1-0-0.blu-96c-1b.ntwk.msn.net [207.46.43.23]
10 77 ms 77 ms 77 ms ten9-2.blu-76c-1a.ntwk.msn.net [207.46.47.155]
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.
Trace complete.
Tracing route to msnbc.com [65.55.53.235] over a maximum of 30 hops:
1 <1 ms <1 ms 1 ms 192.168.1.1
2 5 ms 4 ms 5 ms L100.LSANCA-VFTTP-128.verizon-gni.net [96.229.59.1]
3 8 ms 8 ms 8 ms G0-5-2-4.LSANCA-LCR-22.verizon-gni.net [130.81.183.176]
4 9 ms 8 ms 10 ms so-4-1-0-0.LAX01-BB-RTR2.verizon-gni.net [130.81.151.248]
5 10 ms 10 ms 11 ms 0.so-6-0-0.XT2.LAX9.ALTER.NET [152.63.10.157]
6 60 ms 10 ms 12 ms 0.so-7-1-0.XL4.LAX15.ALTER.NET [152.63.1.242]
7 11 ms 10 ms 12 ms TenGigE0-5-1-0.GW4.LAX15.ALTER.NET [152.63.115.182]
8 10 ms 9 ms 10 ms microsoft.customer.alter.net [152.179.21.26]
9 10 ms 10 ms 10 ms xe-3-0-0-0.lax-96cbe-1a.ntwk.msn.net [207.46.47.10]
10 20 ms 19 ms 19 ms xe-7-0-2-0.by2-96c-1a.ntwk.msn.net [207.46.42.176]
11 43 ms 41 ms 41 ms ge-5-2-0-0.co2-64c-1a.ntwk.msn.net [207.46.40.84]
12 42 ms 40 ms 41 ms xe-0-2-0-0.co2-96c-2b.ntwk.msn.net [207.46.42.203]
13 50 ms 41 ms 41 ms xe-7-3-0-0.co1-96c-1b.ntwk.msn.net [207.46.40.246]
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.
Trace complete.
Tracing route to 65.204.48.118 over a maximum of 30 hops
1 <1 ms <1 ms 1 ms 192.168.1.1
2 4 ms 5 ms 21 ms L100.LSANCA-VFTTP-128.verizon-gni.net [96.229.59.1]
3 6 ms 6 ms 7 ms P9-1.PITBPA-LCR-03.verizon-gni.net [130.81.55.176]
4 6 ms 7 ms 6 ms so-4-1-0-0.LAX01-BB-RTR1.verizon-gni.net [130.81.151.246]
5 6 ms 7 ms 6 ms 0.ae1.BR3.LAX15.ALTER.NET [152.63.2.129]
6 7 ms 7 ms 7 ms 204.255.168.134
7 91 ms 90 ms 92 ms cr2.la2ca.ip.att.net [12.123.30.150]
8 89 ms 91 ms 92 ms cr1.slkut.ip.att.net [12.122.30.29]
9 89 ms 91 ms 91 ms cr2.dvmco.ip.att.net [12.122.30.26]
10 91 ms 91 ms 91 ms cr1.cgcil.ip.att.net [12.122.31.86]
11 111 ms 91 ms 91 ms cr2.cgcil.ip.att.net [12.122.2.54]
12 94 ms 96 ms 94 ms cr1.n54ny.ip.att.net [12.122.1.1]
13 87 ms 87 ms 87 ms cr81.nw2nj.ip.att.net [12.122.105.30]
14 92 ms 92 ms 92 ms 12.122.115.105
15 81 ms 82 ms 82 ms 12.94.85.66
16 * * * Request timed out.
17 81 ms 81 ms 82 ms 65.204.48.118
Trace complete.
Tracing route to 161.221.89.118 over a maximum of 30 hops
1 <1 ms 1 ms <1 ms 192.168.1.1
2 5 ms 4 ms 4 ms L100.LSANCA-VFTTP-128.verizon-gni.net [96.229.59.1]
3 6 ms 6 ms 7 ms P9-1.PITBPA-LCR-03.verizon-gni.net [130.81.55.176]
4 27 ms 7 ms 6 ms 130.81.199.38
5 6 ms 6 ms 7 ms 0.ae1.BR3.LAX15.ALTER.NET [152.63.2.129]
6 7 ms 6 ms 7 ms 192.205.35.161
7 92 ms 91 ms 89 ms cr2.la2ca.ip.att.net [12.123.30.134]
8 89 ms 91 ms 92 ms cr1.slkut.ip.att.net [12.122.30.29]
9 88 ms 87 ms 87 ms cr2.dvmco.ip.att.net [12.122.30.26]
10 89 ms 89 ms 91 ms cr1.cgcil.ip.att.net [12.122.31.86]
11 88 ms 88 ms 91 ms cr2.cgcil.ip.att.net [12.122.2.54]
12 94 ms 95 ms 95 ms cr1.n54ny.ip.att.net [12.122.1.1]
13 85 ms 85 ms 85 ms cr83.n54ny.ip.att.net [12.122.105.50]
14 87 ms 86 ms 86 ms gar1.n54ny.ip.att.net [12.122.105.9]
15 84 ms 82 ms 81 ms 12.94.184.186
16 81 ms 82 ms 82 ms 161.221.88.6
17 81 ms 81 ms 82 ms 161.221.89.118
Trace complete.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@frnkblk: I myself never had this issue or it has been un-noticed. I just noticed routing changed for me a short while ago as I used to ping lower to odd ips now its even again. DNS has some weird policy based routing as they seem to do some of their load balancing just based off destination/source IP being even/odd/etc which is pretty weird to me as I have never seen any other network do that.
I was just showing an example with my traceroutes, of course they went through as I am not having issues. I was curious though if someone else who was having issues if it was just affecting odd or just even ips on the /24 they were trying to access.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@houkou_onchi wrote:<snip>
I was curious though if someone else who was having issues if it was just affecting odd or just even ips on the /24 they were trying to access.
@houkou_onchi: Your theory about odd/even destination IP addresses doesn't seem to hold true, because your traceroutes documented in an earlier post to 4.2.2.2 and 4.2.2.5 both have second-last hops of ae-11-60.car1.LosAngeles1.Level3.net.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Will the lease break/renew IP fix work on a Fios business account that's using a static IP address?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Probably not, as the "new IP" fix requires getting a new/different IP address so that hopefully the return traffic goes over a different (L1/L2) path than the existing path.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Had a feeling. Fabulous. Now I need to see if I can get them to assign me a new static IP address. Thanks for the reply.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Mordeen wrote:Had a feeling. Fabulous. Now I need to see if I can get them to assign me a new static IP address. Thanks for the reply.
Are you assigned a single static IP address? Or are you assigned a range of static IP addresses? Does your router actually use dhcp to get that address? Or is it set to a single IP address?
If you're assigned a range of IP addresses, then you are in the perfect position to change it yourself and test how it works for each individual IP address. It also arms you/us with more information about characteristics of the issue. Things like "every odd IP works" or "every even IP works" or "every 4th IP works" would be most illuminating. Some of the participants in this thread have voices that I believe can reach engineering in Verizon, but only if the details and facts are repeatable. Your physical location might also be part of that equation. So far, we've seen mostly San Diego and Inland Empire reporting this, but one of the participants was also in Long Beach.
If not assigned a range, then yeah, you'll have to get someone on the phone to assign you a new IP. What is it now? During the change to a new IP, do any of them not work? And what IP starts working when you get it changed?
For reference, this IP is working for me: 173.58.47.108 (has been since a tech support agent broke the lease 2 weeks ago and I got this IP)
this is the IP that was not working for me: 173.58.43.42
Here is the conversation when I was troubleshooting with a CDN provider when we figured out that I couldn't reach their west coast node but I could reach their Dallas and Virginia nodes:
<cannonball> Working from office in Los Angeles: Savvis -> Cogent (te0-3-0-7.ccr21) -> 38.99.68.170 -> 38.127.199.74
<cannonball> Not working: Verizon -> AlterNet -> Cogent (te0-3-0-5.ccr2) -> 38.99.68.170 -> 38.127.199.74
<crucially> cannonball: huh, you get all the way to 38.127.199.74?
<cannonball> The tracepath does, yes.
<crucially> ok
<crucially> fun
<cannonball> There are some "no reply" sprinkled in the two paths I pasted into the pastebin. Didn't figure they were that important.
<crucially> mtr gets around those usually
<cannonball> openssl s_client -connect flassets.a.ssl.fastly.net:443 hangs for me, no connection made.
<cannonball> Yeah, you're right, mtr does handle displaying those a lot nicer.
<crucially> mtr will also show you multiple paths
<crucially> better
<crucially> cannonball: can you msg me your end point so i can mtr back/tcpdump
<cannonball> Doing the openssl command from work connects and does the cert exchange.
<cannonball> Sure, do you want work (working) or home (non-working) ip?
<crucially> the non working one
<crucially> cannonball: you are west coast right?
<cannonball> 173.58.43.42 (home, Verizon FIOS) Yes, Los Angeles area (Murrieta, CA)
<cannonball> $ host flassets.a.ssl.fastly.net
<cannonball> flassets.a.ssl.fastly.net has address 38.99.68.180
<cannonball> I assume I'm the only person complaining about this, both for James and for Fastly. So I wouldn't rule out a local issue.
<crucially> I get to 0.ae3.SJC01-BB-RTR1.verizon-gni.NET (tracing from one of the west coast nodes)
<crucially> then it stops
<DocWilco> I can mtr all the wai to 173.58.43.42 from home
<crucially> oh great
<cannonball> does anybody know if enabling ipv6 (miredo, the linux implementation of teredo) would have any adverse effects?
<DocWilco> although, with some heave loss
<crucially> cannonball: if you put in your hsots file
<crucially> 38.99.68.181
<crucially> instead of 180
<crucially> and see if it works
<cannonball> doing
<crucially> cannonball: no, ipv6 should not matter
<DocWilco> a15 can reach him
<crucially> DocWilco: cache-s22 can reach him
<crucially> s1 can not
<DocWilco> yeah
<DocWilco> and 0.ae3.SJC01-BB-RTR1.verizon-gni.NET is where it ends
<DocWilco> I'm going to go with Verizon routing problem
<crucially> from 181 I get 0.ge-2-0-0.SJC01-BB-RTR2.verizon-gni.net
<crucially> from 180 I get 7. 0.ae3.SJC01-BB-RTR1.verizon-gni.NET
<cannonball> Testing with curl from commandline, neither 180 or 181 work. Got a couple other IP's I could try forcing?
<crucially> cannonball: cache-d20.hosts.fastly.net
<cannonball> Yeah, I tend to think it's Verizon too. I'm stumped why the tracepath gets all the wya.
<crucially> cannonball: asymmetric routing
<cannonball> cache-d20 works with curl, testing int he browser.
<crucially> the return path is different
<crucially> cannonball: try varnish-v11
<crucially> and varnish-v12
<cannonball> browser worked, trying varnish-v11 and varnish-v12 next.
<cannonball> ...browser worked with cache-d20
<cannonball> ...curl works with varnish-v11
<cannonball> ..and works with varnish-v12
<crucially> ok
<crucially> so it isn't cogent/verizon
<cannonball> So it's a Verizon issue, not you guys. My apologies for bugging you, I'll call them (though I'm just a FIOS customer, I don't have a NOC contact). If I can figure out who the peer is, that would at least help them.
<crucially> cannonball: d20 is in dallas, v\d is in virginia
<crucially> I will give you the reverse tracepaths
<cannonball> I'm sure I'll go through voicemail hell just to get told I need to reboot my wifi gateway ๐ but I gotta pester them.
<cannonball> thank you for your help guys.
<crucially> hang on
<cannonball> am holding, just being polite ๐
<cannonball> Not enough people say thank you in irc, wanted to make sure I wasn't one of them.
<crucially> cannonball: i want to try a few more things
<cannonball> My typo of "fastfly" also explains why I couldn't find jack about you guys in google. I was beginning to lose faith in the Goo.
<crucially> heheh
<crucially> can you ping 180 and 181
<cannonball> I'm ready to try anything you want.
<crucially> i want to verify we see the requests
<cannonball> pinging 180 right now
<crucially> ok, i see them
<crucially> now one more check
<cannonball> I get responses back from 181.
<cannonball> But I get nothing back from 180.
<crucially> yeah, makes sense
<crucially> though unclear why curl doesn't work to 181
<crucially> ok, both icmp replies leave our netowkr
<DocWilco> friend of mine also on Verizon FIOS can't reach s1 either
<crucially> it is probably a line card that has an error in it for this path
<cannonball> Is .180 the s1 node?
<crucially> yeah
<crucially> 181 is s22
<crucially> you should be able to reach s22
<crucially> if icmp works
<cannonball> I can ping it, mtr shows it being reached, testing with tcptraceroute now.
<DocWilco> nevermind, he could reach them, but cogent routers are very heavily loaded
<cannonball> curl does something weird. It must think it's connected because it displays the download status line, it just never moves.
<crucially> cannonball: what is the curl line you use?
<DocWilco> crucially: see the email I just sent, check out hops 12 and 13, those times are pretty high
<cannonball> curl --verbose https://flassets.a.ssl.fastly.net/stylesheets/base_packaged.css?323c1e4e6e864
<crucially> hmm
<crucially> DocWilco: shouldn't matter, thats just control plane
<crucially> has s1 had a traffic drop
<DocWilco> crucially: the fact that it's responsing slowly means it's heavily loaded though
<DocWilco> not to the point where it's dropping stuff going through, but still
<crucially> DocWilco: on the control plane, not switching plane
<crucially> routing/switching
<crucially> there is no way you can tell if the forward plane is loaded or not
<cannonball> icmp is usually dropped first, so if it's not dropping it, the cpu is probably not near its limits.
<cannonball> but yeah, tells you very little about forwarding.
<cannonball> Nobody has said anything in the Outages mailing list nor NANOG, so I think it's isolated to Verizon, and maybe only my area.
<cannonball> If you're not seeing traffic dips, then I think you guys probably can move on to other, more important things.
<DocWilco> he's getting loss on those routers now
<crucially> DocWilco: using mtr?
<DocWilco> yeah
<crucially> and using tcptrace?
<DocWilco> he said UDP
<crucially> yeah
<crucially> unless we trace back, it is impossible to say where the loss is
<cannonball> As a kick it in the shin test, I can go release and renew the IP on my wifi gateway, trying to get a new IP, and see if the problem persists. But that really doesn't _fix_ anything.
<crucially> cannonball: let us know if it continues and we can start to kick our isps
<crucially> cannonball: if it is the src+dst ip flow that has a error on a line card it might help
I called tech support, got someone who was really clueful and was NOT reading from a script, but was unable to escalate. He said the only thing he could do was break the lease, then I renewed, getting a new IP, and it worked fine. He also sent me a new router (was an old revision Actiontec, Rev A IIRC, and now I have a Rev F IIRC). My IP lease has not changed since I got the new router and replaced the old one.
I posted all of this in the hopes that some of this would be useful to someone in narrowing down the scope of this issue.
...Todd
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Is this still being worked on? It's definitely still a serious problem for me. This morning I used a bookmark testing extension to find out which of my bookmarked pages were failing to connect in the same way, and then tested each of them myself to make sure it was this same problem, and not a legitimate broken web page or other issue.
Cleared browser cache, cookies, and rebooted the router before starting.
All of the sites listed below will give the error: "The server at [address] is taking too long to respond."
In every case I can not get to it via my Verizon FiOS connection.
I can get to each of these via my phone's non-Verizon internet connection, without problem.
downforeveryoneorjustme.com reports all of the sites below are up and running as well during my tests.
costco.com
albertsons.com
club.nintendo.com
portforward.com
eternalmoonwalk.com
nfggames.com
foodily.com
thinkwithportals.com
runicgames.com
tcrf.net
wiki.teamfortress.com
Previously I was unable to get to steampowered.com, but could get to steamcommunity.com
After getting a new IP address, I'm able to get to steampowered.com but not steamcommunity.com
My wife and I both do at least some of our work from home. By what appears to be pure luck none of the websites we need to access for work have been blocked so far, but if that does happen we would be in some serious trouble. We need this issue to be resolved, and soon. It's already been nearly a month so far, our entire time as a Verizon customer.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@PJL: Can you test Scottage's list?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Actually, one correction. I was able to get to steamcommunity.com and steampowered.com both as late as yesterday. So the fact that I can suddenly not get to steamcommunity.com today seems to indicate that this problem seems to be getting worse, not better, and is still spreading.
EDIT 1: Also, testing by using several free proxies (tested by way of proxy.org), I'm able to visit these same sites on the same computer/connection that fails when not using the proxy. But of course that's not a valid fix, just a test.
EDIT 2: As for DNS, I've tried both auto-assigned DNS and another server (opendns.com) and the issue remains either way.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Same problem here now ...steamcommunity.com unavailable (steampowered.com is ok for now). can get there on my phone and at work, not through fios.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@frnkblk wrote:@PJL: Can you test Scottage's list?
Using Verizon's default DNS and then L3 DNS:
costco.com: good response using either DNS
albertsons.com: NO RESPONSE using either DNS
club.nintendo.com: good response using either DNS
portforward.com: NOT RESPONSE using either DNS
eternalmoonwalk.com: good response using either DNS
nfggames.com: NO RESPONSE using either DNS
foodily.com: good respone using either DNS
thinkwithportals.com: good response using either DNS
runicgames.com: NO REPONSE with Verizon, good response with L4
tcrf.net: good response using either DNS
wiki.teamfortress.com: NO RESPONSE using either DNS

