Child Item

Site Totals

61181 Questions Answered 267 Categories

DNS issues in SoCal

Since about Monday I'm unable to access various sites. I have confirmed they are up because I can access them from a nearby Starbucks and a neighbor's connection, neither of which are on FIOS.

This is one site I can't access that I confirmed is online:

http://store.steampowered.com/

When I tried to access thiis site using IE 9 (from my Windows 7 PC), I get this message:

Internet Explorer cannot display the webpage

When I click on the Diagnose Connection Problems button, I get this:

website is online but isn't responding to connection attempts

This is happening on multiple sites.

Comments

DavidCRMLS

Hi shayne571,

We are experiencing the exact same thing since Monday.

Our members on Verizon FIOS and Verizon DSL are having DNS issues. We are based in California and all the issues are based on Verizon's network in California.

Randomally FIOS users cannot get to certain websites and ours is one of them. One of the big ones is www.bankofamerica.com

I have tried for two days to get upper level help with this issue, but Verizon is just not helpful!!! They are too busy stonewalling.

We have over 70,000 members hitting our web servers and this has affect over 300 of them and counting. And it is only the ones on Verizon FIOS or Verizon DSL

We have done some troubleshooting with our members and rebooting their modems have helped on some, but not most.

We believe that Verizon FIOS/DSL has a DNS server that is not functioning properly somewhere in the network that these FIOS/DSL users are using to DNS name resolve.

I just cannot get this thing escalated.

We are about to tell our members to call their Verizon FIOS/DSL tech support and demand a modem replacement or a truck be sent out to check their fiber.

We have no other alternative.

Is this happening to anyone else other than us and shayne571?

shayne571

When I do a tracert to sites, the IP addresses resolve.

This is an example:

C:\>tracert store.steampowered.com

Tracing route to store.steampowered.com [63.228.223.104]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 <1 ms <1 ms <1 ms 192.168.0.1
3 7 ms 6 ms 7 ms L100.LSANCA-VFTTP-91.verizon-gni.net [98.112.217.1]
4 9 ms 9 ms 9 ms G11-0-1-791.LSANCA-LCR-07.verizon-gni.net [130.81.109.230]
5 12 ms 9 ms 11 ms so-7-0-1-0.LAX01-BB-RTR1.verizon-gni.net [130.81.29.140]
6 13 ms 12 ms 11 ms 0.ae1.BR3.LAX15.ALTER.NET [152.63.2.129]
7 12 ms 12 ms 11 ms lap-brdr-03.inet.qwest.net [63.146.26.209]
8 40 ms 39 ms 39 ms tuk-edge-12.inet.qwest.net [67.14.4.10]
9 38 ms 36 ms 39 ms 63-234-174-50.dia.static.qwest.net [63.234.174.50]
10 43 ms 44 ms 43 ms c1-tuk-tuk-b1.valve.net [208.64.202.18]
11 40 ms 39 ms 36 ms store.steampowered.com [63.228.223.104]

Trace complete.

C:\>

vibrentone

Hello. Yes i am having the same issues.

Certain sites are fine and some i cannot access. They just time out.

vibrentone

Who should we notify. Has anyone started a complaint?

shayne571

I just tried changing the Router to use public DNS servera 4.2.2.1 and 4.2.2.2 and it's the same result.

Something is clearly wrong on the FIOS end.

shayne571

Glad it's not just me.

I have not called support because to be honest I expect to get a runaround, I wanted to try here first.

Techman28

let me ask this are you guys able to get the actual client steam up and running?

shayne571

Techman28 - no. The client and the website don't load.

I plugged the computer directly into the FIOS router, rebooted the PC and the Router.

No go.

kesso

I am having the same issues, also in Southern California. The difference is that I can go into alaskaair and steam but I cannot go into:

accounts.craigslist.org

boards.developerforce.com

and others that I routinely use.

The IP / ping / tracert resolve just fine but I cannot browse. Changed DNS multiple times and no change, although I assume this is logical since the IPs resolve (ergo DNS probably not the issue).

I also want to point out that my AT&T iPhone navigates to those sites just fine when not connected to wifi on my network.

Any ideas, techman28?

frnkblk

Has anyone tried running mturoute (http://www.elifulkerson.com/projects/mturoute.php) to each of the problem sites to see if there's some kind of MTU issue? Perhaps those experiencing problems can show the output of mturoute for a working and a few non-working sites.

Also, what does wget (http://gnuwin32.sourceforge.net/packages/wget.htm) show when trying to access the listed sites?

shayne571

Is anyone at Verizon monitoring these posts?

navybuff1

I too, since monday have not been able to connect to 209.242.175.41 (San Diego Credit Union)

When trying to connect through a proxy it connects fine....

Pinging sdccu.com [209.242.175.41] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 209.242.175.41:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

C:\Users\Ken>ping www.sdccu.org

Pinging sdccu.org [209.242.175.41] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 209.242.175.41:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

C:\Users\Ken>ping www.sdccu.net

Pinging sdccu.net [209.242.175.41] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 209.242.175.41:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

I am about ready to trash FIOS

shayne571

We have been in touch with technical support.

They have received calls from customers in SoCal complaining about this same problem and they are looking into it.

Internet might be intermittent for us for the next 12 hours or so as they investigate.

frnkblk

Based on my testing outside of the FiOS network, I don't think any of those three SDCCU sites respond to ICMP. tcping on port 80 works fine. So don't trash FiOS because your credit union blocks ICMP. 😃

homeispv

This is absolutely unacceptable. Another site that is not able to be accessed is the Central Regional Multiple Listing Service. After 3 hours on hold a technician from Fios finally came on, had me run a tracert and claimed this was not a Verizon problem

**bleep**

ONLY Verizon customers are affected of the THOUSANDS of Realtors trying to access the Multiple Listing Service from Ventura County through LA, Riverside, San Bernardino, Orange and San Diego Counties.

How dare Verizon at this late date continue to deny this is NOT their problem ...

I'm disgusted with Verizon and feel hostage to them due to my business "persona" is tied to verizon email address.

JamesLaszko

It's not an issue with SDCCU nor their blocking of ICMP. It is an infrastructure issue related to Verizon network equipment. I have just received this information from the SVP of Technology at SDCCU:

Hi James,

I saw your post on Verizon’s forum regarding not being able to access certain sites. Sorry for the random Email.

I’m the SVP of Technology at San Diego County Credit Union and we have a lot of customers like you who are unable to reach our Internet Banking site, only from Verizon FIOS and only from your area.

We believe that we found the problem by sniffing packets and placing Wireshark on the line. It seems like Verizon change a piece of equipment last Sunday which drops packets larger that 1560 bytes in size. These packets can be generated by various servers. Standard MTU (Maximum Transmission Unit) is 1500 bytes on Windows server, but if the site (like ours) uses EV SSL (Extended Validation SSL) certificate which turns your browser bar green for validation, ads a few more bytes to the header of the packet, making it larger than 1560 bytes.

So in English terms, their equipment was able to pass traffic for packets that were large and carried additional security info, but the new equipment is misconfigured and is dropping the transmission.

We tested this by using PING and reach a couple of destination hosts on Verizon FIOS. When the number of packets got past 1560, it quit responding.

Can you please post this on the forum? I have not been able to reply to any of the posts since I’m coming from Cox network and I believe I’m being blocked or dropped for replying or posting.

I really hope someone from Verizon sees this and corrects the issue. We’ve been trying to reach them since Wednesday without much luck.

Regards,

Sean

We have seen issues connecting all over the place from FIOS users in the Riverside county area, since Monday 1/9/12. Call into VZ tech support have been fruitless. Getting network engineering to even see this issue has been worthless. Someone from Verizon needs to get on here and start dealing with issues and there needs to be some level of professional interaction from higher ups at Verizon that would allow for remediation of issues like this. We're not all just "residential users" that are complaining because our "Internet doesn't work"

Anyone who can do anything or affect anyone at VZ is encouraged to contact me directly and I'd be happy to put the different parties we've been coordinating with together.....

James

JamesLaszko

Shayne-

Do you have a trouble ticket / incident number / outage number you can reference for this "work being performed"?

We have 10 of our engineers and a couple hundred hours involved in this issue so far this week with Verizon and have not had anyone do anything short of rebooting ONT's and renewing DHCP leases.......

James

shayne571

My issues appear to have been resolved. I now can get to all sites I reported as unreachable in earlier posts.

I didn't get a ticket number because the people I spoke to were already aware of the issue.

If anyone is still having issues, keep calling until you get someone who knows about the issue. Point them to this thread. It took us about 1 hr to get through.

DavidCRMLS

We just had one of our Brokers in Long Beach that has been affected since Monday call us. It looks like he used to work for Verizon and gave some inside support numbers that may help get this thing escalated. Worth a shot to slam them with calls and refer them to this post, so they can see it is a wide spread area and only Verizon is having the issue.

Verizon Ontario Switch 909-983-0000 (they will provide you with more internal Verizon phone numbers)

Verizon Long Beach Core Centralized Outage Group 800-367-5801

I have a call back to a Verizon FIOS Tech that is in escalation. I will report back with what I find.

David

frnkblk

@JamesLaszko wrote:
It's not an issue with SDCCU nor their blocking of ICMP.

@JamesLasko: Just to be clear, I wasn't indicating the issue was with SDCCU's blocking of ICMP. 😃 I was pointing out that that the inability to ping that site was normal for the SDCCU site and likely had nothing to do with navybuff's inability to access it.

frnkblk

Sean's note points to an MTU issue -- it's unusual for residential broadband networks to have an MTU size larger than 1500 bytes. It sounds like Verizon's routers are not fragmenting packets when necessary.

If the issue is lack of proper packet fragmentation, SDCCU could help their customers by clamping down their MTU to 1500 (either on the server hosting the website or load balancer or router(s)). I'm assuming, of course, that Verizon's FiOS subscribers use home routers that already have a maximum MTU of 1500, but if they don't, many routers can be manually set to that size.

frnkblk

Another SDCCU access issue posted here: http://forums.techguy.org/web-email/1036088-certain-websites-will-not-load.html

BuzzLightyear1

Same problem here in Moreno Valley. Can not connect to craigslist.org or MSN.com.

Why all the problems guys? I pay my bill .... why do I not get great service?

DavidCRMLS

Looks like our members are still having issues getting to our site. We have not gotten as many calls as last week, but the issue is still there for those on Verizon FIOS or DSL.

Is anyone else still experiencing the issue?

Thanks,

David

Scottage

Definitely still happening when I try to go to steampowered.com

The related steamcommunity.com works just fine.

kpbabln

I'm having the same issue trying to connect to Craigslist. I looked on their help forum, which I'm able to access, and other members have reported the same issue. They said some members had resolution by contacting Verizon via the Live Chat, so I contacted Verizon via the Live Chat this morning. The "Agent" just stated that there is an outage going on with Craigslist with the Verizon server and that the engineers working on it said it will be resolved very soon. I pointed out the DNS server link that I was provided here and stated Craigslist members have had success with getting the issue resolved via the Live Chat. I stated I haven't been able to access Craigslist for at least a week and asked how much longer will it take. His answer: 24-48 hrs. I guess we'll see!

RileyCat

This has been going on in Temecula for over a week now.

I have an open trouble ticket and can't get the issue solved. I spoke with Rick at tech support again this AM and all that he could say was that it was an open issue and that it had been escallated. He assured me that it would be resolved soon, yet when I asked what the issue was and how they planned to fix it he couldn't offer any information.

The orignaltechnicians I spoke with were clueless. They asked me to clear the cache on all my browers and said that would clear it up. I didn't think it was funny.

I can access craigslist.org from the starbucks down the street or from my work provided MiFi device and phone (Verizon) but not from any of my computers via Fios at home.

I can do a trace route to Craigslist..

C:\Users\Scott>tracert www.craigslist.org

Tracing route to www.craigslist.org [208.82.238.130]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms my.router [192.168.2.1]
2 1 ms <1 ms <1 ms myrouter.home [192.168.1.1]
3 7 ms 7 ms 7 ms L100.LSANCA-VFTTP-121.verizon-gni.net [173.58.73
.1]
4 13 ms 12 ms 12 ms G10-0-4-1921.LSANCA-LCR-09.verizon-gni.net [130.
81.138.84]
5 15 ms 11 ms 13 ms so-4-0-0-0.LAX01-BB-RTR1.verizon-gni.net [130.81
.28.72]
6 16 ms 12 ms 12 ms 0.ae1.BR3.LAX15.ALTER.NET [152.63.2.129]
7 12 ms 12 ms 12 ms lap-brdr-03.inet.qwest.net [63.146.26.209]
8 23 ms 21 ms 23 ms svl-edge-23.inet.qwest.net [67.14.12.226]
9 21 ms 22 ms 22 ms 65.113.32.14
10 23 ms 21 ms 22 ms vs-imgs-f.craigslist.org [208.82.238.130]

Trace complete.

Why can't Verizon resolve this?

Scottage

I contacted tech support via chat, had him release/renew my IP address lease, and afterward I was able to get to the sites that I previously could not. So that fix definitely can work on an individual basis, though it's really just steering around the problem rather than properly repairing it... but still worth doing.

JamesLaszko

The higher netblock is part of a seperate, smaller allocation (Handle NET-71-181-128-0-1)

frnkblk

I stand corrrected, thanks.

houkou_onchi

Anyone still having problems with this? Someone posted similar issues on dslreports. I wonder if this has to do with how VZ does its weird load balancing depending on Odd/even IP. If anyone is still having issues with this where the traceroute shows packets being dropped it might be interesting to try some other ips in the same /24 and see if it is specific to odd/even ips (You might notice that your 2nd (real) hop is a different router/IP depending on even/odd IP address. In my case this also causes a significant latency increase as well. Example:

traceroute to vnsc-bak.sys.gtei.net (4.2.2.2), 30 hops max, 46 byte packets
1 router.houkouonchi.jp (1.1.1.1) 0.164 ms 0.133 ms 0.523 ms
2 L103.LSANCA-DSL-29.verizon-gni.net (71.110.63.1) 1.026 ms 0.756 ms 0.724 ms
3 G0-3-1-7.LSANCA-LCR-22.verizon-gni.net (130.81.146.60) 52.357 ms 39.216 ms 39.918 ms
4 so-4-1-0-0.LAX01-BB-RTR2.verizon-gni.net (130.81.151.248) 127.499 ms 36.677 ms 1.965 ms
5 0.ae2.BR3.LAX15.ALTER.NET (152.63.2.133) 2.655 ms 2.285 ms 2.105 ms
6 4.68.63.245 (4.68.63.245) 3.378 ms 2.930 ms 3.413 ms
7 ae-11-60.car1.LosAngeles1.Level3.net (4.69.144.3) 6.407 ms 2.594 ms *
8 vnsc-bak.sys.gtei.net (4.2.2.2) 2.616 ms 2.485 ms 2.517 ms

root@dekabutsu: 07:50 AM :~# traceroute -I 4.2.2.3
traceroute to vnsc-lc.sys.gtei.net (4.2.2.3), 30 hops max, 46 byte packets
1 router.houkouonchi.jp (1.1.1.1) 1.231 ms 0.133 ms 0.106 ms
2 L103.LSANCA-DSL-29.verizon-gni.net (71.110.63.1) 1.419 ms 0.753 ms 0.728 ms
3 G0-3-1-7.LSANCA-LCR-21.verizon-gni.net (130.81.140.224) 5.940 ms 5.479 ms 5.345 ms
4 so-4-1-0-0.LAX01-BB-RTR1.verizon-gni.net (130.81.151.246) 5.749 ms 5.525 ms 5.656 ms
5 0.ae1.BR3.LAX15.ALTER.NET (152.63.2.129) 5.867 ms 6.135 ms 5.869 ms
6 4.68.63.245 (4.68.63.245) 6.303 ms 5.828 ms 5.841 ms
7 ae-21-70.car1.LosAngeles1.Level3.net (4.69.144.67) 6.048 ms 6.154 ms *
8 vnsc-lc.sys.gtei.net (4.2.2.3) 5.309 ms 5.796 ms 5.482 ms

root@dekabutsu: 07:50 AM :~# traceroute -I 4.2.2.4
traceroute to 4.2.2.4 (4.2.2.4), 30 hops max, 46 byte packets
1 router.houkouonchi.jp (1.1.1.1) 0.237 ms 0.433 ms 0.140 ms
2 L103.LSANCA-DSL-29.verizon-gni.net (71.110.63.1) 1.149 ms 1.274 ms 1.022 ms
3 G0-3-1-7.LSANCA-LCR-22.verizon-gni.net (130.81.146.60) 1.891 ms 1.518 ms 1.252 ms
4 so-6-0-0-0.LAX01-BB-RTR2.verizon-gni.net (130.81.29.126) 2.013 ms 1.923 ms 2.208 ms
5 0.xe-11-1-0.BR1.LAX15.ALTER.NET (152.63.112.5) 2.270 ms 3.088 ms 2.369 ms
6 ae6.edge1.LosAngeles9.level3.net (4.68.62.169) 3.105 ms 3.515 ms 3.042 ms
7 ae-31-80.car1.LosAngeles1.Level3.net (4.69.144.131) 105.685 ms 228.988 ms 18.215 ms
8 4.2.2.4 (4.2.2.4) 2.736 ms 2.515 ms 2.485 ms

root@dekabutsu: 07:50 AM :~# traceroute -I 4.2.2.5
traceroute to 4.2.2.5 (4.2.2.5), 30 hops max, 46 byte packets
1 router.houkouonchi.jp (1.1.1.1) 0.247 ms 0.166 ms 0.145 ms
2 L103.LSANCA-DSL-29.verizon-gni.net (71.110.63.1) 1.008 ms 0.814 ms 0.862 ms
3 G0-3-1-7.LSANCA-LCR-21.verizon-gni.net (130.81.140.224) 5.608 ms 5.413 ms 5.338 ms
4 so-4-1-0-0.LAX01-BB-RTR1.verizon-gni.net (130.81.151.246) 5.535 ms 5.540 ms 5.493 ms
5 0.ae1.BR3.LAX15.ALTER.NET (152.63.2.129) 5.950 ms 5.933 ms 5.857 ms
6 4.68.63.245 (4.68.63.245) 36.847 ms 8.010 ms 5.879 ms
7 ae-11-60.car1.LosAngeles1.Level3.net (4.69.144.3) 5.346 ms * *
8 4.2.2.5 (4.2.2.5) 5.478 ms 5.342 ms 5.478 ms

frnkblk

Even though this thread is labeled "DNS issues", there's been nothing shared that actually documents a DNS issue. Everything has pointed to something in Verizon's network that's mishandling packets, packets on the return to the customer.

houkou_onchi: Yes, while your traceroutes show differnet paths depending on the destination IP address, that doesn't matter so much because they all complete. So you haven't actually demonstrated an issue with DNS at all. 😃

If you can share a traceroute with us of sites that fail, that may be helpful, but the solution seems to be, in pretty well every case, getting another IP address that's not affected by whatever issue that Verizon has with their network gear.

PJL

I am seeing issues getting web responses only from certain sites (e.g. msnbc.com, barnsandnoble.com, abcnews.go.com, foxnews.com, etc.) when I use the Verizon's default DNS settings in Windows (obtain DNS server address automatically). (Other sites like dslreports.com and cnn.com are fine.) When I switch from the obtain DNS server address automatically setting (which gave me the 192.168.1.1 primary and 68.238.64.12) to the another DNS server -- Level 3 (4.2.2.2 and 4.2.2.3) - all sites are near-instantenous, including those that I could not get to before. Then I manually tried 68.238.64.12 (the Verizon server added for automatic settings) with no secondary and it failed -- and the Windows diagnostics said I had a DNS issue. I then tried 192.168.1.1 and got the same results. I then went back to the Level 3 DNS and all works perfectly. My router has been factory reset (twice).

My neighbor had the same issue. Verizon released and renewed his IP address to give him a new WAN address -- and they broke his caller ID and remote DVR functions on FiOS TV!. His Internet seems "better" though, but this band-aid fix introduces other problems. I've put in a request on DSL Reports for this issue and I will try to get Verizon's attention Monday.

frnkblk

Can you 'dig' the fauly domain names using the bad DNS servers and share the results? If the DNS servers truly don't work for you, we'll see bad/no results.

i.e.

dig msnbc.com @192.168.1.1

dig msnbc.com @68.238.64.12

houkou_onchi

@frnkblk: I myself never had this issue or it has been un-noticed. I just noticed routing changed for me a short while ago as I used to ping lower to odd ips now its even again. DNS has some weird policy based routing as they seem to do some of their load balancing just based off destination/source IP being even/odd/etc which is pretty weird to me as I have never seen any other network do that.

I was just showing an example with my traceroutes, of course they went through as I am not having issues. I was curious though if someone else who was having issues if it was just affecting odd or just even ips on the /24 they were trying to access.

frnkblk

@houkou_onchi wrote:
<snip>
I was curious though if someone else who was having issues if it was just affecting odd or just even ips on the /24 they were trying to access.

@houkou_onchi: Your theory about odd/even destination IP addresses doesn't seem to hold true, because your traceroutes documented in an earlier post to 4.2.2.2 and 4.2.2.5 both have second-last hops of ae-11-60.car1.LosAngeles1.Level3.net.

Mordeen

Will the lease break/renew IP fix work on a Fios business account that's using a static IP address?

frnkblk

Probably not, as the "new IP" fix requires getting a new/different IP address so that hopefully the return traffic goes over a different (L1/L2) path than the existing path.

Mordeen

Had a feeling. Fabulous. Now I need to see if I can get them to assign me a new static IP address. Thanks for the reply.

mrballcb

@Mordeen wrote:
Had a feeling. Fabulous. Now I need to see if I can get them to assign me a new static IP address. Thanks for the reply.

Are you assigned a single static IP address? Or are you assigned a range of static IP addresses? Does your router actually use dhcp to get that address? Or is it set to a single IP address?

If you're assigned a range of IP addresses, then you are in the perfect position to change it yourself and test how it works for each individual IP address. It also arms you/us with more information about characteristics of the issue. Things like "every odd IP works" or "every even IP works" or "every 4th IP works" would be most illuminating. Some of the participants in this thread have voices that I believe can reach engineering in Verizon, but only if the details and facts are repeatable. Your physical location might also be part of that equation. So far, we've seen mostly San Diego and Inland Empire reporting this, but one of the participants was also in Long Beach.

If not assigned a range, then yeah, you'll have to get someone on the phone to assign you a new IP. What is it now? During the change to a new IP, do any of them not work? And what IP starts working when you get it changed?

For reference, this IP is working for me: 173.58.47.108 (has been since a tech support agent broke the lease 2 weeks ago and I got this IP)

this is the IP that was not working for me: 173.58.43.42

Here is the conversation when I was troubleshooting with a CDN provider when we figured out that I couldn't reach their west coast node but I could reach their Dallas and Virginia nodes:

<cannonball> Working from office in Los Angeles: Savvis -> Cogent (te0-3-0-7.ccr21) -> 38.99.68.170 -> 38.127.199.74

<cannonball> Not working: Verizon -> AlterNet -> Cogent (te0-3-0-5.ccr2) -> 38.99.68.170 -> 38.127.199.74
<crucially> cannonball: huh, you get all the way to 38.127.199.74?
<cannonball> The tracepath does, yes.
<crucially> ok
<crucially> fun
<cannonball> There are some "no reply" sprinkled in the two paths I pasted into the pastebin. Didn't figure they were that important.
<crucially> mtr gets around those usually
<cannonball> openssl s_client -connect flassets.a.ssl.fastly.net:443 hangs for me, no connection made.
<cannonball> Yeah, you're right, mtr does handle displaying those a lot nicer.
<crucially> mtr will also show you multiple paths
<crucially> better
<crucially> cannonball: can you msg me your end point so i can mtr back/tcpdump
<cannonball> Doing the openssl command from work connects and does the cert exchange.
<cannonball> Sure, do you want work (working) or home (non-working) ip?
<crucially> the non working one
<crucially> cannonball: you are west coast right?
<cannonball> 173.58.43.42 (home, Verizon FIOS) Yes, Los Angeles area (Murrieta, CA)
<cannonball> $ host flassets.a.ssl.fastly.net
<cannonball> flassets.a.ssl.fastly.net has address 38.99.68.180
<cannonball> I assume I'm the only person complaining about this, both for James and for Fastly. So I wouldn't rule out a local issue.
<crucially> I get to 0.ae3.SJC01-BB-RTR1.verizon-gni.NET (tracing from one of the west coast nodes)
<crucially> then it stops
<DocWilco> I can mtr all the wai to 173.58.43.42 from home
<crucially> oh great
<cannonball> does anybody know if enabling ipv6 (miredo, the linux implementation of teredo) would have any adverse effects?
<DocWilco> although, with some heave loss
<crucially> cannonball: if you put in your hsots file
<crucially> 38.99.68.181
<crucially> instead of 180
<crucially> and see if it works
<cannonball> doing
<crucially> cannonball: no, ipv6 should not matter
<DocWilco> a15 can reach him
<crucially> DocWilco: cache-s22 can reach him
<crucially> s1 can not
<DocWilco> yeah
<DocWilco> and 0.ae3.SJC01-BB-RTR1.verizon-gni.NET is where it ends
<DocWilco> I'm going to go with Verizon routing problem
<crucially> from 181 I get 0.ge-2-0-0.SJC01-BB-RTR2.verizon-gni.net
<crucially> from 180 I get 7. 0.ae3.SJC01-BB-RTR1.verizon-gni.NET
<cannonball> Testing with curl from commandline, neither 180 or 181 work. Got a couple other IP's I could try forcing?
<crucially> cannonball: cache-d20.hosts.fastly.net
<cannonball> Yeah, I tend to think it's Verizon too. I'm stumped why the tracepath gets all the wya.
<crucially> cannonball: asymmetric routing
<cannonball> cache-d20 works with curl, testing int he browser.
<crucially> the return path is different
<crucially> cannonball: try varnish-v11
<crucially> and varnish-v12
<cannonball> browser worked, trying varnish-v11 and varnish-v12 next.
<cannonball> ...browser worked with cache-d20
<cannonball> ...curl works with varnish-v11
<cannonball> ..and works with varnish-v12
<crucially> ok
<crucially> so it isn't cogent/verizon
<cannonball> So it's a Verizon issue, not you guys. My apologies for bugging you, I'll call them (though I'm just a FIOS customer, I don't have a NOC contact). If I can figure out who the peer is, that would at least help them.
<crucially> cannonball: d20 is in dallas, v\d is in virginia
<crucially> I will give you the reverse tracepaths
<cannonball> I'm sure I'll go through voicemail hell just to get told I need to reboot my wifi gateway 😕 but I gotta pester them.
<cannonball> thank you for your help guys.
<crucially> hang on
<cannonball> am holding, just being polite 🙂
<cannonball> Not enough people say thank you in irc, wanted to make sure I wasn't one of them.
<crucially> cannonball: i want to try a few more things
<cannonball> My typo of "fastfly" also explains why I couldn't find jack about you guys in google. I was beginning to lose faith in the Goo.
<crucially> heheh
<crucially> can you ping 180 and 181
<cannonball> I'm ready to try anything you want.
<crucially> i want to verify we see the requests
<cannonball> pinging 180 right now
<crucially> ok, i see them
<crucially> now one more check
<cannonball> I get responses back from 181.
<cannonball> But I get nothing back from 180.
<crucially> yeah, makes sense
<crucially> though unclear why curl doesn't work to 181
<crucially> ok, both icmp replies leave our netowkr
<DocWilco> friend of mine also on Verizon FIOS can't reach s1 either
<crucially> it is probably a line card that has an error in it for this path
<cannonball> Is .180 the s1 node?
<crucially> yeah
<crucially> 181 is s22
<crucially> you should be able to reach s22
<crucially> if icmp works
<cannonball> I can ping it, mtr shows it being reached, testing with tcptraceroute now.
<DocWilco> nevermind, he could reach them, but cogent routers are very heavily loaded
<cannonball> curl does something weird. It must think it's connected because it displays the download status line, it just never moves.
<crucially> cannonball: what is the curl line you use?
<DocWilco> crucially: see the email I just sent, check out hops 12 and 13, those times are pretty high
<cannonball> curl --verbose https://flassets.a.ssl.fastly.net/stylesheets/base_packaged.css?323c1e4e6e864
<crucially> hmm
<crucially> DocWilco: shouldn't matter, thats just control plane
<crucially> has s1 had a traffic drop
<DocWilco> crucially: the fact that it's responsing slowly means it's heavily loaded though
<DocWilco> not to the point where it's dropping stuff going through, but still
<crucially> DocWilco: on the control plane, not switching plane
<crucially> routing/switching
<crucially> there is no way you can tell if the forward plane is loaded or not
<cannonball> icmp is usually dropped first, so if it's not dropping it, the cpu is probably not near its limits.
<cannonball> but yeah, tells you very little about forwarding.
<cannonball> Nobody has said anything in the Outages mailing list nor NANOG, so I think it's isolated to Verizon, and maybe only my area.
<cannonball> If you're not seeing traffic dips, then I think you guys probably can move on to other, more important things.
<DocWilco> he's getting loss on those routers now
<crucially> DocWilco: using mtr?
<DocWilco> yeah
<crucially> and using tcptrace?
<DocWilco> he said UDP
<crucially> yeah
<crucially> unless we trace back, it is impossible to say where the loss is
<cannonball> As a kick it in the shin test, I can go release and renew the IP on my wifi gateway, trying to get a new IP, and see if the problem persists. But that really doesn't _fix_ anything.
<crucially> cannonball: let us know if it continues and we can start to kick our isps
<crucially> cannonball: if it is the src+dst ip flow that has a error on a line card it might help

I called tech support, got someone who was really clueful and was NOT reading from a script, but was unable to escalate. He said the only thing he could do was break the lease, then I renewed, getting a new IP, and it worked fine. He also sent me a new router (was an old revision Actiontec, Rev A IIRC, and now I have a Rev F IIRC). My IP lease has not changed since I got the new router and replaced the old one.

I posted all of this in the hopes that some of this would be useful to someone in narrowing down the scope of this issue.

...Todd

Scottage

Is this still being worked on? It's definitely still a serious problem for me. This morning I used a bookmark testing extension to find out which of my bookmarked pages were failing to connect in the same way, and then tested each of them myself to make sure it was this same problem, and not a legitimate broken web page or other issue.

Cleared browser cache, cookies, and rebooted the router before starting.
All of the sites listed below will give the error: "The server at [address] is taking too long to respond."
In every case I can not get to it via my Verizon FiOS connection.
I can get to each of these via my phone's non-Verizon internet connection, without problem.
downforeveryoneorjustme.com reports all of the sites below are up and running as well during my tests.

costco.com
albertsons.com
club.nintendo.com
portforward.com
eternalmoonwalk.com
nfggames.com
foodily.com
thinkwithportals.com
runicgames.com
tcrf.net
wiki.teamfortress.com

Previously I was unable to get to steampowered.com, but could get to steamcommunity.com
After getting a new IP address, I'm able to get to steampowered.com but not steamcommunity.com

My wife and I both do at least some of our work from home. By what appears to be pure luck none of the websites we need to access for work have been blocked so far, but if that does happen we would be in some serious trouble. We need this issue to be resolved, and soon. It's already been nearly a month so far, our entire time as a Verizon customer.

frnkblk

@PJL: Can you test Scottage's list?

Ftracy3

I've been having issues for two weeks now. Started with access to Steam forums and/or their store. Reset the router, released the IPaddress and powered everything down for a few hours finally cleared things up. Problem came back later and switched to OpenDNS and I thought that had fixed it.

Now today i can't access speedtest.net or speakeasy.net. Can ping them both. Have flushed dns through ipconfig, rebooted..I may try the IP release, power off router for a few hours again. Verizon, what's going on? I can access these sites from work or through my phone but not through Verizon.

Scottage

Actually, one correction. I was able to get to steamcommunity.com and steampowered.com both as late as yesterday. So the fact that I can suddenly not get to steamcommunity.com today seems to indicate that this problem seems to be getting worse, not better, and is still spreading.

EDIT 1: Also, testing by using several free proxies (tested by way of proxy.org), I'm able to visit these same sites on the same computer/connection that fails when not using the proxy. But of course that's not a valid fix, just a test.

EDIT 2: As for DNS, I've tried both auto-assigned DNS and another server (opendns.com) and the issue remains either way.

Ftracy3

Same problem here now ...steamcommunity.com unavailable (steampowered.com is ok for now). can get there on my phone and at work, not through fios.

Martymcfli

Been having the same issue for almost two days. I just swapped the default Verizon DNS to Google's DNS. Fixed the issues for me.

frnkblk

@Martymcfi:

Which issues? 😉 Your statement makes it sounds like everything is fixed, but you haven't shared which sites weren't working for you.

Can you access these sites, too (they have been noted by some as problematic)

http://store.steampowered.com
http://www.bankofamerica.com
http://www.alaskaair.com
http://www.wpcentral.com
http://www.sdccu.org
http://www.sdccu.net
http://www.mysonicwall.com
http://idp.mrmls.safemls.net/idp/Authn/UserPassword
http://costco.com
http://albertsons.com
http://club.nintendo.com
http://portforward.com
http://eternalmoonwalk.com
http://nfggames.com
http://foodily.com
http://thinkwithportals.com
http://runicgames.com
http://tcrf.net
http://wiki.teamfortress.com

Frank

Scottage

I left my router unplugged overnight. Plugged it back in this morning and got a new IP.

Now it appears I can get to the sites I had trouble with before.

Of course I "fixed" it once before this way and it eventually started to go bad again.

And if it is tied to the IP address as it appears, this means someone else may now have that bad IP.

PJL

@frnkblk wrote:
@PJL: Can you test Scottage's list?

Using Verizon's default DNS and then L3 DNS:

costco.com: good response using either DNS

albertsons.com: NO RESPONSE using either DNS

club.nintendo.com: good response using either DNS

portforward.com: NOT RESPONSE using either DNS
eternalmoonwalk.com: good response using either DNS

nfggames.com: NO RESPONSE using either DNS

foodily.com: good respone using either DNS

thinkwithportals.com: good response using either DNS

runicgames.com: NO REPONSE with Verizon, good response with L4

tcrf.net: good response using either DNS

wiki.teamfortress.com: NO RESPONSE using either DNS

PJL

@Scottage wrote:
I left my router unplugged overnight. Plugged it back in this morning and got a new IP.
Now it appears I can get to the sites I had trouble with before.
Of course I "fixed" it once before this way and it eventually started to go bad again.
And if it is tied to the IP address as it appears, this means someone else may now have that bad IP.

It's tied to a Verizon router cache (not the one in your house) for the IP, which in theory would be cleared when someone else was assigned the IP. Yours if "fixed" because the new IP got a new cache on the Verizon router. It may again become corrupt. Did your FiOS TV Remote DVR and Caller ID break when you were assigned the new IP?

JVMarron

Just got off the phone with Verizon FIOS support. Absolutely no help. He tried to break/renew my IP and that did not help and then he had me use command prompt and trace the route of one of the sites I am having problems with (zune.net) and then told me, it isn't verizon's issue because it fails on the 11th hop and not within the first 6. I told him that I was able to reach this site with my phone when I turned off the wifi and used 3G and also when I used a proxy server on this pc. He didn't have an answer. So has anyone actually found a fix? I tried using google DNS with no success. I can't sync my music player, so I'm not happy right now. Any suggestions would be greatly appreciated

BenM1

I've been experiencing similar problems in Santa Monica. I originally noticed the problem with techmeme.com, speedtest.net, and store.steampowered.com being inaccessible over my FiOS connection unless I was connected to my work VPN.

Changing DNS servers makes no difference in my case. Leaving DNS out of the picture, directly pinging an affected IP (such as techmeme's 173.192.129.24, which does happen to be returned by both Verizon's and Google's DNS servers) shows 100% packet loss. This is compared to 0% packet loss for pings to the same IP when on the VPN.

I just checked the other sites mentioned in this thread:

costco.com (down) 

albertsons.com (down)

 club.nintendo.com (up)

 portforward.com (up)

 eternalmoonwalk.com (down)

 nfggames.com (down)

 foodily.com (down)

 thinkwithportals.com (up)

 runicgames.com (up) 

tcrf.net (up) 

wiki.teamfortress.com (down)

steamcommunity.com (up)
steampowered.com (down) 
store.steampowered.com (down)
www.bankofamerica.com (up)
www.alaskaair.com (up)
www.wpcentral.com (up)
www.sdccu.org (up)
www.sdccu.net (up)
www.mysonicwall.com (up)
idp.mrmls.safemls.net/idp/Authn/UserPassword (up)

techmeme.com (down)
speedtest.net (down)

Ftracy3

I can ping steamcommunity.com but can't get in--I cannnot ping speedtest.net, timed out, 100% loss. Can get to both from my phone and through proxy sites. This is getting old... Released my IP and rebooted the router after a few hours..that worked last week but not this morning.

frnkblk

@PJL wrote:
@Scottage wrote:
I left my router unplugged overnight. Plugged it back in this morning and got a new IP.
Now it appears I can get to the sites I had trouble with before.
Of course I "fixed" it once before this way and it eventually started to go bad again.
And if it is tied to the IP address as it appears, this means someone else may now have that bad IP.
It's tied to a Verizon router cache (not the one in your house) for the IP, which in theory would be cleared when someone else was assigned the IP. Yours if "fixed" because the new IP got a new cache on the Verizon router. It may again become corrupt. Did your FiOS TV Remote DVR and Caller ID break when you were assigned the new IP?

Do you mean the routing tables on Verizon(s)' routers? While rare, it's possible that there could be a bug such that traffic doesn't flow the way that it should, based on the routing tables, or that there is corruption on the routing table. We had a customer several years back that couldn't get to Walgreens, while our other customers could. We eventually narrowed it down to any IP address in their /24. After reaching out the network admins at Walgreens, it turned out that there was a software bug on one or more their routers that led to some kind of corruption that could be manually cleared, but re-occured. I can't recall if it was a configuration change or a software upgrade, but they did get the issue resolved.

Not saying that anything of what I described in the above paragraph applies to Verizon's FiOS network in SoCal -- they haven't shared much detail with their customers and the issue is clearly not resolved.

frnkblk

@BenM1 wrote:
I've been experiencing similar problems in Santa Monica. I originally noticed the problem with techmeme.com, speedtest.net, and store.steampowered.com being inaccessible over my FiOS connection unless I was connected to my work VPN.
Changing DNS servers makes no difference in my case. Leaving DNS out of the picture, directly pinging an affected IP (such as techmeme's 173.192.129.24, which does happen to be returned by both Verizon's and Google's DNS servers) shows 100% packet loss. This is compared to 0% packet loss for pings to the same IP when on the VPN.
I just checked the other sites mentioned in this thread:

@BenM1: When you checked each of those sites, which DNS server(s) did you use, and what were the returned A values for each of those domains? The ability to ping a site from the VPN and not from FiOS is not necessarily indicative that there's an issue with HTTP access, as ICMP could be blocked at one or more hops from certain directions/routes and those blocking hops aren't in the other path.

Perhaps for those domains that have their content hosted on multiple A records (either round-robin, SRV, or geo-based DNS) we need to build a hosts file that can be used by SoCal FiOS subscribers to get around those IPs that don't work.

PJL

@frnkblk wrote:
@PJL wrote:
@Scottage wrote:
I left my router unplugged overnight. Plugged it back in this morning and got a new IP.
Now it appears I can get to the sites I had trouble with before.
Of course I "fixed" it once before this way and it eventually started to go bad again.
And if it is tied to the IP address as it appears, this means someone else may now have that bad IP.
It's tied to a Verizon router cache (not the one in your house) for the IP, which in theory would be cleared when someone else was assigned the IP. Yours if "fixed" because the new IP got a new cache on the Verizon router. It may again become corrupt. Did your FiOS TV Remote DVR and Caller ID break when you were assigned the new IP?

Do you mean the routing tables on Verizon(s)' routers? While rare, it's possible that there could be a bug such that traffic doesn't flow the way that it should, based on the routing tables, or that there is corruption on the routing table. We had a customer several years back that couldn't get to Walgreens, while our other customers could. We eventually narrowed it down to any IP address in their /24. After reaching out the network admins at Walgreens, it turned out that there was a software bug on one or more their routers that led to some kind of corruption that could be manually cleared, but re-occured. I can't recall if it was a configuration change or a software upgrade, but they did get the issue resolved.
Not saying that anything of what I described in the above paragraph applies to Verizon's FiOS network in SoCal -- they haven't shared much detail with their customers and the issue is clearly not resolved.

Yes I mean the routing tables on Verizon's routers. I think what you describe is what we're seeing here is Southern California and what was told to me when a Level 2 network admin spoke with me about my problem. And he said it wasn't just isolated to Southern California, but the corrupt routing tables were occuring elsewhere on the Verizon network. Verizon, can you hear us now?

DiAmore

Okay after grabbing a new IP 71.103.142.* .... I can now access XBL, and steam store works.

store.steam.com (UP)

steampowered.com (down)

steamcommunity.com (down)

techmeme.com (down)

alaskaair.com (down)

speedtest.net (UP)

albertsons.com (UP)

costco.com (UP)

runicgames.com (down)

ChristopheJay

Just another Southern California (Long Beach, specifically) experience the same problem as many people on this forum. I have experienced the issues since Friday 2/3/12 evening.

I called tech support once on Saturday morning and again tonight with little help. Tonight I let the representative know about all the commenter's on the forums but is still unaware of the issue.

I have had problems with facebook (fixed after my second tech support call and router reset.)

I also I had a problem with www.questionpro.com (loading really slow), Alaska airlines (unable to access) xbox live forums and Steam not connections. However after renewing my ip address these issues are fixed.

However I am still unable to connect to Xbox Live.

ChristopheJay

It turns out my video on demand was not working either...I just gave Verizon another call, and got a really helpful representative.

He did something that was a reset of all services. This fixed my issues with video on my demand. Also I can access websites like alaska airlines which I was not able to do before.

The representative also let me know that Xbox Live pushed out a update that changed some port configurations and the Verizon techs are working on making sure their network is compitable. I am still not able to connect to Xbox Live, however Xbox Live my test is now passing the internet section, but failing on the Xbox Live section.

If your having problems, ask your Verizon Tech Support for a remote reset on all home services.

MrShoop

I'm in SoCal, having the same problem! Haven't been able to get to store.steampowered.com since Friday. Facebook has been intermittently up and down. Just tried costco.com and club.nintendo.com and neither are reachable.

From my home w/FIOS:

ping store.steampowered.com
PING store.steampowered.com (63.228.223.104): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Request timeout for icmp_seq 4
Request timeout for icmp_seq 5
^C
--- store.steampowered.com ping statistics ---
7 packets transmitted, 0 packets received, 100.0% packet loss

From a remote pc:

ping store.steampowered.com
PING store.steampowered.com (63.228.223.104) 56(84) bytes of data.
64 bytes from store.steampowered.com (63.228.223.104): icmp_seq=1 ttl=56 time=82.2 ms
64 bytes from store.steampowered.com (63.228.223.104): icmp_seq=2 ttl=56 time=82.1 ms
64 bytes from store.steampowered.com (63.228.223.104): icmp_seq=3 ttl=56 time=82.1 ms
64 bytes from store.steampowered.com (63.228.223.104): icmp_seq=4 ttl=56 time=82.2 ms
64 bytes from store.steampowered.com (63.228.223.104): icmp_seq=5 ttl=56 time=82.2 ms
64 bytes from store.steampowered.com (63.228.223.104): icmp_seq=6 ttl=56 time=82.1 ms
^C
--- store.steampowered.com ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5018ms
rtt min/avg/max/mdev = 82.149/82.217/82.296/0.336 ms

mrballcb

@MrShoop wrote:
I'm in SoCal, having the same problem! Haven't been able to get to store.steampowered.com since Friday. Facebook has been intermittently up and down. Just tried costco.com and club.nintendo.com and neither are reachable.

What's your IP right now?

Regards... Todd

Ftracy3

Relased my IP (again) through the router, shut everything off overnight..Now several sites that I couldn't get yesterday (costco, steam community, alaska air) are working, but speedtest.net (which finally was working yesterday) is not. What you can access seems to be a real crapshoot depending on your IP address or time of day as all of us have common trouble sites but they seem to vary on being either up or down for individuals.

Porch1

http://speedtest.net and http://tandemcal.com have stopped working for anyone on Verizons network over the weekend. Fios or DSL. Even on DSLExtreme.

frnkblk

@Porch wrote:
http://speedtest.net and http://tandemcal.com have stopped working for anyone on Verizons network over the weekend. Fios or DSL. Even on DSLExtreme.

How have you confirmed it doesn't work for anyone on Verizon's network?

ChristopheJay

If we can please begin posting your trace routes (removing personal modem info) on sites you can not access, this will help the Verizon Reps.

For Tandemcal.com

traceroute: Warning: http://tandemcal.com has multiple addresses; using 63.251.179.13

traceroute to http://tandemcal.com (63.251.179.13), 64 hops max, 72 byte packets

[after passing through my local modem]

5 0.so-1-3-0.xl3.lax15.alter.net (152.63.114.145) 5.645 ms 6.214 ms 5.774 ms

6 0.ge-3-3-0.xt1.den4.alter.net (152.63.1.210) 41.231 ms 36.569 ms 36.528 ms

7 pos6-0-0.gw10.den4.alter.net (152.63.89.217) 33.805 ms 33.986 ms 33.599 ms

8 internap-gw.customer.alter.net (152.179.104.90) 46.391 ms 46.678 ms 46.378 ms

9 mpr2.den.ve3-bbnet2.pnap.net (216.52.40.72) 46.139 ms 46.361 ms 45.984 ms

10 allmar-4.mpr2.den.pnap.net (66.151.163.98) 48.958 ms 53.331 ms 62.225 ms

11 * * *

12 * * *

bwren

Just put in a support call, and the tech had no idea about this. I got the typical restart the router, delete your browsing history, etc. He said that no outage has been declared. He did open a ticket for me so hopefully something will come of that. I suggested that they go to this forum to see all the people reporting the problem.

Oh, the first person I got on the phone told me that they determined it was a Microsoft problem. I admit that I was not very polite in my response to that.

Here are a couple of tracerts for sites I can't get to.

my.msn.com

2 13 ms 4 ms 4 ms L100.LSANCA-VFTTP-128.verizon-gni.net [96.229.59.1]
3 7 ms 7 ms 6 ms P9-1.PITBPA-LCR-03.verizon-gni.net [130.81.55.176]
4 39 ms 7 ms 6 ms so-6-0-0-0.LAX01-BB-RTR1.verizon-gni.net [130.81.29.124]
5 8 ms 7 ms 7 ms 0.so-6-2-0.XT1.LAX7.ALTER.NET [152.63.112.45]
6 7 ms 7 ms 7 ms 0.ge-2-0-0.XL3.LAX15.ALTER.NET [152.63.2.66]
7 8 ms 8 ms 7 ms TenGigE0-4-1-0.GW4.LAX15.ALTER.NET [152.63.115.178]
8 8 ms 7 ms 7 ms microsoft.customer.alter.net [152.179.21.26]
9 9 ms 9 ms 9 ms xe-0-0-0-0.lax-96cbe-1a.ntwk.msn.net [207.46.47.4]
10 20 ms 21 ms 20 ms xe-3-0-2-0.bay-16c-1a.ntwk.msn.net [207.46.46.242]

11 * * *

www.speedtest.net

2 6 ms 5 ms 6 ms L100.LSANCA-VFTTP-128.verizon-gni.net [96.229.59.1]
3 7 ms 6 ms 5 ms P9-1.PITBPA-LCR-03.verizon-gni.net [130.81.55.176]
4 6 ms 6 ms 7 ms so-6-0-0-0.LAX01-BB-RTR1.verizon-gni.net [130.81.29.124]
5 6 ms 7 ms 12 ms 0.ae1.BR3.LAX15.ALTER.NET [152.63.2.129]

6 9 ms 10 ms 7 ms 204.255.168.126
7 36 ms 35 ms 36 ms netriver.ge2-16.br01.sea01.pccwbtn.net [63.216.14.162]
8 * * *

Porch1

On Verizon FIOS:
fios@firewall:~$ tracepath tandemcal.com
1: static-71-189-XXX-XXX.lsanca.fios.verizon.net (71.189.XXX.XXX) 0.278ms pmtu 1500
1: L101.LSANCA-VFTTP-14.verizon-gni.net (173.51.37.1) 7.400ms asymm 2
1: L101.LSANCA-VFTTP-14.verizon-gni.net (173.51.37.1) 7.423ms asymm 2
2: G0-3-4-2.LSANCA-LCR-22.verizon-gni.net (130.81.54.218) 7.448ms
3: so-4-1-0-0.LAX01-BB-RTR2.verizon-gni.net (130.81.151.248) 7.497ms
4: 0.ae2.BR3.LAX15.ALTER.NET (152.63.2.133) 7.346ms asymm 3
5: 4.68.63.245 (4.68.63.245) 9.921ms asymm 3
6: vlan80.csw3.LosAngeles1.Level3.net (4.69.144.190) 9.988ms asymm 3
7: ae-92-92.ebr2.LosAngeles1.Level3.net (4.69.137.29) 9.955ms asymm 3
8: ae-6-6.ebr2.SanJose5.Level3.net (4.69.148.202) 22.381ms asymm 4
9: ae-5-5.ebr4.SanJose1.Level3.net (4.69.148.142) 32.493ms asymm 4
10: ae-34-34.ebr2.SanJose1.Level3.net (4.69.153.33) 19.933ms asymm 4
11: ae-7-7.ebr1.Seattle1.Level3.net (4.69.132.50) 37.452ms asymm 8
12: no reply
13: no reply
14: 19pe1-19b1-vlan259.sea.fibercloud.net (216.145.7.126) 36.155ms asymm 10
15: 18ce1-19pe1-vlan257.sea.fibercloud.net (216.145.7.17) 39.936ms asymm 11
16: no reply
17: no reply
18: no reply
19: no reply
^C
fios@firewall:~$ wget tandemcal.com
--2012-02-06 10:30:58-- http://tandemcal.com/
Resolving tandemcal.com... 216.145.24.51
Connecting to tandemcal.com|216.145.24.51|:80... failed: Connection timed out.

On another FIOS line.
root@filter:~# tracepath tandemcal.com
1: static-173-55-XXX-XXX.lsanca.fios.verizon.net (173.55.XXX.XXX) 0.359ms pmtu 1500
1: l300.lsanca-vfttp-131.verizon-gni.net (173.55.22.1) asymm 2 1.690ms
2: g5-0-6-1131.lsanca-lcr-11.verizon-gni.net (130.81.138.104) 5.842ms
3: so-4-3-0-0.lax01-bb-rtr1.verizon-gni.net (130.81.29.244) asymm 5 5.908ms
4: 0.ae1.br1.lax15.br1.lax15.alter.net (152.63.112.1) asymm 6 6.286ms
5: 204.255.168.94 (204.255.168.94) asymm 6 7.843ms
6: sl-crs1-ria-0-3-2-0.sprintlink.net (144.232.24.42) asymm 7 8.507ms
7: sl-crs1-stk-0-2-3-0.sprintlink.net (144.232.9.153) asymm 8 18.200ms
8: sl-crs1-tac-0-9-0-0.sprintlink.net (144.232.24.249) asymm 9 37.272ms
9: no reply
10: sl-whidbeynet-6-0.sprintlink.net (144.228.93.42) 39.384ms
11: 216.145.7.114 (216.145.7.114) 39.737ms
12: no reply
13: no reply
14: no reply
15: no reply
16: no reply

root@filter:~# wget tandemcal.com
--2012-02-06 10:38:35-- http://tandemcal.com/
Resolving tandemcal.com... 216.145.24.51
Connecting to tandemcal.com|216.145.24.51|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `index.html.4'

[ <=> ] 22,426 --.-K/s in 0.1s

2012-02-06 10:38:35 (205 KB/s) - `index.html.4' saved [22426]

I can get to the tamdemcal homepage, but not to the clients calander.

root@filter:~# tracepath bcs.bcscalendar.tandemcal.com
1: static-173-55-XXXX-XXXX.lsanca.fios.verizon.net (173.55.XXXX.XXXX) 0.338ms pmtu 1500
1: l300.lsanca-vfttp-131.verizon-gni.net (173.55.22.1) asymm 2 1.623ms
2: g5-0-6-1131.lsanca-lcr-11.verizon-gni.net (130.81.138.104) 5.563ms
3: so-4-3-0-0.lax01-bb-rtr1.verizon-gni.net (130.81.29.244) asymm 5 5.886ms
4: 0.ae1.br3.lax15.alter.net (152.63.2.129) asymm 6 59.616ms
5: 4.68.63.245 (4.68.63.245) asymm 7 19.714ms
6: vlan60.csw1.losangeles1.level3.net (4.69.144.62) asymm 7 6.293ms
7: ae-92-92.ebr2.losangeles1.level3.net (4.69.137.29) asymm 8 7.640ms
8: ae-6-6.ebr2.sanjose5.level3.net (4.69.148.202) asymm 10 26.118ms
9: ae-5-5.ebr4.sanjose1.level3.net (4.69.148.142) 25.315ms
10: ae-34-34.ebr2.sanjose1.level3.net (4.69.153.33) asymm 8 20.833ms
11: no reply
12: no reply
13: no reply
14: no reply
15: no reply

root@filter:~# wget bcs.bcscalendar.tandemcal.com
--2012-02-06 10:40:21-- http://bcs.bcscalendar.tandemcal.com/
Resolving bcs.bcscalendar.tandemcal.com... 216.145.24.55
Connecting to bcs.bcscalendar.tandemcal.com|216.145.24.55|:80... (hangs forever here)

Porch1

----------------------------------------------------------------
On RR Cable

user@firewall:~$ tracepath tandemcal.com
1: cpe-24-30-XXX-XXX.dc.res.rr.com (24.30.XXX.XXX) 0.296ms pmtu 1500
1: 10.52.64.1 (10.52.64.1) 7.621ms
1: 10.52.64.1 (10.52.64.1) 7.719ms
2: gi2-0-2-bangca1-gsr1.dc.rr.com (66.74.100.20) 8.458ms
3: 66.75.155.71 (66.75.155.71) 14.043ms
4: 107.14.19.32 (107.14.19.32) 10.777ms
5: 107.14.19.138 (107.14.19.138) 12.106ms
6: xe-7-2-0.edge1.LosAngeles6.Level3.net (4.26.0.21) 12.447ms asymm 7
7: vlan70.csw2.LosAngeles1.Level3.net (4.69.144.126) 13.214ms
8: ae-82-82.ebr2.LosAngeles1.Level3.net (4.69.137.25) 12.138ms
9: ae-6-6.ebr2.SanJose5.Level3.net (4.69.148.202) 20.726ms asymm 11
10: ae-5-5.ebr4.SanJose1.Level3.net (4.69.148.142) 26.982ms
11: ae-34-34.ebr2.SanJose1.Level3.net (4.69.153.33) 22.575ms asymm 9
12: ae-7-7.ebr1.Seattle1.Level3.net (4.69.132.50) 41.729ms asymm 10
13: 4.68.105.13 (4.68.105.13) 39.288ms asymm 11
14: FIBERCLOUD.edge2.Seattle3.Level3.net (4.59.234.30) 38.794ms asymm 12
15: 19pe1-19b1-vlan259.sea.fibercloud.net (216.145.7.126) 37.992ms asymm 14
16: 18ce1-19pe1-vlan257.sea.fibercloud.net (216.145.7.17) 39.298ms asymm 15
17: no reply
18: no reply
19: no reply
20: no reply
21: no reply
22: no reply
^C
user@firewall:~$ wget tandemcal.com
--2012-02-06 10:31:42-- http://tandemcal.com/
Resolving tandemcal.com... 216.145.24.51
Connecting to tandemcal.com|216.145.24.51|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `index.html.3'

[ <=> ] 22,426 133K/s in 0.2s

2012-02-06 10:31:43 (133 KB/s) - `index.html.3' saved [22426]

Porch1

This is on my DSL Extreme line, but it uses pppoe to connect to Verizon.

I run my own DNS server and it can't connect to tandemcal.com DNS servers, so here it is by IP.

root@ppoe:~# traceroute 216.145.24.51
traceroute to 216.145.24.51 (216.145.24.51), 30 hops max, 38 byte packets
1 10.35.110.1 (10.35.110.1) 22.500 ms 22.419 ms 23.346 ms
2 P13-1.LSANCA-LCR-01.verizon-gni.net (130.81.35.188) 25.475 ms 25.225 ms 25.445 ms
3 so-7-3-0-0.LAX01-BB-RTR1.verizon-gni.net (130.81.28.217) 25.825 ms 25.638 ms 25.990 ms
4 0.ae1.BR3.LAX15.ALTER.NET (152.63.2.129) 25.289 ms 25.949 ms 25.498 ms
5 4.68.63.245 (4.68.63.245) 25.104 ms 25.489 ms 25.512 ms
6 vlan70.csw2.LosAngeles1.Level3.net (4.69.144.126) 25.804 ms 26.510 ms 25.958 ms
7 ae-92-92.ebr2.LosAngeles1.Level3.net (4.69.137.29) 25.625 ms 26.497 ms ae-62-62.ebr2.LosAngeles1.Level3.net (4.69.137.17) 26.738 ms
8 ae-6-6.ebr2.SanJose5.Level3.net (4.69.148.202) 41.951 ms 36.168 ms 42.740 ms
9 ae-5-5.ebr4.SanJose1.Level3.net (4.69.148.142) 37.404 ms 40.907 ms 37.672 ms
10 ae-34-34.ebr2.SanJose1.Level3.net (4.69.153.33) 37.887 ms 37.707 ms 37.364 ms
11 ae-7-7.ebr1.Seattle1.Level3.net (4.69.132.50) 57.376 ms 63.041 ms 55.175 ms
12 * * *
13 FIBERCLOUD.edge2.Seattle3.Level3.net (4.59.234.30) 55.814 ms 55.539 ms 55.800 ms
14 * * *
15 18ce1-19pe1-vlan257.sea.fibercloud.net (216.145.7.17) 56.400 ms 56.486 ms 56.301 ms
16 * * *
17 *
root@ppoe:~# wget 216.145.24.51
--06:11:16-- http://216.145.24.51/
=> `index.html.1'
Connecting to 216.145.24.51:80... (hangs forever)

Porch1

I am getting all kinds of reports of tandemcal.com not working on Verizon, but it works fine on ATT, T-Mobile, and Comcast cable.

Porch1

root@filter:~# tracepath speedtest.net
1: static-173-55-XXX-XXX.lsanca.fios.verizon.net (173.55.XXX.XXX) 0.441ms pmtu 1500
1: l300.lsanca-vfttp-131.verizon-gni.net (173.55.22.1) asymm 2 1.600ms
2: g5-0-6-1231.lsanca-lcr-12.verizon-gni.net (130.81.140.22) 5.447ms
3: so-4-0-3-0.lax01-bb-rtr2.verizon-gni.net (130.81.29.246) asymm 5 6.968ms
4: 0.xe-11-1-0.BR1.LAX15.ALTER.NET (152.63.112.5) asymm 6 5.846ms
5: ae-7.r05.lsanca03.us.bb.gin.ntt.net (129.250.8.85) asymm 6 7.564ms
6: ae-2.r20.lsanca03.us.bb.gin.ntt.net (129.250.2.229) asymm 7 6.566ms
7: xe-1-1-0.r21.mlpsca01.us.bb.gin.ntt.net (129.250.3.28) asymm 9 15.487ms
8: ae-0.r20.snjsca04.us.bb.gin.ntt.net (129.250.2.96) asymm 7 19.262ms
9: xe-7-1-2.r20.snjsca04.us.bb.gin.ntt.net (129.250.4.39) asymm 7 16.401ms
10: ae-1.r05.sttlwa01.us.bb.gin.ntt.net (129.250.5.47) asymm 9 35.623ms
11: ae-1.r05.sttlwa01.us.bb.gin.ntt.net (129.250.5.47) asymm 9 34.421ms
12: no reply
13: no reply
14: no reply
15: no reply

root@filter:~# wget speedtest.net
--2012-02-06 11:02:46-- http://speedtest.net/
Resolving speedtest.net... 74.209.160.12
Connecting to speedtest.net|74.209.160.12|:80... (hangs)

root@pppoe:~# traceroute speedtest.net
traceroute to speedtest.net (74.209.160.12), 30 hops max, 38 byte packets
1 10.35.110.1 (10.35.110.1) 22.329 ms 22.238 ms 22.289 ms
2 P4-0.LSANCA-LCR-02.verizon-gni.net (130.81.193.98) 23.333 ms 23.378 ms 23.268 ms
3 so-7-1-3-0.LAX01-BB-RTR2.verizon-gni.net (130.81.28.221) 24.063 ms 35.416 ms 94.767 ms
4 0.ae2.BR3.LAX15.ALTER.NET (152.63.2.133) 24.834 ms 24.702 ms 24.768 ms
5 204.255.168.126 (204.255.168.126) 26.397 ms 24.774 ms 24.170 ms
6 netriver.ge2-16.br01.sea01.pccwbtn.net (63.216.14.162) 56.130 ms 56.656 ms 55.333 ms
7 * * *
8 * * *
9 * * *
10 * *
root@pppoe:~# wget speedtest.net
--06:18:03-- http://speedtest.net/
=> `index.html.1'
Resolving speedtest.net... 74.209.160.12
Connecting to speedtest.net|74.209.160.12|:80... (hangs)

user@firewall:~$ tracepath speedtest.net
1: cpe-24-30-XXX-XXXX.dc.res.rr.com (24.30.XXX.XXX) 0.318ms pmtu 1500
1: 10.52.64.1 (10.52.64.1) 7.033ms
1: 10.52.64.1 (10.52.64.1) 8.312ms
2: gi2-0-2-bangca1-gsr1.dc.rr.com (66.74.100.20) 8.448ms
3: 66.75.155.71 (66.75.155.71) 18.456ms
4: 107.14.19.32 (107.14.19.32) 25.296ms
5: 66.109.9.24 (66.109.9.24) 12.558ms
6: 66.109.9.50 (66.109.9.50) 14.368ms asymm 7
7: netriver.ge2-16.br01.sea01.pccwbtn.net (63.216.14.162) 37.141ms asymm 12
8: static-74-209-160-12.lynnwood.netriver.net (74.209.160.12) 38.339ms reached
Resume: pmtu 1500 hops 8 back 51

user@firewall:~$ wget speedtest.net
--2012-02-06 11:06:04-- http://speedtest.net/
Resolving speedtest.net... 74.209.160.12
Connecting to speedtest.net|74.209.160.12|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `index.html.4'

[ <=> ] 14,238 --.-K/s in 0.1s

2012-02-06 11:06:05 (123 KB/s) - `index.html.4' saved [14238]

MrShoop

@mrballcb wrote:
@MrShoop wrote:
I'm in SoCal, having the same problem! Haven't been able to get to store.steampowered.com since Friday. Facebook has been intermittently up and down. Just tried costco.com and club.nintendo.com and neither are reachable.

What's your IP right now?

Regards... Todd

108.23.119.* {edited for privacy}

and still having the connection issues. I was hoping once all the superbowl parties were ove it would get fixed. 🙂

MrShoop

All the sites that were unreachable for me are now working - yay!

ChristopheJay

So after a long call with Verizon Representatives (with some people higher up than your average tech support) it looks like the probelm has been fixed for me. Without Release/Renew ip address.

If you were have problems with certain sites can you go back and try now, and report back.

Fingers crossed everything is working!

DiAmore

Yeah it literally started working later this morning, when I woke up I tried alaskaair.com (previously not working) still didn't work. Around 12 I tried again and it loaded, along with any other sites that were previously down for me.

bwren

Everything working fine for me as well. Would be nice if we could get some acknolwedgement from Verizon that there actually was an issue and some confidence that they consider a three day outage unacceptable. Glad this over the weekend or I would have been working out of the local coffee house all week.

frnkblk

@bwren wrote:
Everything working fine for me as well. Would be nice if we could get some acknolwedgement from Verizon that there actually was an issue and some confidence that they consider a three day outage unacceptable. Glad this over the weekend or I would have been working out of the local coffee house all week.

Three days? This thread was started on January 12, and one or more of the steampowered.com sites has been an issue for most of those experiencing access problems.

mrballcb

@bwren wrote:
Everything working fine for me as well. Would be nice if we could get some acknolwedgement from Verizon that there actually was an issue and some confidence that they consider a three day outage unacceptable. Glad this over the weekend or I would have been working out of the local coffee house all week.

Three days? Most of the people here have been unable to reliably log in to their banking or MLS websites for almost 4 weeks now. It hit me a little under 4 weeks ago but has been clear since then. The issue is, was, and always will be with large companies that there was no way for the people who could reliably produce the troubleshooting info to be connected to the people who could diagnose and fix it. And until enough people started making a stink about it, VZ wouldn't do anything to fix it because it's just not in the best interests of their bottom line to fix it unless more than X% of customers are complaining.

BenM1

@bwren wrote:
Everything working fine for me as well. Would be nice if we could get some acknolwedgement from Verizon that there actually was an issue and some confidence that they consider a three day outage unacceptable. Glad this over the weekend or I would have been working out of the local coffee house all week.

Same here. Back to normal as of Monday afternoon.

Child Item