DNS issues in SoCal
shayne571
Newbie

Since about Monday I'm unable to access various sites. I have confirmed they are up because I can access them from a nearby Starbucks and a neighbor's connection, neither of which are on FIOS.

This is one site I can't access that I confirmed is online:

http://store.steampowered.com/

When I tried to access thiis site using IE 9 (from my Windows 7 PC), I get this message:

Internet Explorer cannot display the webpage

When I click on the Diagnose Connection Problems button, I get this:

website is online but isn't responding to connection attempts

This is happening on multiple sites.

0 Likes
187 Replies
shayne571
Newbie

This is a screenshot for Steam.

image

0 Likes
topdog
Specialist - Level 2

We are able to access the sites mentioned in Baltimore area. Hope this helps. It is a server problem.

0 Likes
frnkblk
Enthusiast - Level 3

@topdogGlad you can access the sites mentioned from Baltitmore.  The issue appears to affect Verizon FiOS customers in SoCal, likely due to some networking equipment in that region.  There's no indication that the issue is a server problem.

0 Likes
mrballcb
Enthusiast - Level 3

@topdog wrote:

We are able to access the sites mentioned in Baltimore area. Hope this helps. It is a server problem.


There has been a lot of discussion about various things here, so I want to provide some clarification:

All of the affected parties are SoCal Verizon FIOS users, both consumer and business grade, who are accessing other sites in SoCal.  San Diego Credit Union (and various other banks), an MLS (real estate) website, Verizon Forums itself, an adult oriented site that uses a CDN that has a node in L.A. serving images and css, etc.  This is a regional issue.  And the issue as far as I was able to determine based on *end-to-end* testing is that the packets left my house and got to the server(s), the server sent it's reply, the reply left the edge of the provider's network, but the packet never made it back to me.

Sometimes the issue can be somewhat negated by using (for example) Google's DNS, because the IP that Google's DNS returns for a geo-located host might be different than the (local to SoCal) Verizon DNS servers.  If that IP is local to, for example, Northern California, then the routing to and from that site works just fine.  But depending on the site and its infrastructure, if the IP that it ultimately routes to is still the same host in SoCal, no connection can be initiated.  It just hangs.

I hope this helps to better define the scope of what we're seeing and fighting.

...Todd

0 Likes
PJL
Master - Level 3

@mrballcb wrote:

@topdog wrote:

We are able to access the sites mentioned in Baltimore area. Hope this helps. It is a server problem.


There has been a lot of discussion about various things here, so I want to provide some clarification:

All of the affected parties are SoCal Verizon FIOS users, both consumer and business grade, who are accessing other sites in SoCal.  San Diego Credit Union (and various other banks), an MLS (real estate) website, Verizon Forums itself, an adult oriented site that uses a CDN that has a node in L.A. serving images and css, etc.  This is a regional issue.  And the issue as far as I was able to determine based on *end-to-end* testing is that the packets left my house and got to the server(s), the server sent it's reply, the reply left the edge of the provider's network, but the packet never made it back to me.

Sometimes the issue can be somewhat negated by using (for example) Google's DNS, because the IP that Google's DNS returns for a geo-located host might be different than the (local to SoCal) Verizon DNS servers.  If that IP is local to, for example, Northern California, then the routing to and from that site works just fine.  But depending on the site and its infrastructure, if the IP that it ultimately routes to is still the same host in SoCal, no connection can be initiated.  It just hangs.

I hope this helps to better define the scope of what we're seeing and fighting.

...Todd



I think you have correctly stated things.  I still have an open ticket with Verizon.  Their level 2 tech contacted my sunday and said that they are aware of the problem and that it is likely a routing table issue, and that he was going to have the router cache cleared fo my routes.  He said it might be at the CO, but when I pointed out to him that there were many others in Southern California experiencing the same issues, he acknowledged it as a bigger problem.  They were supposed to resolve this within 24 hours.  It's not going on 48 since he contacted me and I don't believe it's fixed, but I will monitor and report back.

I told him I didn't want a new IP address for my router node -- when they did that for my neighbor, they broke his FiOS TV caller ID and remote DVR.  He said that was no longer their fix (for the things it broke) and that clearing router caches is now the fix. 

0 Likes
GENOS1
Newbie

I live in Redlands and have had the problem not being able to access craigslist.  Had done all the troubleshooting that Microsoft had available.  Yesterday, I was pretty much on the phone all day off and on with Verizon Tech.  Last resort, they transferred me to a pay tech, which was ridiculous.  Not going to pay for tech.  Supposed to be included in the DSL.  I have the slowest one, which workds fine for me.  This morining, I was prepared to cancel the service, but to my surprise, I was able to access craigslist. I'm sure it must have been a Verizon problem.  A few weeks ago, I also had the KB2585542 problem.  Luckily, I was able to fix by restoring system, and not allowing automatic updates.  By elimination, I four the KB2585542 problem. Now, I am afraid to download any updates.

0 Likes
frnkblk
Enthusiast - Level 3

There's another thread in this forum about Craigslist and someone reported today, too, that they could access Craiglist.

Do you know if your public IP address changed within the last 20 days?  Can you access these sites, too (they have been noted by some as problematic)

0 Likes
PJL
Master - Level 3

@frnkblk wrote:

There's another thread in this forum about Craigslist and someone reported today, too, that they could access Craiglist.

Do you know if your public IP address changed within the last 20 days?  Can you access these sites, too (they have been noted by some as problematic)...



My public IP address has not changed in probably a year.  I am now able to access all of these site, and today things appear fairly "normal."  I had a voicemail from Verizon telling me they've worked the issue.  We'll see how it holds out.

0 Likes
frnkblk
Enthusiast - Level 3

That's good news for the Verizon FiOS subs in SoCal if that's the case.  Hopefully we'll learn what the RFF (reason for failure) was.

0 Likes
mrballcb
Enthusiast - Level 3

@GENOS wrote:

I live in Redlands and have had the problem not being able to access craigslist.  Had done all the troubleshooting that Microsoft had available.  Yesterday, I was pretty much on the phone all day off and on with Verizon Tech.


Do you have any issues accessing the San Diego Credit Union bank website login page?  That seems to be a constant problem for people experiencing this issue.  The main page was ok, it was the login page which had the problem.  I honestly do not recall if craigslist was a problem for me on the day I had my issue (2 weeks ago).

It does kind of sound like your issue was related to your MS update and not necessarily the VZ issue we saw.  I suggest you run, if you haven't already, CCleaner and Spybot Search & Destroy to make sure that there isn't anything in your system that might be lingering from an old virus/spyware/adware infestation (if you've ever had any).

...Todd

0 Likes
DavidCRMLS
Enthusiast - Level 2

Hi Techman28,

I can tell you we have over 300 and counting Verizon FIOS and DSL users having the same issue that shayne571

is having.  It is random on the FIOS/DSL users, but the common theme is that they cannot get to major site and ours happens to be one of them, that is why I have been involved.

Resetting the Verizon modem has only worked on half of the users I test ed with.  5 out of 10

What we need is for you to escalate this to 2nd or 3rd tier or someone who can look at the Verizon California DNS servers.

I have a sampling of source IP's from the affected FIOS/DSL  users that I have been testing with that, I can send (don't want to post out in the open)

I am not sure how to get those to you.

There maybe a common issue, maybe in the Verizon network that the affected are all using the same DNS relay or Name servers or something.

Us being on the outside looking in, all we can is give you what we know.

This will not be resolved for the people affect by just reseting the modem, I can tell you that much.  It needs to be escalated.

I can provide more details to the Verizon tech.  Again, not sure how to get you that information.

Please advise.

Thanks.

0 Likes
JamesLaszko
Enthusiast - Level 3

We are seeing this across all of our FIOS connected customers as well in So Cal.  The issues seem more related to proxy or traffic management issues.  HTTP and HTTPS connections are failing to multiple places, where traceroutes work just fine.  Sites that are next door to each other, both on FIOS, are having different results.  Dynamic IP customers, when they receive a new IP lease, start working again.  We are also seeing HORRIBLE performance of VOIP sessions across multiple VOIP providers.  There is definately something wrong and we are not able to get anyone at Verizon to help.

This is very unacceptable.  Where should we be going to get support from someone that has insight within Verizon?

James Laszko

Mythos Technology Inc

jamesl@mythostech.com

JamesLaszko
Enthusiast - Level 3

To follow up, and perhaps trigger someone's change management lookup capabilities, we started to see these issues start occuring on 1/9/12 at approximately 7:30pm.

James

0 Likes
JamesLaszko
Enthusiast - Level 3

Just received this on the NANOG mailing list - could someone at Verizon please get network engineering involved?? :

> To: {edited for privacy}

> Subject: Verizon FIOS MTU issues in Southern California

>

>Can anyone from the Verizon FiOS NOC contact me off-list. We believe we've identified a network issue in the Southern >California FiOS network impacting your residential subscribers.

>Brent Bowers

>Director, CB/Network/Transport Engineering CCIE #13530 Cox Communications, Inc.

>

> {edited for privacy}

TheMick1
Newbie

Thank you for this forum.  I am also in SoCal and when I try to go to Costco.com, I am redirected to Costco.com.mx (Mexico).  This is the only website I have a problem with.  Costco says its Verizon's problem, but after reading this stream of comments, I feel that it's hopeless to contact Verizon.  I am just letting you know that this problem still exists.  Was there ever a "bottom line" to this problem?

0 Likes
CaptainSTX
Contributor - Level 3

When you suspect a DNS server problem there is a simple solution.

Just look up the public IP for the site that you want to connect manually on the Internet using numerous tools and sites and then type the IP number into your browser.    Failing that call the company using a telephone and ask them what IP you should use.  Not perfect and perhaps not convient but it is better than not being able to reach some site.

Just like  the old days when people used to use phone books instead of dialing 411 and then being auto connected.

0 Likes
louandlaurel
Enthusiast - Level 2

I have Verizon DSL (not FIOS) but can't connect to certain websites either- mainly Xbox and Facebook.  I've tried calling Verizon but am told that they are not aware of any issues and there have been no other reports.  Are you kidding me.  By the looks of the forum entries, there have been several reports!

This is typical of Verizon, VERY poor customer service.  I've had enough and am looking for alternatives in my area. 

It took Verizon a FULL YEAR to find out why I wasn't receiving the internet service I paid for.  I guess that should have been a clue.

Signed - an ex-Verizon user!

0 Likes
Ftracy3
Newbie

Tech support just reset my IP Address..Now I can get to speedtest.net but nowhere on Steam. (before they reset it I could get to the store, just not the community)

0 Likes
Ftracy3
Newbie

Just ran through the previouslists after they reset my IP and my findings below...whatever is going on is only partially consistent for all of us. Am on LONG hold with tech support now. Crappy music, btw.

costco.com (down)


albertsons.com (down)


club.nintendo.com (down)


portforward.com (up)


eternalmoonwalk.com (UP)


nfggames.com (down)


foodily.com (UP)


thinkwithportals.com (up)


runicgames.com (DOWN)


tcrf.net (up)


wiki.teamfortress.com (UP)

steamcommunity.com (DOWN)
steampowered.com (down)

store.steampowered.com (down)
www.bankofamerica.com (up)
www.alaskaair.com (DOWN)
www.wpcentral.com (up)
www.sdccu.org (up)
www.sdccu.net (up)
www.mysonicwall.com (NO)
idp.mrmls.safemls.net/idp/Authn/UserPassword (up)

techmeme.com (down)
speedtest.net (UP)

0 Likes
frnkblk
Enthusiast - Level 3

@Ftracy3 wrote:

Just ran through the previouslists after they reset my IP and my findings below...whatever is going on is only partially consistent for all of us. Am on LONG hold with tech support now. Crappy music, btw.

costco.com (down)


albertsons.com (down)


club.nintendo.com (down)


portforward.com (up)


eternalmoonwalk.com (UP)


nfggames.com (down)


foodily.com (UP)


thinkwithportals.com (up)


runicgames.com (DOWN)


tcrf.net (up)


wiki.teamfortress.com (UP)

steamcommunity.com (DOWN)
steampowered.com (down)

store.steampowered.com (down)
www.bankofamerica.com (up)
www.alaskaair.com (DOWN)
www.wpcentral.com (up)
www.sdccu.org (up)
www.sdccu.net (up)
www.mysonicwall.com (NO)
idp.mrmls.safemls.net/idp/Authn/UserPassword (up)

techmeme.com (down)
speedtest.net (UP)


I wonder what the Verizon tech support person will say when you provide a list of 10 websites that you can't visit.  When we first discussing this problem 5+ weeks ago it was because people only had one or two documented sites they couldn't get to and tech support might try to write it off to something to the customer's PC or home network, but now we have aggregated a sizeable list that people can test against that helps demonstrate the scope of this issue.

0 Likes
Ftracy3
Newbie

OK....Tech support just reset me again..still no access to those sites. When I called first question he asked was if i as in California--he said they know there are issues but said they are being told it is Microsoft's fault...something to do with these sites using the Microsoft Network and the network having problems talking to Verizon. Doesn't make a whole lot of sense to me because why primarily SoCal? Anyway he said "they're working on it" but had no idea when it would be resolved.

0 Likes