Packet loss and lag to AWS

[test421]

Ever since I signed up for 70/70 internet I've noticed slow connection speeds specifically to Amazon's cloud services. Speedtest.net shows that everything is fine, but I know that speedtest.net merely tests PART of your internet speed. Is verizon still throttling (aka refusing to pay for upgrades) to amazon web services? If so, why am I paying for 70/70 internet? My sprint phone on 3G has a better connection to my teamspeak server.

[smith6612]

Many of us in the Northeast have been seeing issues with congestion to and from AWS. My DSLReports Line monitor will show this. It appears to be a peering problem between a transit provider used to connect Verizon and Amazon together, and Verizon. In my case it appears to be Qwest, but I need a reverse traceroute to fully determine that.

Here's my DSLReports line monitor as an exmaple.

The top graph shows a monitor from an East Coast EC2 instance. The bottom one is from a West Coast instance. Blue denotes packet loss - in my case this loss in the past day was due to my IP address changing (I was doing some work on my home network). The East coast instance consistently sees latency at night. West Coast EC2 sees less latency.

Just congestion. Verizon needs to look into their peering with others and fix it up. Has nothing to do with FiOS itself, as I'm on DSL.

[mtgdam]

We have services hosted in AWS N. Virginia. Right now, anywhere from 10-25% packet loss and average 120ms rtt consistantly at multiple clients on FiOS Business Connections. On February 19th, the issue was so bad that services were unreachable from 9:30am to 2:15pm EST.

I notice the same where evenings tend to be worse than normal business hours. Nevertheless, completely unexceptable and quite the headache.

[seitzbg]

I too have noticed this in the form of dropbox sucking.  100kb/sec download when trying to get my friend's video uploads.

I pay for 500/500 and Verizon's peering/routes to major players has started to *SUCK* in the past few months. I am considering dropping my $300/mo elsewhere.  Does Verizon care? Doesn't seem so.  

[Namronorman]

Hey Smith,

I am Norm on DSLR. I see the same thing with my line monitor and actually have an AWS account with some credits on it from trying it out then immediately giving up because of what people are complaining about here. When I tried it out, it was qwest to and fro.

I've tried a lot of cloud providers recently for some projects I'm working on and I haven't found a single one that works reliably with Verizon for an extended period of time. Same thing with CDN providers as well.

- Norm

[Rodder152]

Experienecing the same thing. I visit three Teamspeak3 Servers. The packet loss to two teamspeak servers that I visit is almost ALWAYS sitting at 10-20% packet loss. It's hosted by Typefrag with Amazon AWS. The third that I connect to is light-speed and doesn't use Amazon AWS. 

This is 100% a problem between something Fios and something Amazon related. Not really sure where the problem exists but there definatly a problem somewhere.

[seitzbg]

Yeah this is getting silly.  How Verizon could have bad peering to *AWS* one of the largest cloud providers (or the) in the world is beyond words.  I wish we had a network engineering contact or someone other than script reading level1/2 support.  I want to know if Verizon will be fixing their peering and peering practices or if I'm switching to Comcast.  Can't pay for 500/500 and get 1Mbps.

[CRobGauth]

Verizon could be doing this for the same reason they played hardball with Netflix.

The content providers want the ISPs to pay to continuously upgrade their peering and internal backbone to provde better service to the content providers customers. But the ISP doesn't make any extra money.

So they can either get the content provider to help pay for the extra peering costs, raise rates to the ISP customers or let it sit.

I beleive Verizon (purely a guess based on previous actions) is doing the third.

And Verizon is not the only ISP to do this.
Comcast did the same when it came to Netflix.

Same game as between cable providers and TV content providers.

[test421]

I'm fairly certain it is a peering issue. It was the one that was partly at the root of the netflix issue. Basically verizon is complaining that a particular peering connection is getting more data in one direction than another (verizon is receiving more than sending), and that L3/Cogent/whoever should pay for sending so much more data onto verizons network than they're getting back.

At least that was their argument the last I checked.

either way it makes no sense since the other network is clearly hosting content providers while verizon is hosting consumers. Of course verizon is going to be receiving more than its sending.

Hopefully the complaints to the FCC by Cogent/L3 pay off and verizon is finally forced to upgrade its own hardware.

[CRobGauth]

So when Verizon has to outlay more cash, they need to make it up.
That means our rates will go up.

[test421]

I dunno if rates NEED to go up. It all depends on verizons pricing model and what their profit margins are. Since they're effectively a monopoly I'm fairly certain their current margins are more than enough of absorb a hardware upgrade without impacting us.

[test421]

Actually crobgauth, I would like to know how YOU know that an upgrade would actually result in a rate increase.

[CRobGauth]

Not saying that any one upgrade will cause a rate increase.

But if Verizon (like any other business) is forced to increase their costs, it eventually gets passed on to their customers.

[test421]

And therein lies the problem, these guys aren't like any other business because other businesses have competition. 

Notice how quickly incubment ISP's are able to roll out gigabit service as soon as google announces its moving in? And in addition there's no rate increase to the customers?

[seitzbg]

As soon as other providers offer > 500/500 I'm switching unless Verizon stops peering like a mom and pop ISP.

This is a good resource for testing AWS performance btw:

https://cloudharmony.com/speedtest

Be sure to click advanced and tick 'uplink' as well.

[test421]

Thanks for the speed test. I'll be sure to use it.

In other news,

http://arstechnica.com/business/2015/05/01/verizon-and-cogent-settle-differences-agree-to-boost-inte...

When can we expect to see improvements in the northeast?