Packet loss and lag to AWS

[test421]

Ever since I signed up for 70/70 internet I've noticed slow connection speeds specifically to Amazon's cloud services. Speedtest.net shows that everything is fine, but I know that speedtest.net merely tests PART of your internet speed. Is verizon still throttling (aka refusing to pay for upgrades) to amazon web services? If so, why am I paying for 70/70 internet? My sprint phone on 3G has a better connection to my teamspeak server.

[smith6612]

Many of us in the Northeast have been seeing issues with congestion to and from AWS. My DSLReports Line monitor will show this. It appears to be a peering problem between a transit provider used to connect Verizon and Amazon together, and Verizon. In my case it appears to be Qwest, but I need a reverse traceroute to fully determine that.

Here's my DSLReports line monitor as an exmaple.

The top graph shows a monitor from an East Coast EC2 instance. The bottom one is from a West Coast instance. Blue denotes packet loss - in my case this loss in the past day was due to my IP address changing (I was doing some work on my home network). The East coast instance consistently sees latency at night. West Coast EC2 sees less latency.

Just congestion. Verizon needs to look into their peering with others and fix it up. Has nothing to do with FiOS itself, as I'm on DSL.

[mtgdam]

We have services hosted in AWS N. Virginia. Right now, anywhere from 10-25% packet loss and average 120ms rtt consistantly at multiple clients on FiOS Business Connections. On February 19th, the issue was so bad that services were unreachable from 9:30am to 2:15pm EST.

I notice the same where evenings tend to be worse than normal business hours. Nevertheless, completely unexceptable and quite the headache.

[seitzbg]

I too have noticed this in the form of dropbox sucking.  100kb/sec download when trying to get my friend's video uploads.

I pay for 500/500 and Verizon's peering/routes to major players has started to *SUCK* in the past few months. I am considering dropping my $300/mo elsewhere.  Does Verizon care? Doesn't seem so.  

[Namronorman]

Hey Smith,

I am Norm on DSLR. I see the same thing with my line monitor and actually have an AWS account with some credits on it from trying it out then immediately giving up because of what people are complaining about here. When I tried it out, it was qwest to and fro.

I've tried a lot of cloud providers recently for some projects I'm working on and I haven't found a single one that works reliably with Verizon for an extended period of time. Same thing with CDN providers as well.

- Norm

[Rodder152]

Experienecing the same thing. I visit three Teamspeak3 Servers. The packet loss to two teamspeak servers that I visit is almost ALWAYS sitting at 10-20% packet loss. It's hosted by Typefrag with Amazon AWS. The third that I connect to is light-speed and doesn't use Amazon AWS. 

This is 100% a problem between something Fios and something Amazon related. Not really sure where the problem exists but there definatly a problem somewhere.

[seitzbg]

Yeah this is getting silly.  How Verizon could have bad peering to *AWS* one of the largest cloud providers (or the) in the world is beyond words.  I wish we had a network engineering contact or someone other than script reading level1/2 support.  I want to know if Verizon will be fixing their peering and peering practices or if I'm switching to Comcast.  Can't pay for 500/500 and get 1Mbps.

[CRobGauth]

Verizon could be doing this for the same reason they played hardball with Netflix.

The content providers want the ISPs to pay to continuously upgrade their peering and internal backbone to provde better service to the content providers customers. But the ISP doesn't make any extra money.

So they can either get the content provider to help pay for the extra peering costs, raise rates to the ISP customers or let it sit.

I beleive Verizon (purely a guess based on previous actions) is doing the third.

And Verizon is not the only ISP to do this.
Comcast did the same when it came to Netflix.

Same game as between cable providers and TV content providers.

[test421]

I'm fairly certain it is a peering issue. It was the one that was partly at the root of the netflix issue. Basically verizon is complaining that a particular peering connection is getting more data in one direction than another (verizon is receiving more than sending), and that L3/Cogent/whoever should pay for sending so much more data onto verizons network than they're getting back.

At least that was their argument the last I checked.

either way it makes no sense since the other network is clearly hosting content providers while verizon is hosting consumers. Of course verizon is going to be receiving more than its sending.

Hopefully the complaints to the FCC by Cogent/L3 pay off and verizon is finally forced to upgrade its own hardware.

[CRobGauth]

So when Verizon has to outlay more cash, they need to make it up.
That means our rates will go up.