IPv6 causing 0Mpbs upload speed
gillette2
Enthusiast - Level 2

Just a warning as IPv6 rolls out to all fellow customers of Fios, you may find suddenly that your upload speed goes to zero and/or some sites are no longer working as intended, on WIRED/Ethernet and not WiFi. 

Starting Friday (5/27/2022) morning, I found the following occurring:

  • unable to upload/attach email in Outlook Online (O365).
  • unable to access/stream Apple Music from the web.
  • various speedtest.net locations reported "close enough" Download speeds, but 0Mpbs upload. 
  • other misc issues and browsing headaches.

    How I troubleshooted the issue to determine the IPv6 was the cause:

    1. reset ONT (multiple times)
    2. reset router (multiple times)
    3. new CAT6 Shielded cables
    4. tested iPhone hotspot (Verizon Wireless Service) to PC
    5. swapped between Wired (LAN) and WiFi (WAN) on same router to PC

      Based on #4 and #5, I was able to identify it was a LAN/wired issue as WiFi was not affected.  I thought this may be a router specific issue after isolating the issue.  However, seeing another community member post here about IPv6, I got suspicious. I checked and my IPv6 was fully configured and running in the router and on my ethernet adapter settings. 

      Upon disabling IPv6, the issue was no longer persisting.

      Verizon why?

1 Solution
Cang_Household
Community Leader
Community Leader

According to the Intel technical advisory back in 2017, this sounds about right as both products are listed in the affected products. You can follow their "Recommended action" for a temporary fix. For a real fix you need to replace the affected NIC or the motherboard that contains the affected NIC.

imageIntel Technical Advisory

View solution in original post

35 Replies
gillette2
Enthusiast - Level 2

This is a shared issue with poor communication from Verizon.  They pushed this out and the average person would have no idea why their internet speed went to zero.  It doesn't matter what the issue is, it matters how it was caused - the IPv6 rollout. 

We can only hope people find this thread or others like it to get informed on how to get a resolution.

0 Likes
Reply
AsusMiller
Enthusiast - Level 2

Well said. I agree with you completely. The folks on this forum are intelligent enough to A. recognize there's an issue, and B. proactively seek it out without having to sit on the phone with "tech support" from Verizon for hours with no success. I had the same issue, upgraded to a G3100 last week and all of a sudden my upload speed went to crap. Wireless was fine which told me it was most likely with the LAN. A quick search revealed multiple threads like this one. Disabled IPv6 Checksum for TCP and all was well EXCEPT for terrible YouTube. Disabled UDP checksum as well, now we're all better.

smith6612
Community Leader
Community Leader

@AsusMiller wrote:

 Disabled IPv6 Checksum for TCP and all was well EXCEPT for terrible YouTube. Disabled UDP checksum as well, now we're all better.


Just to add onto your solution. YouTube was still broken, because Google uses the QUIC Protocol to deliver data from YouTube and other Google services. QUIC relies on UDP rather than TCP. So  it's important to disable IPv6 Checksum offload on both TCP and UDP 🙂

Cang_Household
Community Leader
Community Leader

@gillette wrote:

This is a shared issue with poor communication from Verizon.  They pushed this out and the average person would have no idea why their internet speed went to zero. 


Intel released their technical advisory about this IPv6 checksum issue 5 years ago. Do you think it is reasonable for end users who have Intel NICs to know it by now? Technically, from a utilitarian perspective, not only should Intel distribute that advisory on every individual NIC card they sell, Intel should require all networking equipment and motherboard vendors to distribute similar notices as long as the affected Intel NICs are used.


@gillette wrote:

It doesn't matter what the issue is, it matters how it was caused - the IPv6 rollout. 


You are heading to a logical fallacy if you say this, namely post hoc ergo propter hoc. Just because users encounter problems after the IPv6 rollout, it does not mean the IPv6 rollout caused the problems.

gillette2
Enthusiast - Level 2

As a follow up to my original issue, I purchased a new  Ethernet PCI-e NIC (TRENDnet TEG-25GECTX) to test the Realtek vs onboard Intel ethernet. 

Can confirm IPv6 works with this NIC.  This is a good path for only $35, especially if you're like me and don't like having to disable Checksum and potentially put more load on your CPU (albeit minimally but we upgrade parts for frames of second increases in games / processing time), or having to disable IPv6 altogether.

The TRENDnet is also capable of up to 2.5gpbs (whenever Fios 2GB is expanded to your location).  Right now the download is a bit slower (720-750mbps) than the Intel NIC (800-850 mbps).  Upload is the same (900-950 mpbs). I believe download speed variance is due to a variety of factors:

  • IPv6 itself, and/or
  • need to tweak of configuration settings of freshly installed NIC, and/or
  • general saturation network saturation.

    For fairness, Verizon's own Fios speed test did give me high speeds at router and device (over 900Mbps for download/upload).  However, realistically I don't see the same speeds (specially download speeds) anywhere else, so it's more like an advertisement than a useful benchmark.

    image

GregInVa
Enthusiast - Level 1

This same thing happened to me on Friday. Apparently Verizon pushed a firmware update out Thursday night. My router was updated to 3.1.1.17. Friday morning I had the same problems. Websites hanging, incomplete speed tests and the inability to stream video to YouTube or Facebook.

I have 6-7 Intel NICs on my network, mostly the I-219 chipset and a newer I225 chipset. Same issue on all machines. Disabling IPv6 on the NICs is the only thing that really solved the problem. I have also seen some of the same issues on an AQtion 5GB NIC as well.

The question is will Verizon update the ONTs to eliminate the issue?

Greg

dslr595148
Community Leader
Community Leader

@GregInVa wrote:

This same thing happened to me on Friday. Apparently Verizon pushed a firmware update out Thursday night. My router was updated to 3.1.1.17.

The question is will Verizon update the ONTs to eliminate the issue?


Just to clear most ONTs supplied by Verizon do not include a built in NAT router. The few that had a built in NAT router I heard/read were/was used only for some sort of testing (ex testing IPTV - TV over IP).

Cang_Household
Community Leader
Community Leader

Please list all Network Interface cards in your wired devices, as some Intel cards have problems with IPv6 checksum.

gillette2
Enthusiast - Level 2

Hi Cang,

Thank you! Didn't know about the checksum issue, but once you mentioned I searched on my built in NIC (Intel I219-V).

This post then clued me in that the I219-V does suffer a TCP Checksum issue.

The final resolution was:

Disabling TCP-IPv6 Checksum Offload Capability with Intel® 1/10 GbE Controllers

Recommend this as well for anyone having poor internet with IPv6.  So far so good, but will report back if the original issue with upload reoccurs.

Anonymous
Not applicable

Someone over at DSLR found, through a third-party router, that the hardware NAT acceleration that many routers have seemed to contribute to the issue. Something about the hardware acceleration chip in the router doesn't handle the IPv6 packets properly, causing the checksum issue. Considering those chips were designed to speed up the handling of IPv4 NAT packets, this isn't really surprising that there would be issues with IPv6.

The problem is that disabling such hardware acceleration (where it's even possible; I don't think you can disable it in the Verizon G1100) can cause significant performance issues for both IPv4 and IPv6, since the router's CPU needs to handle all of the data passing through the router instead. The whole point of such hardware NAT acceleration was to allow the router to use a lower cost/performance CPU, since it wouldn't need to handle all of the network traffic passing through it.

I don't know if the G3100 suffers from such an issue... it's new enough that the design may have been forward-thinking with IPv6 in mind (even if Verizon wasn't ready for it at that time), as most WiFi 6 routers have faster processers that can handle the data throughput now.

Of course, most software-based routers/firewalls (like pfSense, opnSense, etc.) should not have such an issue, since they don't usually have that kind of hardware acceleration (because they usually run on standard PC hardware).

smith6612
Community Leader
Community Leader

So hardware acceleration is called "Cut-Through Forwarding" and is a hardware level function where, instead of reprocessing packets fully through the CPU, the packet is left mostly untouched and transmitted by the hardware straight to the client. Changes to the packet would mostly be to source/destination information like IP changes (NAT) and MAC Address (L2 Data Link Layer). This has the unintended consequence of "Garbage in, garbage out" since the packet checksum isn't validated and/or changed. The problem device, the ONT, in theory should be doing this as well. However the ONT has buggy software on it that is mangling packets, likely as part of the inspection process it is doing for quality of service or filtering reasons.

Routers which process packets through the CPU entirely should rebuild packets negating the problem. Linux and BSD based routers at least, should. However, routers using the affected NICs and using offload will suffer from the same fate even if they do routing in the CPU.

Wi-Fi is typically unaffected as transmission to/from wireless will typically hit the CPU for packet re-packaging, buffer, and send. There is no fixed function cut-through switching.

gillette2
Enthusiast - Level 2

One quick follow up for anyone trying to determine if this is a cause for them, a very easy way is to go to Verizon's own speed test site.  

If the test is able to full complete (both router and device test runs through) you're not effected.

If the test times out / errors when doing the device test (after router test), try disabling IPv6 in your router settings and running the same speed test again.

0 Likes
Reply
BChappie1
Enthusiast - Level 1

The Verizon speed test does not detect this issue. I started suffering from this issue last week, called Fios Tech Support twice, swapped my G3100 router, wiped my Windows 10 operating system, and none of this resolved my issue. Disabling TCP Checksum Offload for IPv6 resolved the issue. My network card is an Intel I219-V.

Cang_Household
Community Leader
Community Leader

Verizon device speed test can only see the Application Layer of your setup. NIC settings are all the way down to the Link Layer of your setup, of course it could not (and technically infeasible to) detect it.

When techs see an account with an IPv6 address and all of a sudden has speed issues, they should recognize the IPv6 TCP Checksum glitch immediately.

gillette2
Enthusiast - Level 2

This is not entirely true. One of my initial indicators that something was amiss with my Ethernet connection was that the Verizon speed test was only partially working, it would run the router speed test, and then error/time out when attempting to run a speed test on the PC part of the test.


When swapping to WiFi, both parts of the Fios speed test ran fully.  It was one of my confirmations there was something wrong with Ethernet, and I was then able to further troubleshoot that IPv6 was the cause.


Without either disabling IPv6 or changing the Checksum setting, most sites would start to become slow or error out, the Fios Speedtest being a good indicator there was something wrong because it would work at router level but not PC/Ethernet level.

0 Likes
Reply