- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Just a warning as IPv6 rolls out to all fellow customers of Fios, you may find suddenly that your upload speed goes to zero and/or some sites are no longer working as intended, on WIRED/Ethernet and not WiFi.
Starting Friday (5/27/2022) morning, I found the following occurring:
- unable to upload/attach email in Outlook Online (O365).
- unable to access/stream Apple Music from the web.
- various speedtest.net locations reported "close enough" Download speeds, but 0Mpbs upload.
- other misc issues and browsing headaches.
How I troubleshooted the issue to determine the IPv6 was the cause:
- reset ONT (multiple times)
- reset router (multiple times)
- new CAT6 Shielded cables
- tested iPhone hotspot (Verizon Wireless Service) to PC
- swapped between Wired (LAN) and WiFi (WAN) on same router to PC
Based on #4 and #5, I was able to identify it was a LAN/wired issue as WiFi was not affected. I thought this may be a router specific issue after isolating the issue. However, seeing another community member post here about IPv6, I got suspicious. I checked and my IPv6 was fully configured and running in the router and on my ethernet adapter settings.
Upon disabling IPv6, the issue was no longer persisting.
Verizon why?
Solved! Go to Correct Answer
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
According to the Intel technical advisory back in 2017, this sounds about right as both products are listed in the affected products. You can follow their "Recommended action" for a temporary fix. For a real fix you need to replace the affected NIC or the motherboard that contains the affected NIC.
Intel Technical Advisory
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
One quick follow up for anyone trying to determine if this is a cause for them, a very easy way is to go to Verizon's own speed test site.
If the test is able to full complete (both router and device test runs through) you're not effected.
If the test times out / errors when doing the device test (after router test), try disabling IPv6 in your router settings and running the same speed test again.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The Verizon speed test does not detect this issue. I started suffering from this issue last week, called Fios Tech Support twice, swapped my G3100 router, wiped my Windows 10 operating system, and none of this resolved my issue. Disabling TCP Checksum Offload for IPv6 resolved the issue. My network card is an Intel I219-V.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Verizon device speed test can only see the Application Layer of your setup. NIC settings are all the way down to the Link Layer of your setup, of course it could not (and technically infeasible to) detect it.
When techs see an account with an IPv6 address and all of a sudden has speed issues, they should recognize the IPv6 TCP Checksum glitch immediately.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is not entirely true. One of my initial indicators that something was amiss with my Ethernet connection was that the Verizon speed test was only partially working, it would run the router speed test, and then error/time out when attempting to run a speed test on the PC part of the test.
When swapping to WiFi, both parts of the Fios speed test ran fully. It was one of my confirmations there was something wrong with Ethernet, and I was then able to further troubleshoot that IPv6 was the cause.
Without either disabling IPv6 or changing the Checksum setting, most sites would start to become slow or error out, the Fios Speedtest being a good indicator there was something wrong because it would work at router level but not PC/Ethernet level.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please list all Network Interface cards in your wired devices, as some Intel cards have problems with IPv6 checksum.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Cang,
Thank you! Didn't know about the checksum issue, but once you mentioned I searched on my built in NIC (Intel I219-V).
This post then clued me in that the I219-V does suffer a TCP Checksum issue.
The final resolution was:
Disabling TCP-IPv6 Checksum Offload Capability with Intel® 1/10 GbE Controllers
Recommend this as well for anyone having poor internet with IPv6. So far so good, but will report back if the original issue with upload reoccurs.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Someone over at DSLR found, through a third-party router, that the hardware NAT acceleration that many routers have seemed to contribute to the issue. Something about the hardware acceleration chip in the router doesn't handle the IPv6 packets properly, causing the checksum issue. Considering those chips were designed to speed up the handling of IPv4 NAT packets, this isn't really surprising that there would be issues with IPv6.
The problem is that disabling such hardware acceleration (where it's even possible; I don't think you can disable it in the Verizon G1100) can cause significant performance issues for both IPv4 and IPv6, since the router's CPU needs to handle all of the data passing through the router instead. The whole point of such hardware NAT acceleration was to allow the router to use a lower cost/performance CPU, since it wouldn't need to handle all of the network traffic passing through it.
I don't know if the G3100 suffers from such an issue... it's new enough that the design may have been forward-thinking with IPv6 in mind (even if Verizon wasn't ready for it at that time), as most WiFi 6 routers have faster processers that can handle the data throughput now.
Of course, most software-based routers/firewalls (like pfSense, opnSense, etc.) should not have such an issue, since they don't usually have that kind of hardware acceleration (because they usually run on standard PC hardware).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
So hardware acceleration is called "Cut-Through Forwarding" and is a hardware level function where, instead of reprocessing packets fully through the CPU, the packet is left mostly untouched and transmitted by the hardware straight to the client. Changes to the packet would mostly be to source/destination information like IP changes (NAT) and MAC Address (L2 Data Link Layer). This has the unintended consequence of "Garbage in, garbage out" since the packet checksum isn't validated and/or changed. The problem device, the ONT, in theory should be doing this as well. However the ONT has buggy software on it that is mangling packets, likely as part of the inspection process it is doing for quality of service or filtering reasons.
Routers which process packets through the CPU entirely should rebuild packets negating the problem. Linux and BSD based routers at least, should. However, routers using the affected NICs and using offload will suffer from the same fate even if they do routing in the CPU.
Wi-Fi is typically unaffected as transmission to/from wireless will typically hit the CPU for packet re-packaging, buffer, and send. There is no fixed function cut-through switching.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This same thing happened to me on Friday. Apparently Verizon pushed a firmware update out Thursday night. My router was updated to 3.1.1.17. Friday morning I had the same problems. Websites hanging, incomplete speed tests and the inability to stream video to YouTube or Facebook.
I have 6-7 Intel NICs on my network, mostly the I-219 chipset and a newer I225 chipset. Same issue on all machines. Disabling IPv6 on the NICs is the only thing that really solved the problem. I have also seen some of the same issues on an AQtion 5GB NIC as well.
The question is will Verizon update the ONTs to eliminate the issue?
Greg
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@GregInVa wrote:
This same thing happened to me on Friday. Apparently Verizon pushed a firmware update out Thursday night. My router was updated to 3.1.1.17.
The question is will Verizon update the ONTs to eliminate the issue?
Just to clear most ONTs supplied by Verizon do not include a built in NAT router. The few that had a built in NAT router I heard/read were/was used only for some sort of testing (ex testing IPTV - TV over IP).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
As a follow up to my original issue, I purchased a new Ethernet PCI-e NIC (TRENDnet TEG-25GECTX) to test the Realtek vs onboard Intel ethernet.
Can confirm IPv6 works with this NIC. This is a good path for only $35, especially if you're like me and don't like having to disable Checksum and potentially put more load on your CPU (albeit minimally but we upgrade parts for frames of second increases in games / processing time), or having to disable IPv6 altogether.
The TRENDnet is also capable of up to 2.5gpbs (whenever Fios 2GB is expanded to your location). Right now the download is a bit slower (720-750mbps) than the Intel NIC (800-850 mbps). Upload is the same (900-950 mpbs). I believe download speed variance is due to a variety of factors:
- IPv6 itself, and/or
- need to tweak of configuration settings of freshly installed NIC, and/or
- general saturation network saturation.
For fairness, Verizon's own Fios speed test did give me high speeds at router and device (over 900Mbps for download/upload). However, realistically I don't see the same speeds (specially download speeds) anywhere else, so it's more like an advertisement than a useful benchmark.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This just happened to me. Yesterday I had turned on the IPv6 and it was OK. Today I was at the desktop and I had all kinds of issues.
Then I saw this post.
My NIC is Intel I210 Gigabit Network Connection. I have another one in my Dell, an Intel 82579LM, maybe I'll try that one...
LE: It seems that the 82579LM is not affected too.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
According to the Intel technical advisory back in 2017, this sounds about right as both products are listed in the affected products. You can follow their "Recommended action" for a temporary fix. For a real fix you need to replace the affected NIC or the motherboard that contains the affected NIC.
Intel Technical Advisory
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I can confirm that this just happened to me overnight.
I have a Gigabyte Z390 Aorus Master Mobo and it utilizes the I219-V NIC
I woke up and noticed that my upload speeds were < 1 when running a speed test. Disabled Firewalls, VPNs, reinstalled drivers, but nothing resolved my issue. When switching to WiFi my speeds were what I expected them to be. Tested the cable on another laptop to rule out any sort of hardware issue. I stumbled upon a reddit thread with a solution of disabling IPV6 in the network adapter which worked. I re-enabled it and the speeds went back to being <1 mbps.
I tried the Powershell script in this thread which resolved the speeds when running a speed test, but Youtube videos and other similar things were still not loading properly. I went back to disabling IPV6 and all is well. Wouldn't mind another solution besides disabling IPV6 or updating my MOBO 🙂
Hope this helps someone at least because it was driving me crazy for the day.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Look, checksum offloading is a hardware feature. If it has an issue, it is a HARDWARE issue. Firmware only modifies the software and does nothing to hardware. So to fix this issue once for all, these affected adapters need to be replaced one way or the other.
OR get a cheap 2.5GbE RTL8156B dongle and call it a day along with a link speed upgrade.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is a shared issue with poor communication from Verizon. They pushed this out and the average person would have no idea why their internet speed went to zero. It doesn't matter what the issue is, it matters how it was caused - the IPv6 rollout.
We can only hope people find this thread or others like it to get informed on how to get a resolution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@gillette wrote:
This is a shared issue with poor communication from Verizon. They pushed this out and the average person would have no idea why their internet speed went to zero.
Intel released their technical advisory about this IPv6 checksum issue 5 years ago. Do you think it is reasonable for end users who have Intel NICs to know it by now? Technically, from a utilitarian perspective, not only should Intel distribute that advisory on every individual NIC card they sell, Intel should require all networking equipment and motherboard vendors to distribute similar notices as long as the affected Intel NICs are used.
@gillette wrote:
It doesn't matter what the issue is, it matters how it was caused - the IPv6 rollout.
You are heading to a logical fallacy if you say this, namely post hoc ergo propter hoc. Just because users encounter problems after the IPv6 rollout, it does not mean the IPv6 rollout caused the problems.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Well said. I agree with you completely. The folks on this forum are intelligent enough to A. recognize there's an issue, and B. proactively seek it out without having to sit on the phone with "tech support" from Verizon for hours with no success. I had the same issue, upgraded to a G3100 last week and all of a sudden my upload speed went to crap. Wireless was fine which told me it was most likely with the LAN. A quick search revealed multiple threads like this one. Disabled IPv6 Checksum for TCP and all was well EXCEPT for terrible YouTube. Disabled UDP checksum as well, now we're all better.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@AsusMiller wrote:
Disabled IPv6 Checksum for TCP and all was well EXCEPT for terrible YouTube. Disabled UDP checksum as well, now we're all better.
Just to add onto your solution. YouTube was still broken, because Google uses the QUIC Protocol to deliver data from YouTube and other Google services. QUIC relies on UDP rather than TCP. So it's important to disable IPv6 Checksum offload on both TCP and UDP 🙂
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Cang_Household wrote:
Look, checksum offloading is a hardware feature. If it has an issue, it is a HARDWARE issue. Firmware only modifies the software and does nothing to hardware. So to fix this issue once for all, these affected adapters need to be replaced one way or the other.
Only true for these particular models of Intel Ethernet chips. There are many networking chips that use firmware to control data flows and process packets. Those chips can have issues like this resolved with a firmware upgrade. Without reading the full data sheet for one of the Intel chips, I can't tell how the checksum function is performed; but I have worked with plenty of other chips that do work like this in firmware or a combination of firmware with low level hardware calculators.
