Just a warning as IPv6 rolls out to all fellow customers of Fios, you may find suddenly that your upload speed goes to zero and/or some sites are no longer working as intended, on WIRED/Ethernet and not WiFi.
Starting Friday (5/27/2022) morning, I found the following occurring:
- unable to upload/attach email in Outlook Online (O365).
- unable to access/stream Apple Music from the web.
- various speedtest.net locations reported "close enough" Download speeds, but 0Mpbs upload.
- other misc issues and browsing headaches.
How I troubleshooted the issue to determine the IPv6 was the cause:
- reset ONT (multiple times)
- reset router (multiple times)
- new CAT6 Shielded cables
- tested iPhone hotspot (Verizon Wireless Service) to PC
- swapped between Wired (LAN) and WiFi (WAN) on same router to PC
Based on #4 and #5, I was able to identify it was a LAN/wired issue as WiFi was not affected. I thought this may be a router specific issue after isolating the issue. However, seeing another community member post here about IPv6, I got suspicious. I checked and my IPv6 was fully configured and running in the router and on my ethernet adapter settings.
Upon disabling IPv6, the issue was no longer persisting.
Solved! Go to Correct Answer
According to the Intel technical advisory back in 2017, this sounds about right as both products are listed in the affected products. You can follow their "Recommended action" for a temporary fix. For a real fix you need to replace the affected NIC or the motherboard that contains the affected NIC.
One quick follow up for anyone trying to determine if this is a cause for them, a very easy way is to go to Verizon's own speed test site.
If the test is able to full complete (both router and device test runs through) you're not effected.
If the test times out / errors when doing the device test (after router test), try disabling IPv6 in your router settings and running the same speed test again.
Thank you! Didn't know about the checksum issue, but once you mentioned I searched on my built in NIC (Intel I219-V).
This post then clued me in that the I219-V does suffer a TCP Checksum issue.
The final resolution was:
Recommend this as well for anyone having poor internet with IPv6. So far so good, but will report back if the original issue with upload reoccurs.
Someone over at DSLR found, through a third-party router, that the hardware NAT acceleration that many routers have seemed to contribute to the issue. Something about the hardware acceleration chip in the router doesn't handle the IPv6 packets properly, causing the checksum issue. Considering those chips were designed to speed up the handling of IPv4 NAT packets, this isn't really surprising that there would be issues with IPv6.
The problem is that disabling such hardware acceleration (where it's even possible; I don't think you can disable it in the Verizon G1100) can cause significant performance issues for both IPv4 and IPv6, since the router's CPU needs to handle all of the data passing through the router instead. The whole point of such hardware NAT acceleration was to allow the router to use a lower cost/performance CPU, since it wouldn't need to handle all of the network traffic passing through it.
I don't know if the G3100 suffers from such an issue... it's new enough that the design may have been forward-thinking with IPv6 in mind (even if Verizon wasn't ready for it at that time), as most WiFi 6 routers have faster processers that can handle the data throughput now.
Of course, most software-based routers/firewalls (like pfSense, opnSense, etc.) should not have such an issue, since they don't usually have that kind of hardware acceleration (because they usually run on standard PC hardware).
So hardware acceleration is called "Cut-Through Forwarding" and is a hardware level function where, instead of reprocessing packets fully through the CPU, the packet is left mostly untouched and transmitted by the hardware straight to the client. Changes to the packet would mostly be to source/destination information like IP changes (NAT) and MAC Address (L2 Data Link Layer). This has the unintended consequence of "Garbage in, garbage out" since the packet checksum isn't validated and/or changed. The problem device, the ONT, in theory should be doing this as well. However the ONT has buggy software on it that is mangling packets, likely as part of the inspection process it is doing for quality of service or filtering reasons.
Routers which process packets through the CPU entirely should rebuild packets negating the problem. Linux and BSD based routers at least, should. However, routers using the affected NICs and using offload will suffer from the same fate even if they do routing in the CPU.
Wi-Fi is typically unaffected as transmission to/from wireless will typically hit the CPU for packet re-packaging, buffer, and send. There is no fixed function cut-through switching.
This same thing happened to me on Friday. Apparently Verizon pushed a firmware update out Thursday night. My router was updated to 220.127.116.11. Friday morning I had the same problems. Websites hanging, incomplete speed tests and the inability to stream video to YouTube or Facebook.
I have 6-7 Intel NICs on my network, mostly the I-219 chipset and a newer I225 chipset. Same issue on all machines. Disabling IPv6 on the NICs is the only thing that really solved the problem. I have also seen some of the same issues on an AQtion 5GB NIC as well.
The question is will Verizon update the ONTs to eliminate the issue?
This same thing happened to me on Friday. Apparently Verizon pushed a firmware update out Thursday night. My router was updated to 18.104.22.168.
The question is will Verizon update the ONTs to eliminate the issue?
Just to clear most ONTs supplied by Verizon do not include a built in NAT router. The few that had a built in NAT router I heard/read were/was used only for some sort of testing (ex testing IPTV - TV over IP).
As a follow up to my original issue, I purchased a new Ethernet PCI-e NIC (TRENDnet TEG-25GECTX) to test the Realtek vs onboard Intel ethernet.
Can confirm IPv6 works with this NIC. This is a good path for only $35, especially if you're like me and don't like having to disable Checksum and potentially put more load on your CPU (albeit minimally but we upgrade parts for frames of second increases in games / processing time), or having to disable IPv6 altogether.
The TRENDnet is also capable of up to 2.5gpbs (whenever Fios 2GB is expanded to your location). Right now the download is a bit slower (720-750mbps) than the Intel NIC (800-850 mbps). Upload is the same (900-950 mpbs). I believe download speed variance is due to a variety of factors:
- IPv6 itself, and/or
- need to tweak of configuration settings of freshly installed NIC, and/or
- general saturation network saturation.
For fairness, Verizon's own Fios speed test did give me high speeds at router and device (over 900Mbps for download/upload). However, realistically I don't see the same speeds (specially download speeds) anywhere else, so it's more like an advertisement than a useful benchmark.
The Verizon speed test does not detect this issue. I started suffering from this issue last week, called Fios Tech Support twice, swapped my G3100 router, wiped my Windows 10 operating system, and none of this resolved my issue. Disabling TCP Checksum Offload for IPv6 resolved the issue. My network card is an Intel I219-V.