Just a warning as IPv6 rolls out to all fellow customers of Fios, you may find suddenly that your upload speed goes to zero and/or some sites are no longer working as intended, on WIRED/Ethernet and not WiFi. Starting Friday (5/27/2022) morning, I found the following occurring: * unable to upload/attach email in Outlook Online (O365). * unable to access/stream Apple Music from the web. * various speedtest.net locations reported "close enough" Download speeds, but 0Mpbs upload. * other misc issues and browsing headaches.How I troubleshooted the issue to determine the IPv6 was the cause: * reset ONT (multiple times) * reset router (multiple times) * new CAT6 Shielded cables * tested iPhone hotspot (Verizon Wireless Service) to PC * swapped between Wired (LAN) and WiFi (WAN) on same router to PCBased on #4 and #5, I was able to identify it was a LAN/wired issue as WiFi was not affected. I thought this may be a router specific issue after isolating the issue. However, seeing another community member post here about IPv6, I got suspicious. I checked and my IPv6 was fully configured and running in the router and on my ethernet adapter settings. Upon disabling IPv6, the issue was no longer persisting. Verizon why?

IPv6 causing 0Mpbs upload speed

Just a warning as IPv6 rolls out to all fellow customers of Fios, you may find suddenly that your upload speed goes to zero and/or some sites are no longer working as intended, on WIRED/Ethernet and not WiFi.

Starting Friday (5/27/2022) morning, I found the following occurring:

unable to upload/attach email in Outlook Online (O365).
unable to access/stream Apple Music from the web.
various speedtest.net locations reported "close enough" Download speeds, but 0Mpbs upload.
other misc issues and browsing headaches.
How I troubleshooted the issue to determine the IPv6 was the cause:
1. reset ONT (multiple times)
2. reset router (multiple times)
3. new CAT6 Shielded cables
4. tested iPhone hotspot (Verizon Wireless Service) to PC
5. swapped between Wired (LAN) and WiFi (WAN) on same router to PC
  Based on #4 and #5, I was able to identify it was a LAN/wired issue as WiFi was not affected. I thought this may be a router specific issue after isolating the issue. However, seeing another community member post here about IPv6, I got suspicious. I checked and my IPv6 was fully configured and running in the router and on my ethernet adapter settings.
  
  Upon disabling IPv6, the issue was no longer persisting.
  
  Verizon why?

Accepted answers

Cang_Household

According to the Intel technical advisory back in 2017, this sounds about right as both products are listed in the affected products. You can follow their "Recommended action" for a temporary fix. For a real fix you need to replace the affected NIC or the motherboard that contains the affected NIC.

Intel Technical Advisory

All comments

gillette2

One quick follow up for anyone trying to determine if this is a cause for them, a very easy way is to go to Verizon's own speed test site.

If the test is able to full complete (both router and device test runs through) you're not effected.

If the test times out / errors when doing the device test (after router test), try disabling IPv6 in your router settings and running the same speed test again.

Cang_Household

Please list all Network Interface cards in your wired devices, as some Intel cards have problems with IPv6 checksum.

gillette2

Hi Cang,

Thank you! Didn't know about the checksum issue, but once you mentioned I searched on my built in NIC (Intel I219-V).

This post then clued me in that the I219-V does suffer a TCP Checksum issue.

The final resolution was:

Disabling TCP-IPv6 Checksum Offload Capability with Intel® 1/10 GbE Controllers

Recommend this as well for anyone having poor internet with IPv6. So far so good, but will report back if the original issue with upload reoccurs.

unknown

Someone over at DSLR found, through a third-party router, that the hardware NAT acceleration that many routers have seemed to contribute to the issue. Something about the hardware acceleration chip in the router doesn't handle the IPv6 packets properly, causing the checksum issue. Considering those chips were designed to speed up the handling of IPv4 NAT packets, this isn't really surprising that there would be issues with IPv6.

The problem is that disabling such hardware acceleration (where it's even possible; I don't think you can disable it in the Verizon G1100) can cause significant performance issues for both IPv4 and IPv6, since the router's CPU needs to handle all of the data passing through the router instead. The whole point of such hardware NAT acceleration was to allow the router to use a lower cost/performance CPU, since it wouldn't need to handle all of the network traffic passing through it.

I don't know if the G3100 suffers from such an issue... it's new enough that the design may have been forward-thinking with IPv6 in mind (even if Verizon wasn't ready for it at that time), as most WiFi 6 routers have faster processers that can handle the data throughput now.

Of course, most software-based routers/firewalls (like pfSense, opnSense, etc.) should not have such an issue, since they don't usually have that kind of hardware acceleration (because they usually run on standard PC hardware).

smith6612

So hardware acceleration is called "Cut-Through Forwarding" and is a hardware level function where, instead of reprocessing packets fully through the CPU, the packet is left mostly untouched and transmitted by the hardware straight to the client. Changes to the packet would mostly be to source/destination information like IP changes (NAT) and MAC Address (L2 Data Link Layer). This has the unintended consequence of "Garbage in, garbage out" since the packet checksum isn't validated and/or changed. The problem device, the ONT, in theory should be doing this as well. However the ONT has buggy software on it that is mangling packets, likely as part of the inspection process it is doing for quality of service or filtering reasons.

Routers which process packets through the CPU entirely should rebuild packets negating the problem. Linux and BSD based routers at least, should. However, routers using the affected NICs and using offload will suffer from the same fate even if they do routing in the CPU.

Wi-Fi is typically unaffected as transmission to/from wireless will typically hit the CPU for packet re-packaging, buffer, and send. There is no fixed function cut-through switching.

GregInVa

This same thing happened to me on Friday. Apparently Verizon pushed a firmware update out Thursday night. My router was updated to 3.1.1.17. Friday morning I had the same problems. Websites hanging, incomplete speed tests and the inability to stream video to YouTube or Facebook.

I have 6-7 Intel NICs on my network, mostly the I-219 chipset and a newer I225 chipset. Same issue on all machines. Disabling IPv6 on the NICs is the only thing that really solved the problem. I have also seen some of the same issues on an AQtion 5GB NIC as well.

The question is will Verizon update the ONTs to eliminate the issue?

Greg

dslr595148

@GregInVa wrote:
This same thing happened to me on Friday. Apparently Verizon pushed a firmware update out Thursday night. My router was updated to 3.1.1.17.

The question is will Verizon update the ONTs to eliminate the issue?

Just to clear most ONTs supplied by Verizon do not include a built in NAT router. The few that had a built in NAT router I heard/read were/was used only for some sort of testing (ex testing IPTV - TV over IP).

gillette2

As a follow up to my original issue, I purchased a new Ethernet PCI-e NIC (TRENDnet TEG-25GECTX) to test the Realtek vs onboard Intel ethernet.

Can confirm IPv6 works with this NIC. This is a good path for only $35, especially if you're like me and don't like having to disable Checksum and potentially put more load on your CPU (albeit minimally but we upgrade parts for frames of second increases in games / processing time), or having to disable IPv6 altogether.

The TRENDnet is also capable of up to 2.5gpbs (whenever Fios 2GB is expanded to your location). Right now the download is a bit slower (720-750mbps) than the Intel NIC (800-850 mbps). Upload is the same (900-950 mpbs). I believe download speed variance is due to a variety of factors:

IPv6 itself, and/or
need to tweak of configuration settings of freshly installed NIC, and/or
general saturation network saturation.
For fairness, Verizon's own Fios speed test did give me high speeds at router and device (over 900Mbps for download/upload). However, realistically I don't see the same speeds (specially download speeds) anywhere else, so it's more like an advertisement than a useful benchmark.

BChappie1

The Verizon speed test does not detect this issue. I started suffering from this issue last week, called Fios Tech Support twice, swapped my G3100 router, wiped my Windows 10 operating system, and none of this resolved my issue. Disabling TCP Checksum Offload for IPv6 resolved the issue. My network card is an Intel I219-V.

Cang_Household

Verizon device speed test can only see the Application Layer of your setup. NIC settings are all the way down to the Link Layer of your setup, of course it could not (and technically infeasible to) detect it.

When techs see an account with an IPv6 address and all of a sudden has speed issues, they should recognize the IPv6 TCP Checksum glitch immediately.

gillette2

This is not entirely true. One of my initial indicators that something was amiss with my Ethernet connection was that the Verizon speed test was only partially working, it would run the router speed test, and then error/time out when attempting to run a speed test on the PC part of the test.

When swapping to WiFi, both parts of the Fios speed test ran fully. It was one of my confirmations there was something wrong with Ethernet, and I was then able to further troubleshoot that IPv6 was the cause.

Without either disabling IPv6 or changing the Checksum setting, most sites would start to become slow or error out, the Fios Speedtest being a good indicator there was something wrong because it would work at router level but not PC/Ethernet level.

SoNic671

This just happened to me. Yesterday I had turned on the IPv6 and it was OK. Today I was at the desktop and I had all kinds of issues.

Then I saw this post.

My NIC is Intel I210 Gigabit Network Connection. I have another one in my Dell, an Intel 82579LM, maybe I'll try that one...

LE: It seems that the 82579LM is not affected too.

Cang_Household

Intel Technical Advisory

PrestonK

I can confirm that this just happened to me overnight.

I have a Gigabyte Z390 Aorus Master Mobo and it utilizes the I219-V NIC

I woke up and noticed that my upload speeds were < 1 when running a speed test. Disabled Firewalls, VPNs, reinstalled drivers, but nothing resolved my issue. When switching to WiFi my speeds were what I expected them to be. Tested the cable on another laptop to rule out any sort of hardware issue. I stumbled upon a reddit thread with a solution of disabling IPV6 in the network adapter which worked. I re-enabled it and the speeds went back to being <1 mbps.

I tried the Powershell script in this thread which resolved the speeds when running a speed test, but Youtube videos and other similar things were still not loading properly. I went back to disabling IPV6 and all is well. Wouldn't mind another solution besides disabling IPV6 or updating my MOBO 🙂

Hope this helps someone at least because it was driving me crazy for the day.

Cang_Household

Look, checksum offloading is a hardware feature. If it has an issue, it is a HARDWARE issue. Firmware only modifies the software and does nothing to hardware. So to fix this issue once for all, these affected adapters need to be replaced one way or the other.

OR get a cheap 2.5GbE RTL8156B dongle and call it a day along with a link speed upgrade.

stoggy1

This did not work for me. 3 routers later and I went from great wifi to crap upload on the wifi to 0 upload. The only way I can get good internet to my PC is via the wan port and nothing else. Not the wifi ports or Lan ports and Tech just shrugged his head and said "test fine on my phone" and left.

I am not in the slightest please with this situation. I went from having amazing internet and a great wifi to garbage and nobody seems to know why but my issue is identical to people in here complaining as of july 2022

stoggy1

I've found it. It just didn't solve the problem.

Taco77

same! turn off ipv6 and all works well again

Cang_Household

@Taco77 wrote:
same! turn off ipv6 and all works well again

This is the wrong approach. To resolve a fire alarm is not to rip the alarm off the wall/ceiling.

TheTimeIsChow

So if I may ask - what would be the solution then?

TheTimeIsChow

It did. This drove me bananas.

I tested and troubleshooted in every way possible. It took me all night trying to figure out what was going on because everything else besides my gaming PC seemed to work properly.

That said - My upload speeds across the board are still pretty bad. What used to be 500+ on gigabit is now <120 despite disabling IPv6. Much better than the 0.1 that it was before though.

It's very irritating to be honest. I wish Verizon would have sent out a notice ahead of time letting people know the problem might arise.

BobBloomGeek

For a month and a half I chased this. To add to the discussion, I found a easy way to discover the problem - run a Google (M-Lab) speed test. Uploads are 1% (!) of the download speed. Third party speed tests show full speed down and uploads. Fully reproducible.

Disabled IPV6 and instant gratification. M-Lab now shows equivalent results. Until something else breaks, that's how it will stay. (Sorry Win7's Homegroups, you're EOL anyway.)

Adaminion (https://forums.verizon.com/t5/fios-internet/a-few-websites-not-working-with-lan/td-p/921287) has it right ... it's Verizon's, Intel's, Microsoft's, and Google's fault.

I'm *** happy I found this discussion.

-bb

strouli

This was driving me crazy. I can't believe Fios pushed out a changed like this and did not warn all of their customers.

I wasted a ton of time and money trying to fix this. Here is what I ended up doing

I could not figure out for the life of me why from only wired devices my upload speeds to the internet went to 0 (basically the internet would break for me). I tried everything
- replacing ethernet cables to the 48 port Gigabit Ether switch
- changing port on the Fios G3100 router ethernet cable used
- changing port on the 48 port Gigabit switch the ethernet cable used
- rewiring my 2nd switch (an 11 port PoE switch) to go directly to router vs directly to the 48 port Ethernet switch next to it
- bought a new second switch (power of ethernet switch) and replaced it
- rebooted primary 48 port tp-link 48 port gigabit ethernet switch and Fios G3100 router
- rebooted Fios optical network termination (ont)
- bought a brand new 48 port gigabit ethernet switch to see if it was my switch and rewired my entire house to use this new switch

None of this worked. '

Then I remembered on a project 3 years ago a piece of software (mongodb) would not work in Kubernetes if it had IPv6 enabled and the operating system did not have IPv6 at all.

So I disabled IPv6 on the Windows 10 and 11 PCs having this issue and the problem went away.

Then I found this thread and then disabled IPv6 on the WAN on my Fios G3100 router.

Also On this which describes the issue
https://www.intel.com/content/www/us/en/download/19174/27263/disabling-tcp-ipv6-checksum-offload-capability-with-intel-1-10-gbe-controllers.html?

I noticed my Gigabit Ethernet chips on my motherboards are I225 and I211 which are not on the list of affected gigabiit chips but these also have the issue

https://forums.verizon.com/t5/fios-internet/ipv6-causing-0mpbs-upload-speed/td-p/918778
This was exactly my problem!!!! There is a Fios and or Intel Gigabit Ethernet bug!!!! I then went into my router and disabled IPv6 on the Wide Area Network as that should fix this for anyone without a person having to do fancy stuff on their device

This was bothering me for months and today I decided to go all in and try and solve it.

craphead123

I've turned off IPv6 on the network card (Intel I219-LM) and am still having the issue.

Router is a G1100 with firmware 02.03.00.14, it just started happening last week.

IPv6 was already disabled on the G1100.

Not sure what to try next. Rebooting the router seems to help it for a day maybe two days sometimes, but not always.

Krl2

New set box months ago then Google Chrome and Amazon and Youtube started choking.... Tried literally everything. Diagnostics all good... Deleted everything possible on Google thinking that was problem... So much pain.... Amazon not working. Nothing working. As of today, nothing working. No email function, Chrome nonfunctional with anything, not possible to upload files to Chrome email or shared folders... Not possible to send an email. Shared folders not functional... Youtube nonfunctional.... Could not make an order on Amazon... Incredible pain and disruption of life patterns... Then finally at wits end called Verizon support today. After terrible automated assistant firewall ... forced them to put an agent on the line. Great guy. Human. After several diagnostics that all showed everything peachy aok, the fellow suggested disablign ipv6. Suddenly, everthing worked. Lightning fast. Back in business. Long live ipv4.

craphead123

At least that fixed yours, it didn't fix mine, I had to turn Home Network Protection off in the Fios App before min was fixed.

[Deleted User]

In order to keep discussion on the community current, this topic has been locked to prevent new replies. If you have a similar question or issue that you wish to discuss, then please feel free to post a new message on the most relevant board. Thanks!