CR1000A FW version 3.2.0.7 "[FW] IPTABLES [Pkt_Illegal]" Blocks TCP over Static Route

[atroph]

Got a firmware push last night that appears to have broken any forwarded TCP packets to my static route.

Network information:

Local network: 192.168.0.0/24

Default GW: 192.168.0.1 (Verizon CR1000A)

VPN Server: 192.168.0.105 (Linux box w/ OpenVPN Access Server)

Static route: 192.168.10.0/24 GW 192.168.0.105

VPN tunnel: 172.27.224.0/20

Remote network: 192.168.10.0/24

VPN Client: 192.168.10.10 (another Linux box w/ OpenVPN as a client)

Static route: 192.168.0.0/24 GW 192.168.10.10

The FW push:

2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download File Type: 1 Firmware Upgrade Image

2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download URL: https://cpe-ems34.verizon.com/firmware/chr2fa_fw_3.2.0.7.bin

2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download DelaySeconds: 1

2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download Start Time: 2023-02-07T03:45:55

2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download Complete Time: 2023-02-07T03:45:55

2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] TR069: Sending DownloadResponse

2023 Feb 7 03:45:57 info arc_tr69: [TR69.6][ADV] Tr69_DownloadTask Transfer Start Time: 2023-02-07T03:45:57

2023 Feb 7 03:45:57 info arc_tr69: [TR69.6][ADV] Tr69_ExecuteDownload FILEType: 1

2023 Feb 7 03:45:57 info arc_tr69: [TR69.6][ADV] Tr69_ExecuteDownload FileName: chr2fa_fw_3.2.0.7.bin

2023 Feb 7 03:45:57 info arc_tr69: [TR69.6][ADV] Tr69_ExecuteDownload URL=[https://cpe-ems34.verizon.com/firmware/chr2fa_fw_3.2.0.7.bin], FileSize=[0]

2023 Feb 7 03:46:06 info arc_tr69: [TR69.6][ADV] ExecuteDownload:ht_client_send return [61186832], content-len=[61186832]

2023 Feb 7 03:46:06 info arc_tr69: [TR69.6][ADV] S=[2023-02-07T03:45:57], E=[2023-02-07T03:46:06]

2023 Feb 7 03:46:06 info arc_tr69: [TR69.6][ADV] System upgrade /data/chr2fa_fw_3.2.0.7.bin

2023 Feb 7 03:46:06 info arc_tr69: [TR69.6][ADV] sys_openwrt_upgrade

And then the errors begin:

2023 Feb 7 03:49:57 warning kernel: [FW] IPTABLES [Pkt_Illegal] IN=br-lan OUT=br-lan MAC=*edit* SRC=192.168.0.51 DST=192.168.10.12 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=7396 DF PROTO=TCP SPT=35650 DPT=554 WINDOW=229 RES=0x00 ACK URGP=0

2023 Feb 7 03:50:57 warning kernel: [FW] IPTABLES [Pkt_Illegal] IN=br-lan OUT=br-lan MAC=*edit* SRC=192.168.0.51 DST=192.168.10.12 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=19341 DF PROTO=TCP SPT=35768 DPT=554 WINDOW=229 RES=0x00 ACK URGP=0

2023 Feb 7 03:51:57 warning kernel: [FW] IPTABLES [Pkt_Illegal] IN=br-lan OUT=br-lan MAC=*edit* SRC=192.168.0.51 DST=192.168.10.12 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=61327 DF PROTO=TCP SPT=35834 DPT=554 WINDOW=229 RES=0x00 ACK URGP=0

192.168.0.51 is a local NAS and 192.168.10.12 is a remote camera.

I can ICMP all day long local to remote and vice versa but nothing greater than layer 3 will pass the firewall.

Tracing route to 192.168.10.1 over a maximum of 30 hops

1 * <1 ms <1 ms 192.168.0.1 --local router

2 <1 ms <1 ms <1 ms 192.168.0.105 -- VPN gateway

3 92 ms 69 ms 69 ms 172.27.234.3 -- VPN tunnel (172.27.224.0/20 network)

4 143 ms 64 ms 73 ms 192.168.10.1 -- remote router

Trace complete.

Talked to "advanced support" and the tech suggested to disable IPv6 (it was already) and set firewall to the lowest setting. No change.

Last suggestion is to factory reset the router and see if that fixes it... I'll write down all the settings and try that later. 

I find it quite interesting that IPTABLES is filtering packets to a static route on a private IP space (on br-lan). 

Worst case I'll replace this box and install my own equipment. 

[atroph]

Thanks for the workaround!

I chatted w/ Cang in PM and hopefully the engineers can tap in to my router and see what the problem is. 

[DuplicateUser]

Can I ask where you found this firmware update info? I also experienced a problem starting 2/2 that I was convinced was from a firmware update. I spoke with 5 different 'support' people and no one had any access to when firmware updates were pushed, so no one could help. Their answer was to replace the router, which of course did squat.

[atroph]

What I provided up above is logs from the device itself. 

You will find firmware updates in:

Advanced > Diagnostics & Monitoring > System Logging > System Log

I was fortunate enough to have a rough timestamp of when my remote connection went down so I could sift through the log data and see that a FW update was pushed at that time. 

Check your FW version at:

Advanced > System > System Status - Router - Firmware Version

It is probably safe to say that if you have 3.2.0.7 you got a FW update at some point unless they are shipping with that now. 

[DuplicateUser]

Thanks for the info. It's amazing that of the 5 different support staff I spoke with no one had any idea where to find this and they had no access to it on their systems as well. It's pathetic. In my case they ended up sending me a new router. I think I installed it around the 6th or 7th. Unfortunately, none of the logs go back past yesterday, even if I change the Options to a custom date range. I was just curious to see if this router updated itself right away when I installed it.  

Is the person who responded to you an actual verizon employee? I'm trying to get someone to take a bug report on what they broke.

[atroph]

Im not sure if Cang is an employee or not. The likelihood is high IMHO. 

As for the firmware I had an older quantum router in the closet as a dumb AP that I put as the main. As soon as I gave it a WAN connection it got updated. 

Either way if you have the version I posted it is the latest and greatest. 

Also discovered another potential bug. This device on all of the various menu screens etc doesn't not show that I am connected to the WAN. It pulls and IP and seems to work just fine but in the status screen is shows connecting. 

Using Ethernet to ONT and IPv4 only, all IPv6 is disabled. 

[FunnyFarm2020]

Has anyone found a solution to this issue? or has a new router resolved it? Sounds like we are waiting for a firmware update or hotfix? 

Thank you all 

[FunnyFarm2020]

Did the Replacement router work? Or are you still having issues. 

[DuplicateUser]

The replacement router did not work and I knew it wouldn't. I accepted it because I knew they wouldn't continue to troubleshoot if I didn't. The bottom line is that they have NO ONE that is qualified to handle firmware issues. They introduced a significant bug in that last release and out of 5 support people I spoke with, no one knew what to do about it. 

[FunnyFarm2020]

We have been experiencing network drops to random devices throughout the house. Investigating I find this error on the firewall pertaining to IPTABLES. My CR1000A Router is on 3.2.0.7 firmware currently. I tried to roll back my firmware to the previous version available, but the router will not take it. 

I have rebooted the router, Disabled IPV6, and do not know what would be next. Any thoughts? 

Here is the error. 

[FW] IPTABLES [Pkt_Illegal] IN=br-lan OUT=eth1 MAC=78:67:0e:59:fb:62:d4:6a:91:b0:df:c5:08:00 src=192.168.2.253 DST=3.213.172.111 LEN=993 TOS=0x00 PREC=0x00 TTL=62 ID=8300 DF PROTO=TCP SPT=55744 DPT=443 WINDOW=1365 RES=0x00 ACK PSH FIN URGP=0 MARK=0x262