CR1000A FW version 3.2.0.7 "[FW] IPTABLES [Pkt_Illegal]" Blocks TCP over Static Route
atroph
Enthusiast - Level 2

Got a firmware push last night that appears to have broken any forwarded TCP packets to my static route.

 

Network information:

Local network: 192.168.0.0/24

Default GW: 192.168.0.1 (Verizon CR1000A)

VPN Server: 192.168.0.105 (Linux box w/ OpenVPN Access Server)

Static route: 192.168.10.0/24 GW 192.168.0.105

 

VPN tunnel: 172.27.224.0/20

 

Remote network: 192.168.10.0/24

VPN Client: 192.168.10.10 (another Linux box w/ OpenVPN as a client)

Static route: 192.168.0.0/24 GW 192.168.10.10

 

The FW push:

2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download File Type: 1 Firmware Upgrade Image

2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download URL: https://cpe-ems34.verizon.com/firmware/chr2fa_fw_3.2.0.7.bin

2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download DelaySeconds: 1

2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download Start Time: 2023-02-07T03:45:55

2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download Complete Time: 2023-02-07T03:45:55

2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] TR069: Sending DownloadResponse

2023 Feb 7 03:45:57 info arc_tr69: [TR69.6][ADV] Tr69_DownloadTask Transfer Start Time: 2023-02-07T03:45:57

2023 Feb 7 03:45:57 info arc_tr69: [TR69.6][ADV] Tr69_ExecuteDownload FILEType: 1

2023 Feb 7 03:45:57 info arc_tr69: [TR69.6][ADV] Tr69_ExecuteDownload FileName: chr2fa_fw_3.2.0.7.bin

2023 Feb 7 03:45:57 info arc_tr69: [TR69.6][ADV] Tr69_ExecuteDownload URL=[https://cpe-ems34.verizon.com/firmware/chr2fa_fw_3.2.0.7.bin], FileSize=[0]

2023 Feb 7 03:46:06 info arc_tr69: [TR69.6][ADV] ExecuteDownload:ht_client_send return [61186832], content-len=[61186832]

2023 Feb 7 03:46:06 info arc_tr69: [TR69.6][ADV] S=[2023-02-07T03:45:57], E=[2023-02-07T03:46:06]

2023 Feb 7 03:46:06 info arc_tr69: [TR69.6][ADV] System upgrade /data/chr2fa_fw_3.2.0.7.bin

2023 Feb 7 03:46:06 info arc_tr69: [TR69.6][ADV] sys_openwrt_upgrade

 

And then the errors begin:

2023 Feb 7 03:49:57 warning kernel: [FW] IPTABLES [Pkt_Illegal] IN=br-lan OUT=br-lan MAC=*edit* SRC=192.168.0.51 DST=192.168.10.12 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=7396 DF PROTO=TCP SPT=35650 DPT=554 WINDOW=229 RES=0x00 ACK URGP=0

2023 Feb 7 03:50:57 warning kernel: [FW] IPTABLES [Pkt_Illegal] IN=br-lan OUT=br-lan MAC=*edit* SRC=192.168.0.51 DST=192.168.10.12 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=19341 DF PROTO=TCP SPT=35768 DPT=554 WINDOW=229 RES=0x00 ACK URGP=0

2023 Feb 7 03:51:57 warning kernel: [FW] IPTABLES [Pkt_Illegal] IN=br-lan OUT=br-lan MAC=*edit* SRC=192.168.0.51 DST=192.168.10.12 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=61327 DF PROTO=TCP SPT=35834 DPT=554 WINDOW=229 RES=0x00 ACK URGP=0

 

192.168.0.51 is a local NAS and 192.168.10.12 is a remote camera.

I can ICMP all day long local to remote and vice versa but nothing greater than layer 3 will pass the firewall.

 

Tracing route to 192.168.10.1 over a maximum of 30 hops

1 * <1 ms <1 ms 192.168.0.1 --local router

2 <1 ms <1 ms <1 ms 192.168.0.105 -- VPN gateway

3 92 ms 69 ms 69 ms 172.27.234.3 -- VPN tunnel (172.27.224.0/20 network)

4 143 ms 64 ms 73 ms 192.168.10.1 -- remote router

Trace complete.

 

Talked to "advanced support" and the tech suggested to disable IPv6 (it was already) and set firewall to the lowest setting. No change.

Last suggestion is to factory reset the router and see if that fixes it... I'll write down all the settings and try that later. 

 

I find it quite interesting that IPTABLES is filtering packets to a static route on a private IP space (on br-lan). 

 

Worst case I'll replace this box and install my own equipment. 

Re: CR1000A FW version 3.2.0.7 "[FW] IPTABLES [Pkt_Illegal]" Blocks TCP over Static Route
Cang_Household
Community Leader
Community Leader

This might have to do with static routing. Please see PM for further instructions.

Re: CR1000A FW version 3.2.0.7 "[FW] IPTABLES [Pkt_Illegal]" Blocks TCP over Static Route
filonovd
Enthusiast - Level 2


Cang_Household -

Why don't you share your instructions publicly? I have the very same issue. The whole office was stuck without access to the satelite office today. All because of the FW upgrade on a router.

0 Likes
Re: CR1000A FW version 3.2.0.7 "[FW] IPTABLES [Pkt_Illegal]" Blocks TCP over Static Route
filonovd
Enthusiast - Level 2

Hi  atroph -

 just in case other option suggested wouldn't work, I got the following workaround working fine in my case:

 

1) set all  computers  but the VPN server default GW to 192.168.0.105

2) Keep VPN server's default GW at 192.168.0.1

3) Profit.

 

This way local computers will go to VPN server first, if they need to go over tunnel they will go from there and if they need Internet then they will be router further to Verizon.
It's an additional point of failure. If VPN server goes down then you lose internet access as well.

The other option is just add static routes to each local device. which is a pain and not always possible.

 

Fil

Re: CR1000A FW version 3.2.0.7 "[FW] IPTABLES [Pkt_Illegal]" Blocks TCP over Static Route
atroph
Enthusiast - Level 2

Thanks for the workaround!

I chatted w/ Cang in PM and hopefully the engineers can tap in to my router and see what the problem is. 

0 Likes
Re: CR1000A FW version 3.2.0.7 "[FW] IPTABLES [Pkt_Illegal]" Blocks TCP over Static Route
DuplicateUser
Enthusiast - Level 2

Can I ask where you found this firmware update info? I also experienced a problem starting 2/2 that I was convinced was from a firmware update. I spoke with 5 different 'support' people and no one had any access to when firmware updates were pushed, so no one could help. Their answer was to replace the router, which of course did squat.

0 Likes
Re: CR1000A FW version 3.2.0.7 "[FW] IPTABLES [Pkt_Illegal]" Blocks TCP over Static Route
atroph
Enthusiast - Level 2

What I provided up above is logs from the device itself. 

You will find firmware updates in:

Advanced > Diagnostics & Monitoring > System Logging > System Log

I was fortunate enough to have a rough timestamp of when my remote connection went down so I could sift through the log data and see that a FW update was pushed at that time. 

 

Check your FW version at:

Advanced > System > System Status - Router - Firmware Version

It is probably safe to say that if you have 3.2.0.7 you got a FW update at some point unless they are shipping with that now. 

0 Likes
Re: CR1000A FW version 3.2.0.7 "[FW] IPTABLES [Pkt_Illegal]" Blocks TCP over Static Route
DuplicateUser
Enthusiast - Level 2

Thanks for the info. It's amazing that of the 5 different support staff I spoke with no one had any idea where to find this and they had no access to it on their systems as well. It's pathetic. In my case they ended up sending me a new router. I think I installed it around the 6th or 7th. Unfortunately, none of the logs go back past yesterday, even if I change the Options to a custom date range. I was just curious to see if this router updated itself right away when I installed it.  

Is the person who responded to you an actual verizon employee? I'm trying to get someone to take a bug report on what they broke.

Re: CR1000A FW version 3.2.0.7 "[FW] IPTABLES [Pkt_Illegal]" Blocks TCP over Static Route
atroph
Enthusiast - Level 2

Im not sure if Cang is an employee or not. The likelihood is high IMHO. 

 

As for the firmware I had an older quantum router in the closet as a dumb AP that I put as the main. As soon as I gave it a WAN connection it got updated. 

 

Either way if you have the version I posted it is the latest and greatest. 

Also discovered another potential bug. This device on all of the various menu screens etc doesn't not show that I am connected to the WAN. It pulls and IP and seems to work just fine but in the status screen is shows connecting. 

Using Ethernet to ONT and IPv4 only, all IPv6 is disabled. 

0 Likes
Re: CR1000A FW version 3.2.0.7 "[FW] IPTABLES [Pkt_Illegal]" Blocks TCP over Static Route
FunnyFarm2020
Newbie

Did the Replacement router work? Or are you still having issues. 

0 Likes