Got a firmware push last night that appears to have broken any forwarded TCP packets to my static route.
Network information:
Local network: 192.168.0.0/24
Default GW: 192.168.0.1 (Verizon CR1000A)
VPN Server: 192.168.0.105 (Linux box w/ OpenVPN Access Server)
Static route: 192.168.10.0/24 GW 192.168.0.105
VPN tunnel: 172.27.224.0/20
Remote network: 192.168.10.0/24
VPN Client: 192.168.10.10 (another Linux box w/ OpenVPN as a client)
Static route: 192.168.0.0/24 GW 192.168.10.10
The FW push:
2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download File Type: 1 Firmware Upgrade Image
2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download URL: https://cpe-ems34.verizon.com/firmware/chr2fa_fw_3.2.0.7.bin
2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download DelaySeconds: 1
2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download Start Time: 2023-02-07T03:45:55
2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download Complete Time: 2023-02-07T03:45:55
2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] TR069: Sending DownloadResponse
2023 Feb 7 03:45:57 info arc_tr69: [TR69.6][ADV] Tr69_DownloadTask Transfer Start Time: 2023-02-07T03:45:57
2023 Feb 7 03:45:57 info arc_tr69: [TR69.6][ADV] Tr69_ExecuteDownload FILEType: 1
2023 Feb 7 03:45:57 info arc_tr69: [TR69.6][ADV] Tr69_ExecuteDownload FileName: chr2fa_fw_3.2.0.7.bin
2023 Feb 7 03:45:57 info arc_tr69: [TR69.6][ADV] Tr69_ExecuteDownload URL=[https://cpe-ems34.verizon.com/firmware/chr2fa_fw_3.2.0.7.bin], FileSize=[0]
2023 Feb 7 03:46:06 info arc_tr69: [TR69.6][ADV] ExecuteDownload:ht_client_send return [61186832], content-len=[61186832]
2023 Feb 7 03:46:06 info arc_tr69: [TR69.6][ADV] S=[2023-02-07T03:45:57], E=[2023-02-07T03:46:06]
2023 Feb 7 03:46:06 info arc_tr69: [TR69.6][ADV] System upgrade /data/chr2fa_fw_3.2.0.7.bin
2023 Feb 7 03:46:06 info arc_tr69: [TR69.6][ADV] sys_openwrt_upgrade
And then the errors begin:
2023 Feb 7 03:49:57 warning kernel: [FW] IPTABLES [Pkt_Illegal] IN=br-lan OUT=br-lan MAC=*edit* SRC=192.168.0.51 DST=192.168.10.12 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=7396 DF PROTO=TCP SPT=35650 DPT=554 WINDOW=229 RES=0x00 ACK URGP=0
2023 Feb 7 03:50:57 warning kernel: [FW] IPTABLES [Pkt_Illegal] IN=br-lan OUT=br-lan MAC=*edit* SRC=192.168.0.51 DST=192.168.10.12 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=19341 DF PROTO=TCP SPT=35768 DPT=554 WINDOW=229 RES=0x00 ACK URGP=0
2023 Feb 7 03:51:57 warning kernel: [FW] IPTABLES [Pkt_Illegal] IN=br-lan OUT=br-lan MAC=*edit* SRC=192.168.0.51 DST=192.168.10.12 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=61327 DF PROTO=TCP SPT=35834 DPT=554 WINDOW=229 RES=0x00 ACK URGP=0
192.168.0.51 is a local NAS and 192.168.10.12 is a remote camera.
I can ICMP all day long local to remote and vice versa but nothing greater than layer 3 will pass the firewall.
Tracing route to 192.168.10.1 over a maximum of 30 hops
1 * <1 ms <1 ms 192.168.0.1 --local router
2 <1 ms <1 ms <1 ms 192.168.0.105 -- VPN gateway
3 92 ms 69 ms 69 ms 172.27.234.3 -- VPN tunnel (172.27.224.0/20 network)
4 143 ms 64 ms 73 ms 192.168.10.1 -- remote router
Trace complete.
Talked to "advanced support" and the tech suggested to disable IPv6 (it was already) and set firewall to the lowest setting. No change.
Last suggestion is to factory reset the router and see if that fixes it... I'll write down all the settings and try that later.
I find it quite interesting that IPTABLES is filtering packets to a static route on a private IP space (on br-lan).
Worst case I'll replace this box and install my own equipment.
This might have to do with static routing. Please see PM for further instructions.
Cang_Household -
Why don't you share your instructions publicly? I have the very same issue. The whole office was stuck without access to the satelite office today. All because of the FW upgrade on a router.
I am having the same issue. Can you share your solution publically?
There should have been a firmware push to revert back to a working previous version. Unless this firmware somehow also prevents a new firmware to be downloaded, which is a bigger problem and requires very-involved troubleshooting.
It works. Thank you for letting us know!
I guess I spoke too soon. Something is still not working right. Don't have time atm to investigate, had to revert back to using special routing table on the end devices. ๐
I changed my static IP to dynamic IP and rebooted the router. It all works now.
Static IPs and Static Routes are two separate things. Previously we saw the 3.2.x firmware broke the static routing per customer reports. As stated earlier, this has been addressed through an automatic firmware rollback. Static IPs were not the subject of the firmware issue.
Could you be more clear on what exactly is the "static" referring to?
Not sure how this is relevant to my comments by I wasn't using any static IPs other than for tor the dhcp server (in my case dhcp server is not the router). But the router has some static routes that don't work as expected. Some tcp packets were still blocked by the router's firewall.
Sorry don't have much info on hand. Don't have time for a deep investigation atm
Fil
I was replying to @rsheftel 's comment about static IP.
But still, @filonovd , since you mentioned the DHCP server in the context of static IPs, why are DHCP servers relevant to static IPs? Static IPs are IPs that still exist on the hosts even in the absence of a DHCP server. Not address lease reservations. Reservations are not static IPs. They will go down with the DHCP server.
If you don't have time for the investigation, you could supply me with your router's serial number through PM so I could ask the engineering to remote into your router to take a look.
Hi -
Yes, I know Static IPs have nothing to do with static routes. The reason I mentioned DHCP server is that you have to use static IP for the DHCP server itself.
Two little things to add -
1) ICMP (ping) goes through no problem. So it's not routing that is causing the problem. It's the firewall that blocks TCP
2) some systems (non-windows) seem to work. So it might be something windows -specific, that is added to the packet and makes router's firewall unhappy.
I will PM you my routers's serial but am not sure what engineers can do without me troubleshooting it at the same time.
Why would a DHCP server need a static IP? DHCP Discover is sent through broadcast.
To my understanding Windows machines do not respond to cross-subnet pings by default, unless it is joined to a domain.
Engineering could SSH into your router and do everything you could do in the web GUI and more. They can also review the logs and the connected devices on your network.
Because DHCP server needs an IP address to start and before it starts there's no DHCP server to offer DHCP address to the server ๐
am not sure how ping to windows is relevant. In our setup ping actually works, tcp (ssh, for example) doesn't.
I have having similar errors in my CR1000A router.
[FW] IPTABLES [Pkt_Illegal] IN=eth1 OUT= MAC=[redacted for privacy] src=108.xxx.2x.xx DST=100.xx.xx2.xx4 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=0 DF PROTO=TCP SPT=443 DPT=60862 WINDOW=0 RES=0x00 RST URGP=0
To the best of my knowledge, I'm not using any static routes. All I know is that this router has more drops in it than the 5 year old one it replaced.
Any suggestions on how I can get rid of these errors?
Same problem on my CR1000B. Port forwarding is suddenly totally broken...firewall log is full of "[FW] IPTABLES [Pkt_Illegal]" errors. I've tried every fix suggested here and more with no luck. I'm running firmware 3.1.1.20. How can I make the router check for a firmware update?
Could you define what is โsuddenly,โ did it break overnight after a firmware refresh or some sort?
DM sent
Well, my forwarded ports just decided to start working again today. I didn't reboot the router or touch anything else since yesterday ๐. Sorry for the noise I guess.
I am having a similar firewall log events.
I am only able to go back to the end of the previous month, so March 31, 2024.
Every few minutes there's a new log entry and they all appear like this:
[FW] IPTABLES [Pkt_Illegal] IN=br-lan OUT=eth1 MAC= src=192.168.1.235 DST=100.26.12.99 LEN=40 TOS=0x00 PREC=0x00 TTL=127 ID=6934 DF PROTO=TCP SPT=5253 DPT=443 WINDOW=0 RES=0x00 ACK RST URGP=0 MARK=0x262
With different local source IPs and different destination IPs, inbound and outbound connections.
I can't figure out what is happening.
CR1000A, no static ips or port forwarding enabled, upnp enabled.
Can anyone assist?
Hi atroph -
just in case other option suggested wouldn't work, I got the following workaround working fine in my case:
1) set all computers but the VPN server default GW to 192.168.0.105
2) Keep VPN server's default GW at 192.168.0.1
3) Profit.
This way local computers will go to VPN server first, if they need to go over tunnel they will go from there and if they need Internet then they will be router further to Verizon.
It's an additional point of failure. If VPN server goes down then you lose internet access as well.
The other option is just add static routes to each local device. which is a pain and not always possible.
Fil
Start a new topic or ask questions in the open forum.
