Got a firmware push last night that appears to have broken any forwarded TCP packets to my static route.
Network information:
Local network: 192.168.0.0/24
Default GW: 192.168.0.1 (Verizon CR1000A)
VPN Server: 192.168.0.105 (Linux box w/ OpenVPN Access Server)
Static route: 192.168.10.0/24 GW 192.168.0.105
VPN tunnel: 172.27.224.0/20
Remote network: 192.168.10.0/24
VPN Client: 192.168.10.10 (another Linux box w/ OpenVPN as a client)
Static route: 192.168.0.0/24 GW 192.168.10.10
The FW push:
2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download File Type: 1 Firmware Upgrade Image
2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download URL: https://cpe-ems34.verizon.com/firmware/chr2fa_fw_3.2.0.7.bin
2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download DelaySeconds: 1
2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download Start Time: 2023-02-07T03:45:55
2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] Tr69Rpcmethod_Download Complete Time: 2023-02-07T03:45:55
2023 Feb 7 03:45:55 info arc_tr69: [TR69.6][ADV] TR069: Sending DownloadResponse
2023 Feb 7 03:45:57 info arc_tr69: [TR69.6][ADV] Tr69_DownloadTask Transfer Start Time: 2023-02-07T03:45:57
2023 Feb 7 03:45:57 info arc_tr69: [TR69.6][ADV] Tr69_ExecuteDownload FILEType: 1
2023 Feb 7 03:45:57 info arc_tr69: [TR69.6][ADV] Tr69_ExecuteDownload FileName: chr2fa_fw_3.2.0.7.bin
2023 Feb 7 03:45:57 info arc_tr69: [TR69.6][ADV] Tr69_ExecuteDownload URL=[https://cpe-ems34.verizon.com/firmware/chr2fa_fw_3.2.0.7.bin], FileSize=[0]
2023 Feb 7 03:46:06 info arc_tr69: [TR69.6][ADV] ExecuteDownload:ht_client_send return [61186832], content-len=[61186832]
2023 Feb 7 03:46:06 info arc_tr69: [TR69.6][ADV] S=[2023-02-07T03:45:57], E=[2023-02-07T03:46:06]
2023 Feb 7 03:46:06 info arc_tr69: [TR69.6][ADV] System upgrade /data/chr2fa_fw_3.2.0.7.bin
2023 Feb 7 03:46:06 info arc_tr69: [TR69.6][ADV] sys_openwrt_upgrade
And then the errors begin:
2023 Feb 7 03:49:57 warning kernel: [FW] IPTABLES [Pkt_Illegal] IN=br-lan OUT=br-lan MAC=*edit* SRC=192.168.0.51 DST=192.168.10.12 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=7396 DF PROTO=TCP SPT=35650 DPT=554 WINDOW=229 RES=0x00 ACK URGP=0
2023 Feb 7 03:50:57 warning kernel: [FW] IPTABLES [Pkt_Illegal] IN=br-lan OUT=br-lan MAC=*edit* SRC=192.168.0.51 DST=192.168.10.12 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=19341 DF PROTO=TCP SPT=35768 DPT=554 WINDOW=229 RES=0x00 ACK URGP=0
2023 Feb 7 03:51:57 warning kernel: [FW] IPTABLES [Pkt_Illegal] IN=br-lan OUT=br-lan MAC=*edit* SRC=192.168.0.51 DST=192.168.10.12 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=61327 DF PROTO=TCP SPT=35834 DPT=554 WINDOW=229 RES=0x00 ACK URGP=0
192.168.0.51 is a local NAS and 192.168.10.12 is a remote camera.
I can ICMP all day long local to remote and vice versa but nothing greater than layer 3 will pass the firewall.
Tracing route to 192.168.10.1 over a maximum of 30 hops
1 * <1 ms <1 ms 192.168.0.1 --local router
2 <1 ms <1 ms <1 ms 192.168.0.105 -- VPN gateway
3 92 ms 69 ms 69 ms 172.27.234.3 -- VPN tunnel (172.27.224.0/20 network)
4 143 ms 64 ms 73 ms 192.168.10.1 -- remote router
Trace complete.
Talked to "advanced support" and the tech suggested to disable IPv6 (it was already) and set firewall to the lowest setting. No change.
Last suggestion is to factory reset the router and see if that fixes it... I'll write down all the settings and try that later.
I find it quite interesting that IPTABLES is filtering packets to a static route on a private IP space (on br-lan).
Worst case I'll replace this box and install my own equipment.