My issue is resolved. I'm appending here to help others with the same issue that are using log records to search for clues. Long (painful) story made short ...
During the past couple of months I had been experiencing intermittent FIOS internet outages. Somedays there would be no outage, on other days there would be multiple outages. Outages would range from minutes to hours. I have access to three Verizon routers, outages would occur on all three.
On the G1100 the log records indicating failure included records of the form:
Jan 11 20:44:31 2022 local0.info<134> dhcpcd[3430]: eth1: carrier lost
Jan 11 14:12:31 2022 local1.err<139> named[8498]: client @0x1cc0f10 192.168.1.16#36716 (www.google.com😞 view internal-clients: query failed (SERVFAIL) for www.google.com/IN/A at query.c:8629
On the G3100 the log records indicating failure included records of the form:
2022 Feb 8 12:18:41 warning arc_cloud: [CLOUD.4][ADV] ping WAN failed, try again.
2022 Feb 9 02:10:07 err arc_cloud: [CLOUD.3][ADV] curl_easy_perform() failed:(6) Error.
2022 Feb 9 02:38:30 warning arc_cloud: [CLOUD.4][ADV] cloud test link failed, try again
2022 Feb 9 02:54:45 info wan_detection: [WAN.6][WDHCP] eth4 arping to 100.2.248.1 fail.
Using diagnostics on both routers I could not ping the gateway during an outage.
Root cause: In my neighborhood Verizon prefers to mount the ONT outside, drill a small hole, and fabricate a ethernet cable to connect the ONT to a Verizon terminus inside the house. Turns out the outside section of the Verizon cable was bad. After the cable was replaced everything has worked fine.
Observations from my experience:
- Keep a spare router around for testing.
- When testing for intermittent errors always connect directly to the ONT, don't trust any ethernet cable in the path (my fault).
I found working with Verizon 1st level phone support took a lot of time and was hit-or-miss, mostly miss, in general all they want to do is remotely reset the router and when that fails place the blame on customer. Seems to me that Verizon should be able to detect failures between their router and their gateway. Also, it bugs me that neither level one phone support nor the field technician were interested in analyzing the (Verizon) router logs.