If you have a smartwatch turn off the remote Bluetooth settings. It appears to be waiting for you to speak into your smartwatch microphone. This is why Active edge only worked when I was holding the phone with my LEFT hand, which is the arm where I wear my S3 Frontier. I was too dumb to notice!
settings/ Google/ settings/ Google Assistant/ Phone/ other voice settings/ hands free
Google couldn't have buried this menu better if they tried!