The fitbit may have to be paired through the fitbit app. I have the Jawbone UP24 and that's how it works.
From the blueant manual:
Pair the S4 with your Phone To use the S4 you need to pair it with your phone. 1. Slide the power switch on the side of the S4 to the ON position. 2. If this is the fi rst time you have used the S4 it will go straight to pairing mode. The voice prompts will start to talk you through the pairing process and the light will fl ash blue. For subsequent pairings, say BlueAnt Speak To Me, wait for the S4 to ask you to Say A Command and then say Pair Me. 3. Follow the voice prompts to pair the S4 with your phone. The pairing code is 0000 (four zeros). Transfer your Phonebook Once the pairing is complete your S4 will attempt to transfer the contacts from your phonebook so that it can announce the names of incoming callers. Note: If your phone prompts you to allow a connection to the S4, you must select “YES” or “OK”
Sounds like you need to get it into pairing mode first.