That doesn't sound very functional, Aesexton9383! Let's get this figured out. How long have you had your Note 4? When exactly did this start happening? Has the phone been dropped or wet at all? What happens if you use Headphones or Bluetooth? Do you happen to have a case on the phone? If so, please try removing it to see if there's a difference. Please elaborate so we can better assist you.
Follow us on Twitter @VZWSupport
Sorry if my previous post wasn't specific. My followup might not add much.
I think the Note4 has 3 microphones. Probably the one at the top of the phone is for noise cancellation / voice enhancement if you have that feature enabled. Don't know if it's active in speakerphone mode. It might be, and maybe that's how your callers are hearing you.
Maybe you could try running the voice recorder app and seeing if it picks up your voice when you cover each of the three mic locations with a finger through experimentation.
I'm thinking your problem is hardware related. You could call Samsung and ask to speak with technical support, and ask them on the Note4 what microphones are in use when speakerphone is enabled and when it's not.