I have not messages but the icon will not go away. I have taken the battery out to try that but no luck.
Turn your phone off...then call your mobile phone number from another phone and leave a message. Turn the phone on and then go to your voicemail and listen to the message. Listen and then delete it; then press the appropriate key to disconnect from voicemail, don't just hang up. That should clear the "voicemail" icon, if not, turn the phone off and then back on.
My icon stays on the screen if I listen to a message then hang up (as Morgan posted above). It disappears once I select either 7 or 9, to delete or save.