Do it from within the app. I use K-9 Mail instead of the native Mail app so it might be a slightly different menu, but it should be similar to this path: Open Mail, then go to More > Settings > Account Settings > Notifications. Go to New Mail Ringtone and choose Silent. You can also check the Vibrate box if you want that. Hope this helps.