I often watch cartoons muted because those screechy japanese voices can get on your nerves sometimes (though muting will also kill the sound effects). A cartoon loses something if you mute it because it doesn't attempt to bring the dialogue across in another way. A game can be played muted without losing anything as long as it's not made by Ubisoft. Hell, I play all handheld games muted.
Seriously, hard or not, you're not defusing bombs, working in a war zone or otherwise risking your life. That's what deserves a LOT of money.
There are MANY jobs that require no qualification you can take instead, nobody forces you to work exclusively as a voice actor. If VA doesn't pay the bills learn to flip burgers. Better payment, better treatment, better job stability. Low job security and payment is what you get for taking a highly competitive job like that.