For me, the fear was always about being seen as immature, in that I'd be seen as one of those people who obsessed over violence. My greater interest is for platforming, action-oriented games, cartoon-themed games moreso than realism (though I still like my Battlefield, and Silent Hill 3). But some people hear "video games" and think in terms of guns, violence, etc.
I made an effort to show my parents and grandmother innovative games like Donkey Kong Jungle Beat and Katamari Damacy, to show them that games are often light-hearted, playful and fun, unlike the violent stereotype the media loves to put on them.
Of course, the constant sound of gunfire coming from my younger brother's bedroom doesn't help the image any. Thank goodness he's open-minded as well, and also enjoys his Marios, Sonics, etc. It's just that his greater interests lean towards FPSs and MMORPGs.
The problem that I have right now is not only that "gamer" can yield images of the Grand Theft Auto series, but also that gamers tend to refer to such immature, juvenile antics as "mature", "sophisticated", and other such meaningless buzzwords to try to make themselves seem more adult, but instead have the opposite effect to the outside world, kind of like what a 12 year old who smokes would look like to an adult.