I drank whole milk for all my life, but started to cut back on calories in grad school and decided to go down to 2%.  I'm now all the way down to 1%, and it tastes fine... although I still like a sip of whole once in a while.  (I totally drink the little cups of half & half at restaurants.)  However, I will never go down to skim milk.  There's a huge gulf of taste and texture between some fat and no fat.