The visuals are worse in a lot of ways. The art style seems lacking. The character creator actually seems more limited, despite giving you a myriad of sliders. The graphics are VERY dark. The font throughout is VERY tiny, like it was in GH: World Tour. The microphone has numerous problems with recognizing cowbells and initiating overdrive. Every single person has to have their own character account set up to really play right, and assigning stand-ins is confusing. The list goes on.
From what I can tell, the only things they actually improved were: Added harmonies for vocals, finally introduced SDHC support, created a nice "overshell" to reduce the amount of backtracking through menus, lifted the restrictions on how many similar instruments are played at once. I have no intentions of purchasing a keyboard, so I can't tell you how that works... I'm going to say not too well though, simply because of the lack of truly keyboard-centric music worth listening to.
In a lot of ways, RB3 could have benefited from ripping off GH5's interface more. The running "jukebox" and mini-challenges worked so well there.