I've only played for a few days, but as far as I can tell Nintendo seems to have done a bangup job on giving the Tennis game some depth.
The direction you swing the Wiimote really does matter, going from low to high on a swing will give the ball some altitude once its been hit, and just oppositely going from high to low will typically cause the ball to either hit the net, or skim right over its surface. This it particularly useful when you're the guy close to the net, as it'll allow you to spike the ball right over the net.
The angle of the Wiimote appears to have an effect on the ball's overall direction when its been hit. Keep the buttons facing up, and the ball will be inclined to go forward. Tilt the Wiimote clockwise, so the buttons are facing the right, and the ball will fly(quite sharply, depending on your angle) in the opposite direction. So if you tilt to the right, it'll fly left, tilt to the left, it'll fly right, etc.
Be sure to be swinging in the appropriate direction when the ball has reached you, if you do a left swing while the ball is coming from your left, you're unlikely to get a succesful hit.
I really wish Nintendo had some more four player games like Tennis on Wii Sports, it's been a great deal of fun for me and my siblings. Good luck!