maybe when you're transporting it, the buttons are being pressed, which also drains the batteries because its trying to detect the Wii.
The remote battery lasts a pretty long time when using only motion sensing (60 hours), continuous use of the pointer, like in a game like trauma center uses up a lot more power (30 hours). But 5 hours? I dunno man. I found the batteries that came with the remote to last a really short time. So I guess it depends on brand too.
EDIT: Another theory. Maybe you need to get Elebits (Or play more of it, if you already have it). Your Remote is trying to tell you something.