In the opening scene of Jedi, Luke calmly walks in, uses the Jedi mind-trick to get in to see Jabba and states his request for his friends to be set free. The Luke at the end of Empire would have stormed the fortress and used force as his primary method of rescuing Han and Leia.
Also, it took Luke a year and a half to get around to saving his friends? WTF was he doing all that time? I have zero exposure to the expanded universe, so if it's not readily explained in the 6 movies I'm in the dark.
The scene at Jabba's palace is just Luke putting on a show for Jabba's benefit, hoping to get Han back without a fight, but also getting all the pieces in place for his backup plan. Lando's spent months getting in as a guard, to give you an idea of how much work they've put into it.
If that seems too mature a plan, consider that the Luke at the end of ESB had just learned a painful lesson in rushing in unprepared. He learned that he was the fruit of Darth Vader's loins, lost a hand (along with that youthful sense of immortality all kids have), and didn't even save his friends. They saved
him.
Between movies, they spent some time trying to intercept Boba Fett before he could arrange payment and make delivery to Jabba. When that failed, they were faced with a rescue mission far harder than freeing Leia from the Death Star (the Empire actually helped them infiltrate the station
and escape it).