When you ADVANCE your Bros. Atacks, you attack differently and do more damage to your enemies. The button presses are different combinations too.
For example, for Mario's Chopper Bros. attack, the button combination is A B A (repeatedly). This sends Mario flying into the air and come down spinning his hammer on the enemy.
The ADVANCED version of the move is much harder to perform. The button combo is:
A A -- this sends Luigi into the ground. When he moves under the enemy, press B to make him jump up. Then have Mario press A again to knock Luigi back into the ground, then press B to have him jump up again. Then just repeat till you mess up.
So, it is essentially A A B A B A B A B, etc....