It depends on the situation. With max doors a pair of humans beaming into an double airlock and you immediately expose them to vacuum, they are dead. Mantis will break out if they attack the same door otherwise die, but be weakened and killed in the next room as that goes into vacuum. Just make sure you keep the door behind them open.
If four mantis beam into the engine room of the Kestral Type A, evacuate it, open it to space. They might do one point of engine damage, but they will then be too busy trying to get out of the airless room to do more harm. You then drive them towards the weapons room making sure the back third is spaced as they break through each door. You can then fight them there as they have been heavily weakened or you can open the middle airlocks and continues to flush them to the front of the ship towards the medbay.
It gets real messy if they beam into the front of the ship in the type A as flushing them towards the medbay is a lot more involved and abandoning the pilot seat is last resort in a fire fight.
The key to dealing with boarders to to set it up so that they fight on your terms. Even if you don't kill them, if they came from a ship they will retreat and not come back. Yeah it sucks a little to lose the crew bonus, but it's better than losing the crew or the system. Also remember the air in the ship has a rudimentarily phyisics system. More doors you open, quicker air escapes. To get air back into a room faster, use the open all doors button, but remember to press the close all doors button first then open all and only once as clicking twice will open all doors which will almost immediately space the ship.
Even if you don't space them, doors gives your guys time to get back to the medbay to heal and fight them for another round. Max doors can actually contain a fire long enough for it to burn itself out.
Of course if you have a Mantis team of your own, you can counter board them before they can board you keeping the fight on their ship. Or a preigniter to blow out that system, but they will try to board once the system is repaired and if the boarding party is still alive.