Rovers manager Darrell Clarke took the bold move of replacing goalkeeper Will Puddy with Steve Mildenhall at the end of extra time but the former Mariners stopper was not needed as Jon-Paul Pittman blazed his penalty over.
With every other penalty put away after the sides could not be separated following a 1-1 draw in 120 minutes, Mansell had the opportunity to win it for Rovers and he made no mistake to spark wild scenes at Wembley.
The Mariners needed more than a slice of fortune to take an early advantage as Lenell John-Lewis' shot rebounded off Puddy and onto his face before rolling across the line.
Barcelona consider Martinez as Koeman replacement - Paper Round
John-Lewis saw another shot saved by the Rovers stopper, who was then lucky to be shown only a yellow card after charging out of his box and handling Ollie Palmer's goal-bound effort.
Arnold curled a 25-yard effort just wide as Grimsby continued to turn the screw, but it was Rovers who got the all-important second goal.
Aristote Nsiala sliced a clearance over his own bar and from the resultant corner Ellis Harrison punished more bad defending by firing the ball home from close range.
Grimsby Town's Jon-Paul Pitman stands dejected after missing his spot kick
Image credit: Press Association
The momentum was suddenly with Rovers and they thought they had a penalty when Taylor rounded McKeown and went to ground after capitalising on a mistake by Shaun Pearson, only for the referee to blow for a dive.
The Pirates had the first chance of the second half as Taylor shot over after good work by Jake Gosling, but the opportunities dried up with both sides desperate not to make a mistake.
Danny Parslow saw a cross flash across the Rovers six-yard box and Mansell sliced a shot over at the other end late on in normal time.
Extra-time was every bit as cagey as the second period, with Chris Lines' looping header and Pittman's cleared cut-back the only chances of note.
With Rovers leading 3-2 in the shootout, Pittman sent his spot-kick high over the bar before Mansell settled the contest and booked the Pirates' return to League Two.
Carabao Cup: City down Wycombe as Everton and Watford knocked out
Women's World Cup Qualification UEFA
England hit 10 as they crush Luxembourg