The American team of Cassidy Cox, Paige Gore and Lexi Keller pulled four points ahead in the second end after a tie in the first.
Britain came back to within one point of their opponents in the third end but the USA went on to shoot three tens and two nines to claim the gold medal.
Despite losing out on the top spot, O'Sullivan, Mason and Greenwood were delighted to be travelling home with silverware and already have their sights set on the World Championships in Mexico this October.
"It was great, I found this to be one of the funniest teams I have been on for so many years," O'Sullivan said.
"The weather was awful today but we went out there to have fun and smile.
"We still have to choose our team for the World Championships, but we are hoping to be better in Mexico."
It looked as if the recurve team of Sarah Bettles, Naomi Folkard and Bryony Pitman would add to the medal haul, but they lost out 6-2 to Russia in the bronze-medal match.
The recurvers defeated number two seeds India in the quarter-finals and came within touching distance of the gold-medal match when they took an early lead over Mexico in the semis.
But the Mexicans clawed their way back into the second and third ends, winning them by a point to take the overall victory 5-3.