Over the course of 100 episodes of Gossip Girl, Penn Badgley has had his share of encounters with fans. But none were quite as intense as his first.

The cast was shooting at New York’s Metropolitan Museum of Art a few days after the show’s premiere, Badgley, 25, recounted to PEOPLE on Saturday at the CW show’s 100th episode party. He wasn’t shooting, but his costars were texting him to come down, as the set was being besieged by fans.

“I tried to come down and wasn’t able to get to the Met because probably, like, 40 or 50 Catholic schoolgirls all swarmed me and pinned me up against a chain-link fence,” Badgley says.

“At first, it was entertaining,” says the actor, who plays Brooklynite Dan Humphrey on the show. “And then they started pulling at my clothes and it was a little aggressive. Then I realized I couldn’t get out, even though they’re all [smaller].”

Luckily, he was saved at the last moment.

“This bodyguard came out of nowhere, reached out like I was holding on to a rock in a raging rapids, and pulled me through it,” he says. “[It] was bizarre.”

While the incident left him a little shaken up, Badgley is still content with his level of fame.

“It means I have access to the world’s most beautiful women, [so] I wouldn’t say that it’s bad,” says the actor, who dated costar Blake Lively for three years and is now seeing Zoe Kravitz.

“That also means you have access to the most volatile, crazy social situations, so you know, sometimes it’s a little nuts,” he adds. “Sometimes it’s a little bizarre. But, ultimately, I would recommend it.”

Source, thanks to @roughrider92!