Clay Aiken Says He Wasn’t ‘Too Nervous’ When He Joined ‘American Idol’ Since the Show ‘Was Not Big’ Yet (Exclusive)

Today, American Idol is one of the most popular music competition series of all time — but back when Clay Aiken was a contestant, it was just getting started.

Speaking with PEOPLE at Project Angel Food’s annual fundraising special, Lead with Love: Going the Distance on June 27, the 47-year-old American Idol alum, who finished in second place on the second season in 2003, reflected on his time on the show and his subsequent career in show business.

Aiken, who opened the event with a performance of the Hercules song “Go the Distance,” tells PEOPLE that “the show was not big when I was on it, before I was on it, so we all went into it kind of thinking this is going to be a fun little thing to do.”

“Nobody thought that anybody was going to … Even Kelly [Clarkson] had not had anything happen for her yet, right? Her album had not even come out when we auditioned,” he continues, referring to the season 1 winner. “So none of us realized that the stakes were there.”

“So yeah, we were nervous because we wouldn’t want to go home. Nobody wants to get cut each time,” Aiken recalls of his time on the show. “But I remember thinking when we found out we were going to be on the lot, we were going to go to the live show — the top 32 — oh, we thought our lives were changed. We were so excited.”

Still, he adds, “We didn’t know what we were getting into, so it was hard to be too nervous because we didn’t know what we could be losing if we lost.”

Asked what it’s like to take a gamble by auditioning for American Idol now, after decades in the public eye, Aiken says, “It’s hard to answer that because it’s been so long.”

In fact, while chatting with PEOPLE, he realized that he has lived more life since leaving the reality show than before he appeared on it.

“So I’m now having a realization. …I am coming up close to the point where I will have lived more of my life post-Idol than pre-Idol,” he says.

“But when I look back at it without and try to close my eyes and try to remember what it was like to be 23 years old coming here, it is surreal, a little bit, to think about,” he admits.

Thinking back on his experience on the show in 2003, he tells PEOPLE, “I mean, CBS isn’t even in the same building. We stayed at the Meridian, which isn’t the Meridian anymore.”

“So much has changed, so it’s hard to know,” he continues, “but it is sort of surreal to think when people say … People still ask me, ‘How much has your life changed since?’ I’m like, ‘Well, I don’t know. It’s been so long.’ But if you look at it objectively, I mean, it’s crazy. It’s crazy to think of the sliding doors thing of what could have happened if I hadn’t shown up in that audition and how life would be different.”

“I’m sitting here, Eric McCormack is there, and I was watching him on Will & Grace the week before [auditioning], right? You know what I mean?” Aiken adds at the Lead with Love event.

McCormack also attended the fundraiser for Project Angel Food, which, per its website, “nourishes the health and spirit of people facing critical and life-threatening illnesses by preparing and delivering medically tailored meals with love, care and dignity.” 

The charity has provided more than 20 million meals to critically ill individuals across Los Angeles County since 1989, according to the Project Angel Food website.