Gabrielle Unionisn’t the onlyAmerica’s Got Talentemployee who had grievances withjudge and executive producerSimon Cowell.

In the wake of Union’s controversial andunexpected firing from the NBC reality competition series, a source tells PEOPLE that Cowell, 60, “runs the show” and gets what he asks for a majority of the time.

“It’s true thatAmerica’s Got Talentis a boy’s club. Simon runs the show and as long as he’s there, he will continue to do so,” says the source.

PEOPLE is out to Cowell’s rep for comment.

On Monday,Howard Stern,who previously served as a judge onAGTfor four seasons from 2012 through 2015, hinted at Cowell’s allegedly questionable behavior on the show, both slamming and accusing him of having “orchestrated” Union’s firing.

“How is it that Simon Cowell has orchestrated this?” said Stern, 65. “He sets it up that the men stay, no matter how ugly they are, no matter how old they are, no matter how fat they are, no matter how talentless they are.”

“But what he manages to do on all his shows is he constantly replaces the hot chicks with hotter chicks and younger chicks. Which is so obvious,” Stern added.

A rep for Cowell did not immediately respond to PEOPLE’s request for comment regarding Stern’s remarks.

On Sunday, Syco Entertainment, which producesAGT,released a joint statement with NBC and production company Fremantle. “We remain committed to ensuring a respectful workplace for all employees and take very seriously any questions about workplace culture,” they said.

Trae Patton/NBC

Howie Mandel, Gabrielle Union, Julianne Hough, Simon Cowell

“We are working with Ms. Union through her representatives to hear more about her concerns, following which we will take whatever next steps may be appropriate,” the statement concluded.

Union and Hough, 31, were also both subject to “excessive notes” on their physical appearance, sources alleged inVariety‘s report.

Representatives for Hough, Union and Leno did not respond to PEOPLE’s request for comment.

BlogLove B Scottwas first to report about complaints Union had allegedly made prior to her departure, with an NBC source saying that she had spoken out about “problematic” situations on set.

Frazer Harrison/Getty Images; Tibrina Hobson/WireImage

Gabrielle Union, Simon Cowell

Sources on the series told the outlet that Leno allegedly joked the dogs looked like something one would find “on the menu at a Korean restaurant,” alluding to the stereotype that all Asian people eat dog meat.

In another instance, sources toldVarietythat Union found herself again speaking up against racial insensitivity when a white male contestant auditioned with an act where he portrayed people of color and impersonated singers through multiple quick costume changes.

During his performance, the man impersonatedBeyoncéand, according toVariety‘s sources, emerged with black hands. The audition didn’t sit well with Union and she allegedly addressed the matter with a segment producer and an executive producer, explaining that the audience should not have to watch something so “racially insensitive.”

After the actress requested that they remove the audition from the show, the producers allegedly told her, “We’ll take it out later.” Like Leno’s dog joke, the contestant’s audition was also reportedly cut from the show.

Variety‘s report went on to describe claims that Union and Hough often received “excessive notes” on their appearance — noting that Union was allegedly told that her ever-changing hairstyles were “too black” forAGT‘s audience, while Hough reportedly received constant criticism on her hair, makeup and wardrobe.

However, in a statement toVariety, Hough denied that she had a negative experience working on the show. “I had a wonderful time onAmerica’s Got Talent, I loved working with the cast, crew, and producers. I am happy to continue my working relationship with NBC,” said Hough, who is set to star in two upcoming NBC Christmas specials. “I’m looking forward to what the future holds.”

When it came to Union’s critique, four sources toldVarietythat the actress allegedly received the note “over half a dozen times.” But one NBC insider denied the frequency of that feedback and claimed that it was given to both judges due to their hair continuity throughout the episodes.

Gabrielle Union and Jay Leno.Steve Granitz/WireImage; Vivien Killilea/Getty

Gabrielle Union and Jay Leno

Additionally, according to multiple reports, Union allegedly complained that Cowell was smoking indoors, which is illegal in California whereAGTfilms. She is allergic to cigarette smoke.

Amid the drama, Union is leaning on her husbandDwyane Wadeand family for support. On Friday, the actressshared a selfie of herself and Wade, in which the NBA legend could be seen sweetly kissing her forehead as she leaned back on his chest.

“Lord, you KNOW I’m tryin…” Union captioned on Instagram. “Whew ? and breathe. Support is everything. ❤.”

On Monday,Leno spoke out about Union’s exitfor the first time, tellingTMZ,“I love Gabrielle Union. She’s a great girl. I really enjoyed working with her.”

“She’s really good,” he told the outlet.

When asked if she was treated fairly on the hit competition series, Leno said, “I don’t know … but I think she’s a great girl.”

source: people.com