Rumors of the music mogul stepping down from American Idol were spread across the Internet today after an interview with a British newspaper.

Cowell was talking about his high workload when he made the following comment: “One [of the 3 shows he presents] may have to go. I don’t know which one. That is the problem, I like all of them. I can’t imagine not doing Britain’s Got Talent and I can’t imagine not doing X Factor.”

Some may say this is high time to get rid of the rude and critical producer from the show, but really, could you imagine American Idol without Cowell’s dry sense of humor and often hilarious take on some of the contestants?