Cowell admitted he is "very sad" about leaving the Fox reality hit that made him an American household name and revealed how the decision had not been finalized until moments before Monday's announcement.
"We were driving down to do the press conference and we didn't have an agreement," Cowell said. "We signed this agreement in front of all this people."
Responding to rumors he suggested the host fill his spot at the judging table, Cowell joked that he thought of Seacrest in his "darkest moments" by possibly hooking him up with another job.
As for why he is jumping ship to launch the American version of The X-Factor, he said: "You have to know when it is time to go...at one point I wanted to do both shows." So why isn't he pulling a television one-two punch?
"I thought people would be sick and tired of me."
Asked what he would miss most about being on Idol, the normally sarcastic Brit had a moment on sincere reflection.
"Everything," he responded. "We had fun didn't we?"
He was more tight-lipped about the rumor that Paula Abdul would be a judge on his new show.
"You'll have to wait and see."