Empire spoke with Michael Rosenbaum to promote tonight’s season premiere of his new show Impastor on TV Land, now entering its second season, but the big highlight is that Rosenbaum spoke a bit about his departure and series finale return to Smallville several years ago.

Rosenbaum played Lex Luthor on the series and is generally remembered as one of the best aspects of the show. He departed after the show’s seventh season, but in the Empire profile, he recalled Warner Bros. Television President Peter Roth‘s taking him out to dinner, trying to get him to do two more seasons of the show. This is a story Rosenbaum has never really told.

“He tried to get me to do two more seasons of Smallville. I was very polite and respectful. I said, ‘Peter, my grandma thinks I’m funny and I’ve always wanted to do comedy, and I started out in comedy, and I was doing tons of comedy, and then I was catapulted into this role that I love and it’s been great, but I was contracted for six years to play Lex Luthor, I did seven, and I’m just ready to move on and I’m just ready to take a new step’,” Rosenbaum recalled. “He looked at me and says, ‘You know, Julianna Margulies, she turned down millions of dollars to stay with ER and look where she is now.’ It wasn’t two or three years later where she just made a fortune with The Good Wife and all of that, and her career just took off. I said, ‘I’m going to bank on my talent. I’m just going to take a chance on me. I think I’ve done this long enough, I did this character for seven years and I just don’t feel like shaving my head for two more years.’ I came back for the finale, but at the time I just wanted to take a chance,” he explained.

After leaving the show, Michael grew out his hair, met with casting directors, and landed roles including a fun one in Season 1 of Adam F. Goldberg’s Breaking In. In 2014, Rosenbaum directed a feature, Back In The Day, with Morena Baccarin, Nick Swardson, and Harland Williams.

Michael assured that his leaving Smallville was not driven by ego. It wasn’t, ‘I’m not doing Smallville because I’m too good for it.’ It was more, ‘Hey, I’ve got more to offer.’ Look, luck is a commodity of preparation and opportunity and I feel like I’m always prepared when that moment comes. I think it comes down to just believing in yourself,” the actor recalled.

Also revealed in the piece: Rosenbaum did not see what happened in the later years of the show after he left. “Here’s the thing: I didn’t watch the last three seasons, because I wasn’t in it,” Michael admitted. “Call me egotistical, call me whatever, but that’s the reason I didn’t watch the show. I was working and getting my shit together. But I finally called them up and said, ‘Hey, look, it’s the last episode ever. I’ll do it, you’ve got me for one day next week.’ When I got there I was, like, ‘What’s happened since I left?’ I had no idea what was going on. There were moments where I just didn’t know what the f— I was doing. I liked my scenes with Tom Welling, but I felt like the show was, for me, done when I left in season seven. Then I sort of did it for the fans and did it for me for closure and to say, ‘Hey, I did come back.’ I did do it, and that’s ultimately why,” he said.

“I’m proud of it,” Rosenbaum said about his experience on the show. “I have fans all over the world because of that show and I love them. I go to Australia, I go to England…people just embrace it. You can’t be luckier as an actor or as a human being to feel that sort of accomplishment, and if that’s all I did — if I was just Lex Luthor — it would be enough. It really would be enough to go back home to New Berg, Indiana, where there are, like, 3,000 people in the town and where I wasn’t supposed to do anything. To say you were this iconic, legendary character for seven years. I would’ve mowed my lawn with a smile on my face,” he said.

You can read the entire Empire interview with Michael Rosenbaum here.