The 44-year-old was left evaluating his life after the sudden death of a stranger, he told presenters Piers Morgan and Susanna Reid.
James commented: "I was chatting to him and he stood up on stage, and collapsed and passed away in front of me. That was over a year ago."
The northerner continued: "That was when I made the decision to change. I looked back at all the work I'd done. I'd had four days off that year and five days off the year before.
Saturday Kitchen: James Martin has revealed what pushed him to quit the BBC show
Saturday Kitchen: James Martin left the show after a fan died in front of him
Saturday Kitchen: The guest presenters
Sun, November 20, 2016
Saturday Kitchen: When James Matin left the show in March, he paved the way for these famous chefs to presenter the popular cookery show.
1 of 9
Saturday Kitchen chefs and guest presenters
That was when I made the decision to change.
Get Quotes on Home Insurance
"That's when I thought things would have to change. It's partly the reason I gave up Saturday Kitchen."
Speculation as to his reasons for leaving has been rife since James announced he was stepping down, with many claiming he had other television offers.
"I've bet on myself that I’ll be the new [Great British] Bake Off host, because everyone’s always saying I'm going to be," he said.
Saturday Kitchen: James Martin said he was left evaluating his life
Saturday Kitchen: James Martin was appearing on ITV's Good Morning Britain
James added: "I didn’t leave because we didn’t have a sink, or because of Top Gear, or budget cuts. All these rumours!
"I left because 10 years is a long time. That’s every Friday and Saturday you’re giving up. I had to be in bed by 9pm and up at 3am. It’s a tough show!"
Saturday Kitchen Live airs airs Saturdays from 10am on BBC One.