Mr Bercow was left fighting to save his job amid 'impartiality' row over Brexit
Tory MPs warned that the parliamentary official was "no longer impartial" after a recording of his remarks revealed his backing for the Remain side in last year's EU referendum.
One former minister claimed Mr Bercow was already "getting close to the end" of his term in the key Commons post.
And Tory Cabinet minister David Lidington predicted that the Speaker will face a "strong reaction" from MPs for his remarks while insisting that the Government would keep out of the row about his future.
I wouldn't expect him to stay for much longer
An Early Day Motion proposing that the Commons had lost confidence in Mr Bercow was tabled last week following concern at his outspoken attempt to block US President Donald Trump from delivering an address to MPs and peers during his proposed state visit this summer.
The row deepened today when a video recording of the Speaker revealing his views about Brexit to Reading University students in February last year.
"Personally, I voted to Remain. I thought it was better to stay in the European Union than not," he told them.
Critics claim his remarks breach parliamentary rules on impartiality and raise questions over his ability to fairly chair debates.
A recording of Mr Bercow revealed his backing for the Remain side in last year's EU referendum
Mr Lidington, who post as Commons Leader involves liaising between the Government and Commons authorities, told the BBC1 Andrew Marr Show: "There will be strong reaction among some MPs to what he said at Reading, particularly after what he said about the state visit earlier in the week.
"Ultimately, the Speaker has to command the confidence of the House of Commons as a whole.
Get Quotes on Home Insurance
"John has his very strong supporters as well as his strong critics in the House of Commons, but we shall have to see how members as a whole respond.
"It is really important for the very independence of the Speaker's office that the Speaker, whether they start as a Conservative MP, a Labour MP, or whatever, is independent of Government. Speakers, if anything, should be towards the people who are not in Government, as, actually John Bercow probably has done in the way that he has used urgent questions that we have found inconvenient."
Former Tory culture secretary John Whittingdale expected Mr Bercow to stand down soon.
Mr Whittingdale expected Mr Bercow to stand down soon
Mr Whittingdale told ITV's Peston on Sunday: "John was elected with a very firm pledge that he wouldn't stay for more than eight or nine years, and we are pretty much getting close to the end of the that period.
"So, I wouldn't expect him to stay for much longer."
A spokeswoman for the Speaker insisted his private views about Brexit had no effect on his ability to do his job.
"Mr Bercow voted in the EU referendum, along with millions of others," the spokeswoman said.
"The record shows that he has rigorously facilitated the raising of concerns of those on both sides of this argument, as he does on every other issue.
"The Speaker's impartiality is required on matters of debate before the House, and he has been scrupulous in ensuring that both sides of the argument are always heard."
James Duddridge tabled the no-confidence motion last week
Tory MP James Duddridge, who tabled the no-confidence motion last week, yesterday insisted Mr Bercow could no longer impartially oversee Commons debates.
"I think there will be a vote of no confidence and I think he will go," Mr Duddrige said.
"There's absolutely no way Speaker Bercow can sit in the chair on European issues.
"When you become Speaker you must be impartial. He's no longer impartial – he's no longer able to continue to do the role, which is why I think the House will vote him down in a vote of no confidence. In reality he may see the lie of the land and go before he's pushed."
Critics claim his remarks raise questions over his ability to fairly chair debates
He added he had been "amazed" by the number of people to privately voice to him their support for his motion.
Professor Vernon Bogdanor, a constitutional historian at King's College London, said Mr Bercow's position would become untenable if he was seen as a divisive figure in the Commons.
He said: "Just as the Queen is a servant of the state and therefore cannot speak out on any public issues because that would divide the state, so John Bercow is a servant of the Commons and can't speak out in public on any political issues because that would divide MPs."
Downing Street sources confirmed that ministers will be given a free vote if the motion of no confidence in Mr Bercow comes before the Commons.
The move is a signal that Theresa May intends to remain neutral in the debate about the Speaker's future and will not make any effort to defend him the Tory attempt to oust him.