A Tory MP claimed he has support of more than half of the Cabinet in his bid to unseat John Bercow
Conservative MP James Duddridge said Mr Bercow must either quit or face a no confidence vote in the chamber whose debates he chairs.
Doubts erupted over Mr Bercow's ability to be as politically impartial as his job requires, after he effectively vetoed "racist and sexist" US President Donald Trump from addressing Parliament when he visits the UK.
Mr Bercow was then revealed to have told students recently that he voted Remain in last June's referendum, that "untruths" were told in the Brexit campaign and that he hoped European Union workplace and equality rights continue after Britain leaves the bloc.
He was originally elected as a Conservative but all Speakers are meant to be strictly neutral from the moment they are elected to the Commons chair by fellow MPs.
Mr Duddridge has tabled a motion of no confidence in Mr Bercow, which dozens of MPs could add their names to when the Commons returns next week from its half-term break.
The MP said he had been assured that Parliamentary rules will require a vote to take place unless Mr Bercow resolves the matter himself by quitting.
Downing Street says it is a decision for the Commons and that ministers can vote as they wish.
I would expect a majority of Cabinet members to vote for the no confidence motion
MP James Duddridge
Former Foreign Office minister Mr Duddridge said he was responding to colleagues' concerns rather than actively running a campaign against Mr Bercow and he had received supportive messages from ministers and MPs across the House.
He added to the Daily Express: "I would expect a majority of Cabinet members to vote for the no confidence motion – indeed, I would be flabbergasted if you could find any member of the Cabinet who is expressing full-blown support for John Bercow."
Members of various parties are unhappy with Mr Bercow's stewardship although frustration is keenest among the Speaker's former Tory colleagues.
Earlier, Brexit backer Mr Duddridge told BBC Radio 4: "He's used the Speaker's chair to pontificate on international affairs.
Get Quotes on Home Insurance
Bercow is also accused of telling students how a voted in the referendum
"In the Commons, he quite frequently calls his friends rather than being totally impartial.
"But critically now, he can no longer be impartial given he has expressed views so publicly and so clearly.
"He should resign and if he doesn't, the vote of no confidence will go ahead."
If he survives this row, Mr Bercow would step down in June next year if he kept his 2009 promise to serve nine years but he is now thought to be determined to stay until the 2020 general election.
Ironically, he owes his continuance so far to a revolt by Tory MPs just before the 2015 election.
The Speaker has been plagued with doubts since his statement about Donald Trump
They joined Labour to defeat a bid by Tory ministers to have the Speaker picked after that election by secret ballot, which the Government thought would make it easier to block Mr Bercow than if MPs had to vote publicly.
Mr Bercow has since strained many MPs' patience – including ministers who are increasingly being dragged to Parliament to answer MPs' questions because he accepts Opposition parties' demands for statements on issues of the day.
But UK Independence Party MP Douglas Carswell – who as a Tory in 2009 tabled a confidence motion which triggered the resignation of the last Speaker, Labour's Michael Martin – warned toppling Mr Bercow could endanger Brexit by causing upheaval.
Mr Carswell wrote in his online blog that Mr Bercow was guilty of "an absurd over-reach … student politics" for trying to keep Mr Trump out of Parliament.
But he added: "Parliament has some big Brexit battles ahead. We need to win them.
"The one thing we don't need now is Commons chaos.
"Remainer MPs will continue to frustrate the referendum result.
"Brexiteers should be exclusively focused on winning those fights, not not picking others. Nothing else matters."
Government sources declined to say if Prime Minister Theresa May would take part in a vote, stressing it was hypothetical as none was yet scheduled.
James Duddridge said Mr Bercow must either quit or face a no confidence vote
Justice Secretary Liz Truss said when asked at an event how she would vote that it was “a matter for Parliament”.
Long-term Bercow supporter and Tory MP Conor Burns told BBC Radio 4's The Westminster Hour on Sunday night that the Speaker had indulged in needless public "grandstanding" about Mr Trump and should stick to his plan to step down next year, to let his successor be chosen by experienced MPs rather than a new post-election House.
Mr Bercow's team insists he has a record of "rigorously" ensuring MPs on both sides of the EU and all other arguments can raise their concerns, regardless of how he personally had voted in any election.
Mr Bercow himself this week became the first Commons Speaker to pay an official visit to Israel. He was pictured looking emotional at Jerusalem's Yad Vashem Holocaust Remembrance Centre.