Former cabinet minister Amber Rudd has accused Downing of using language that “does incite violence”.
Ms Rudd told the Evening Standard that the language used was the sort “people think legitimises a more aggressive approach”.
It comes after a stormy week in the House of Commons, where several MPs criticised the PM’s use of language, including the term “surrender bill”.
Boris Johnson has insisted he “deplores any threats to anybody”.
In the interview with the Evening Standard, Ms Rudd said: “The sort of language I’m afraid we’ve seen more and more of coming out from Number 10 does incite violence.
“It’s the sort of language people think legitimises a more aggressive approach and sometimes violence.”
Ms Rudd said some of the rhetoric used over Brexit was “immoral”.
She told the newspaper she was “disappointed and stunned” when Mr Johnson dismissed “genuine fear that a lot of women have” following the 2016 murder of Labour MP Jo Cox.
On Thursday, several MPs said the prime minister should apologise for saying the best way to honour Mrs Cox was to “get Brexit done”.
In interviews with the BBC, Mr Johnson acknowledged that “tempers need to come down” in Parliament.
But he added: “I do think in the House of Commons it is important I should be able to talk about the surrender bill, the surrender act, in the way that I did.”