In light of the weekend's London Bridge attack, the former soldier told the GMB hosts that there is currently issues on how to tackle a terrorist threat.
He explained: "The ideal thing to do is prosecute them and put them behind bars before they get to the stage of an attack.
"But clearly, as we've seen in virtually every attack that's taken place in Europe, we've known who these people are, but we haven't done anything about it.
"That's partly because the evidence we have against them is not evidence we would be able to use in court.
"Therefore we have to do something else, and that would be getting them out of the country; if they are non-British citizens they have to be deported back to where they came from.
Are we not straying into Guantanamo Bay territory here?
"They can't be able to stay here and plot against us and those people who are British suspects have to be detained without trial if the intelligence is strong enough," the former head of COBRA added.
However, host Morgan asked: "Are we not straying into Guantanamo Bay territory here?"
Kemp continued: "I quite agree with that and I'm not suggesting we lock people up and throw away the key.
"What I'm saying is we use the intelligence we have, put it in front of clear judges who can asses whether or not these people represent a threat.
Colonel Kemp argued over how to tackle a threat in the UK London Bridge terror: Scenes as armed police respond Sun, June 4, 2017
At least two people have died after being after reportedly being struck by a van on London Bridge on Saturday night.
GETTY 1 of 7
Get Quotes on Home Insurance
Armed police responding to the incident at London Bridge
"They can then intern them and keep them under review," he argued.
However, Brigadier Paul Gibson suggested that the country is falling on preventing potential attacks, to which Kemp suggested: "It wasn't a lack of resources, it was a lack of political will."
This comes after Morgan destroyed the culture secretary Karen Bradley over Theresa May's previous police cuts.