The crossbench peer said the move would help to "cut out" net migration by 100,000 a year in the UK after Britain unshackles itself from the beleagured bloc.
He said: "We would like to see free access for tourists, students and the self sufficient.
"We would like to maximise the long term reduction of migration by intorducing work permits at the level for the highly skilled."
The MigrationWatch boss added: "This would cut net migration by 100,000 a year becasye the low paid, low skilled, mainly easterm Europeans who would be cut out by a work permit system."
Lord Green set out his ideas and plans to control immigration across the EU following Britain's exit.
Speaking about EU citizens who are self-employed in Britain during this afternoon's Home Affairs Committee, he said: "We need to ensure they are genuine.
"We all know the Romanians and Bulgarians are good at being self-employed.
"There has to be a pretty high threshold of income in order to be considered as self-employed."
More to follow…