The crossbench peer said the move would help to "cut out" net migration by 100,000 a year in the UK after Britain unshackles itself from the beleagured bloc.
He said: "We would like to see free access for tourists, students and the self sufficient.
"We would like to maximise the long term reduction of migration by intorducing work permits at the level for the highly skilled."
The MigrationWatch boss added: "This would cut net migration by 100,000 a year becasye the low paid, low skilled, mainly easterm Europeans who would be cut out by a work permit system."
Lord Green set out his ideas and plans to control immigration across the EU following Britain's exit.
Speaking about EU citizens who are self-employed in Britain during this afternoon's Home Affairs Committee, he said: "We need to ensure they are genuine.
"We all know the Romanians and Bulgarians are good at being self-employed.
"There has to be a pretty high threshold of income in order to be considered as self-employed."
Lord Green of Dennington speaks during a Home Affairs Committee
Associate Director for Migration, Integration and Communities for the Institute for Public Policy Research that different industries relied to varying degrees on migrants from the European Union.
In farming and agriculture, profit margins for farmers had been squeezed to very small levels, she said, due to supermarket price wars. While government cuts in health care costs had put pressure on residential care.
Ms Griffith also added that migrants were also more susceptible to being exploited by unscrupulous employers.
Phoebe Griffith, Associated Director at the Institute for Public Policy Research
However when Lord Green was questioned by the Labour MP for Streatham Chuka Umunna over how the goverment would be able to identify and then remove non-British citizens from the EU who had not been here for five years he said: "[It was] Not possible to identify them."
But Lord Green added that it might be possible to do this through the HMRC although he stipulated that the situation of having to remove these people would be unlikely to arise.
Chair of the Committee Yvette Cooper MP
Lord Green welcomed the suggestion that a regional system for migration could be introduced, telling the Committee: "Whe should look at that." But he added that the current system should be able to cope with varying demands in different areas of the country.
Ms Griffith also agreed and said: " We should explore the idea of a regional system."
She added that the UK had diverse requirements across the country and that needed a "pragmatic response to regional differences."
However while both accepted that targets could in future be set by the regions they still advocated centralised control over things like visa control.
This was heavily criticised by Mr Umunna who savaged the idea as "patronising centralised Whitehall control" and said the policy had been "rubbish".
Prof Alan Manning (right) and Madeleine Sumption
Lord Green also slammed the EU for having a "fixation" on free movement.
Ms Griffith also agreed and added: "UK concerns over EU migration have not been shared on the continent".
She stated that the UK had acted as "a pressure valve" for the EU labour market, especially for the southern countries who had been hit by economic problems.
Madeleine Sumption gives evidence to the Committee
Giving evidence are Professor Alan Manning, Chair, Migration Advisory Committee, and Madeleine Sumption, Director, Migration Observatory.
Madeleine Sumption told the panel that it was difficult to generalise over the impact of new migration in the UK labour market as it was dependent on a number of factors such as technology.
However she did indicate that what studies had been done had shown a "relatively small" impact in general.
Figures quoted by Prof Manning, from the last census indicated that 15 percent of care workers in the UK came from outside the EU while just three percent came from within the union.
Prof Alan Manning at the Committee hearing
Ms Sumption aired a note of caution over introducing a quota system for different industry sectors.
She said: "Most countries do distinguish between high and low-skilled workers" but added in some areas it was hard to identify which sectors some jobs fell into.