Quebec Police spokesman Etienne Doyon also confirmed two suspects have been arrested, adding "there are many victims… there are deaths".
At least 20 emergency services vehicles were at the scene in Quebec city as a heavily armed tactical squad entered the mosque to deal with the incident which is now under control, according to police.
Mosque president Mohamed Yangui said: "Why is this happening here? This is barbaric.”
Mr Yangui was not inside the mosque when the incident occurred but said he received a flurry of calls from people attending evening prayers.
One of the suspects is believed to be 27-years-old and they are both from Quebec and of Arab region.
Emergency services were quick to attend the scene in Quebec, Canada
He added he was unsure of the number of victims because they’ve have been taken to different hospitals in the city.
Police have set up a perimeter around the mosque and a number of ambulances are outside the Islamic cultural centre of Quebec in the Sainte-Foy neighbourhood as paramedics try to help the wounded.
A witness claimed there were about 40 people inside the mosque at the time of the shooting.
Police have closed roads around the city and claim they have arrested two suspects.
They do not believe anyone else was involved in the attack.
Canadian prime minister Justin Trudeau tweeted his condolences to those affected by what he described as a "cowardly attack".
In a statement, he said: "We condemn this terrorist attack on Muslims in a centre of worship and refuge.”
Meanwhile, the country's Public Safety Minister, Ralph Goodale, tweeted: "Deeply saddened by loss of life & injured in Ste-Foy."
There is a strong police presence across the city amid fears of another attack
Police have set up a cordon around the mosque
Tonight, Canadians grieve for those killed in a cowardly attack on a mosque in Quebec City. My thoughts are with victims & their families
Canadian PM Justin Trudeau
Speaking to Radio-Canada, a witness said two masked individuals entered the mosque during prayers on Sunday evening.
"It seemed to me that they had a Quebecois accent. They started to fire, and as they shot they yelled, 'Allahu Akbar!'
"The bullets hit people that were praying… a bullet passed right over my head.
Armed police units were called to the scene amid fears the gunmen could still be at large
"There were even kids. There was even a three-year-old who was with his father."
New York Mayor Bill de Blasio said additional security for mosques was being provided by police in the aftermath of the shooting.
He tweeted: "All New Yorkers should be vigilant. If you see something, say something."
Canadian prime minister Justin Trudeau tweeted his condolences and support
Armed police patrol the city following the terror attack
Incidents of Islamophobia have increased in the city in recent years amid a political debate over banning the niqab, or Muslim face covering.
In June last year, a pig's head was left on the doorstep of the Islamic cultural center.
In 2015, a mosque in the neighboring province of Ontario was set on fire in 2015, just hours after the terrorist attacks in Paris.
The incident occured in the Sainte-Foy neighbourhood in Quebec city, close to the US border
Shocking moment thugs set fire to a mosque in Essex
Thu, January 12, 2017
The men initially tried to break into Al Falah Braintree Islamic Centre in Essex by throwing bricks and kicking the door. But when that failed the pair, believed to be intoxicated at the time, set fire to the plastic corrugated roof on the building. After starting the fire they fled the scene at around 2.30am on Sunday (8/12), as melted plastic dripped down and the fire spread. The centre's secretary Sikander Sleemy said: "I think they had a drink or were drunk, just by the way they were walking and the fact that the pubs closed just before.
1 of 8
The news comes just hours after a mosque in Texas was torched as the US deals with the fallout from Donald Trump’s ‘Muslim ban’.The blaze broke out at the Islamic Centre of Victoria in the early hours of Saturday morning.
A clerk spotted the fire at 2am as smoke and flames billowed out of the building.
Firefighters arrived on scene and battled to get the fire under control for four hours.
Shahid Hashmi, the Islamic Center’s president said: “It [the building] looks completely destroyed.”