The number of children doing an hour of exercise a day falls by nearly 40% between the ages of five and 12.
Figures suggest that by the final year of primary school, just 17% of pupils are doing the recommended 60 minutes of physical activity every day.
A spokesman for Public Health England described the drop in activity levels as "concerning".
More than a third of children in England are overweight by the time they leave primary school.
A new survey from Public Health England and Disney looked at the effects of physical activity on children's emotional wellbeing
More than 1,000 children aged five to 11 were questioned, with their parents acknowledging that being active made their children feel happier (79%), more confident (72%), and more sociable (74%).
But the survey also found that children's overall happiness declined with age, with 64% of five-and six-year-olds saying they always felt happy, compared with just 48% of 11-year-olds.
"Children's physical activity levels in England are alarmingly low, and the drop in activity from the ages of five to 12 is concerning," said Public Health England's Eustace de Sousa.
"Children who get enough physical activity are mentally and physically healthier, and have all-round better development into adulthood – getting into the habit of doing short bursts of activity early can deliver lifelong benefits."
Currently, just 23% of boys and 20% of girls, between the ages of five and 15, meet the national recommended level of activity, according to an NHS report published last December.
"Not being very good" was cited by many children as the reason they did not take part in some physical activities, with older children more likely to be self-conscious than their younger counterparts: 29% of 11-year-olds compared with 17% of five-year-olds.
As part of the Change4Life campaign, Sport England and Disney have joined forces to launch a 10 Minute Shake Ups programme, encouraging children to take part in accessible activities across the school holidays.
"The 10 Minute Shake Ups provide a load of fun activities to get kids moving more," said Olympic marathon swimmer Keri-anne Payne, who is backing the campaign.
"Being active is not just for Olympians, it's for everyone. "
Get Quotes on Home Insurance