Hamilton topped the timing sheets during the first practice session of the weekend on Thursday but he and his Mercedes team-mate Valtteri Bottas to finish eighth and 10th in the afternoon, the first warning sign for the team.
Boss Toto Wolff admitted afterwards that the team had gone in the wrong direction on set-up but nothing appeared to have changed two days later when it came to qualifying.
Hamilton was anything but comfortable in the first session but avoided the humiliation of falling at the first hurdle with a 10th-placed lap.
But the three-time world champion was not settled and nearly lost the car into the barriers on a number of occasions.
Jose Mourinho enjoys F1 Monaco Grand Prix Sat, May 27, 2017
Jose Mourinho caught the action from the pit at the Monaco Grand Prix
Play slideshow Getty Images 1 of 6
Jose Mourinho is loving the F1 action in Monaco and dropped a Griezmann hint…
During Q2, he repeatedly radioed back to the pits, complaining of problems with his car and traffic on the short, narrow Monaco track.
With just seconds remaining, he appeared to be on a lap that might sneak him into the top 10 and the final qualifying session.
However, Stoffel Vandoorne crashed into the barriers ahead of him and the Brit was forced to pull out of hot lap, condemning him to a start on the sixth row of the grid in 12th, up one from the 14th in which he qualified thanks to Jenson Button and Vandoorne's penalties.
Lewis Hamilton has struggled for a couple of days at the Monaco Grand Prix
Lewis Hamilton was unable to get his ultra-soft tyres up to optimal temperature during Q2
Get Quotes on Home Insurance
"We got unfortunate with the yellow flag at the end but it is what it is," Hamilton said afterwards, having missed out on the top 10 in qualifying for the first time since the 2013 Italian Grand Prix.
"It was an odd feeling in the car.
"Clearly [we didn't make progress overnight].
"That's why I'm 14th or whatever it is.
"That lap at the end may have got me into the top 10 but I probably would have struggled to be in the top 5 with the pace that I had and whatever issue there was with the car.
"It's great to see that Valtteri [Bottas, 3rd] was able to extract one sense of what the performance of the car was.
"It shows that we're not terrible here.
Monaco Grand Prix 2017 qualifying results in full
"We'll just have to figure out why I wasn't able to be up there with him."
Bottas will start third in Sunday's race behind the two Ferrari's although it had been Sebastian Vettel who was the pre-session favourite for pole.
Instead, his Finnish team-mate Kimi Raikkonen secured his first No1 slot in nine years, less than five hundredths of a second clear of his Ferrari rival.