Caldwell converted a corner-kick from Paul Hartley into the back of the net in the 67th minute to seal the win for the hosts.
Before the goal, France captain Patrick Vieira headed a Franck Ribery free-kick home but his effort was disallowed for offside.
Thierry Henry also failed to score as his curling free-kick was denied by the post.
Scotland now have nine points from three matches in first place of Group B while the World Cup finalists are second, three points behind.
In another match, Andrius Skerla scored a last-gasp winner as Lithuania edged Faroe Islands 1-0.
Skerla poked home in the 90th minute to mark his 50th international game.
In Roma, Italy scored two second-half goals to see off Ukraine 2-0.
Massimo Oddo broke the deadlock in the 71st minute from the spot after Luca Toni was brought down inside the area.
Toni himself added the second nine minutes later when he controlled Antonio Di Natale's lofted pass on the edge of the area before firing into the top corner.
The World champions remained third on four points as Ukraine lie fifth with one point less.