Milan Jovanovic scored the match's only goal on 38 minutes from point-blank range after Nikola Zigic headed a Milos Krasic cross towards him.
The goal came seconds after Miroslav Klose was sent off for two bookable offences.
Germany had more than one chance to salvage a draw but were notably jinxed.
Sami Khedira could have brought the score level late in the first half but the post denied his powerful drive.
A follow-up overhead kick from Thomas Mueller was cleared off the line by the defenders too.
On the hour mark, Serbia goalie Vladimir Stojkovic, who was named man of the match, saved a tame penalty kick from Lukas Podolski.
Now Serbia, Germany and Ghana are all on three points in Group D. The latter team will face Australia, who have only one point, on Saturday.