The ex-Germany coach was under immense pressure lately, following Bayern's humiliating 5-1 aggregate loss to Barcelona in the UEFA Champions League.
An early exit from the German Cup, as well as poor Bundesliga form which saw the club lie third in the competition led the Bayern board to lose patience with Klinsi.
"We did not come to this decision lightly," Bayern chairman Karl-Heinz Rummenigge said.
"But the results of the past weeks, the way in which they came about and above all, the situation we find ourselves in five games before the end of the season forced us to act out of a sense of responsibility to the club."
A 1-0 loss to Schalke on Saturday left Bayern in desperate need to hang on to their league position in order to secure Champions League spot next season.
The sacking decision comes less than nine months from Klinsmann's appointment at the Allianz Arena.
Caretaker
In the meantime, former Germany and Borussia Monchengladbach legend Jupp Heynckes was appointed as Bayern's caretaker coach until the end of the season.
Heynckes previously coached Bayern between 1987 and 1991, where he won two Bundesliga titles for the club.
Later, he moved to Spain to coach Atletico Bilbao, before winning the Champions League with Real Madrid in 1998.