I think it was just that those were the four main classes in 1st and 2nd edition, with others seen as variations of those (e.g. paladins / rangers were basically fighter subclasses). In the Basic D&D line, they were the only four human classes.

Due to this, adventure designers would include challenges that took advantage of thief skills, or Turn Undead, or whatever, to help all players contribute. Therefore, a smart party would have a mix of the main classes, because they knew they’d encounter obstacles that needed them (although it was never hard and fast, and a good adventure had multiple routes to victory.)