D&D comes from chain mail, a heroic fantasy miniature battle game inspired by naval miniature battle game. Hence the concept of character class (instead of boat class) and armour class (where the lower the better). But based on that, slowly the idea of playing a single character appeared and it turned into D&D and then early RPG

Modern RPG got (Somehow) rid of character class (even though some archetype/Playbooks are classes witha different name) and armour place. But D&D kept it by traditipn

I think it was just that those were the four main classes in 1st and 2nd edition, with others seen as variations of those (e.g. paladins / rangers were basically fighter subclasses). In the Basic D&D line, they were the only four human classes.

Due to this, adventure designers would include challenges that took advantage of thief skills, or Turn Undead, or whatever, to help all players contribute. Therefore, a smart party would have a mix of the main classes, because they knew they’d encounter obstacles that needed them (although it was never hard and fast, and a good adventure had multiple routes to victory.)

The first three character classes to appear in D&D were the Fighting-Man (now called the fighter), Magic-User (now called the wizard) and Cleric. The first supplement introduced Thief (now called Rogue) and Paladin (as a Fighter sub class). The list grew a bit with f.e. druid and ranger but then with adnd 2e it was re-organized again with all character classes falling under the four traditional abstract archetypes: Warrior, Wizard, Priest, and Rogue.

So it came from the humble beginnings I’d say. It also makes the game more fun when you have your own specialty. Having two bards argue about who’s the better one at haggling does sound like a fun rp opportunity though.