To be honest, I don't have a good answer to that. I've read (but not played) games that had special rules for "mooks" or "mobs". For example, Wushu, a game of cinematic Kung-Fu action, was like that, because I guess it's very in the spirit of the setting. I've played computer games that throw hordes of enemies at you, also. Invariably there's a progression from mooks, to mini-bosses, to bosses to big bads. Then there's games like Call of Cthulhu (CoC) were there are no goblins and even the weakest enemies can drive your character insane (well, ish. I think Ghouls are relatively safe...).
I have no clue what's the right way to achieve balance. It seems to me that the only way to avoid what you describe is to have no weakling enemies and make every encounter a matter of life-and-death, with a substantial risk for at least one character to die (like CoC). But not for a Total Party Kill probably.
I kind of have Nethack in mind, here, and other roguelikes (I think a lot about them lately). In Nethack you fight some weak enemies at first, like slime molds, but you can at least eat them, so they're a resource. Then as you progress down the dungeon, the chance of dying increases very steeply. And then your character dies. And the next one. And the next one too. And the one after that. So there's no boring encounters, but I personally want to throw the keyboard after a while and I end up savescumming because I just want to get a bit deeper in the dungeon and see what's there.
(The goblins realize they're going to lose and start fleeing, they ambush in an interesting way, they are a distraction for something else, one of them has a bomb, the party has to fight them in their underwear using camping gear as weapons, etc etc etc)
or you do something else to preserve the fun such as simplifying the encounter
(You easily crush the puny goblins while only losing (1d6 health), After seeing the might of the wizards powerful spell the goblins start bowing down before him, you quickly make a game to see who can knock a goblin the farthest and the rest run off, etc etc etc)
The thing that I love like crazy about dnd is that you don't have to do anything that you don't wish to. Anything that is boring, annoying, frustrating, etc can be easily discussed in the group and house ruled away. The only limits and annoyances are the ones that you have chosen to allow to continue.