To be honest, I don't have a good answer to that. I've read (but not played) games that had special rules for "mooks" or "mobs". For example, Wushu, a game of cinematic Kung-Fu action, was like that, because I guess it's very in the spirit of the setting. I've played computer games that throw hordes of enemies at you, also. Invariably there's a progression from mooks, to mini-bosses, to bosses to big bads. Then there's games like Call of Cthulhu (CoC) were there are no goblins and even the weakest enemies can drive your character insane (well, ish. I think Ghouls are relatively safe...).
I have no clue what's the right way to achieve balance. It seems to me that the only way to avoid what you describe is to have no weakling enemies and make every encounter a matter of life-and-death, with a substantial risk for at least one character to die (like CoC). But not for a Total Party Kill probably.
I kind of have Nethack in mind, here, and other roguelikes (I think a lot about them lately). In Nethack you fight some weak enemies at first, like slime molds, but you can at least eat them, so they're a resource. Then as you progress down the dungeon, the chance of dying increases very steeply. And then your character dies. And the next one. And the next one too. And the one after that. So there's no boring encounters, but I personally want to throw the keyboard after a while and I end up savescumming because I just want to get a bit deeper in the dungeon and see what's there.