What's wrong with vBulletin and phpBB? It's a genuine question, mostly naive - I've used but never admin'd them.
I do like them, and have great memories of them, they were my childhood. But I do think we're overdue for a modern forum stack. Discourse was a step in that direction, but a lot of people strongly dislike it, and it doesn't seem to have gotten the reach of others before it.
A real contender for "new forum software" would have to address the fact that Discord is its competitor, not other forum software, and it would have to have a very compelling answer for "why this instead of Discord".
Genuinely curious, what would that be?
From user perspective is there anything else to the forum than Read/Post/Search?
Or are you referring to administration and moderation tasks?
Anyone wanting to launch new forum software needs to compete with that.
And on top of that, "just self-host" isn't enough in an era where developers don't even want to self-host their own projects anymore, let alone some third-party forum. That's another thing that has -deeply- changed in the last 20 years.
phpBB is pretty badly dated. It still works but it's clunky and development has been very slow. (The last major release was four years ago; everything since then has been minor maintenance.)
vBulletin 4 is okay, but 5 and 6 are poorly coded, extremely inefficient and full of bugs even today.
Some of the original team went off to create XenForo, which has basically replaced vBulletin for most large forums now, though that itself has its own issues.