Interesting idea. Well done and good luck with it.
IMHO, the use case for async audio notes is not entertainment "podcast". People listen in a passive and linear way.
But the use case is team planning and note taking for on-the-go and hybrid WFH teams.
Add some ability to "call-in", listen and add comments and you've got something like an audio Kanban board.
Also add voice2text to dump a threaded transcript.