I don't think you can dismiss feedback because it only came from a handful of people. The brutal fact is, if someone doesn't like you or your podcast, they're almost guaranteed to unsubscribe, close the tab, and never contact you about it. If you're getting multiple negative feedbacks on the same topic, you can bank on 10x more people thinking the same thing, but never telling you.

I ran a podcast with friends for 13 years. We took it on the chin, and paid (via patreon subs) for an audio engineer to fix the things people complained about in our show. It made a world of difference, but we'll never be able to break the "first impression" of people who didn't appreciate it, and unsubbed as a result.