Depends on the type of audience, I think.

Most podcasts will probably never get a huge audience. But the audience you do get is a motivated, probably very specific one. That's a valuable audience.