I'm coming from a similar place of intense public speaking anxiety as the parent poster, and so attended my local Toastmasters, and at the first session mostly just observed. The hour was ~20 people who have been doing this for years giving highly refined talks to each other.
It was somewhat valuable seeing good speakers in their element, but to someone like me who's coming in with unrefined speaking manners and an anxiety problem, it was a total no-go. It probably varies by chapter, but I realized that Toastmasters wasn't a support group for bad speakers, it was a group of good speakers working to become great speakers.
That's not to say they wouldn't have been supportive and all, but in my opinion Toastmasters isn't an environment for overcoming anxiety at the low end. Of course, your mileage may vary.