I don't know what your gifted education programs were like, but I never recall feeling like I missed out on my childhood. Occasionally I would take different classes to some of my friends (but with others), and that was about it.
Far from keeping people from being free to develop and think for themselves, most of the gifted classes I took actively encouraged that, with open-ended problem solving and free-form classes often led by student interest.
But children need to be reined in, they don't have the reasoning skills or understanding of adults; the one time I was allowed to fully control my own syllabus it led almost directly to me losing the ability to speak Cantonese, which I'd been fluent in up to that point. As a kid who'd spoken it all his life, I simply didn't believe all of the people who told me I'd forget it without practice. Because I was a child.
I don't know exactly how free-form Sudbury Schools are, but I'd have extreme reservations about sending my children to one.