Where I grew up in a liberal big city, the only home-schooled kids I met were super well-educated children of professionals, and were way ahead of both public and private school kids. I was a lot like you - always finished my homework before class was over, read all the materials and was bored in school most of the time. I was jealous of the home-schooled kids. My parents would say, "well but it's good to be around people your own age," but the home-schooled kids had plenty of friends. That gave me the idea that homeschooling was a better way.
That was before I met home-schooled kids from the rest of the country, where it had nothing to do with giving them a better education; it had a lot to do with limiting their education, and forcing them into a fundamentalist Christian or Mormon ideology. Those kids really didn't get out to socialize much, and they didn't get a good education, either.
I think parents have a right to homeschool their kids, and a government ban on that scares me. But I also think the parents need to meet or exceed the qualifications of the teachers who would otherwise be teaching, and [edit][strikethrough]the government[/strikethrough] *society* has some interest in preventing someone from having 48 children and forming a cult.
So like a lot of other stuff extremist wackos do, "this is why we can't have nice things." The goal is to walk a fine line in the center where government can stop crazy people from using freedoms to do crazy things, while still allowing sane people to have the rights and privileges they need as free individuals within the framework of society.