As someone who was both home-schooled and went to public school, you are buying into the mythos around school too much. I had my own experiences when I was homeschooled that were far better than "enduring the horrors of PE".
And life without a little horror is not much of a life if you ask me.