As someone who was homeschooled through 6th grade and then private schooled and considering homeschool for early elementary (I have a 4 year old) I would say my main concern is actually in my wife's ability to maintain a parent/teacher balance right now.
What I mean by this is simply wearing two hats in the household is a tricky business and can be overwhelming especially when our young students have younger siblings that must be cared for. I have enormous respect for my wife but handling these responsibilities is no easy feat!
Of course my kids are very young. I have other concerns beyond elementary, mostly around an increasing need for expertise in sciences/math and properly rigorous testing ensuring mastery over subjects without access to a realistic grade curve.
My personal experience was very good but a bit unrealistic - I was taught to read by my mom who has taught kindergarten for decades. Not everyone has a professional educator as their homeschool teacher!