1) If you are non-religious, the canned curriculum, trade shows, support groups are very few and far between. You will be much more lonely than your religious home-schooling counterparts.
2) As your kids get older, it is harder and harder to give them a proper education. By the time your kids are in middle-school, subjects get harder for a parent (or pairs of parents) to cover in a way that does each subject justice.
In the end, most of my friends who home-schooled eventually sent their kids to middle-school and or high-school... and that timing seemed about right.