I don't think it's so much a privacy concern (it certainly is), but more with regards to accidental triggers by actions that are passive and flaky.

Location accuracy isn't always the greatest, particularly indoors; in-and-out connections to wifi/bluetooth are possible when you're on the fringe of its range.

These issues need to be addressed in some form before _true_ automation is given to them.