The bad news is as you say: having mapping data of your private residence siphoned off by third parties, be it government or private industry.
I would say I'm as excited as I am worried about the potential for this technology. That said, I think the privacy implications are very similar to Kinect, which is already here. The only difference perhaps being mobility.
However, my recollection is that there was a reasonably vocal reaction to the privacy implications of Xbox One / Kinect 2 in gaming and tech media, although they mostly related to the timing of the first Snowden leaks. This article is a pretty good example of the media hype around it: http://www.theverge.com/2013/7/16/4526770/will-the-nsa-use-t... .
I think the "always-on" nature was more criticized than the point-cloud/time-of-flight abilities of the camera, but pushback against more and more ways to collect data attached to equipment made by big tech corporations isn't exclusive to Google.
Having point cloud data of your home in general is a complete non-issue, I'm sure most of the time the government already has floor plan data for your home.
The possible benefits of this tech are tremendous, and far outweigh any crazy conspiracy theory downsides.