I can imagine a lot of cases in the future where this goes awry and is abused, probably not directly by Google Duplex, but by homemade versions.

I think there should be a kind of non-jarring way for AIs to identify themselves - maybe playing three quick tones before the AI speaks - to identify that it isn't a human on the other hand and therefore shouldn't be able to say, order thousands of dollars of food. This is quick, unobtrusive, easy for an AI to do, and easy for an AI to detect.

  That would defeat the purpose.

If the purpose is to robocall a hair salon and make an appointment, I don't see why identifying itself would defeat the purpose?