Regardless of whether you thing the return value should be True or False, this behavior violates the Principle of Least Astonishment [0] for a significant number of people.
Such violations in software inevitably lead to bugs that could have been avoided. The real question in my mind isn't what the return value should be. The question is "How can this problem be wholly avoided?"
Some have suggested requiring a default return value when the iterator argument is empty. Others have suggested throwing an error. Perhaps others will suggest the all() function shouldn't exist, although that may be throwing the baby out with the bathwater.
[0] https://en.wikipedia.org/wiki/Principle_of_least_astonishmen...