> Answering questions like "did fleet battery life get worse with yesterday's update?" get really really hard when you start managing thousands of devices.

To me that only seems like a question you could answer by managing thousands of devices. Not that it would be easy, just that you couldn't really answer that very accurately with only a handful of devices.