> It sounds like Prometheus needs to run properly on a single box for simplicity but over time needs to be broken up and made scalable beyond the bounds of a single server.

I can understand the reasoning behind them building for a single node. Building distributed systems is hard, not everyone is capable of building these systems. They also require languages and frameworks suited to working in clusters. Golang might not scale too well, but its great for the simple things.

> The chosen approach is that we'll interface with something else such as OpenTSDB that'll do the long term storage, and we'll support seamlessly graphing across it. That's much easier to make reliable (just add a timeout), and if it does go down you'd still have the last few weeks of data sitting on the Prometheus box.

Doesn't OpenTSDB require zookeeper? So if your "network falls apart" you have no historical data? I guess thats fine if you're not alerting on data trends, and also have 10 minutes for your graphs to render.