> Most SaaS providers would rather push data than have it pulled, is what I'd imagine. Are you hearing otherwise from folks you've been speaking with? For instance, in serverless environments (which is the poison of choice for me, at least), pull is much harder to accomplish, even where possible.
We totally offer Push based as well, all of our workers are just using the same API endpoint to push the data we collect to, its just not the strong highlight since other providers have push offered.
We went down the pull path since during our discovery process about 6 months ago, we were chatting with some DB and Infra companies who just built out and integrated with a billing provider. All of them mentioned being annoyed with the amount of engineering commitment it took for them to measure, persist, and then transmit the usage data to the provider for them to handle the rest. So we wanted to offer a pull based solution to help with this need.
You're totally right that its architecture dependent, and we don't want to cause a huge load (and cost) on a serverless platform. So for some dimensions, push is definitely an option.
> Apparently, a BigCloud, in perhaps a case of NIH, ended up creating a highly-parallel event-queue as a direct result of the scale it was dealing with: https://archive.is/IUKvT Curious to hear how you deal with the barrage of multi-dimensional events?
So to dive into more technical detail, we have an event queue where our workers drop data off to and then it gets persisted into our ledger, by workers reading from the queue. We have the queue hosted by a major cloud platform and its offered as a managed service, similar to kinesis.
For the different dimensions, we have a standard dataformat they need to be in before we can persist them, this transformation typically occurs on the client side, though in some cases we can transform the data from an open standard format (Prometheus https://prometheus.io/docs/concepts/data_model/) to our backend format.
At a criminally high level, this data format consists of a measurement, a value, a field, and a set of metadata tags. Our ledger is built on a schema-less Timeseries DB, so it doesn't matter if the same measurement has a different set of metadata from another. This gives us a boat ton of flexibility when it comes to how we want to query data.
For the different types of dimensions and their different data types, it becomes an issue when wanting to aggregate on them. For instance, a you may want the Total of one dimension, however Average and Count wouldn't make any sense.
To get by this, clients need to tell us what Aggregation method to use per dimension.
This really isn't present in the demo, since its a fairly simplistic version of the whole app, but its a requirement we have implemented into the API.
> 2 cents: Fly.io Machines is a tremendous platform
Thanks for the hot tip, looks awesome! I'll definitely check this out and how we can sneak it into our product.