> Object and Array cannot be streamed when writing. They require a 'count' at the beginning
Most languages know exactly how many elements a collection has (to say nothing of the number of members in a struct).
Most languages know exactly how many elements a collection has (to say nothing of the number of members in a struct).
For example, consider serializing something like [fetch(url1), join(fetch(url2), fetch(url3))]. The outer count is knowable, but the inner isn't. Even if the size of fetch(url2) and fetch(url3) are known, evaluating a join function may produce an unknown number of matches in its (streaming) output.
JSON, Protobuf, etc. can be very efficiently streamed, but it sounds like MessagePack is not designed for this. So processing the above would require pre-rendering the data in memory and then serializing it, which may require too much memory.
Protobuf yes, JSON no: you can't properly deserialize a JSON collection until it is fully consumed. The same issue you're highlighting for serializing MessagePack occurs when deserializing JSON. I think MessagePack is very much written with streaming in mind. It makes sense to trade write-efficiency for read-efficiency. Especially as the entity primarily affected by the tradeoff is the one making the cut, in case of msgpack. It all depends on your workloads but Ive done benchmarks for past work where msgpack came up on top. It can often be a good fit for when you need to do stuff in Redis.
(If anyone thinks to counter with JSONL, well, there's no reason you can't do the same with msgpack).
JSON Lines is particularly helpful for JavaScript clients where streaming JSON parsers tend to be much slower than JSON.parse.
For example, pseudo code in a sub-function:
if (that) write_field('that'); if (these) write_field('these');
With messagepack you have to go and apply the logic to count, then again to write. And keep a state for each levels etc.