Backpressure – the resisted flow of data through software
medium.com
medium.com
> Backpressure Strategies: control, buffer, drop
One of my favorite hacks for Gluster (at Facebook) was in this area. We were using bog-standard NFS as the connection protocol, so neither explicit control nor dropping were options. (People who have always had that luxury with their own thrift/grpc/whatever protocols and libraries don't realize how much of a pain it is not to have it.) That left buffering, which led to memory pressure and latency bubbles. What I ended up implementing was implicit control. We'd just stop reading on a particular client's socket for increasing numbers of milliseconds if overall latency was too high and its queue was too long. This leveraged information we already had, and "borrowed" TCP's flow/buffer control as our own. Clients couldn't ignore it; they'd block in send() or somewhere equivalent. It worked really well to smooth out many of the worst latency spikes. If you're ever stuck in a similar situation, I highly recommend this approach.
- https://blog.booking.com/using-riak-as-events-storage-part4....
https://hexdocs.pm/gen_stage/GenStage.html
and these additional abstactions are built on top of it:
I wish there was an easier built in way to do it, but a producer could read the backlog length of it's consumers and try to throttle itself.
What's different now, as the article points out, is the growth of micro-service architectures. Effectively creating a lot of tiny producer/consumer problems across the solution space.
Sometimes, I'm glad I don't write code anymore, but only sometimes :-)
[1] https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_prob...
Surely in most business scenarios letting data back up on queues for a while suffices?
In the situations where it doesn’t, add some more capacity.
Back pressure is something I’ve never had to deal with even in HFT and electronic trading situations.
Sometimes "add more capacity" just costs too much.
OTOH, I'm sort of amazed by the people who reach for queues to solve loading issues without first assuring that the system works properly at full load without any queuing. Queue's only serve to smooth out busty behavior and come with their own problems (buffer bloat related latency).
Well, two things.
First, our clients didn't have a "simple" back off, retry system. They tended to have more sophisticated back-off and retry systems that didn't increase pressure on services that were overloaded. There are a number of different techniques you use to accomplish this and you can wrap up several of these techniques in a library, use the library in your client code, and have that client code (a fat client) be the official API for your service.
Second, this also assumes that front-end load is significant to begin with. In our system, it definitely wasn't. Requests were super cheap. You could hammer the front-end extremely hard and it would just respond with "try again later" until there was some capacity available, and then it would let in some work. The size of an actual request was quite small, it was the size of the work represented by that request that was large.
> OTOH, I'm sort of amazed by the people who reach for queues to solve loading issues without first assuring that the system works properly at full load without any queuing. Queue's only serve to smooth out busty behavior and come with their own problems (buffer bloat related latency).
Queues in our system were a necessary part of the design. The system would not operate efficiently without them, because work items needed to be batched in order to be processed efficiently. I don't think there's a way that you can indict a system for using queues unless you know something about the design requirements.
There are absolutely other reasons to use queues other than smoothing out bursty behavior.
I've worked in the embedded space now for about a decade (network, storage, databases). Everywhere I've been where the product has no backpressure, the product inevitably suffers from the above issues, which translate directly into customer dissatisfaction. And it's never easy to add backpressure to a system which doesn't have it.
It's also not always an option to just "add more capacity" -- there are many situations where that's not possible, or requires time to enact. Meanwhile, your queues are growing without bound.
Build backpressure in from the start.
1) Queues should be of finite size (exactly what size is entirely application dependent, but the point is that it's almost never ok to let a queue grow indefinitely).
2) When a queue is full, producers should often be blocked (backpressure).
Obviously it depends a lot on the particular use case, but IME it's usually risky to think that a 'sufficiently large' queue is going to be an acceptable solution.
> In the situations where it doesn’t, add some more capacity.
That obviously has its limits too. Some people actually care about cost. Also, added capacity might be totally wasted if there's any sort of affinity between servers and data. Maybe it works if you're in a stateless middle tier, but not e.g. in a storage system. You can add capacity to those, sure, but rebalancing the data to take advantage of the new capacity can be a lengthy process and compete with user traffic (making user-perceived latency even worse) if you're not careful. I've also seen too many cases where rebalancing undid the careful work done during initial data placement to maximize data safety. Oops.
> Back pressure is something I’ve never had to deal with
Count your blessings.
afaik the only time a modern software developer needs to worry about backpressure is if they're using asynchronous or otherwise non-flow-controlled i/o. neither of these cases nor even a discussion of synchronous vs. asynchronous i/o were mentioned.
Also eventually you might need a worker pool pulling from a queue. Just guessing, because they seem to appear eventually in/near web servers. You can spin up a bajillion lambdas simultaneously like a madman if you want, but odds are there’s a resource constraint on the source and/or destination data store side at least. So again you’ve got async execution, a growing queue, and bottlenecks to think about still.
But maybe with threaded code it’s like a “pit of success” for traffic shaping. You only have so many threads you can use, so you configure a pool size and think about what happens when you’re at capacity, and define some nice predictable failure behavior and test it. (As opposed to the node.js server that will handle more connections but surprise you with a weird bottleneck that you don’t handle gracefully). So then the trad server developer can go longer (or forever) without being bitten by the queues that remain elsewhere in their system.
I worked on a system a while back which processed petabytes of data... autoscaling was out of the question (we had a fixed amount of hardware to process the data, and the lead time for getting new hardware was not especially short), and buffering the data would require eye-watering amounts of disk space. We just buffered as much as we could and then returned a "try again later" error code. We made sure that data accepted by the system was processed within a short enough window. We made sure that the cost of submitting a work item was so small that you could really hammer the front-end as much as you wanted, and it would just say "try again later" or accept work items.
I think one of the lessons here is that you need to think long and hard about what error conditions you want to expose upstream. The farther you propagate an error condition, the more different ways you can solve it, but the more complicated your clients get. For example, disks sometimes fail. You can decide to not propagate disk failure beyond an individual machine, load up RAID 1 (mirror) in all your file servers, and back everything up. Or you can push disk failures farther up the stack, and recover from failed disks at a higher level, with lower hardware cost but higher implementation complexity. And if you build a bunch of systems assuming that each part is always working, you run very serious risks of cascading failures once one part does inevitably fail.
Obviously, small enough systems running on cloud hardware can usually be autoscaled just fine.
Or can we add autoscale? I encountered this exact problem today as I were looking for a simple to setup and maintain autoscaler solution. The programs feed in the backpressure metrics to monitor, the autoscaler would tell me how many pods/consumers there needs to be, based on naive estimation or prediction by yesterday's traffic.
K8s seems to be an over kill and with stateful pods problems, KEDA looks promising, but I wish there's much simpler ones.
I don't (try to) control or manage backpressure. Backpressure controls - or rather, steers - me/my code. Getting backpressure right is massively important and frequently under-considered design aspect.
The most significant thing to understand IMHO is that you need to look at backpressure in the context of the larger system. More concretely, you need to ask yourself "is this telling me that another task should run right now"? Both for synchronously driven sources (e.g. blocking write() syscall) as well as asynchronously driven sources (event loop with "can send" flag), you need to make sure you (a) don't block other progress (held locks on DB) and (b) actually pull up other tasks (particularly important for partially-asynchronous code).
And this leads straight to the single most impactful design/scaling decision for these systems: is the source maintaining its outbound scheduling at a high enough level? This is best clarified with an example: say you have a key/value store that sends out change notifications per key. Does your application need every single step, or just the most recent value? And if it's the latter: you shouldn't even have a "true" queue! Instead, the code needs to track which keys are pending for which receiver, and whenever the receiver can accept data, it needs to send the most recent data for pending keys. This is the difference between an unbounded queue and a bounded by number of keys queue. The former can fail disastrously, the latter will just start lagging but once it reaches the worst case, it just stays there.
> For example, if someone says: “I made this new library with built-in backpressure…” they probably mean it somehow helps you manage backpressure, not that it actually intrinsically is the cause of it. Backpressure is not “desirable” except when it’s inescapable and you want to protect something else from receiving it.
… and this really encapsulates my disagreement. Backpressure is absolutely desirable. I want my libraries to provide me with backpressure when there is reason to do so, and do it in a way I can actually "consume" the backpressure!
And lastly:
> Backpressure Strategies: control, buffer, drop
"buffer" and "drop" are in most cases just bugs. And very hard to debug & track down, due to their load dependency.
Explaining backpressure was a lot of work. With zero reward. Less so for fixing it.
I was bemused that the entire nodejs ecosystem, community was oblivious to this issue. Hard to take any of it seriously.