Hacker News Reader

Comment by kroolik | Hacker News Reader

I have a feeling the root cause is not in FastAPI or Flask, but in the architecture of the system itself.

Why? You are doing the inference in the same request, which is synchronous from the perspective of the caller. The request can be memory-intensive or cpu-intensive. And the issue is you can't efficiently consider all the workloads for a single machine without being bottlenecked by Python.

I would say that the problem is in your approach trying to use the webapp hammer for all the different flavors of nails in your system, using a language that isn't suited for concurrency. What I would do is decoupling the validation/interface logic from your models via a queue. This way you can scale your capacity according to workload and make sure the workload runs on hardware most relevant to the job.

I have a feeling trying to throw a webapp at the problem might not solve your root issue, only delay it in time.

View on HN

Having designed similar systems, I agree completely. This is not a language or framework problem, but system architecture. I have used RabbitMQ to great success as the message queuing system, and there are many other good alternatives.

I wouldn't put the model under fastapi or any similar framework, i would serve it from a different process to also allow me to serve multiple versions of the model as well (similar to tf serving). But eventually we have an API call to some web framework calling this different process and requiring a response with the model recommendations to be returned in a few milliseconds, how is a message queue appropriate for such a real-time use case, could you elaborate?

Would you describe getting response from the model within few milliseconds cpu or memory intensive? What I assumed as to the characteristics of inference is a process that takes multiple seconds to minutes.

In a system that requires a response to the customer within a few milliseconds through web api, how do you ensure the performance in Python? I'm genuinely interested as that sounds outside of what stock CPython can do except trivial logic.

Another aspect that's completely ignored are the requirement - you might need the response time in milliseconds, couple of times per day. Others might need to serve hundreds of requests. You also need to consider the budget available.

Just to add some ballparks: I have a yolov5 model that I run in PyTorch on a 3080 and the inference time is roughly 20ms. And inference with a 200estimator random forest is a few ms on a cpu. I can imagine that in some scenarios it’s not enough though.

Indeed it might not be enough, and there is no one size that fits them all ;) For inference times of 50ms I wouldn't bother complicating the setup that much. Again, it all depends on your requirement and trade offs available.

If I had my way both Python runners and the runner webapp would not be long for this world.

I will say, though: the runner webapp MVP exists to do basically what you're describing (and keeps an internal queue). Yatai is architected so that the runner instances run on a separate Kubernetes cluster to the pre- and post-processing code that that's run in the main webapp itself, and can be scaled separately.

Regardless of the architecture, you would still need the model server for online serving use cases. And you’re right that it’s best to decouple the validation/preprocessing logic from the model runtime from architecture perspective, and that’s exactly what BentoML does but a regular web framework can not do.

Apologies for the noob question but how would FastAPI/Flask know that the job has been successfully completed? Would the worker have to persist ml inference results somewhere and the FastAPI server poll it periodically?

This is a very good question, with a lot of different answers depending on your use-case.

One approach is to translate the synchronous call into an async call plus polling on the webapp side. You push onto a queue, with the callback queue in the message body. But that gives you problems when you want deploy a new version of your webapp - existing connection will be disrupted and the state lost.

Since you need to deal with retries, anyway, you can move the logic into the client itself. It will get the request id on the initial response and then ask the service for results.

You see, this solution can vary wildly depending on your scalability, durability and resiliency requirements. And on your budget. Its not wild to expect the response to be big, so you might want to upload it to s3. You might use websockets, too. Technology gives you a lot of options here, of different levels of complexity and scalability ;)

Thank you for the response. Would this pattern be appropriate for something like ML inference if I’m looking for sub 200ms response time and the results objects are small? The polling interval would need to be very short, and would that become an issue?

Depending on how much time it takes to query your model. If it takes significant portion of your time budget, you could just put a reverse proxy in front of your models and make it route the traffic. I would architect your workers around homogenous traffic as it would be easier to calculate the capacity, and route the traffic, via paths, on the proxy.

In that case, you trade off safety whilst the setup is still simple. But if querying the model takes less that 100ms I would argue whether thats cpu/memory intensive at all. Remember that you still need to allocate and populate the memory, and Python doesn't free the memory back to OS.

Might be a good application of WebSockets or server-sent events with a job queue on the backend. That or polling on the client-side.

Reply on news.ycombinator.com