Hacker News Reader

Comment by jononor | Hacker News Reader

Having designed similar systems, I agree completely. This is not a language or framework problem, but system architecture. I have used RabbitMQ to great success as the message queuing system, and there are many other good alternatives.

View on HN

I wouldn't put the model under fastapi or any similar framework, i would serve it from a different process to also allow me to serve multiple versions of the model as well (similar to tf serving). But eventually we have an API call to some web framework calling this different process and requiring a response with the model recommendations to be returned in a few milliseconds, how is a message queue appropriate for such a real-time use case, could you elaborate?

Would you describe getting response from the model within few milliseconds cpu or memory intensive? What I assumed as to the characteristics of inference is a process that takes multiple seconds to minutes.

In a system that requires a response to the customer within a few milliseconds through web api, how do you ensure the performance in Python? I'm genuinely interested as that sounds outside of what stock CPython can do except trivial logic.

Another aspect that's completely ignored are the requirement - you might need the response time in milliseconds, couple of times per day. Others might need to serve hundreds of requests. You also need to consider the budget available.

Just to add some ballparks: I have a yolov5 model that I run in PyTorch on a 3080 and the inference time is roughly 20ms. And inference with a 200estimator random forest is a few ms on a cpu. I can imagine that in some scenarios it’s not enough though.

Indeed it might not be enough, and there is no one size that fits them all ;) For inference times of 50ms I wouldn't bother complicating the setup that much. Again, it all depends on your requirement and trade offs available.

Reply on news.ycombinator.com