Breaking up with Flask and FastAPI: Why they don’t scale for ML model serving | Hacker News Reader

Breaking up with Flask and FastAPI: Why they don’t scale for ML model serving

modelserving.com

38 pointsyubozhao42 comments

I have a feeling the root cause is not in FastAPI or Flask, but in the architecture of the system itself.

Why? You are doing the inference in the same request, which is synchronous from the perspective of the caller. The request can be memory-intensive or cpu-intensive. And the issue is you can't efficiently consider all the workloads for a single machine without being bottlenecked by Python.

I would say that the problem is in your approach trying to use the webapp hammer for all the different flavors of nails in your system, using a language that isn't suited for concurrency. What I would do is decoupling the validation/interface logic from your models via a queue. This way you can scale your capacity according to workload and make sure the workload runs on hardware most relevant to the job.

I have a feeling trying to throw a webapp at the problem might not solve your root issue, only delay it in time.

Having designed similar systems, I agree completely. This is not a language or framework problem, but system architecture. I have used RabbitMQ to great success as the message queuing system, and there are many other good alternatives.

I wouldn't put the model under fastapi or any similar framework, i would serve it from a different process to also allow me to serve multiple versions of the model as well (similar to tf serving). But eventually we have an API call to some web framework calling this different process and requiring a response with the model recommendations to be returned in a few milliseconds, how is a message queue appropriate for such a real-time use case, could you elaborate?

Would you describe getting response from the model within few milliseconds cpu or memory intensive? What I assumed as to the characteristics of inference is a process that takes multiple seconds to minutes.

In a system that requires a response to the customer within a few milliseconds through web api, how do you ensure the performance in Python? I'm genuinely interested as that sounds outside of what stock CPython can do except trivial logic.

Another aspect that's completely ignored are the requirement - you might need the response time in milliseconds, couple of times per day. Others might need to serve hundreds of requests. You also need to consider the budget available.

Just to add some ballparks: I have a yolov5 model that I run in PyTorch on a 3080 and the inference time is roughly 20ms. And inference with a 200estimator random forest is a few ms on a cpu. I can imagine that in some scenarios it’s not enough though.

Indeed it might not be enough, and there is no one size that fits them all ;) For inference times of 50ms I wouldn't bother complicating the setup that much. Again, it all depends on your requirement and trade offs available.

Apologies for the noob question but how would FastAPI/Flask know that the job has been successfully completed? Would the worker have to persist ml inference results somewhere and the FastAPI server poll it periodically?

This is a very good question, with a lot of different answers depending on your use-case.

One approach is to translate the synchronous call into an async call plus polling on the webapp side. You push onto a queue, with the callback queue in the message body. But that gives you problems when you want deploy a new version of your webapp - existing connection will be disrupted and the state lost.

Since you need to deal with retries, anyway, you can move the logic into the client itself. It will get the request id on the initial response and then ask the service for results.

You see, this solution can vary wildly depending on your scalability, durability and resiliency requirements. And on your budget. Its not wild to expect the response to be big, so you might want to upload it to s3. You might use websockets, too. Technology gives you a lot of options here, of different levels of complexity and scalability ;)

Thank you for the response. Would this pattern be appropriate for something like ML inference if I’m looking for sub 200ms response time and the results objects are small? The polling interval would need to be very short, and would that become an issue?

Depending on how much time it takes to query your model. If it takes significant portion of your time budget, you could just put a reverse proxy in front of your models and make it route the traffic. I would architect your workers around homogenous traffic as it would be easier to calculate the capacity, and route the traffic, via paths, on the proxy.

In that case, you trade off safety whilst the setup is still simple. But if querying the model takes less that 100ms I would argue whether thats cpu/memory intensive at all. Remember that you still need to allocate and populate the memory, and Python doesn't free the memory back to OS.

Might be a good application of WebSockets or server-sent events with a job queue on the backend. That or polling on the client-side.

If I had my way both Python runners and the runner webapp would not be long for this world.

I will say, though: the runner webapp MVP exists to do basically what you're describing (and keeps an internal queue). Yatai is architected so that the runner instances run on a separate Kubernetes cluster to the pre- and post-processing code that that's run in the main webapp itself, and can be scaled separately.

Regardless of the architecture, you would still need the model server for online serving use cases. And you’re right that it’s best to decouple the validation/preprocessing logic from the model runtime from architecture perspective, and that’s exactly what BentoML does but a regular web framework can not do.

The jump of going from model -> webserver by placing the webserver in the same process as the model is enticing because you can get it to work in under an hour by adding flask/django/fastapi to the env and decorating a function. The problem is that that your model and webserver do NOT scale in the same way, and if you don't realize this fast, you are going to be trying to fit a square peg through a round hole once you have adoption trying to make it work.

All models at scale eventually need to be executed by an async queue processor which is fundamentally different from a request response REST API. For simplicity managing this outside of the process making the web request will help you debug issues when people start asking why they are getting 502 responses. If you are forced to use python for this, I would always suggest of going to celery/huey/dramatiq as an immediate next step after the REST API MVP. I hear Celery is getting better but I have ran into issues over the year so it pains me to recommend it.

Exactly this.

Prime example of the difference is whether you accept disruption of inference when you deploy a new version of your webapp.

Very high chances you don't, thus you will start implementing a queueing mechanism without realizing it.

"FastAPI is not perfect for ML serving"

Yup. There's a huge amount of work that you need to do to do the whole ML lifecycle, and FastAPI doesn't support that out of the box like a full fledged ML Platform.

But you probably don't actually want a full ML Platform because they're all opinionated and if you try and fight them it's often worse than just serving it as an API via FastAPI...

Yeah... I mean the author is blatantly biased, but his low level discussion is cogent.

Exactly! A vertical solution focused on model serving and deployment would make more sense if it can easily integrate with the ML training platform for CICD

Forgive me, I don't mean this flippantly, but it sounds like you implemented queuing and multiprocessing consumers on a Starlette webserver. "micro batching" is a feature enabled by the queueing. The GPU/CPU abstraction is nice, but I feel it's buried by the "FastAPI isn't good enough" digression. If it were framed as "here's what we added to the Starlette ecosystem", I would have approached it much more agreeably.

It would've been delightful to see "instantiate a runner in your existing Starlette application". I don't want to instantiate a Bento service. Perhaps I can mount the bento service on the Starlette application?

Apologies if I am still grossly misunderstanding. I tried to look through some of the _internal codebase to see how the Runner is implemented, the constructor signatures are very complex and the indirection to RunnerMethod had me cross-eyed.

You can absolutely mount a BentoML service into your own Starlette (or any ASGI framework); `svc.asgi_app` is all you'd need.

Instantiating and using a runner can be done anywhere with `init_local`, but it's really the runner ASGI app that does the work of queuing and batching. We've thought about allowing users to spin that app up separately but it's not a focus right now; instead we're trying to ensure that the system is as easy to use for data scientists as possible and have that workflow fully ironed out before we support the more advanced use-cases.

The whole runner situation is quite complex because we wanted to support user-created runners in the nicest way possible, and also leave the space open for non-python runners (and service app) in the future.

That is helpful clarification, thank you.

> We've thought about allowing users to spin that app up separately but it's not a focus right now; instead we're trying to ensure that the system is as easy to use for data scientists as possible and have that workflow fully ironed out before we support the more advanced use-cases.

I like this, and I understand. I might like to take a stab at implementing this more modular interface.

> While FastAPI does support async calls at the web request level, there is no way to call model predictions in an async manner.

This confuses me. How is that FastAPI’s fault? Can’t you just asynchronously delegate them to a concurrent.futures.ThreadPoolExecutor or concurrent.futures.ProcessPoolExecutor? What does Starlette provide here that FastAPI doesn’t? If the FastAPI limitations are due to ASGI, shouldn’t Starlette have the same limitations?

> While there are several different methods to use your own executor pools or potentially use shared memory for a large model, all of these solutions are not first-class solutions for ML use cases...

Definitely not FastAPI's fault and yes Starlette has the same limitations. BentoML builds additional ML features/abstractions on top of Starlette. We introduced a "runner" concept which automatically creates separate processes for models to run in.

Great—but then one ought to be able to performantly use that same runner equally well from FastAPI, Starlette, Quart (ASGI port of Flask), or any other ASGI framework. You’ve decided to build a convenient integration with Starlette instead of the others, but it’s weird to frame this as an argument that other frameworks are ill suited for this domain.

That’s is true, it would be more accurate to say FastAPI (or any ASGI) framework along is not enough for ML model serving, you need batching and runner for performance. And some additional Ml focused features for integration with other ML tools

Advert disguised as experience report.

Disclaimer: I'm the author.

We've been using Flask for years as the foundation for our 0.13 version. Our choice to move away from Flask and FastAPI as a core part of our library is based on our experience with hundreds of users and use cases

So why not Quart if you used Flask?

Nice advertorial but what about a a queue and some machines running torchserve?

We have been using async python for GPU pydata , including fronting dask/dask_cuda for sharing and bigger-than-memory scenarios, so a lot rings true.

For model serving, we were thinking Triton (native vs python server) as it is a tightly scoped problem and optimized: any perf comparison there?

FastAPI is collapsing under the weight of its github issues (1,100) and pull requests (483).

IIRC FastAPI is built on starlette, can some of these issues belong to them ? just hoping the numbers don't reflect a whole reality.

Could you clarify what you mean by "collapsing under the weight"?

Will a project be abandoned because users are pointing out ways to improve it? Does work stop at some point because "we can't get to every issue"? Sure, the maintainer could get burned out, but that is not a given.

Yeah the issues look fairly healthy, even thought there are a lot of them.

Simply could also mean it's thriving. FastAPI has nearly 50k stars. It is very popular and very effective in what it promises. Active repos with similar popularity have way, WAY more issues and pull requests, projects like Apache/echarts, Godot, Redis or Grafana.

Projects with similar stars, momentJS or Jquery don't see much active development have very few issues and pull requests.

Wait... so you're tell me FastAPI is slow...

FastAPI is great building block but can't expect it to work for model serving out of box.

Load your pickle file and it works?

I know, pickle is insecure, etc

Most team don't really concern about the security side of things when using pickle for model deployment, more about performance and resource utilization. It works ok for very light weight models, but doesn't scale.

Actually it's very slow. Wait until you have 50 concurrent customers hitting it.

That's weird, it is supposed to be faster than alternatives. Do you have some benchmarks to support that critique?

Reply on news.ycombinator.com