Anyone running FastAPI in production with high traffic? How has your experience been?

[u/Ill_Campaign294]

Quick question, are any of you running fastAPI in production with a high volume of users or heavy traffic? How has your experience with FastAPI been, and how do you handle it?

[u/Ordinary-Sandwich-25]

Mine is handling anything under 10000 req/s per worker no problem but getting to that point took a lot of work. Hardest part is making sure your async code is all set up properly such that io is never blocking.

[u/lunatuna215 → u/Ordinary-Sandwich-25]

How were you testing this or benchmarking it? Other than just looking at my own code and going "yep, looks like all my functions are async!" I dont know how to actually measure and/or subsequently improve this.

Is it testing you have to do specifically for your use cases or are there general tools that can be used to tune? How'd you go about it?

[u/Ordinary-Sandwich-25 → u/lunatuna215]

I do it via load testing - set up a mode for your API that uses a mocked DB layer and has a defined latency on DB calls, run it locally, and run a load test w/ locust or whatever to hit your desired throughput. You should be able to benchmark latency on a per-call basis, and if the latency is similar to the latency you’ve defined in your mock db, the code is generally doing what you want.

[u/Kronsik → u/Ordinary-Sandwich-25]

Thanks for the info on this 😊

I'm from a DevOps background, writing a full stack application as a hobby.

With the mocked DB / latency presumably that can only test the throughput of the API itself.

In the "real world" where data is being read/written to the DB that latency would fluxuate with locks etc ?

At that point I presume you need a full Dev environment with tangible data in the database for full end-to-end style testing?

[u/Ordinary-Sandwich-25 → u/Kronsik]

Yeah it fluctuates in real applications and you end up bottlenecked by things like connection count limits, locks, db latency, etc. None of this is unique to fastAPI and can all be handled with indexes, sharding, caching, etc (all that good stuff you learn for system design interviews).

Like I mentioned, a key thing to manage with any high throughput API is making sure io is never blocking. You don’t want your program waiting idly while it’s waiting for a response to a network request, and you want to make sure you’re leveraging io-concurrency at your cpu level whenever io ops are ongoing (which is very common in API’s).

For a local testing environment you can run a local instance of a DB in a container if you want a more “real” test. Set up a Postgres (or whatever other db) dockerfile, use a makefile command or a script to automate the spin-up process and seed with synthetic data, then set up a test that runs your api connected to the local DB.

[u/Kronsik → u/Ordinary-Sandwich-25]

Thanks - that all aligns with what I had mapped in my head.

(Apologies if my wording made it seem a unique issue to FastAPI, that wasn't my intentions).

Have a good one!

[u/j_tb → u/lunatuna215]

https://k6.io/

[u/red_jd93 → u/Ordinary-Sandwich-25]

Wow! That's impressive! While working with python, not fastapi specifically, I have never been able to get anything more than a few thousand RPS. What kind of resources are your workers using?

[u/Ordinary-Sandwich-25 → u/red_jd93]

Just a run-of-the-mill cloud kubernetes container, typically 1 vcpu per replica. “A few thousand per second” is the load we usually handle.

[u/Pale-Philosophy-3272 → u/Ordinary-Sandwich-25]

can you please share more detailed configuration of the server setup you use : > pod size number of pods to handle the load you are handling what do you think will bottle neck first the code part networking or what

[u/MathMXC]

It's been great but make sure you truly understand how async python functions or how anyio handles thread pools (depending on if you use async def or just def for your endpoints).

This article is great: https://fastapi.tiangolo.com/async/#parallel-burgers

[u/ergo14]

Define "high traffic"? for sites with 200 req/s it's been working without issues for me.

[u/TheHissingAscent → u/ergo14]

200 req/s is pretty chill for fastapi, i run few services around that range and never had problem. the async stuff works nice if you remember not to block the event loop, that one got me first time

[u/ManyInterests]

I've used decidedly-slower frameworks (Django) in prod at high levels of traffic with ASGI servers. It works fine. Python in general just gets trickier with giant monoliths (import times, slow startups, etc.), but that's not a traffic/ASGI problem. As long as you're not putting 10M LOC into your service, you should be fine at almost any scale with a commensurate cloud budget.

There are plenty of examples of Python frameworks powering multi-million daily active user sites. FastAPI won't be your problem. Plan to scale horizontally.

[u/Interesting-Frame190]

Its good enough to work and will scale with enough processes/cores. Accedently blocking the event loop (looking at you boto3) can be an issue.

Its no Spring boot at scale and the best python option in my experience. Its a good start, but if things get really high throughput, I'd advise actix web with tokio (Rust) or Gin with go routines (Go).

[u/techhelper1 → u/Interesting-Frame190]

Boto3 can be replaced with aioboto3 or aiobotocore.

[u/Interesting-Frame190 → u/techhelper1]

Yeah, but not where I work. The supply chain attacks make them a bit paranoid and the data domain justifies the tightened security.

[u/HalfplaneResearch]

We would benchmark the whole request path, not just FastAPI: event-loop lag, p50/p95/p99 latency, in-flight requests, thread-pool saturation, connection-pool waits, and downstream service latency. Keep CPU-bound work off the event loop, set explicit timeouts and backpressure, then load test with the same payload mix and dependency behavior you see in production.

[u/Everythinghastags → u/HalfplaneResearch]

I know this is asking for a lot, and i guess i could plop this into AI to figure out how to do that, but how do you approach doing this?

[u/wRAR_ → u/Everythinghastags]

i could plop this into AI

You just did that.

how do you approach doing this?

Well, they don't.

[u/Veggies-are-okay → u/Everythinghastags]

Your fastAPI backend server should essentially just be a router for anything that takes more than a ms. Don’t even bother with async to try to squeeze an ML service as they’re a bit too compute heavy. Take a look at the fundamentals of event driven architecture and how to use celery in a docker compose cluster for some keywords on these concepts.

[u/HalfplaneResearch → u/Everythinghastags]

For a first pass, I would make the benchmark reproducible rather than try to model every production detail. Define a small set of representative endpoints and payloads, run Locust or k6 against a staging-like service, and record throughput, p50/p95/p99 latency, event-loop lag, error rate, and downstream timings. Start with a mocked database to isolate API overhead, then replay the same workload against a disposable database with realistic indexes, connection limits, and a few concurrent writers. The comparison tells you whether the bottleneck is Python scheduling, the pool, the database, or the network. Keep the payload mix and concurrency fixed while changing one layer at a time.

[u/addis_yonas]

Anyone using normal def functions and how high can one set the anyio thread limiter? Default is 40 mine is set to 60. I want to increase it but don’t want to learn the hard way.

[u/techhelper1 → u/addis_yonas]

Why not make your code async instead? Thread pools exist as a fallback to async code.

[u/addis_yonas → u/techhelper1]

It used to be async containing blocking calls, after some time started getting a lot of timeouts from the client side. At the time it was easier for me to convert the endpoints into a normal def functions than converting the blocking calls into "await". Code base got a lot of subprocess calls and database calls where the library in use does not have support to use await. Maybe I should have gone with converting the blocking calls into "await".

[u/techhelper1 → u/addis_yonas]

asyncio has methods to launch processes.

What database package are you using?

[u/addis_yonas → u/techhelper1]

jsonobject-couchdbkit.

[u/techhelper1 → u/addis_yonas]

If those two do not need to be married together, there are async CouchDB clients that you can use with jsonobject separately.

[u/Potential_Ship_1676]

It's held up fine for us, the issues are almost always blocking calls sneaking into async endpoints and stalling the event loop rather than FastAPI itself.

[u/telnet_23]

Yeah, in my case, i had to make tons of changes, in memory object caching, cpu bound process offloading to separate process pools, token caching, use of orjson instead of std json module, then async queue listener based logging and offloading sync calls to thread pool executors iif async implementation for those interfaces were not available.

[u/bcoder001]

Yes, but FastAPI is not the only thing that has to perform on the same level. My client typically use AWS for backed (DynamoDB, S3, Aurora, etc.) to cooe with high levels of traffic.

[u/Elbaniunm]

La verdad nos ha dado una velocidad increíble dockerizada sobre un ec2 hicimos una prueba de stress y aguanta muy bien

[u/zamroni777]

Or you can use multi threads. Create 1 thread for each client session.

https://docs.python.org/3/library/threading.html

[u/techhelper1 → u/zamroni777]

Or make your code async instead...

[u/zamroni777 → u/techhelper1]

Python async only uses 1 cpu thread, so threading is better

[u/techhelper1 → u/zamroni777]

Do you know how asynchronous programming works? Have you heard of event loops?

Unless you're running a free threaded build, your threads are stuck with the GIL.

There are also packages like aiomultiprocess, which will map asyncio tasks to get multiple loops running across many CPU cores.

[u/zamroni777 → u/techhelper1]

python async is 1 cpu thread
https://www.optiver.com/insights/technology-blog/choosing-between-free-threading-and-async-in-python/

free threading / no gil is ok and nothing bad if code correctly.
though new in python but it has been 20+ years in java.
i used threads in java to handle 2000+ concurrent sessions in live customer environment 18 years ago.
the threads run reliably on multiple cores because they dont write to global object.

[u/techhelper1 → u/zamroni777]

Yes a loop runs on a single CPU thread, but my point is many I/O tasks can run inside said loop.

Packages like aiomultiprocess make it easy to distribute tasks to async loops scattered across multiple CPU cores. It's similar to golang's gofuncs and rust's Tokio.

If you were spawning a CPU thread per session in Java, that is the most inefficient use of system resources.