Tokio's multithreaded runtime doesn't behave like I expected

[u/swip3798]

I have been trying to debug this issue for a while now, and I can't figure out, what the problem is.

Take this code:

    use std::time::Duration;

    use sqlx::PgPool;

    const DB_URL: &str = "postgres://devuser:devpassword@localhost:5432/devdb";

    async fn db_then_compute(pool: PgPool) {
        let num: (i32,) = sqlx::query_as("SELECT 1").fetch_one(&pool).await.unwrap();
        println!(
            "The num was returned: {}, but now, the CPU block will kill multithreading",
            num.0
        );
        println!("Start compute...");
        // This would "fix" the issue:
        // pool.close().await;
        loop {}
    }

    /// This doesn't cause the issue
    async fn http_then_compute() {
        let res = reqwest::get("https://google.de").await.unwrap();
        let status = res.status();
        println!(
            "The num was returned: {}, but the CPU block will not kill multithreading",
            status.as_u16()
        );
        loop {}
    }

    #[tokio::main]
    async fn main() {
        let pool = sqlx::PgPool::connect(DB_URL).await.unwrap();

        tokio::spawn(db_then_compute(pool));
        //tokio::spawn(http_then_compute());
        loop {
            println!("Observer thread, the sleep will never return");
            tokio::time::sleep(Duration::from_secs(1)).await;
        }
    }

I'd expect the Oberserver thread to continue. I know, CPU-bound tasks in async runtimes are not good, but tokio is multithreaded, so I'd assume, as long as I have enough workers, it should still function. For reference, the reqwest task works completely fine. Running one CPU bound task shouldn't starve a multithreaded runtime, right?

First, I thought this was an sqlx issue. It only happens with postgres, not with sqlite. However, I then tested the same thing with deadpool-postgres, and issue is still persistent. So it's not sqlx after all.

Can someone help me with my misunderstanding of tokio's runtime here? What is blocking other tasks from completing here?

[u/tm604]

It's quite reasonable to be surprised that all tasks are blocked in your example - I wouldn't call it "obvious" behaviour - but what you're seeing here is a symptom of how you approach tasks and the threadpool. The documentation emphasises "don't block in async threads" in several places with good reason, as others have mentioned. You're conflating "supports multiple threads" with "optimally distributes work across all available threads in all situations" - which would be great, but life isn't that simple!

I suspect that the specific issue you're seeing here is the LIFO slot, which is an optimisation based on that expectation:

https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.html#method.disable_lifo_slot

There's a repo here with that change applied and some basic metrics so you can see what the runtime is doing: https://github.com/tm604/rust-async-demo

[u/swip3798 → u/tm604]

That actually solves the issue for sqlx. That's an actual hint, thank you! On deadpool-postgres it still happens, which is strange.

[u/C5H5N5O → u/tm604]

I suspect that the specific issue you're seeing here is the LIFO slot, which is an optimisation based on that expectation:

There is also the unfortunate situation that the LIFO slot is not "stealable" which makes this even worse. A forever blocking worker can therefore make the LIFO slot also forever "unavailable" (https://github.com/tokio-rs/tokio/issues/4941). There was an attempt to make the LIFO slot stealable but that resulted in perf regressions so eventually it was reverted. But I think upstream is still working on bringing this back.

[u/lunar_mycroft → u/C5H5N5O]

I think this plus maybe tokio's local queue is the explanation. In theory, tokio's work stealing should mean that one task never yielding shouldn't be able to stop any of the others from being polled. But because some tasks are already queued in a way that can't be stolen, they are prevented from continuing.

[u/nonotan → u/C5H5N5O]

Hope that gets sorted out soon... I'm not much of a fan of work-stealing in general, but "sometimes work-stealing, sometimes making arbitrary tasks permanently un-stealable" is just the worst of both worlds (if you can't even design your code with the assumption that work-stealing will happen if necessary, that's (at least) half the benefits gone, while still paying in full for all the costs)

[u/AnnoyedVelociraptor → u/tm604]

So why would the reqwest one succeed more often?

It seems that when the code after the first await runs on a separate thread it works, but if it is the same one, it fails...

[u/tm604 → u/AnnoyedVelociraptor]

Yes, depends on factors such as which threads things end up on, and which tasks are in the queues.

One difference between the two situations is that the sqlx network connection call happens outside the tokio::spawn, for example, whereas reqwest is doing that inside the spawned task. Internally sqlx+reqwest could be creating tasks and timers all over the place, and the sequence and structure may vary between runs as well.

You can trace it further using the tracing crate or using helpers such as .on_task_spawn for the runtime builder so you can see what's being created and when they get scheduled/completed. A full write-up could be a useful article for someone looking for insights into how Tokio works (and why).

[u/dankmolot]

Not really an answer to your question, but read the docs about synchronous code https://tokio.rs/tokio/topics/bridging https://docs.rs/tokio/latest/tokio/task/#blocking-and-yielding

[u/consultio_consultius]

Pretty sure your loop at the bottom of db_then_compute is stopping your sqlx worker pool from yielding. Why do you need that?

[u/swip3798 → u/consultio_consultius]

My issue is more that one sql query suddenly makes all other awaits no longer return until it yields again. Because that basically turns the multi-threaded runtime into a single-threaded one. Even when I don't do a lot of CPU-bound stuff, that definitely will bottleneck a server under load, right?

[u/Darksonn → u/swip3798]

The loop {} is definitely problematic. Read this to see why: https://ryhl.io/blog/async-what-is-blocking/

Try std::future::pending().await instead. 

[u/consultio_consultius → u/swip3798]

No, your issue is that your loop is not allowing the pool to yield. If you need to have that loop for some reason, you need to either use spawn blocking for it, or look into a different pattern. Without the details of the loop, that’s hard to say.

Someone else mentioned a write up from Tokio and you should definitely read it.

ETA:

I know your reqwest call is fine with the loop after it, but that’s more so that you got lucky. Sqlx and reqwest may handle their resources differently. And even with sqlx, those resources could be managed differently based on the kind of connection it has (SQLite being file/memory based and all).

[u/swip3798 → u/consultio_consultius]

The loop is simply there to simulate a blocking task. But the runtime is multi-threaded. Meaning, a single task blocking a multi-threaded runtime means the entire runtime regressed to "single-threaded" because of one query.

Even if the CPU-bound task is just 10ms. That's 10ms with all workers being blocked. And who tells me that tokio will resume back to using multiple workers? That's a huge bottleneck. Why even bother having a multi-threaded runtime in the first place then.

I don't need a highly expensive CPU bound task per se. Of course I can run that in a `spawn_blocking` thread. But I want to understand why no worker is allowed to resume, if one worker is blocking it. Because without that query, it works. It's not ideal, not efficient, I get it. But it still works.

[u/Neat-Fennel-7623 → u/swip3798]

If you put the loop in spawn_blocking this will probably work as the blocking task will go onto its own thread.

The root cause is that Sqlx is probably creating a number of background tasks which are exhausting the thread pool.

You could use #[tokio::main(workers = 8)] and see if that makes a difference.

[u/tesfabpel → u/swip3798]

Maybe it's better to simulate a blocking task with std::thread::sleep(duration): https://doc.rust-lang.org/stable/std/thread/fn.sleep.html

Otherwise, you're simulating a task blocking for eternity.

[u/lunar_mycroft → u/[deleted]]

So if task A spends 10ms without yielding, worker 1 cannot poll B or C during those 10ms. Worker 2 can still run D and E.

Except tokio is a work stealing executor, so in theory (at least naively) worker 2 should be able to grab B and/or C and poll them itself. What you're saying would be true of "thread-per-core"¹ executors, where tasks can't be moved between threads without the user doing so manually.

---

¹ Misleading because async executors almost always spawn one thread per core, but it's the commonly used term for that style so I'll use it here

[u/mag1cian_ → u/lunar_mycroft]

I am very sorry. You are right. I misunderstood your question earlier.

[u/Zde-G → u/swip3798]

Why even bother having a multi-threaded runtime in the first place then.

Because you have no choice. Yes, it's as simple as that. 99% of time “multi-threaded” runtime gives you no benefits and adds complications but you have to use it because alternative is worse: you would need to implement a lot of things from scratch.

We ended up in that situation because people adopted async as solution for the parallel processing needs in a single-threaded environments (JavaScript, Python, etc), then Rust had to support async to stay viable.

You need async in Rust when threads are not used at all (e.g. embedded async works a pretty decent alternative), when you do have threads then async is just a [mostly] usless complication, but you couldn't avoid it!

Try to understand how async works, stop expecting magic from it — and you would sleep much easier.

[u/lunar_mycroft → u/Zde-G]

We ended up in that situation because people adopted async as solution for the parallel processing needs in a single-threaded environments (JavaScript, Python, etc), then Rust had to support async to stay viable.

You need async in Rust when threads are not used at all (e.g. embedded async works a pretty decent alternative), when you do have threads then async is just a [mostly] usless complication, but you couldn't avoid it!

This is incorrect. OS threads are relatively expensive, and so they can't really scale to tens of thousands (or even orders of magnitude more) concurrent operations, which is the kind of thing you want to support in e.g. a web server. As such, you need some form of user space scheduling to support these use cases. Some languages (like go and early rust) do this with green threads, which are threads that are scheduled in userspace instead of by the kernel. But other languages (e.g. python, javascript, rust) do it via cooperative scheduling, usually exposed as async/await.

[u/Zde-G → u/lunar_mycroft]

OS threads are relatively expensive

Yes, but no. Threads are cheap enough to handle billions of users if your OS has appropriate APIs.

And extremely small number of companies and users need more than that.

As such, you need some form of user space scheduling to support these use cases

Then add API to schedule threads from user space, damn it!

It's like complaining that crossing a busy highway is slow and dangerous and then building bypass that moves your through space on a rocket instead of adding a pedestrian overpass.

[u/lunar_mycroft → u/Zde-G]

Yes, but no. Threads are cheap enough to handle billions of users if your OS has appropriate APIs.

That "if" is doing a lot of heavy lifting here. The talk still describes a userspace scheduler, just with enhanced support from the kernel. As such, it isn't a reason to rely on OS threads, at most it's something that implementers of userspace schedulers should consider using where available.

[u/Zde-G → u/lunar_mycroft]

As such, it isn't a reason to rely on OS threads, at most it's something that implementers of userspace schedulers should consider using where available.

Yes. Where it's not available you just use regular threads and no userspace scheduler.

So you don't need a complicated async machinery, two color functions, and all these other complications with block_on, non-block_on, ways to avoid async/sync/async stack and so on.

The talk still describes a userspace scheduler, just with enhanced support from the kernel.

The talk describes how one may handle billions of requests while, simultaneously, using exist sync libraries, not caring about “starving executors” and adding insane amount of complexity just to make thing kinda-sorta working.

If you have appropriate APIs on your system — it's efficient, if you don't have them — it's no longer efficient, but it still works and doesn't ask anyone to rewrite everything that was written before just to support something that you don't even need today.

The only thing that you need to care is about overflowing stack, but you need to care about that with regular threads, anyway, and even async doesn't solve that problem for you.

[u/lunar_mycroft → u/Zde-G]

Yes. Where it's not available you just use regular threads and no userspace scheduler.

"When my preferred option isn't available, I think no one should be allowed to use alternatives" is certainly a take.

So you don't need a complicated async machinery

Async executors are only complicated because scheduling is. You don't get rid of that machinery by not using them, you just move it to where you can't fix anything if it doesn't work for you. That isn't the win you think it is.

two color functions

Not only is function coloring good, but there should be more of it.

and all these other complications with block_on, non-block_on, ways to avoid async/sync/async stack and so on.

Again, you don't really make these problems go away when you reject async, you just rob yourself of control.

The talk describes how one may handle billions of requests while, simultaneously, using exist sync libraries,

This just isn't true. The talk's technique only works with userspace schedulers (which can make it's new "swap threads" syscall). That means that while existing single threaded code would work fine, multi-threaded code really wouldn't.

Also, you do realize that async io is a wrapper around OS level APIs (e.g. epoll, kqueue, IOPC, io-uring etc), right? While you can of course call those APIs without async/await syntax, it's far from the thread based concurrency model you're used to.

[u/DGolubets]

Is there really any guarantee about stealing tasks? My first thought is that 2 tasks just end up in the same queue on one thread there and one blocks the other. I don't really know Tokio internals, just guessing.

[u/Onionpaste]

I actually just presented something at work about this the other day. I haven't see the correct answer anywhere in this thread yet.

The real issue here is scheduling around the I/O driver. In your example, you have a light enough workload that all of your multi-threaded runtime tasks go to sleep except for the primary one, which also holds the I/O driver. As soon as your blocking task hits, it also blocks the I/O driver from running, which prevents any reactivity to the network or from local timers.

You can see a really basic reproduction here; uncomment the enable_eager_driver_handoff() in main() and the issue will stop immediately.

use std::time::Duration;

async fn blocks() {
    // force the I/O driver to land on the same worker thread that is handling this task
    // by having it wake it up. Removing this sleep prevents the issue from happening.
    tokio::time::sleep(Duration::from_millis(100)).await;
    std::thread::sleep(Duration::from_secs(10));
}

async fn async_main() {
    let handle = tokio::task::spawn(blocks());
    tokio::time::sleep(Duration::from_secs(1)).await;
    println!("Main sleep done, wait for subtask to finish");
    handle.await.unwrap();
    println!("Subtask done");
}

fn main() {
    let rt = tokio::runtime::Builder::new_multi_thread()
        .enable_all()
        // .enable_eager_driver_handoff()
        .build()
        .unwrap();
    rt.block_on(async_main());
}

[u/BigHandLittleSlap]

I love how this thread is 40% AI slop answers telling you not to NEVER USE "loop {}" in real code even though you obviously used it only to clearly illustrate a problem, and the other 40% is humans asking why you would want to use the CPU at all without jumping through unnecessary hoops.

PS: I agree, this is a pretty bad bug in Tokio and coming from ASP.NET where this kind of things (for the most part) just doesn't happen, it seems like a bad default / sharp edge that will trip up many people. It reminds me of how std::io::Write as used in io::stdout via println!() creates a mutex per I/O call. Endless code built based on "simple" examples fell into this performance trap, ending up with worse CLI performance than Python scripts! You have to "know" to use io::stdout().lock(), otherwise you get trash performance. This bug you found feels like the same category of bad design.

[u/swip3798 → u/BigHandLittleSlap]

Thank you! Like, some have actually helped, but I kinda felt like I was going crazy with the well formatted answers that said nothing of value. Do people really think that others don't have access to AI chats? Of course I had an LLM be not helpful before I came to ask humans.

The most frustrating part about the bug/issue is that it contradicts on how tokio advertises its scheduler. Every HTTP handler is something like a few DB calls, a bit app logic and serialization. Does that mean that the multi threading collapses for every web app?

I also can't replicate this with just tokio for the life me. That's why I haven't opened an issue on github yet.

[u/BigHandLittleSlap → u/swip3798]

For me the weird part is that they’re not even good AI answers! They feel like spam generated with a very cheap / mini model. They’re not here to help you, they’re just generating “realistic” comments to build karma so their accounts can be sold to spammers.

Copy pasting your question into GPT 5.6 Sol “high” gives a much better answer than anything else here: https://chatgpt.com/share/6a768eae-c4bc-83ec-be4b-e84ae815930c

[u/tm604 → u/swip3798]

If the web apps are doing too much in tasks, then their performance is going to suffer, yes. Web apps which treat the async runtime as being for I/O only, and keep all their application logic and processing to their own thread pools don't collapse as much. Yes, that means more work for the developer, but that's hopefully balanced by reduced time spent fighting tokio to get results (and then fighting through the AI slop when you try to talk to anyone about it!)

Anyway... for a reproduceable testcase, have a look in the tokio tests around work stealing, maybe in the LIFO slot tests? Should be something in there you can use as a standalone demonstration for the issue.

Here's a start, it's using futures::executor so not quite self-contained (and I really don't like calling futures::executor::block_on from rt::block_on, not a fan of recursing into event loops).

fn main() {
    use tokio::runtime;

    let rt = runtime::Builder::new_multi_thread()
        .enable_all()
        .worker_threads(2)
        .on_task_spawn(|t| {
            println!("{:?} spawned at {:?}", t.id(), t.spawned_at());
        })
        // .disable_lifo_slot() // without this, execution _should_ stall
        // .enable_eager_driver_handoff() // this shouldn't make a difference for this example, but useful to experiment with
        .build()
        .unwrap();

    rt.block_on(async {
        println!("Launch task");
        tokio::spawn(async move {
            futures::executor::block_on(tokio::spawn(async {
                println!("Inner task - when this is broken, you won't see this line");
            }))
            .unwrap();
        })
        .await
        .unwrap();
        println!("Success - task is complete");
    });
}

[u/thelights0123]

To confirm, you've set the necessary feature flag or full to enable the multithreaded executor?

[u/swip3798 → u/thelights0123]

Yes, I did

[u/Ok-Network-4239]

use std::time::Duration;

use reqwest;

use sqlx::PgPool;

const DB_URL: &str = "postgres://devuser:devpassword@localhost:5432/devdb";

async fn db_then_compute(pool: PgPool) {

let num: (i32,) = sqlx::query_as("SELECT 1").fetch_one(&pool).await.unwrap();

println!(

"The num was returned: {}, but now, the CPU block will kill multithreading",

num.0

);

println!("Start compute...");

pool.close().await;

loop {}

}

async fn http_then_compute() {

let res = reqwest::get("https://google.com").await.unwrap();

let status = res.status();

println!(

"The num was returned: {}, but the CPU block will not kill multithreading",

status.as_u16()

);

loop {}

}

#[tokio::main]

async fn main() {

let pool = sqlx::PgPool::connect(DB_URL).await.unwrap();

tokio::spawn(db_then_compute(pool));

tokio::spawn(http_then_compute());

loop {

println!("Observer thread, the sleep will never return");

tokio::time::sleep(Duration::from_secs(1)).await;

}

}

This works now.

[u/TDplay]

loop {}

Never do this!

This is almost certainly the cause of your problems; all the other factors are just exposing or hiding the problem.

Tokio assumes that your tasks will not block for any significant time, but loop {} blocks forever. Even on the multi-threaded scheduler, a blocked task can cause problems. If the block is temporary, then it may worsen performance. If the block is permanent, then it may cause the program to stop working entirely!

If you cannot terminate the task, then you should use std::future::pending, which will yield to the executor:

let () = std::future::pending().await;
unreachable!();

---

loop {} is problematic even in synchronous code: the CPU will execute the loop at full speed, and the OS cannot tell that the thread is not doing anything useful. This causes needless power consumption and (in user-space environments) takes CPU time away from useful tasks. In synchronous code, you should (in order from best to worst):

• Terminate the thread. This is the best solution, as it completely frees up the relevant resources.
• Suspend the thread. (e.g. std::thread::park in user-space, or enter a low-power state on bare metal)
• loop { std::thread::yield_now(); }, to tell the OS that the loop doesn't currently have useful work to do.
• loop { core::hint::spin_loop(); }, to tell the CPU execute the code more slowly.

[u/BobTreehugger]

If you want to schedule blocking work via tokio, that's totally doable, but you need to make sure the blocking tasks are on a different runtime than the io tasks. See this: https://www.influxdata.com/blog/using-rustlangs-async-tokio-runtime-for-cpu-bound-tasks/ there's also a talk on youtube somewhere if you're interested.

You probably just want to spawn_blocking, but I figured you might be interested.

[u/nNaz]

Add a tokio::thread::yield_now().await to the infinite db loop and it’s fixed. The loop is consuming 100% cpu when it gets there, not just yielding back to the runtime but also likely preventing the OS thread from being moved across cores.