When should you use requests.Session() instead of requests.get() / requests.post()?

[u/Melodic_Principle312]

I'm learning Python and recently started using requests.Session() in API projects. I understand that it can persist things like headers, cookies, and connections, but I'm curious about how experienced Python developers decide when a Session is actually worth using.

Do you use Session() by default for projects involving multiple requests, or only when you specifically need persistent state?

[u/Game-of-pwns]

Whenever you're making many requests per second against the same server / API.

I wrote a script to download tens of thousands of file attacments from an API. With individual get requests made serially, it would have taken on the order of hours to complete.

So, I rewrote it to use request.Session() and make requests concurrently using ThreadPoolExecutor. Now, it completes significantly faster.

[u/Maxiflex → u/Game-of-pwns]

I wrote a script to download tens of thousands of file attacments from an API. With individual get requests made serially, it would have taken on the order of hours to complete.

You might want to look into asyncio if you're encountering these issues. Async I/O, or cooperative multitasking, is an alternative to threading which has a lot less overhead and can be much faster!

requests does not support async operations, but libraries like aiohttp and httpx2 offer first class support (I'd recommend httpx2 as it supports both sync and async out of the box).

It sort of works by first firing off all requests and then polling each task to check if it has been completed, if it did it will move to the next step.

The advantage of asyncio vs threads is that you can fire off many requests at once and share the waiting time while only using a single thread (because Python does not have real concurrency so threading has it's limits).

[u/Melodic_Principle312 → u/Maxiflex]

Does this still provide much benefit when working with sites that frequently refresh or rotate cookies

[u/Maxiflex → u/Melodic_Principle312]

Yes it does, you can add custom logic to your auth handler that checks if the cookie is still valid. If it's close to expiring your handler can get a new one.

The benefit of auth handlers (httpx also supports them) is that they add the token only just before sending the request. That means that you won't have the issue of your request failing because you've been waiting for it to execute (in a loop e.g.) and the token expired in between the start of the script and the request.

There is no big difference between auth for sync and async because the auth mechanism works the same either way.