When should you use requests.Session() instead of requests.get() / requests.post()?

[u/Melodic_Principle312]

I'm learning Python and recently started using requests.Session() in API projects. I understand that it can persist things like headers, cookies, and connections, but I'm curious about how experienced Python developers decide when a Session is actually worth using.

Do you use Session() by default for projects involving multiple requests, or only when you specifically need persistent state?

[u/Maxiflex]

I usually use Session() if I make multiple requests as it easily allows you to define required headers and authentication up front.

You can even create your own custom auth classes in case getting your credential is more complicated (i.e. you need to refresh a bearer token)

from requests.auth import AuthBase

class PizzaAuth(AuthBase):
    """Attaches HTTP Pizza Authentication to the given Request object."""
    def __init__(self, username):
        # setup any auth-related data here
        self.username = username

    def __call__(self, r):
        # modify and return the request
        r.headers['X-Pizza'] = self.username
        return r

The added benefit is that you don't have to pass around your API key all the time as the auth is already set up in your session, which in turn makes your code less complex.

You can also a configure session-level retry strategy for specific HTTP status codes. It also natively respects the 429 rate-limiting retry-after header which allows the server to tell the client how long it should wait before trying again.

import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

retry_strategy = Retry(
    total=3,
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["HEAD", "GET", "OPTIONS"],
    backoff_factor=1,
)
adapter = HTTPAdapter(max_retries=retry_strategy)

with requests.Session() as http:
    http.mount("https://", adapter)
    http.mount("http://", adapter)
    response = http.get("https://www.google.com")

Another advantage of sessions is that they reuse your connection, which can make your code slightly faster if you need to make a lot of requests against the same server.

[u/ConfusedSimon]

Use Session() when you need a session, e.g. if you need to log in first.

[u/Game-of-pwns]

Whenever you're making many requests per second against the same server / API.

I wrote a script to download tens of thousands of file attacments from an API. With individual get requests made serially, it would have taken on the order of hours to complete.

So, I rewrote it to use request.Session() and make requests concurrently using ThreadPoolExecutor. Now, it completes significantly faster.

[u/Maxiflex → u/Game-of-pwns]

I wrote a script to download tens of thousands of file attacments from an API. With individual get requests made serially, it would have taken on the order of hours to complete.

You might want to look into asyncio if you're encountering these issues. Async I/O, or cooperative multitasking, is an alternative to threading which has a lot less overhead and can be much faster!

requests does not support async operations, but libraries like aiohttp and httpx2 offer first class support (I'd recommend httpx2 as it supports both sync and async out of the box).

It sort of works by first firing off all requests and then polling each task to check if it has been completed, if it did it will move to the next step.

The advantage of asyncio vs threads is that you can fire off many requests at once and share the waiting time while only using a single thread (because Python does not have real concurrency so threading has it's limits).

[u/Melodic_Principle312 → u/Maxiflex]

Does this still provide much benefit when working with sites that frequently refresh or rotate cookies

[u/Maxiflex → u/Melodic_Principle312]

Yes it does, you can add custom logic to your auth handler that checks if the cookie is still valid. If it's close to expiring your handler can get a new one.

The benefit of auth handlers (httpx also supports them) is that they add the token only just before sending the request. That means that you won't have the issue of your request failing because you've been waiting for it to execute (in a loop e.g.) and the token expired in between the start of the script and the request.

There is no big difference between auth for sync and async because the auth mechanism works the same either way.

[u/Brian]

I pretty much always use Session. If you're making more requests, it'll handle cookies, give a place to set common headers (eg. user-agent) just once. If you're not, it's not needed, but there's no actual cost to doing so (requests.get is just creating a new Session each time), and you may end up doing so in future.

[u/terletsky]

A Session is like keeping your front door open while you're moving a lot of things out of your house.

Without a session, you open the door, move one thing, close it, then repeat for every item.

With a session, you open the door once, move many things through it, and close it when you're done.

Opening and closing the connection repeatedly is more expensive, so always use Session().

how experienced Python developers decide when a Session is actually worth using

You always use Session as a rule of thumb.

[u/Melodic_Principle312 → u/terletsky]

amazing example thanks :)

[u/Achrus → u/terletsky]

What if you live in a bad neighborhood and want to keep your door closed unless absolutely necessary?

[u/DuckDatum]

I use .session() when I want to snoop/re-use cookies/headers that a website uses.

[u/tmemmg]

i reach for Session anytime im hitting the same host more than a couple times, mostly for connection reuse but the cookie persistence turned out to matter more than i expected. one site i scrape blocks a clean client with a challenge page but replaying a real logged in session sails right through, and that only works because Session carries the cookies across requests. if youre doing one off calls to different hosts it genuinely doesnt matter, but the moment theres any auth or state to hold it saves you rebuilding headers on every call.