Use HTTP Keep-alives with the Python Requests module



Published: 2021-06-17 15:31:27 +0000
Categories: Python3,

Language

Python3

Description

If you're wanting to make multiple requests against a server (scraping pages, calling an API, whatever) then you may want to re-use connections rather than establishing a new one for each request (incurring the overhead of a TCP 3-way and SSL handshake each time).

The requests module has support for this in it's Session module. It pools connections, so can re-use an existing connection where one is available (it can also persist auth, cookies, proxy settings etc between requests).

Snippet

s = requests.Session()
s.get(url)
s.post(url)

Usage Example

import requests
s = requests.Session()
r1 = s.get('https://snippets.bentasker.co.uk/')
r2 = s.get('https://snippets.bentasker.co.uk/sitemap.html')
r3 = s.get("https://recipebook.bentasker.co.uk/page-2105201510-Steak-Sandwich-Misc.html")

Keywords

python, requests, keep-alive, connection, persistence, pipelining,

Latest Posts


Copyright © 2021 Ben Tasker | Sitemap | Privacy Policy
Available at snippets.bentasker.co.uk and http://phecoopwm6x7azx26ctuqcp6673bbqkrqfeoiz2wwk36sady5tqbdpqd.onion