Use HTTP Keep-alives with the Python Requests module (Python3)

Category: Python3

If you're wanting to make multiple requests against a server (scraping pages, calling an API, whatever) then you may want to re-use connections rather than establishing a new one for each request (incurring the overhead of a TCP 3-way and SSL handshake each time).

The requests module has support for this in it's Session module. It pools connections, so can re-use an existing connection where one is available (it can also persist auth, cookies, proxy settings etc between requests).

Details

  • Language: Python3

Snippet

s = requests.Session()
s.get(url)
s.post(url)

Usage Example

import requests
s = requests.Session()
r1 = s.get('https://snippets.bentasker.co.uk/')
r2 = s.get('https://snippets.bentasker.co.uk/sitemap.html')
r3 = s.get("https://recipebook.bentasker.co.uk/page-2105201510-Steak-Sandwich-Misc.html")