How to save and load cookies in Python requests?

by scrapecrow Dec 08, 2022

HTTP cookies in web scraping play a vital role across several factors, including blocking and localization. Hence, pausing and resuming request sessions is often useful. For this, we'll explain how to save and load Python requests cookies.

Save Cookies

Let's start by saving cookies in Python requests. For this, we'll use the requests.cookiejar utility package:

from pathlib import Path from requests.utils import dict_from_cookiejar import json import requests # create a session to presists cookies session = requests.session() # send cookies through a response object session.get("https://httpbin.dev/cookies/set?key1=value1&key2=value2") # turn cookiejar into dict cookies = dict_from_cookiejar(session.cookies) # save them to file as JSON Path("cookies.json").write_text(json.dumps(cookies))

Above, we create a new session object and request a URL to accept few cookie values. Then, we save the retrieved Python request cookies to an object using the dict_from_cookiejar method. Finally, the cookie data is saved to a JSON file:

{"key1": "value1", "key2": "value2"}

Load Cookies

Now that we saved the cookie object using Python cookiejar, let's load it:

from pathlib import Path from requests.utils import cookiejar_from_dict import json import requests # create a session session = requests.session() # load the JSON file cookies = json.loads(Path("cookies.json").read_text()) # turn the object into a a cookie jar cookies = cookiejar_from_dict(cookies) # load cookiejar to current session session.cookies.update(cookies) # validate the cookies print(session.get("https://httpbin.dev/cookies").text) {"key1": "value1", "key2": "value2"}

Here, we started by loading the JSON file and importing the cookie values using the cookiejar_from_dict. Next, we let the requests set cookie values by loading them into the session, which will automatically set cookie header values.

For further details on utilizing cookies for requests in Python, refer to our dedicated guide.