My program runs multiple threads of a function performing a get request, and although the number of threads remains constant throughout the program, the memory usage and object count gradually increases over time until no free memory is left on the system. After doing some digging I found that base requests library has some memory leaking issues with the response object that can be fixed by using .close() method on the session and response like so:

 r = session.get(url=url, headers=headers) r.close() session.close() 

I can't seem to find a solution to solving this issue within this library.

Sample code:

def func1(): #populate list while True: threads = [] for item in list: t = threading.Thread(target=func2, args=(item, v, q)) threads.append(t) t.start() for thread in threads: thread.join() print(f"Retrying... {retry}") retry += 1 def func2(i, v, q): headers = {'':''} session = Session(client_identifier='chrome_105') r = session.get('url', headers=headers) response = json.loads(r.text) #do stuff with response return