Web Scraping efficiency improvement

[HiImNew]

Is there any way to speed up a web-scraper by having multiple computers contribute to processing a list of urls? Like computer A takes urls 1 - 500 and computer B takes urls 501 - 1000, etc. I am looking for a way to build the fastest possible web scraper with resources available to everyday people.

I am already using multiprocessing from the grequests module. Which is gevent + requets combined. If anyone knows any way to help at all I will greatly appreciate it.

This scraping does not need to be run constantly, but at a specific time each day in the morning (6 A.M.), and be done near as soon as it starts. (if that info helps). I am looking for something quick and punctual.

Also I am looking through urls for retail stores (i.e.: target, bestbuy, newegg, etc), and using it to check what items are in stock for the day.

this is a code segment for grabbing those urls in the script I'm trying to put together:

import datetime import grequests thread_number = 20 nnn = int(len(product_number_list)/100) float_nnn = (len(product_number_list)/100) # Product number list is a list of product numbers, too big for me to include the full list. Here are like three: product_number_list = ['N82E16820232476', 'N82E16820233852', 'N82E16820313777'] base_url = 'https://www.newegg.com/Product/Product.aspx?Item={}' url_list = [] for number in product_number_list: url_list.append(base_url.format(product_number_list)) # The above three lines create a list of urls. results = [] appended_number = 0 for x in range(0, len(product_number_list), thread_number):	attempts = 0	while attempts < 10:	try:	rs = (grequests.get(url, stream=False) for url in url_list[x:x+thread_number])	reqs = grequests.map(rs, stream=False, size=20)	append = 'yes'	for i in reqs:	if i.status_code != 200:	append = 'no'	print('Bad Status Code. Nothing Appended.') attempts += 1	break	if append == 'yes':	appended_number += 1	results.extend(reqs)	break except:	print('Something went Wrong. Try Section Failed.')	attempts += 1	time.sleep(5)	if appended_number % nnn == 0:	now = datetime.datetime.today()	print(str(int(20*appended_number/float_nnn)) + '% of the way there at: ' + str(now.strftime("%I:%M:%S %p"))) if attempts == 10:	print('Failed ten times to get urls.')	time.sleep(3600) if len(results) != len(url_list): print('Results count is off. len(results) == "' + str(len(results)) + '". len(url_list) == "' + str(len(url_list)) + '".')

Any way to speed this up (and yes, I am removing the print statements later) is appreciated.

P.S. This is not my code, I grabbed part of it from here: https://stackoverflow.com/questions/4620...-grequests
and here: https://stackoverflow.com/questions/2197...-get-max-r