Need help to correct my python function for fetching full data!

[PrateekG]

I am trying to fetch product data from an api using Python3.6.

By default this api returns 20 products and in a single request the api can return max 500 products if we use api's parameter Limit=500.

So for fetching all products we need to use one more parameter with Limit- Offset(Number of products to skip).

I have written following function to achieve this but in case of full data my function is not working well and it's giving me error like- Login failed, Signature mismatching.

def get_data(userid, api_key, action, pagination=True): timeformat = datetime.datetime.now().replace(microsecond=0).isoformat() + '+08:00' endpoint = 'https://example.com' page_json = {} # set required parameters for this api parameters = { 'UserID': userid, 'Version': '1.0', 'Action': action, 'Format': 'JSON', 'Timestamp': timeformat } if pagination: page = 0 parameters['Limit'] = 500 while True: parameters['Offset'] = 500 * page # set the required cryptographic signature concatenated = urllib.parse.urlencode(sorted(parameters.items())) parameters['Signature'] = HMAC(api_key, concatenated.encode('utf-8'), sha256).hexdigest() page += 1 try: response = requests.get(endpoint, params=parameters) page_json = response.json() except requests.exceptions.ConnectionError: print("Connection refused!") sleep(5) else: try: concatenated = urllib.parse.urlencode(sorted(parameters.items())) # set the required cryptographic signature parameters['Signature'] = HMAC(api_key, concatenated.encode('utf-8'), sha256).hexdigest() response = requests.get(endpoint, params=parameters) page_json = response.json() except requests.exceptions.ConnectionError: print("Connection refused!") sleep(5) return page_json

It looks like I am not fitting my signature parameter line correctly in case of full data.

Can you please look into my function and help me to find out what I have written wrong and what it should be like?

[killerrex]

Hi,

The 2 things I see there are that when you put pagination = True you enter in an endless loop and that you overwrite the results of the previous query.

I have not way to try it, specially confirm that the json object returned is a list of results and there is not more wrappers... but try this:

def get_data(userid, api_key, action, limit=500): timeformat = datetime.datetime.now().replace(microsecond=0).isoformat() + '+08:00' endpoint = 'https://example.com' page_json = {} # set required parameters for this api parameters = { 'UserID': userid, 'Version': '1.0', 'Action': action, 'Format': 'JSON', 'Timestamp': timeformat } if limit > 0: parameters['Limit'] = limit page = 0 while True: parameters['Offset'] = limit * page # set the required cryptographic signature concatenated = urllib.parse.urlencode(sorted(parameters.items())) parameters['Signature'] = HMAC(api_key, concatenated.encode('utf-8'), sha256).hexdigest() try: response = requests.get(endpoint, params=parameters) except requests.exceptions.ConnectionError: print("Connection refused!") sleep(5) # As the page was not recovered, just try it again, # do not increase the page counter continue # Ok, one page recovered, append to the total result page += 1 # Here I assume the answer is a dictionary of entries like: # { 501: value, 502: value, ...} # you might need to look for the right element to accumulate. # i.e.: if response is something like: # {'result': [value, value, value...]} # The following lines will not be valid delta = response.json() page_json.update(delta) # This might be the end, no more pages... but this criteria again depends # on the api you are using if len(delta) % limit == 0: # Here we enter only if the number of results are 0 or 500 break else: try: concatenated = urllib.parse.urlencode(sorted(parameters.items())) # set the required cryptographic signature parameters['Signature'] = HMAC(api_key, concatenated.encode('utf-8'), sha256).hexdigest() response = requests.get(endpoint, params=parameters) page_json = response.json() except requests.exceptions.ConnectionError: print("Connection refused!") sleep(5) return page_json

Notice that I changed your pagination flag with the limit number (so you can request 50 by 50, 100 by 100...) use limit=0 for the equivalent of your pagination=False.

[PrateekG]

Thanks killerrex!
I am able to find issue in my code.