I do web scraping on my daily basis. I was looking for the way to deal with the infinite scrolling using Selenium Python. One of the method I used while dealing with infinite scrolling data from chrome web store was to loop around the screen size.
number = 0 number2 = 2500 for i in range(105): script_scroll = "window.scrollTo("+str(number)+","+str(number2)+")" print(script_scroll) contents = browser.find_elements_by_xpath("//div[@role='row']") urls_list = ['urls'] for content in contents: link = content.find_element_by_xpath(".//a") link_url = link.get_attribute('href') urls_list.append(link_url) time.sleep(4) browser.execute_script(script_scroll) number = number2 number2 = number2 + number2 
Top comments (0)