Scraping Issue with BS

[snippsat]

Can test attribute of button,in BS can do this with .attrs
Example i would use .get() as .attrs return a dictionary,then can add as default value eg more pages.

import requests from bs4 import BeautifulSoup import time # First url = "https://www.homeadvisor.com/c.Garage-Garage-Doors.Atlanta.GA.-12036.html" # Last #url = "https://www.homeadvisor.com/c.Garage-Garage-Doors.Atlanta.GA.-12036.html?startingIndex=50" headers = { "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)\ AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" } response = requests.get(url, headers=headers) soup = BeautifulSoup(response.content, "lxml") button = soup.select('button.page-next.\@px-1.\@ml-1')

Test

# First page,or all pages before last page will return this >>> button[0].attrs.get('disabled', 'More pages') 'More pages' >>> # Last page >>> button[0].attrs.get('disabled', 'More pages') 'disabled'

[muzikman]

Fantastic!

This does the job. The next thing I need to do is grab the city, state and category for each listing. But it only appears in two places, the URL and at the top of the page. Then I will add them to the dictionary using RegEx.

PAGE = 0 while True: html = get_html(session, BASE_URL, PAGE) listings = get_listings(html) for listing in listings: print(listing['company'], listing['phone'], listing['rating'], end='\n') button = html.select('button.page-next.\@px-1.\@ml-1') if button[0].attrs.get('disabled') == 'disabled': break PAGE += 25

I also have to read each URL from a file instead of hard coding it. In the following code, I am appending each state > city > category > listings then writing the rows to a CSV file. One question, how do I only write the column names one time?

def save_csv(listings, filename): filename = 'home-advisor-data-{}.csv'.format(state) with open(filename, 'a', encoding='utf-8', newline='') as file: writer = csv.writer(file, delimiter=',') writer.writerow(['Company', 'Phone Number', 'Rating']) #While paginating through each page of results, it will write these literal columns. #How do I avoid this? I only want these at the top, once. for listing in listings: writer.writerow( [listing['company'], listing['Phone_Number'], listing['Rating']])