Contents: intro, imports, what will be scraped, code, links, outro.
Intro
This blog post is a continuation of Google's web scraping series. Here you'll see examples of how you can scrape Google Organic Search Results using Python. An alternative SerpApi solution will be shown.
Imports
import lxml, requests from bs4 import BeautifulSoup from serpapi import GoogleSearch What will be scraped
Title, link, displayed link, snippet
Inline sitelinks
Expanded sitelinks
Headers
Make sure you specified HTTP header user-agent so Google won't block your requests, otherwise it will block it eventually. Why? Without a headers it might think that your request is a request from a bot (script) which it is.
Example of passing headers into request:
headers = { 'User-agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582" } requests.get('https://www.google.com/search', headers=headers) # other code Code
import requests, lxml from bs4 import BeautifulSoup headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582" } params = { "q": "apple", "hl": "en", } html = requests.get("https://www.google.com/search", params=params, headers=headers) soup = BeautifulSoup(html.text, 'lxml') # container with all needed data for result in soup.select('.tF2Cxc'): title = result.select_one('.DKV0Md').text link = result.select_one('.yuRUbf a')['href'] displayed_link = result.select_one('.TbwUpd.NJjxre').text snippet = result.select_one('#rso .lyLwlc').text # inline sitelinks for i in soup.select('.HiHjCd a'): print(i.text) # expanded sitelinks for t in soup.select('.usJj9c'): text = t.select_one('.r').text text_link = t.select_one('.r a')['href'] snippet = t.select_one('.st').text print(f'{text}\n{text_link}\n{snippet}\n') Using Google Organic Results API
SerpApi is a paid API with a free trial of 5,000 searches.
The difference is that all that needs to be done is just to iterate over a ready made, structured JSON instead of coding everything from scratch, and selecting correct selectors which could be time consuming at times.
from serpapi import GoogleSearch import json # just for prettier output params = { "api_key": "YOUR_API_KEY", "engine": "google", "q": "buy trampoline", "hl": "en" } search = GoogleSearch(params) results = search.get_dict() params = { "api_key": os.getenv("API_KEY"), "engine": "google", "q": "cyber security", "hl": "en", } search = GoogleSearch(params) results = search.get_dict() for result in results['organic_results']: title = result['title'] link = result['link'] displayed_link = result['displayed_link'] snippet = result['snippet'] try: inline_sitelinks = result['sitelinks']['inline'] except: inline_sitelinks = None try: expanded_sitelinks = result['sitelinks']['expanded'] except: expanded_sitelinks = None print(f"{title}\n{link}\n{displayed_link}\n{snippet}\n") print(json.dumps(expanded_sitelinks, indent=2, ensure_ascii=False)) print(json.dumps(inline_sitelinks, indent=2, ensure_ascii=False)) ----------------- # organic ''' Trampolines - Sports & Outdoors - The Home Depot https://www.homedepot.com/b/Sports-Outdoors-Trampolines/N-5yc1vZc455 https://www.homedepot.com › Sports & Outdoors Get free shipping on qualified Trampolines or Buy Online Pick Up ... Round Trampoline with Safety Enclosure Basketball Hoop and Ladder. ''' # expanded sitelinks ''' [ { "title": "Store", "link": "https://m.dji.com/", "snippet": "Mavic Series - Refurbished Products - Buy Osmo Series - ..." } ] ''' # inline sitelinks ''' [ { "title": "Pure Fun", "link": "https://www.homedepot.com/b/Sports-Outdoors-Trampolines/Pure-Fun/N-5yc1vZc455Zdeo" } ] ''' 
Top comments (0)