This Python package is meant to scrape and parse Google, Google Scholar, Bing, Baidu, Yandex, Yahoo, Ebay results using SerpApi.
The following services are provided:
- Search API
- Search Archive API
- Account API
- Location API (Google Only)
SerpApi provides a script builder to get you started quickly.
Python 3.7+
pip install google-search-resultsLink to the python package page
from serpapi import GoogleSearchResults search = GoogleSearchResults({ "q": "coffee", "location": "Austin,Texas", "api_key": "<your secret api key>" }) result = search.get_dict()This example runs a search about "coffee" using your secret api key.
The SerpApi service (backend)
- searches on Google using the search: q = "coffee"
- parses the messy HTML responses
- return a standardizes JSON response The GoogleSearchResults class
- Format the request
- Execute GET http request against SerpApi service
- Parse JSON response into a dictionary Et voila..
Alternatively, you can search:
- Bing using BingSearchResults class
- Baidu using BaiduSearchResults class
- Yahoo using YahooSearchResults class
- Ebay using EbaySearchResults class
- Yandex using YandexSearchResults class
- GoogleScholar using GoogleScholarSearchResults class
See the playground to generate your code.
- Google Search Results in Python
- Installation
- Quick start
- Summary
- Google Search API capability
- How to set SERP API key
- Example by specification
- Location API
- Search Archive API
- Account API
- Search Bing
- Search Baidu
- Search Yandex
- Search Yahoo
- Search Ebay
- Search Google Scholar
- Generic search with SerpApiClient
- Search Google Images
- Search Google News
- Search Google Shopping
- Google Search By Location
- Batch Asynchronous Searches
- Change log
- Conclusion
Source code.
params = { "q": "coffee", "location": "Location Requested", "device": "desktop|mobile|tablet", "hl": "Google UI Language", "gl": "Google Country", "safe": "Safe Search Flag", "num": "Number of Results", "start": "Pagination Offset", "api_key": "Your SERP API Key", # To be match "tbm": "nws|isch|shop", # To be search "tbs": "custom to be search criteria", # allow async request "async": "true|false", # output format "output": "json|html" } # define the search search search = GoogleSearchResults(params) # override an existing parameter search.params_dict["location"] = "Portland" # search format return as raw html html_results = search.get_html() # parse results dict_results = search.get_dict() json_results = search.get_json()Link to the full documentation
see below for more hands on examples.
You can get an API key here if you don't already have one: https://serpapi.com/users/sign_up
The SerpApi api_key can be set globally:
GoogleSearchResults.SERP_API_KEY = "Your Private Key"The SerpApi api_key can be provided for each search:
query = GoogleSearchResults({"q": "coffee", "serp_api_key": "Your Private Key"})We love true open source, continuous integration and Test Drive Development (TDD). We are using RSpec to test our infrastructure around the clock to achieve the best QoS (Quality Of Service).
The directory test/ includes specification/examples.
Set your api key.
export API_KEY="your secret key"Run test
make testfrom serpapi import GoogleSearchResults search = GoogleSearchResults({}) location_list = search.get_location("Austin", 3) print(location_list)it prints the first 3 location matching Austin (Texas, Texas, Rochester)
[ { 'canonical_name': 'Austin,TX,Texas,United States', 'country_code': 'US', 'google_id': 200635, 'google_parent_id': 21176, 'gps': [-97.7430608, 30.267153], 'id': '585069bdee19ad271e9bc072', 'keys': ['austin', 'tx', 'texas', 'united', 'states'], 'name': 'Austin, TX', 'reach': 5560000, 'target_type': 'DMA Region'}, ...]The search result are stored in temporary cached. The previous search can be retrieve from the the cache for free.
from serpapi import GoogleSearchResults search = GoogleSearchResults({"q": "Coffee", "location": "Austin,Texas"}) search_result = search.get_dictionary() search_id = search_result.get("search_metadata").get("id") print(search_id)Now let retrieve the previous search from the archive.
archived_search_result = GoogleSearchResults({}).get_search_archive(search_id, 'json') print(archived_search_result.get("search_metadata").get("id"))it prints the search result from the archive.
from serpapi import GoogleSearchResults search = GoogleSearchResults({}) account = search.get_account()it prints your account information.
from serpapi import BingSearchResults search = BingSearchResults({"q": "Coffee", "location": "Austin,Texas"}) data = search.get_json()this code prints baidu search results for coffee as JSON.
https://serpapi.com/bing-search-api
from serpapi import BaiduSearchResults search = BaiduSearchResults({"q": "Coffee"}) data = search.get_json()this code prints baidu search results for coffee as JSON. https://serpapi.com/baidu-search-api
from serpapi import YandexSearchResults search = YandexSearchResults({"text": "Coffee"}) data = search.get_json()this code prints yandex search results for coffee as JSON.
https://serpapi.com/yandex-search-api
from serpapi import YahooSearchResults search = YahooSearchResults({"p": "Coffee"}) data = search.get_json()this code prints yahoo search results for coffee as JSON.
https://serpapi.com/yahoo-search-api
from serpapi import EbaySearchResults search = EbaySearchResults({"_nkw": "Coffee"}) data = search.get_json()this code prints ebay search results for coffee as JSON.
https://serpapi.com/ebay-search-api
from serpapi import GoogleScholarSearchResults search = GoogleScholarSearchResults({"q": "Coffee"}) data = search.get_json()this code prints Google Scholar search results.
from serpapi import SerpApiClient query = {"q": "Coffee", "location": "Austin,Texas", "engine": "google"} search = SerpApiClient(query) data = search.get_json()This class enables to interact with any search engine supported by SerpApi.com
from serpapi import GoogleSearchResults search = GoogleSearchResults({"q": "coffe", "tbm": "isch"}) for image_result in search.get_json()['images_results']: link = image_result["original"] try: print("link: " + link) # wget.download(link, '.') except: passthis code prints all the images links, and download image if you un-comment the line with wget (linux/osx tool to download image).
This tutorial covers more ground on this topic. https://github.com/serpapi/showcase-serpapi-tensorflow-keras-image-training
from serpapi import GoogleSearchResults search = GoogleSearchResults({ "q": "coffe", # search search "tbm": "nws", # news "tbs": "qdr:d", # last 24h "num": 10 }) for offset in [0,1,2]: search.params_dict["start"] = offset * 10 data = search.get_json() for news_result in data['news_results']: print(str(news_result['position'] + offset * 10) + " - " + news_result['title'])this script prints the first 3 pages of the news title for the last 24h.
from serpapi import GoogleSearchResults search = GoogleSearchResults({ "q": "coffe", # search search "tbm": "shop", # news "tbs": "p_ord:rv", # last 24h "num": 100 }) data = search.get_json() for shopping_result in data['shopping_results']: print(shopping_result['position']) + " - " + shopping_result['title'])this script prints all the shopping results order by review order.
With SerpApi, we can build Google search from anywhere in the world. This code is looking for the best coffee shop per city.
from serpapi import GoogleSearchResults for city in ["new york", "paris", "berlin"]: location = GoogleSearchResults({}).get_location(city, 1)[0]["canonical_name"] search = GoogleSearchResults({ "q": "best coffee shop", # search search "location": location, "num": 1, "start": 0 }) data = search.get_json() top_result = data["organic_results"][0]["title"]We do offer two ways to boost your searches thanks to async parameter.
- Blocking - async=false - it's more compute intensive because the search would need to hold many connections. (default)
- Non-blocking - async=true - it's way to go for large amount of query submitted by batch (recommended)
# Python 3.6+ (tested) # # Operating system import os # regular expression library import re # safe queue (named Queue in python2) from queue import Queue # Time utility import time # SerpApi search from serpapi import GoogleSearchResults # store searches search_queue = Queue() # SerpApi search search = GoogleSearchResults({ "location": "Austin,Texas", "async": True }) # loop through a list of companies for company in ['amd','nvidia','intel']: print("execute async search: q = " + company) search.params_dict["q"] = company search = search.get_dict() print("add search to the queue where id: " + search['search_metadata']['id']) # add search to the search_queue search_queue.put(search) print("wait until all search statuses are cached or success") # Create regular search search = GoogleSearchResults({"async": True}) while not search_queue.empty(): search = search_queue.get() search_id = search['search_metadata']['id'] # retrieve search from the archive - blocker print(search_id + ": get search from archive") search_archived = search.get_search_archive(search_id) print(search_id + ": status = " + search_archived['search_metadata']['status']) # check status if re.search('Cached|Success', search_archived['search_metadata']['status']): print(search_id + ": search done with q = " + search_archived['search_parameters']['q']) else: # requeue search_queue print(search_id + ": requeue search") search_queue.put(search) # wait 1s time.sleep(1) # self.assertIsNotNone(results["local_results"][0]["title"]) print('all searches completed')This code shows how to run searches asynchronously. The search parameters must have {async: True}. This indicates that the client shouldn't wait for the search to be completed. The current thread that executes the search is now non-blocking which allows to execute thousand of searches in seconds. The SerpApi backend will do the processing work. The actual search result is defer to a later call from the search archive using get_search_archive(search_id). In this example the non-blocking searches are persisted in a queue: search_queue. A loop through the search_queue allows to fetch individual search result. This process can be easily multithreaded to allow a large number of concurrent search requests. To keep thing simple, this example does only explore search result one at a time (single threaded).
2020-06-30 @ 1.8.3
- simplify import
- improve package for python 3.5+
- add support for python 3.5 and 3.6 2020-03-25 @ 1.8
- add support for Yandex, Yahoo, Ebay
- clean-up test 2019-11-10 @ 1.7.1
- increase engine parameter priority over engine value set in the class 2019-09-12 @ 1.7
- Change namespace "from lib." instead: "from serpapi import GoogleSearchResults"
- Support for Bing and Baidu 2019-06-25 @ 1.6
- New search engine supported: Baidu and Bing
SerpApi supports all the major search engines. Google has the more advance support with all the major services available: Images, News, Shopping and more.. To enable a type of search, the field tbm (to be matched) must be set to:
- isch: Google Images API.
- nws: Google News API.
- shop: Google Shopping API.
- any other Google service should work out of the box.
- (no tbm parameter): regular Google search.
The field tbs allows to customize the search even more.