Contents: intro, imports, what will be scraped, process, code, links, outro.
Intro
Well, hello there to people who came from the last Bing series! This blog post is a continuation of Bing's web scraping series and contains info about how to scrape Bing News results using Python. An alternative solution will be shown after the first block of code.
Imports
import requests import lxml from bs4 import BeautifulSoup from serpapi import GoogleSearch What will be scraped
Process
The process is straight-forward. SelectorGadget Chrome extension was to grab CSS selectors.
The following GIF illustrates how to get CSS selectors of the Title, URL, Snippet, Source website, and when news has been posted.
Code
from bs4 import BeautifulSoup import requests, lxml headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582" } html = requests.get('https://www.bing.com/news/search?q=faze+clan', headers=headers) soup = BeautifulSoup(html.text, 'lxml') for result in soup.select('.card-with-cluster'): title = result.select_one('.title').text link = result.select_one('.title')['href'] snippet = result.select_one('.snippet').text source = result.select_one('.source a').text date_posted = result.select_one('#algocore span+ span').text print(f'{title}\n{link}\n{source}\n{date_posted}\n{snippet}\n') # part of the output: ''' FaZe Clan shows off new execute for Mirage against Furia Esports https://win.gg/news/8521/faze-clan-shows-off-new-execute-for-mirage-against-furia-esports WIN.gg 2h During a match against Team Furia in the Gamers Without Borders Cup, the camera spotted an interesting interaction between ... ''' Using Bing News Engine Results API
SerpApi is a paid API with a free trial of 5,000 searches.
from serpapi import GoogleSearch import json params = { "api_key": "YOUR_API_KEY", "engine": "bing_news", "q": "faze clan" } search = GoogleSearch(params) results = search.get_dict() for result in results['organic_results']: print(json.dumps(result, indent=2, ensure_ascii=False)) # part of the output: ''' { "title": "FaZe Clan shows off new execute for Mirage against Furia Esports", "link": "https://win.gg/news/8521/faze-clan-shows-off-new-execute-for-mirage-against-furia-esports", "snippet": "During a match against Team Furia in the Gamers Without Borders Cup, the camera spotted an interesting interaction between ...", "source": "WIN.gg", "date": "2h", "thumbnail": "https://serpapi.com/searches/60d82f308ccee022b4ab7525/images/62e054f4209c882415dd75f5245f96d23bd4c1538d707fb513a0918671c831d7.jpeg" } ''' Link
Code in the online IDE • Bing News Engine Results API
Outro
If you have any questions or something isn't working correctly or you want to write something else, feel free to drop a comment in the comment section or via Twitter at @serp_api.
Yours,
Dimitry, and the rest of SerpApi Team.
Top comments (0)