Same Data Showing Several Times With Beautifulsoup Query

[eddywinch82]

Hi there,

I have the following Python Code :-

import pandas as pd import requests import numpy as np from bs4 import BeautifulSoup import xlrd import re pd.set_option('display.max_rows', 500) pd.set_option('display.max_columns', 500) pd.set_option('display.width', 1000) res3 = requests.get("https://web.archive.org/web/20220521203053/https://www.military-airshows.co.uk/press22/bbmfschedule2022.htm") soup3 = BeautifulSoup(res3.content,'lxml') BBMF_2022 = [] #BBMF_elem = soup3.find_all('a', string=re.compile(r'between|Flypast')) for item in soup3.find_all('a', string=re.compile(r'between|Flypast')): li1 = item.find_parent().text #li2 = li1.find_previous().font #print(link) print(li1) #print(li2) #BBMF_2022.append(li1) #check if links are in dataframe #df = pd.DataFrame(BBMF_2022, columns=['BBMF_2022']) #df

The issue I have is when I run the Code, the Data is printed for 15 Entries from May 28th to May 29th, several times,
I am not sure why that is the case ? Could someone suggest for me the reason why ? And tell me what I need to change in the Code, so
that that Data is printed only once and not several times ? I have tried to Scrape Data from a Website, where entries contain the word between or Flypast.

When I use the following piece of Code instead :-

for item in soup3.find_all('a', string=re.compile(r'between|Flypast')): li1 = item.find_parent().text #li2 = li1.find_previous().font #print(link) #print(li1) #print(li2) BBMF_2022.append(li1) df = pd.DataFrame(BBMF_2022, columns=['BBMF_2022']) df

The first entry for the 28th May, is printed out in the DataFrame 15 times ! instead of 15 seperate Entries I mentioned before.

Any help would be much appreciated.

Best Regards

Eddie Winch ))

[Larz60+]

You are using a redirected url, instead use: https://python-forum.io/thread-37342.html ?

This code will get all data and save as a json file, without any filtering. You can add filters, and any other data you need

import requests from bs4 import BeautifulSoup import os import json import sys class airshowdata: def __init__(self): self.airshow_details = {} self.cd = CreateDict() self.jsonfile = 'airshow.json' def get_links(self): url = 'https://www.military-airshows.co.uk/press22/bbmfschedule2022.htm' res3 = requests.get(url) if res3.status_code == 200: soup3 = BeautifulSoup(res3.content,'lxml') else: print(f"Cannot load page {url}") sys.exit(-1) links = soup3.find_all('a') for link in links: anode = self.cd.add_node(self.airshow_details, link.text.strip()) self.cd.add_cell(anode, 'url', link.get('href')) with open(self.jsonfile, 'w') as fp: json.dump(self.airshow_details, fp) # following not needed and can be removed (displays dictionary contents) self.cd.display_dict(self.airshow_details) class CreateDict: """ CreateDict.py - Contains methods to simplify node and cell creation within a dictionary Usage: new_dict(dictname) - Creates a new dictionary instance with the name contained in dictname add_node(parent, nodename) - Creates a new node (nested dictionary) named in nodename, in parent dictionary. add_cell(nodename, cellname, value) - Creates a leaf node within node named in nodename, with a cell name of cellname, and value of value. display_dict(dictname) - Recursively displays a nested dictionary. Requirements: Python standard library: os Author: Larz60+ -- May 2019. """ def __init__(self): os.chdir(os.path.abspath(os.path.dirname(__file__))) def new_dict(self, dictname): setattr(self, dictname, {}) def add_node(self, parent, nodename): node = parent[nodename] = {} return node def add_cell(self, nodename, cellname, value): cell = nodename[cellname] = value return cell def display_dict(self, dictname, level=0): indent = " " * (4 * level) for key, value in dictname.items(): if isinstance(value, dict): print(f'\n{indent}{key}') level += 1 self.display_dict(value, level) else: print(f'{indent}{key}: {value}') if level > 0: level -= 1 def main(): airs = airshowdata() airs.get_links() if __name__ == '__main__': main()

[eddywinch82]

Many thanks for that Code Larz60+, its very much appreciated by me, thankyou for taking the time to type it. I chose
the web.archive link, because the Data is from a week ago, from that Website, the 21st May Data was removed from the Website the other day.

Does anyone have any idea, how I can change my Code, to solve the issue I am having with it ?

Any help would be very much appreciated.

Regards

Eddie Winch ))