(Apr-23-2024, 08:49 AM)Winfried Wrote: Some of the files could be Windows (latin1, iso9959-1, cp1252), others could be utf-8.
As these are
.html some advice if you are making or saving this these files,then there is a way if using Requests and BS to always save as utf-8.
If files are already made then as bye Gribouillis there is
chardet.
So eg if i have one .html file which(i make to be latin-1) and one in utf-8.
λ chardetect page_latin.html page_latin.html: ISO-8859-1 with confidence 0.73 G:\div_code\html_utf λ chardetect page_utf8.html page_utf8.html: utf-8 with confidence 0.7525
from bs4 import BeautifulSoup with open('page_latin.html', encoding='latin-1') as fp: soup = BeautifulSoup(fp, 'lxml') h1_tag = soup.find('h1') print(h1_tag) # Utf-8 the default with open('html_new.html') as fp: soup = BeautifulSoup(fp, 'lxml') h1_tag = soup.find('h1') print(h1_tag)Output:
<h1>Jalapeñod je pèle</h1> <h1>Jalapeñod je pèle</h1>
So all works as it should,if take away
encoding='latin-1' it break and get
UnicodeDecodeError.
Can also convert to utf-8 as this happens when open a file in Beautiful Soup:
Bs4 Doc Wrote:Any HTML or XML document is written in a specific encoding like ASCII or UTF-8.
But when you load that document into Beautiful Soup, you’ll discover it’s been converted to Unicode.
Beautiful Soup uses a sub-library called Unicode, Dammit to detect a document’s encoding and convert it to Unicode.
So from latin-1 to utf-8.
from bs4 import BeautifulSoup with open('page_latin.html', 'rb') as fp,open('html_new.html', 'w', encoding='utf-8') as fp_out: file_out = fp.read() # When open a file in BS it will be Unicode soup = BeautifulSoup(file_out, 'lxml') fp_out.write(soup.prettify())λ chardetect html_new.html html_new.html: utf-8 with confidence 0.7525
File used in test,same just with different encoding.
<html lang="en"> <head> <title>Here is site title</title> </head> <body> <h1>Jalapeñod je pèle</h1> </body> </html>
