Consider using a library like trafilatura to do the fetching and extraction of content. It should be quite a bit more reliable, although it will also struggle with many JS based sites.