feat: adds UnstructuredURLLoader for loading data from urls

[MthwRobinson]

Summary

Adds a UnstructuredURLLoader that supports loading data from a list of URLs.

Testing

from langchain.document_loaders import UnstructuredURLLoader urls = [ "https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-8-2023", "https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-9-2023" ] loader = UnstructuredURLLoader(urls=urls) raw_documents = loader.load()

[hwchase17]

looks awesome! thanks

[Boscop]

@MthwRobinson Can you please adjust UnstructuredURLLoader to allow loading text/plain files from URLs? (Ideally also other mime types like text/markdown.)
Currently I'm getting Error fetching or processing https://raw.githubusercontent.com/[...], exeption: Expected content type text/html. Got text/plain; charset=utf-8.

[MthwRobinson]

@Boscop - Thanks for flagging, we added an issue in the unstructured library to support other MIME types and will pick it up as soon as we can.

[IqraShahid-dev]

@MthwRobinson Got error for pdf while fetching from s3 bucket,

raise ValueError(f"Expected content type text/html. Got {content_type}.") ValueError: Expected content type text/html. Got application/pdf. 

[MthwRobinson]

#2793 will allow for processing non HTML resources with the URL loader. @Boscop - you can pass the content_type kwarg into the loader to force it to use a specific MIME type.