feat: added element metadata to unstructured loader (#1068)

### Summary Adds tracked metadata from `unstructured` elements to the document metadata when `UnstructuredFileLoader` is used in `"elements"` mode. Tracked metadata is available in `unstructured>=0.4.9`, but the code is written for backward compatibility with older `unstructured` versions. ### Testing Before running, make sure to upgrade to `unstructured==0.4.9`. In the code snippet below, you should see `page_number`, `filename`, and `category` in the metadata for each document. `doc[0]` should have `page_number: 1` and `doc[-1]` should have `page_number: 2`. The example document is `layout-parser-paper-fast.pdf` from the [`unstructured` sample docs](https://github.com/Unstructured-IO/unstructured/tree/main/example-docs). ```python from langchain.document_loaders import UnstructuredFileLoader loader = UnstructuredFileLoader(file_path=f"layout-parser-paper-fast.pdf", mode="elements") docs = loader.load() ```

Author: MthwRobinson

langchain/document_loaders/unstructured.py

@@ -33,12 +33,19 @@ def _get_elements(self) -> List:
  def load(self) -> List[Document]:
  """Load file."""
  elements = self._get_elements()
- metadata = {"source": self.file_path}
  if self.mode == "elements":
- docs = [
- Document(page_content=str(el), metadata=metadata) for el in elements
- ]
+ docs: List[Document] = list()
+ for element in elements:
+ metadata = {"source": self.file_path}
+ # NOTE(MthwRobinson) - the attribute check is for backward compatibility
+ # with unstructured<0.4.9. The metadata attributed was added in 0.4.9.
+ if hasattr(element, "metadata"):
+ metadata.update(element.metadata.to_dict())
+ if hasattr(element, "category"):
+ metadata["category"] = element.category
+ docs.append(Document(page_content=str(element), metadata=metadata))
  elif self.mode == "single":
+ metadata = {"source": self.file_path}
  text = "\n\n".join([str(el) for el in elements])
  docs = [Document(page_content=text, metadata=metadata)]
  else: