Streamline the Document API #38
Conversation
Add TextFormatter and DefaultTextFormatter concept that can filter the metadata and format the Document metadata and text according to predefined templates.
When the splitter breaks the parent Document into multiple chunks (e.g. into a list of children Documents) copy the source content formatter to the chunks by default. Use the copyContentFormatter flag to enable/disable copping.
Add useFormattedContent field to control the formatted vs. raw content being used for indexing.
| | ||
| enum MetadataMode { | ||
| | ||
| ALL, EMBED, LLM, NONE; |
There was a problem hiding this comment.
I think we should change LLM to something like INFERENCE of MODEL as I've removed use of LLM generally speaking in naming things.
| /** | ||
| * Metadata keys that are excluded from text for the LLM. | ||
| */ | ||
| private final List<String> excludedLlmMetadataKeys; |
There was a problem hiding this comment.
similar comment wrt to naming , excludedInferenceMetadataKeys ?
spring-ai-core/src/main/java/org/springframework/ai/document/Document.java Outdated Show resolved Hide resolved
spring-ai-core/src/main/java/org/springframework/ai/document/Document.java Show resolved Hide resolved
| if (this.copyContentFormatter) { | ||
| // Copies the parent content formatter to the chucks documents it was | ||
| // spelt into. | ||
| newDoc = newDoc.updateContentFormatter(formatters.get(i)); |
There was a problem hiding this comment.
multiple formatters indexed by List of strings seems unusual to me.
There was a problem hiding this comment.
It is because the createDocuments takes a list of texts, one per source document, and merged metadata of all source documents. We can not merge the content formatters, so we have to pass in a list of formatters (one per source document) that corresponds in order to the list of texts.
We can simplify the createDocuments to take one document at a time, but will loose the metadata merging, or introduce an internal record class like Cuhnk(String text, Map<String, Object> metadata, ContentFormatter formatter)?
There was a problem hiding this comment.
Actually, why de we merge the metadata from all parent documents? Shouldn't the chunked docs inherit only the metadata their own parent? Not a mixture of metadata of all different parents?
Later can make sense if we can determine that all parent documents come from the same initial source for example. But, currently, we don't have a convention to reliably recognise this.
- Make the metadata mode configurable for the EmbeddingClient implementations. - Use the EMBED mode by default.
| Replaced #45 |
2 participants
Add TextFormatter and DefaultTextFormatter concept that can filter the metadata and format the Document metadata and text according to predefined templates.