FilterExpressionTextParser does not support 'null' values #3694
Description
Bug description
FilterExpressionTextParser does not support 'null' values, so I can't write
SearchRequest searchRequest = SearchRequest.builder() .query("Spring AI") .similarityThresholdAll() .topK(3) .filterExpression("meta1 == null") .build();At the same time, FilterExpressionBuilder do support it.
var b = new FilterExpressionBuilder(); SearchRequest searchRequest = SearchRequest.builder() .query("Spring AI") .similarityThresholdAll() .topK(3) .filterExpression(b.eq("meta1", null).build()) .build();Environment
Spring AI v1.0.0, Java 17, Vector Store - ClickHouse (but it doesn't depend on vector store type)
Steps to reproduce
SearchRequest searchRequest = SearchRequest.builder() .query("Spring AI") .similarityThresholdAll() .topK(3) .filterExpression("meta1 == null") .build();Expected behavior
I expect that Filter.Expression::right will return null when it called from filter expression converter (FilterExpressionConverter::convertExpression)
Minimal Complete Reproducible example
https://github.com/linarkou/spring-ai-clickhouse-store/blob/main/spring-ai-clickhouse-store/src/test/java/org/springframework/ai/vectorstore/clickhouse/ClickhouseVectorStoreIT.java
private static List<Document> documents() { return List.of( new Document("1", getText("classpath:/test/data/spring.ai.txt"), Map.of("meta1", "meta1","intMeta", 456)), new Document("2", getText("classpath:/test/data/time.shelter.txt"), Map.of()), new Document("3", getText("classpath:/test/data/great.depression.txt"), Map.of("meta2", "meta2", "intMeta", 123))); } private static String getText(String uri) { var resource = new DefaultResourceLoader().getResource(uri); try { return resource.getContentAsString(StandardCharsets.UTF_8); } catch (IOException e) { throw new RuntimeException(e); } } public static Stream<Arguments> filterExpressions() { var b = new FilterExpressionBuilder(); return Stream.of( Arguments.of( b.ne("meta1", null).build(), "meta1 != null", List.of("1")), Arguments.of( b.eq("meta1", null).build(), "meta1 == null", List.of("2", "3")) ); } @MethodSource("filterExpressions") @ParameterizedTest void testSimilaritySearchWithFilters(Filter.Expression filterExpression, String nativeFilterExpression, List<String> expectedIds) { this.contextRunner.run(context -> { VectorStore vectorStore = context.getBean(AbstractObservationVectorStore.class); List<Document> originalDocuments = documents(); vectorStore.doAdd(originalDocuments); if (filterExpression != null) { SearchRequest searchRequest = SearchRequest.builder() .query("Spring AI") .similarityThresholdAll() .topK(3) .filterExpression(filterExpression) .build(); List<Document> documents = vectorStore.doSimilaritySearch(searchRequest); assertEquals(expectedIds, documents.stream().map(Document::getId).collect(Collectors.toList())); } if (nativeFilterExpression != null) { SearchRequest searchRequest = SearchRequest.builder() .query("Spring AI") .similarityThresholdAll() .topK(3) .filterExpression(nativeFilterExpression) .build(); List<Document> documents = vectorStore.doSimilaritySearch(searchRequest); assertEquals(expectedIds, documents.stream().map(Document::getId).collect(Collectors.toList())); } vectorStore.doDelete(originalDocuments.stream().map(Document::getId).toList()); vectorStore.close(); }); } Metadata
Metadata
Assignees
Labels
No labels