聊聊Spring AI Alibaba的BilibiliDocumentReader

序

本文主要研究一下Spring AI Alibaba的BilibiliDocumentReader

BilibiliDocumentReader

community/document-readers/spring-ai-alibaba-starter-document-reader-bilibili/src/main/java/com/alibaba/cloud/ai/reader/bilibili/BilibiliDocumentReader.java

public class BilibiliDocumentReader implements DocumentReader { private static final Logger logger = LoggerFactory.getLogger(BilibiliDocumentReader.class); private static final String API_BASE_URL = "https://api.bilibili.com/x/web-interface/view?bvid="; private final String resourcePath; private final ObjectMapper objectMapper; private static final int MEMORY_SIZE = 5; private static final int BYTE_SIZE = 1024; private static final int MAX_MEMORY_SIZE = MEMORY_SIZE * BYTE_SIZE * BYTE_SIZE; private static final WebClient WEB_CLIENT = WebClient.builder() .defaultHeader(HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE) .codecs(configurer -> configurer.defaultCodecs().maxInMemorySize(MAX_MEMORY_SIZE)) .build(); public BilibiliDocumentReader(String resourcePath) { Assert.hasText(resourcePath, "Query string must not be empty"); this.resourcePath = resourcePath; this.objectMapper = new ObjectMapper(); } @Override public List<Document> get() { List<Document> documents = new ArrayList<>(); try { String bvid = extractBvid(resourcePath); String videoInfoResponse = fetchVideoInfo(bvid); JsonNode videoData = parseJson(videoInfoResponse).path("data"); String title = videoData.path("title").asText(); String description = videoData.path("desc").asText(); Document infoDoc = new Document("Video information", Map.of("title", title, "description", description)); documents.add(infoDoc); String documentContent = fetchAndProcessSubtitles(videoData, title, description); documents.add(new Document(documentContent)); } catch (IllegalArgumentException e) { logger.error("Invalid input: {}", e.getMessage()); documents.add(new Document("Error: Invalid input")); } catch (IOException e) { logger.error("Error parsing JSON: {}", e.getMessage(), e); documents.add(new Document("Error parsing JSON: " + e.getMessage())); } catch (Exception e) { logger.error("Unexpected error: {}", e.getMessage(), e); documents.add(new Document("Unexpected error: " + e.getMessage())); } return documents; } private String extractBvid(String resourcePath) { return resourcePath.replaceAll(".*(BV\\w+).*", "$1"); } private String fetchVideoInfo(String bvid) { return WEB_CLIENT.get().uri(API_BASE_URL + bvid).retrieve().bodyToMono(String.class).block(); } private JsonNode parseJson(String jsonResponse) throws IOException { return objectMapper.readTree(jsonResponse); } private String fetchAndProcessSubtitles(JsonNode videoData, String title, String description) throws IOException { JsonNode subtitleList = videoData.path("subtitle").path("list"); if (subtitleList.isArray() && subtitleList.size() > 0) { String subtitleUrl = subtitleList.get(0).path("subtitle_url").asText(); String subtitleResponse = WEB_CLIENT.get().uri(subtitleUrl).retrieve().bodyToMono(String.class).block(); JsonNode subtitleJson = parseJson(subtitleResponse); StringBuilder rawTranscript = new StringBuilder(); subtitleJson.path("body").forEach(node -> rawTranscript.append(node.path("content").asText()).append(" ")); return String.format("Video Title: %s, Description: %s\nTranscript: %s", title, description, rawTranscript.toString().trim()); } else { return String.format("No subtitles found for video: %s. Returning an empty transcript.", resourcePath); } } }

BilibiliDocumentReader使用WebClient去请求B站接口，它从url解析bvid，再根据bvid去请求接口，解析json获取title、description，通过fetchAndProcessSubtitles再去请求subtitle_url获取字幕内容作为document的内容

示例

public class BilibiliDocumentReaderTest { private static final Logger logger = LoggerFactory.getLogger(BilibiliDocumentReader.class); @Test void bilibiliDocumentReaderTest() { BilibiliDocumentReader bilibiliDocumentReader = new BilibiliDocumentReader( "https://www.bilibili.com/video/BV1KMwgeKECx/?t=7&vd_source=3069f51b168ac07a9e3c4ba94ae26af5"); List<Document> documents = bilibiliDocumentReader.get(); logger.info("documents: {}", documents); } }

小结

spring-ai-alibaba-starter-document-reader-bilibili提供了BilibiliDocumentReader用于解析B站的视频url，它请求两次接口，一次获取title和description，一次获取字幕。

doc

• java2ai