feat(agent): support multiple flow log senders #10177
Open
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
This PR is for:
Feature:
背景问题
在专属采集器或高速数据面(如 2 Mpps、L4 Flow 限速 4 万条/s)下,现有链路中仅包含单个发送队列和单个
UniformSenderThread,很容易出现写入过快、队列被写满、日志覆盖丢弃的问题,最终影响日志完整性与可观测性。本次改动实现
✅ 主要改动
在 agent/crates/public/src/queue/ 下新增:mpmc_queue.rs:实现基于 crossbeam_channel 的 Sender、Receiver 和 StatsHandle,支持 MPMC 模型;
修改 queue/mod.rs: 引入并导出 mpmc_queue,增加 bounded_mpmc() 构造函数
✅ 新增功能:
引入可配置的 日志发送并发度参数,包括:
l4_flow_sendersl7_flow_sendersmetric_senderspcap_senders等每类日志类型按配置值创建多个:
DebugSender队列(queue::bounded_with_debug)UniformSenderThread实例所有 sender 命名后缀追加编号(如
3-flowlog-to-collector-sender-0~-3)✅ 日志发送改为并发分发:
负载均衡策略:目前支持:
发送线程存入
Vec<UniformSenderThread<_>>,在Trident::start()中统一启动,优雅退出时统一停止✅ 配置示例(YAML):