[SOT] Add unsafe cache fast path #73372
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
SigureMo reviewed Jun 16, 2025
python/paddle/jit/sot/opcode_translator/executor/executor_cache.py Outdated Show resolved Hide resolved
python/paddle/jit/sot/opcode_translator/executor/executor_cache.py Outdated Show resolved Hide resolved
| 加个单测 |
SigureMo reviewed Jun 17, 2025
python/paddle/jit/sot/opcode_translator/executor/executor_cache.py Outdated Show resolved Hide resolved
Comment on lines 78 to 85
| cache: dict[ | ||
| types.CodeType, tuple[GuardedFunctions, paddle.framework.core.GuardTree] | ||
| ] | ||
| translate_count: int | ||
| code_symbolic_inputs: dict[types.CodeType, dict[str, None | dict[int, int]]] | ||
| compile_time_stats: dict[types.CodeType, float] | ||
| consecutive_cache_hit_count: dict[types.CodeType, int] | ||
| last_cache_index: dict[types.CodeType, int | None] |
There was a problem hiding this comment.
TODO: 太多 code 相关状态，后续单独拆出来一个 class 用来管理这些状态，OpcodeExecutorCache 用来管理 code 到这个状态的映射
SigureMo changed the title huangjiyi pushed a commit to huangjiyi/Paddle that referenced this pull request Jun 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
PR Category
Execute Infrastructure
PR Types
Improvements
Description
在推理场景下，通常只会命中同一个 cache，但目前每次都需要重新执行 guard 检查，这其实没有必要。因此，我们可以启用一种不安全的优化策略：如果同一个 cache 被连续命中超过一定次数（例如 32 次），则可以认为输入数据没有发生变化，直接返回该 cache，跳过 guard 检查，以提升推理效率。
这个策略通过环境变量
SOT_UNSAFE_CACHE_FASTPATH开启收益：（稳定后）某模型单Token时间从 35.9ms -> 29.8ms
cc @SigureMo
PCard-66972