Skip to content

Conversation

sobychacko
Copy link
Contributor

…citly cache tools

This commit fixes a bug where the SYSTEM_ONLY cache strategy was incorrectly placing cache_control on tool definitions, making it functionally identical to SYSTEM_AND_TOOLS.

Problem:

  • SYSTEM_ONLY and SYSTEM_AND_TOOLS both placed cache breakpoints on tools AND system messages, using 2 breakpoints
  • This violated the documented behavior and wasted breakpoints
  • The strategies were functionally identical

Root Cause:

  • CacheEligibilityResolver.resolveToolCacheControl() checked if eligible message types included SYSTEM, which both strategies have
  • It should have checked the strategy directly instead

Solution:

  • Changed resolveToolCacheControl() to use allowlist approach
  • Only SYSTEM_AND_TOOLS and CONVERSATION_HISTORY explicitly cache tools
  • SYSTEM_ONLY now places cache_control only on system message
  • Tools still get cached implicitly via Anthropic's cache hierarchy (tools → system → messages)

Impact:

  • SYSTEM_ONLY: Uses 1 breakpoint (system only) instead of 2
  • SYSTEM_AND_TOOLS: Uses 2 breakpoints (last tool + system) as before
  • Users can now optimize breakpoint usage more effectively

Changes:

  • Fix CacheEligibilityResolver.resolveToolCacheControl() to check strategy directly instead of eligible message types
  • Enhance testSystemOnlyCacheStrategy() to include tools and verify they don't have cache_control
  • Update toolCacheControlRespectsStrategy() test to verify all strategies behave correctly

Thank you for taking time to contribute this pull request!
You might have already read the contributor guide, but as a reminder, please make sure to:

  • Add a Signed-off-by line to each commit (git commit -s) per the DCO
  • Rebase your changes on the latest main branch and squash your commits
  • Add/Update unit tests as needed
  • Run a build and make sure all tests pass prior to submission

For more details, please check the contributor guide.
Thank you upfront!

@sobychacko
Copy link
Contributor Author

@adase11 I made some slight semantic changes here. Could you take a look and see what you think?

@adase11
Copy link
Contributor

adase11 commented Oct 2, 2025

@adase11 I made some slight semantic changes here. Could you take a look and see what you think?

Can do, sorry I am traveling until next week but I'll try to get to it in a timely manner. I also saw your other question and am giving it some thought. Thanks!

…citly cache tools This commit fixes a bug where the SYSTEM_ONLY cache strategy was incorrectly placing cache_control on tool definitions, making it functionally identical to SYSTEM_AND_TOOLS. Problem: - SYSTEM_ONLY and SYSTEM_AND_TOOLS both placed cache breakpoints on tools AND system messages, using 2 breakpoints - This violated the documented behavior and wasted breakpoints - The strategies were functionally identical Root Cause: - CacheEligibilityResolver.resolveToolCacheControl() checked if eligible message types included SYSTEM, which both strategies have - It should have checked the strategy directly instead Solution: - Changed resolveToolCacheControl() to use allowlist approach - Only SYSTEM_AND_TOOLS and CONVERSATION_HISTORY explicitly cache tools - SYSTEM_ONLY now places cache_control only on system message - Tools still get cached implicitly via Anthropic's cache hierarchy (tools → system → messages) Impact: - SYSTEM_ONLY: Uses 1 breakpoint (system only) instead of 2 - SYSTEM_AND_TOOLS: Uses 2 breakpoints (last tool + system) as before - Users can now optimize breakpoint usage more effectively Changes: - Fix CacheEligibilityResolver.resolveToolCacheControl() to check strategy directly instead of eligible message types - Enhance testSystemOnlyCacheStrategy() to include tools and verify they don't have cache_control - Update toolCacheControlRespectsStrategy() test to verify all strategies behave correctly Signed-off-by: Soby Chacko <soby.chacko@broadcom.com>
@sobychacko sobychacko force-pushed the fix/anthropic-system-only-cache-strategy branch from 82d63b2 to 56e2e0a Compare October 3, 2025 17:38
 This commit introduces the TOOLS_ONLY caching strategy, enabling scenarios where large tool definitions should be cached while system prompts remain dynamic and uncached. Key Changes: - Add TOOLS_ONLY enum to AnthropicCacheStrategy with comprehensive javadocs - Update CacheEligibilityResolver to support TOOLS_ONLY strategy - Enhance all strategy javadocs with detailed use cases and token guidance - Add comprehensive unit tests covering all caching scenarios - Update documentation with TOOLS_ONLY examples and cascade invalidation Use Cases: - Multi-tenant SaaS applications with shared tools but per-tenant system prompts - A/B testing scenarios with stable tools but variable system instructions - Applications with large tool sets (5000+ tokens) and dynamic contexts Technical Details: - TOOLS_ONLY uses 1 cache breakpoint on the last tool definition - System messages are NOT cached, processed fresh on each request - Supports Anthropic's cache hierarchy (tools → system → messages) - Compatible with cascade invalidation behavior Documentation: - Added strategy comparison table with breakpoint usage - Added multi-tenant SaaS use case example - Updated best practices with cascade invalidation explanation - Clarified SYSTEM_AND_TOOLS independence behavior Signed-off-by: Soby Chacko <soby.chacko@broadcom.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants