Skip to main content
Assertions use a single entrypoint and an expressive catalog. Prefer structural checks for stability, and combine them with judges only when necessary.
Use one entrypoint: 
await session.assert_that(Expect.content.contains("Example"), response=resp)
Catalog source: catalog.py

​
Immediate vs deferred

  • Immediate: content and judge (given response)
  • Deferred (requires final metrics): tools, performance, path

​
Content

  • Expect.content.contains(text, case_sensitive=False)
  • Expect.content.not_contains(text, case_sensitive=False)
  • Expect.content.regex(pattern, case_sensitive=False)
Examples: 
resp = await agent.generate_str("Fetch https://example.com") await session.assert_that(Expect.content.contains("Example Domain"), response=resp) await session.assert_that(Expect.content.not_contains("Stack trace"), response=resp) await session.assert_that(Expect.content.regex(r"Example\\s+Domain"), response=resp)

​
Tools

  • Expect.tools.was_called(name, min_times=1)
  • Expect.tools.called_with(name, {args})
  • Expect.tools.count(name, expected_count)
  • Expect.tools.success_rate(min_rate, tool_name=None)
  • Expect.tools.failed(name)
  • Expect.tools.output_matches(name, expected, field_path?, match_type?, case_sensitive?, call_index?)
  • Expect.tools.sequence([names], allow_other_calls=False)
Notes:
  • field_path supports nested dict/list access, e.g. content[0].text or content.0.text
Examples: 
# Verify a fetch occurred with expected output pattern in first content block await session.assert_that(  Expect.tools.output_matches(  tool_name="fetch",  expected_output=r"use.*examples",  match_type="regex",  case_sensitive=False,  field_path="content[0].text",  ),  name="fetch_output_match", )  # Verify sequence and counts await session.assert_that(Expect.tools.sequence(["fetch"], allow_other_calls=True)) await session.assert_that(Expect.tools.count("fetch", 1)) await session.assert_that(Expect.tools.success_rate(1.0, tool_name="fetch"))

​
Performance

  • Expect.performance.max_iterations(n)
  • Expect.performance.response_time_under(ms)
Example: 
await session.assert_that(Expect.performance.max_iterations(3)) await session.assert_that(Expect.performance.response_time_under(10_000))

​
Judge

  • Expect.judge.llm(rubric, min_score=0.8, include_input=False, require_reasoning=True)
  • Expect.judge.multi_criteria(criteria, aggregate_method="weighted", require_all_pass=False, include_confidence=True, use_cot=True, model=None)
Examples: 
# Single-criterion rubric judge = Expect.judge.llm(  rubric="Response should identify JSON format and summarize main points",  min_score=0.8,  include_input=True, ) await session.assert_that(judge, response=resp, name="quality_check")  # Multi-criteria from mcp_eval.evaluators import EvaluationCriterion criteria = [  EvaluationCriterion(name="accuracy", description="Factual correctness", weight=2.0, min_score=0.8),  EvaluationCriterion(name="completeness", description="Covers key points", weight=1.5, min_score=0.7), ] judge_mc = Expect.judge.multi_criteria(criteria, aggregate_method="weighted", use_cot=True) await session.assert_that(judge_mc, response=resp, name="multi_criteria")

​
Path

  • Expect.path.efficiency(optimal_steps?, expected_tool_sequence?, allow_extra_steps=0, penalize_backtracking=True, penalize_repeated_tools=True, tool_usage_limits?, default_tool_limit=1)
Evaluators source: evaluators/

​
Examples

# Enforce exact order and single use of tools await session.assert_that(  Expect.path.efficiency(  expected_tool_sequence=["validate", "process", "save"],  tool_usage_limits={"validate": 1, "process": 1, "save": 1},  allow_extra_steps=0,  penalize_backtracking=True,  ),  name="golden_path", )  # Allow a retry of a fragile step without failing the whole path await session.assert_that(  Expect.path.efficiency(  expected_tool_sequence=["fetch", "parse"],  tool_usage_limits={"fetch": 2, "parse": 1},  allow_extra_steps=1,  penalize_repeated_tools=False,  ),  name="robust_path", )

​
Tips

  • Prefer structural checks (output_matches) for tool outputs when possible for stability
  • Use name in assert_that(..., name="...") to label checks in reports
  • Combine judge + structural checks for high confidence
Combine a minimal judge (e.g., rubric with min_score=0.8) with one or two structural checks (like output_matches) for resilient tests.
Deferred assertions are evaluated when the session ends or when when="end" is used. If an assertion depends on final metrics (e.g., success rate), defer it.