Skip to main content
Version: Next

mellea.stdlib.sampling.sofai

SOFAI (Slow and Fast AI) Sampling Strategy.

This module implements a two-solver sampling strategy that uses:

  1. S1 Solver (fast model) - Iterative solving with feedback-based repair
  2. S2 Solver (slow model) - Single attempt escalation when S1 fails or shows no improvement

The strategy leverages ValidationResult.reason fields to provide targeted feedback for repair, enabling more effective iterative improvement.

Classes​

CLASS SOFAISamplingStrategy ​

SOFAI (Slow and Fast AI) two-solver sampling strategy.

Uses S1 Solver (fast model) in a loop with targeted feedback from validation results. If S1 Solver fails after exhausting the budget or shows no improvement, escalates to a single attempt with S2 Solver (slow model).

The strategy leverages ValidationResult.reason fields to provide targeted feedback for repair, enabling more effective iterative improvement.

Args:

  • s1_solver_backend: Backend for the fast S1 solver used in the iterative repair loop.
  • s2_solver_backend: Backend for the slow S2 solver used as a final escalation step.
  • s2_solver_mode: How to invoke the S2 solver when S1 fails.
  • loop_budget: Maximum number of S1 repair attempts before escalating to S2. Must be greater than 0. Defaults to 3.
  • judge_backend: Optional backend for LLM-as-Judge validation. If None, falls back to the session backend.
  • feedback_strategy: Detail level of repair feedback provided to the S1 solver.

Methods:

FUNC repair ​

repair(old_ctx: Context, new_ctx: Context, past_actions: Sequence[SampleActionType], past_results: list[ComputedModelOutputThunk], past_val: list[list[tuple[Requirement, ValidationResult]]]) -> tuple[SampleActionType, Context]

Create targeted feedback message from validation results.

Extracts failed requirements and uses their ValidationResult.reason fields to provide specific, actionable feedback for the next attempt.

Args:

  • old_ctx: The context WITHOUT the last action + output.
  • new_ctx: The context including the last action + output.
  • past_actions: List of actions executed.
  • past_results: List of generation results.
  • past_val: List of validation results.

Returns:

  • Tuple of (Message component with repair feedback, new context).

FUNC select_from_failure ​

select_from_failure(sampled_actions: Sequence[SampleActionType], sampled_results: list[ComputedModelOutputThunk], sampled_val: list[list[tuple[Requirement, ValidationResult]]]) -> int

Select the most informed attempt (last) when all fail.

Returns the last attempt as it has benefited from the most feedback.

Args:

  • sampled_actions: List of actions that have been executed (without success).
  • sampled_results: List of (unsuccessful) generation results for these actions.
  • sampled_val: List of validation results for the results.

Returns:

  • The index of the result that should be selected as .value.