Skip to main content

​
mellea.stdlib.sampling.base

Base Sampling Strategies.

​
Classes

​
BaseSamplingStrategy

Base class for multiple strategies that rejects samples based on given instructions. Methods:

​
repair

repair(old_ctx: Context, new_ctx: Context, past_actions: list[Component], past_results: list[ModelOutputThunk], past_val: list[list[tuple[Requirement, ValidationResult]]]) -> tuple[Component, Context]
Repair function that is being invoked if not all requirements are fulfilled. It should return a next action component. Args:
  • old_ctx: The context WITHOUT the last action + output.
  • new_ctx: The context including the last action + output.
  • past_actions: List of actions that have been executed (without success).
  • past_results: List of (unsuccessful) generation results for these actions.
  • past_val: List of validation results for the results.
Returns:
  • The next action component and context to be used for the next generation attempt.

​
select_from_failure

select_from_failure(sampled_actions: list[Component], sampled_results: list[ModelOutputThunk], sampled_val: list[list[tuple[Requirement, ValidationResult]]]) -> int
This function returns the index of the result that should be selected as .value iff the loop budget is exhausted and no success. Args:
  • sampled_actions: List of actions that have been executed (without success).
  • sampled_results: List of (unsuccessful) generation results for these actions.
  • sampled_val: List of validation results for the results.
Returns:
  • The index of the result that should be selected as .value.

​
sample

sample(self, action: Component, context: Context, backend: Backend, requirements: list[Requirement] | None) -> SamplingResult
This method performs a sampling operation based on the given instruction. Args:
  • action : The action object to be sampled.
  • context: The context to be passed to the sampling strategy.
  • backend: The backend used for generating samples.
  • requirements: List of requirements to test against (merged with global requirements).
  • validation_ctx: Optional context to use for validation. If None, validation_ctx = ctx.
  • format: output format for structured outputs.
  • model_options: model options to pass to the backend during generation / validation.
  • tool_calls: True if tool calls should be used during this sampling strategy.
  • show_progress: if true, a tqdm progress bar is used. Otherwise, messages will still be sent to flog.
Returns:
  • A result object indicating the success or failure of the sampling process.
Raises:
  • AssertionError: Asserts that all required components (repair, select_from_failure, validate, and generate) are provided before proceeding with the sampling.

​
RejectionSamplingStrategy

Simple rejection sampling strategy that just repeats the same call on failure. Methods:

​
select_from_failure

select_from_failure(sampled_actions: list[Component], sampled_results: list[ModelOutputThunk], sampled_val: list[list[tuple[Requirement, ValidationResult]]]) -> int
Always returns the 0th index. Args:
  • sampled_actions: List of actions that have been executed (without success).
  • sampled_results: List of (unsuccessful) generation results for these actions.
  • sampled_val: List of validation results for the results.
Returns:
  • The index of the result that should be selected as .value.

​
repair

repair(old_ctx: Context, new_ctx: Context, past_actions: list[Component], past_results: list[ModelOutputThunk], past_val: list[list[tuple[Requirement, ValidationResult]]]) -> tuple[Component, Context]
Always returns the unedited, last action. Args:
  • old_ctx: The context WITHOUT the last action + output.
  • new_ctx: The context including the last action + output.
  • past_actions: List of actions that have been executed (without success).
  • past_results: List of (unsuccessful) generation results for these actions.
  • past_val: List of validation results for the results.
Returns:
  • The next action component and context to be used for the next generation attempt.

​
RepairTemplateStrategy

A sampling strategy that adds a repair string to the instruction object. Methods:

​
select_from_failure

select_from_failure(sampled_actions: list[Component], sampled_results: list[ModelOutputThunk], sampled_val: list[list[tuple[Requirement, ValidationResult]]]) -> int
Always returns the 0th index. Args:
  • sampled_actions: List of actions that have been executed (without success).
  • sampled_results: List of (unsuccessful) generation results for these actions.
  • sampled_val: List of validation results for the results.
Returns:
  • The index of the result that should be selected as .value.

​
repair

repair(old_ctx: Context, new_ctx: Context, past_actions: list[Component], past_results: list[ModelOutputThunk], past_val: list[list[tuple[Requirement, ValidationResult]]]) -> tuple[Component, Context]
Adds a description of the requirements that failed to a copy of the original instruction. Args:
  • old_ctx: The context WITHOUT the last action + output.
  • new_ctx: The context including the last action + output.
  • past_actions: List of actions that have been executed (without success).
  • past_results: List of (unsuccessful) generation results for these actions.
  • past_val: List of validation results for the results.
Returns:
  • The next action component and context to be used for the next generation attempt.

​
MultiTurnStrategy

Rejection sampling strategy with (agentic) multi-turn repair. Methods:

​
select_from_failure

select_from_failure(sampled_actions: list[Component], sampled_results: list[ModelOutputThunk], sampled_val: list[list[tuple[Requirement, ValidationResult]]])
Always returns the last index. The last message from the model will always be returned if all results are failures. Args:
  • sampled_actions: List of actions that have been executed (without success).
  • sampled_results: List of (unsuccessful) generation results for these actions.
  • sampled_val: List of validation results for the results.
Returns:
  • The index of the result that should be selected as .value.

​
repair

repair(old_ctx: Context, new_ctx: Context, past_actions: list[Component], past_results: list[ModelOutputThunk], past_val: list[list[tuple[Requirement, ValidationResult]]]) -> tuple[Component, Context]
Returns a Message with a description of the failed requirements. Args:
  • old_ctx: The context WITHOUT the last action + output.
  • new_ctx: The context including the last action + output.
  • past_actions: List of actions that have been executed (without success).
  • past_results: List of (unsuccessful) generation results for these actions.
  • past_val: List of validation results for the results.
Returns:
  • The next action component and context to be used for the next generation attempt.
⌘I