mellea.core.sampling

Abstract interfaces for sampling strategies and their results.

SamplingStrategy defines the contract for all sampling algorithms: an async sample method that takes an action, context, backend, and requirements, and returns a SamplingResult. SamplingResult records the chosen generation alongside the full history of intermediate samples, their validation outcomes, and associated contexts — enabling detailed post-hoc inspection of the sampling process.

Classes

CLASS SamplingResult

Stores the results from a sampling operation. This includes successful and failed samplings.

Args:

• result_index: Index into sample_generations identifying the chosen final output.
• success: Whether the sampling operation produced a passing result.
• sample_generations: All output thunks generated during sampling.
• sample_validations: Per-generation validation results; each inner list contains one tuple per requirement evaluated.
• sample_actions: The actions used to produce each generation.
• sample_contexts: The contexts associated with each generation.

Attributes:

• result_index: Index into sample_generations identifying the chosen final output.
• success: Whether the sampling operation produced a passing result.
• sample_generations: All output thunks generated during sampling; always a list (None input is normalised to []).
• sample_validations: Per-generation validation results; always a list (None input is normalised to []).
• sample_actions: The actions used to produce each generation; always a list (None input is normalised to []).
• sample_contexts: The contexts associated with each generation; always a list (None input is normalised to []).

Methods:

FUNC result

result(self) -> ComputedModelOutputThunk[S]

The final output or result from applying the sampling strategy.

FUNC result_ctx

result_ctx(self) -> Context

The context of the final output or result from applying the sampling strategy.

FUNC result_action

result_action(self) -> SampleActionType

The action that generated the final output or result from applying the sampling strategy.

FUNC result_validations

result_validations(self) -> list[tuple[Requirement, ValidationResult]]

The validation results associated with the final output or result from applying the sampling strategy.

CLASS SamplingStrategy

A SamplingStrategy class defines an abstract base class for implementing various sampling strategies.

This class provides a template for creating concrete sampling strategies that can be used to generate model outputs based on given instructions. It allows setting custom validation and generation functions through properties.

Attributes:

• loop_budget: Maximum number of generate/validate cycles. Defaults to 1.
• requirements: Global requirements evaluated on every sample. When set, overrides per-call requirements. Defaults to None.

Methods:

FUNC sample

sample(self, action: Component[S] | CBlock | ModelOutputThunk, context: Context, backend: Backend, requirements: list[Requirement] | None) -> SamplingResult[S]

Concrete wrapper: owns the sampling lifecycle and fires loop start/end hooks.

Mints a sampling_id, dispatches sampling_loop_start (which may modify loop_budget), delegates to _sample, and dispatches sampling_loop_end on every exit path — success, budget exhaustion, and raised exceptions.

Args:

• action: The action object to be sampled. A Component, CBlock, or ModelOutputThunk.
• context: The context to be passed to the sampling strategy.
• backend: The backend used for generating samples.
• requirements: List of requirements to test against (merged with global requirements).
• validation_ctx: Optional context to validate over. If None, each sample is validated over its own post-generation context.
• format: output format for structured outputs.
• model_options: model options to pass to the backend during generation / validation.
• tool_calls: True if tool calls should be used during this sampling strategy.
• show_progress: If true, a tqdm progress bar is used. Otherwise, messages will still be sent to flog.

Returns:

• SamplingResult[S]: A result object indicating the success or failure of the sampling process.

Raises:

• ValueError: If a SAMPLING_LOOP_START hook returns a non-positive loop_budget.