Skip to main content

mellea.core.sampling

Abstract interfaces for sampling strategies and their results.

SamplingStrategy defines the contract for all sampling algorithms: an async sample method that takes an action, context, backend, and requirements, and returns a SamplingResult. SamplingResult records the chosen generation alongside the full history of intermediate samples, their validation outcomes, and associated contexts — enabling detailed post-hoc inspection of the sampling process.

Classes​

CLASS SamplingResult ​

Stores the results from a sampling operation. This includes successful and failed samplings.

Args:

  • result_index: Index into sample_generations identifying the chosen final output.
  • success: Whether the sampling operation produced a passing result.
  • sample_generations: All output thunks generated during sampling.
  • sample_validations: Per-generation validation results; each inner list contains one tuple per requirement evaluated.
  • sample_actions: The actions used to produce each generation.
  • sample_contexts: The contexts associated with each generation.

Attributes:

  • result_index: Index into sample_generations identifying the chosen final output.
  • success: Whether the sampling operation produced a passing result.
  • sample_generations: All output thunks generated during sampling; always a list (None input is normalised to []).
  • sample_validations: Per-generation validation results; always a list (None input is normalised to []).
  • sample_actions: The actions used to produce each generation; always a list (None input is normalised to []).
  • sample_contexts: The contexts associated with each generation; always a list (None input is normalised to []).

Methods:

FUNC result ​

result(self) -> ComputedModelOutputThunk[S]

The final output or result from applying the sampling strategy.

FUNC result_ctx ​

result_ctx(self) -> Context

The context of the final output or result from applying the sampling strategy.

FUNC result_action ​

result_action(self) -> Component[S]

The action that generated the final output or result from applying the sampling strategy.

FUNC result_validations ​

result_validations(self) -> list[tuple[Requirement, ValidationResult]]

The validation results associated with the final output or result from applying the sampling strategy.

CLASS SamplingStrategy ​

A SamplingStrategy class defines an abstract base class for implementing various sampling strategies.

This class provides a template for creating concrete sampling strategies that can be used to generate model outputs based on given instructions. It allows setting custom validation and generation functions through properties.

Methods:

FUNC sample ​

sample(self, action: Component[S], context: Context, backend: Backend, requirements: list[Requirement] | None) -> SamplingResult[S]

This method is the abstract method for sampling a given component.

It must be implemented by any concrete subclasses to provide specific sampling logic.

Args:

  • action : The action object to be sampled.
  • context: The context to be passed to the sampling strategy.
  • backend: The backend used for generating samples.
  • requirements: List of requirements to test against (merged with global requirements).
  • validation_ctx: Optional context to use for validation. If None, validation_ctx = ctx.
  • format: output format for structured outputs.
  • model_options: model options to pass to the backend during generation / validation.
  • tool_calls: True if tool calls should be used during this sampling strategy.

Returns:

  • SamplingResult[S]: A result object indicating the success or failure of the sampling process.