Skip to main content

mellea.core.backend

Abstract Backend interface and generation-walk utilities.

Defines the Backend abstract base class whose two key abstract methods — generate_from_context (context-aware single-action generation) and generate_from_raw (context-free batch generation) — all concrete backends must implement. Also provides generate_walk, which traverses a Component tree to find un-computed ModelOutputThunk leaves that need to be resolved before rendering.

Functions​

FUNC generate_walk ​

generate_walk(c: CBlock | Component | ModelOutputThunk) -> list[ModelOutputThunk]

Return all uncomputed ModelOutputThunk leaves reachable from c.

Args:

Returns:

  • A flat list of uncomputed ModelOutputThunk instances in the order
  • they need to be resolved (depth-first over Component.parts()).

Raises:

Classes​

CLASS Backend ​

Abstract base class for all inference backends.

All concrete backends must implement generate_from_context (context-aware single-action generation) and generate_from_raw (context-free batch generation). The do_generate_walk / do_generate_walks helpers can be used to pre-compute any unresolved ModelOutputThunk leaves before rendering.

Methods:

FUNC generate_from_context ​

generate_from_context(self, action: Component[C] | CBlock, ctx: Context) -> tuple[ModelOutputThunk[C], Context]

Generates a model output from a context. May not mutate the context. This must be called from a running event loop as it creates a task to run the generation request.

Args:

  • action: The last item of the context should be passed in as an action instead of as part of the ctx. See docs/dev/generate_signature_decisions.md.
  • ctx: The rest of the context.
  • format: A response format to used for structured outputs / constrained decoding.
  • model_options: Any model options to upsert into the defaults for this call.
  • tool_calls: If True, then tool calls are extracts from the action Component. Assumption: if tool_calls is enabled, then the action Component has a TemplateRepresentation

Returns:

  • a tuple of (ModelOutputThunk, Context) where the Context is the new context after the generation has been completed.

FUNC generate_from_raw ​

generate_from_raw(self, actions: list[Component[C]], ctx: Context) -> list[ModelOutputThunk[C]]

FUNC generate_from_raw ​

generate_from_raw(self, actions: list[Component[C] | CBlock], ctx: Context) -> list[ModelOutputThunk[C | str]]

FUNC generate_from_raw ​

generate_from_raw(self, actions: Sequence[Component[C] | CBlock], ctx: Context) -> list[ModelOutputThunk]

Generates a model output from the provided input. Does not use context or templates.

Args:

  • actions: list of actions to generate responses for. Each action is separate.
  • ctx: context passed to generation. Currently not used in generate_from_raw
  • format: A response format to used for structured outputs / constrained decoding. Note: some backends do not support this parameter. They will log warnings and continue to generate.
  • model_options: Any model options to upsert into the defaults for this call.
  • tool_calls: Always set to false unless supported by backend.

Returns:

  • list[ModelOutputThunk]: A list of output thunks, one per action, in the same order as actions.

FUNC do_generate_walk ​

do_generate_walk(self, action: CBlock | Component | ModelOutputThunk) -> None

Awaits all uncomputed ModelOutputThunk leaves reachable from action.

Traverses the component tree rooted at action via generate_walk, collects any uncomputed ModelOutputThunk nodes, and concurrently awaits them all.

Args:

  • action: The root node to traverse.

FUNC do_generate_walks ​

do_generate_walks(self, actions: list[CBlock | Component | ModelOutputThunk]) -> None

Awaits all uncomputed ModelOutputThunk leaves reachable from each action in actions.

Traverses the component tree of every action in the list via generate_walk, collects all uncomputed ModelOutputThunk nodes across all actions, and concurrently awaits them.

Args:

  • actions: The list of root nodes to traverse.