Skip to main content

mellea.telemetry.metrics

OpenTelemetry metrics instrumentation for Mellea.

Provides metrics collection using OpenTelemetry Metrics API with support for:

  • Counters: Monotonically increasing values (e.g., request counts, token usage)
  • Histograms: Value distributions (e.g., latency, token counts)
  • UpDownCounters: Values that can increase or decrease (e.g., active sessions)

Metrics Exporters:

  • Console: Print metrics to console for debugging
  • OTLP: Export to OpenTelemetry Protocol collectors (Jaeger, Grafana, etc.)
  • Prometheus: Register metrics with prometheus_client registry for scraping

Configuration via environment variables:

General:

  • MELLEA_METRICS_ENABLED: Enable/disable metrics collection (default: false)
  • OTEL_SERVICE_NAME: Service name for metrics (default: mellea)

Console Exporter (debugging):

  • MELLEA_METRICS_CONSOLE: Print metrics to console (default: false)

OTLP Exporter (production observability):

  • MELLEA_METRICS_OTLP: Enable OTLP metrics exporter (default: false)
  • OTEL_EXPORTER_OTLP_ENDPOINT: OTLP endpoint for all signals (optional)
  • OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: Metrics-specific endpoint (optional, overrides general)
  • OTEL_METRIC_EXPORT_INTERVAL: Export interval in milliseconds (default: 60000)

Prometheus Exporter:

  • MELLEA_METRICS_PROMETHEUS: Enable Prometheus metric reader (default: false)

Pricing (for cost counter):

  • MELLEA_PRICING_FILE: Path to a JSON file with custom model pricing overrides (optional)

Multiple exporters can be enabled simultaneously.

Example - Console debugging: export MELLEA_METRICS_ENABLED=true export MELLEA_METRICS_CONSOLE=true

Example - OTLP production: export MELLEA_METRICS_ENABLED=true export MELLEA_METRICS_OTLP=true export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

Example - Prometheus monitoring: export MELLEA_METRICS_ENABLED=true export MELLEA_METRICS_PROMETHEUS=true

Example - Multiple exporters: export MELLEA_METRICS_ENABLED=true export MELLEA_METRICS_CONSOLE=true export MELLEA_METRICS_OTLP=true export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 export MELLEA_METRICS_PROMETHEUS=true

Built-in metrics (auto-recorded via plugins when metrics are enabled):

  • Token counters: mellea.llm.tokens.input, mellea.llm.tokens.output (unit: tokens)
  • Latency histograms: mellea.llm.request.duration (unit: s), mellea.llm.ttfb (unit: s, streaming only)
  • Error counter: mellea.llm.errors (unit: {error}), categorized by semantic error type
  • Cost counter: mellea.llm.cost.usd (unit: USD), estimated cost when pricing data is available
  • Sampling counters: mellea.sampling.attempts, mellea.sampling.successes, mellea.sampling.failures (unit: {attempt}/{sample}/{failure})
  • Requirement counters: mellea.requirement.checks (unit: {check}), mellea.requirement.failures (unit: {failure})
  • Tool counter: mellea.tool.calls (unit: {call}), tagged by tool name and status

Programmatic usage: from mellea.telemetry.metrics import create_counter, create_histogram

request_counter = create_counter( "mellea.requests", description="Total number of LLM requests", unit="1" ) request_counter.add(1, {"backend": "ollama", "model": "llama2"})

latency_histogram = create_histogram( "mellea.request.duration", description="Request latency distribution", unit="s" ) latency_histogram.record(1.5, {"backend": "ollama"})

Functions​

FUNC is_metrics_enabled ​

is_metrics_enabled() -> bool

Check if metrics collection is enabled.

Returns:

  • True if MELLEA_METRICS_ENABLED is truthy AND OpenTelemetry is installed.

FUNC create_counter ​

create_counter(name: str, description: str = '', unit: str = '1') -> Any

Create a counter instrument for monotonically increasing values.

Counters are used for values that only increase, such as:

  • Total number of requests
  • Total tokens processed
  • Total errors encountered

Args:

  • name: Metric name (e.g., "mellea.requests.total")
  • description: Human-readable description of what this metric measures
  • unit: Unit of measurement (e.g., "1" for count, "ms" for milliseconds)

Returns:

  • Counter instrument (or no-op if metrics disabled)

FUNC create_histogram ​

create_histogram(name: str, description: str = '', unit: str = '1') -> Any

Create a histogram instrument for recording value distributions.

Histograms are used for values that vary and need statistical analysis:

  • Request latency
  • Token counts per request
  • Response sizes

Args:

  • name: Metric name (e.g., "mellea.request.duration")
  • description: Human-readable description
  • unit: Unit of measurement (e.g., "ms", "tokens", "bytes")

Returns:

  • Histogram instrument (or no-op if metrics disabled)

FUNC create_up_down_counter ​

create_up_down_counter(name: str, description: str = '', unit: str = '1') -> Any

Create an up-down counter for values that can increase or decrease.

UpDownCounters are used for values that go up and down:

  • Active sessions
  • Items in a queue
  • Memory usage

Args:

  • name: Metric name (e.g., "mellea.sessions.active")
  • description: Human-readable description
  • unit: Unit of measurement

Returns:

  • UpDownCounter instrument (or no-op if metrics disabled)

FUNC record_token_usage_metrics ​

record_token_usage_metrics(input_tokens: int | None, output_tokens: int | None, model: str, provider: str) -> None

Record token usage metrics following OpenTelemetry Gen-AI semantic conventions.

This is a no-op when metrics are disabled, ensuring zero overhead.

Args:

  • input_tokens: Number of input tokens (prompt tokens), or None if unavailable
  • output_tokens: Number of output tokens (completion tokens), or None if unavailable
  • model: Model identifier (e.g., "gpt-4", "llama2:7b")
  • provider: Provider name (e.g., "openai", "ollama", "watsonx")

FUNC record_request_duration ​

record_request_duration(duration_s: float, model: str, provider: str, streaming: bool = False) -> None

Record total LLM request duration.

This is a no-op when metrics are disabled, ensuring zero overhead.

Args:

  • duration_s: Request duration in seconds
  • model: Model identifier (e.g., "gpt-4", "llama2:7b")
  • provider: Provider name (e.g., "openai", "ollama", "watsonx")
  • streaming: Whether the request used streaming mode

FUNC record_ttfb ​

record_ttfb(ttfb_s: float, model: str, provider: str) -> None

Record time-to-first-token for streaming LLM requests.

This is a no-op when metrics are disabled, ensuring zero overhead. Should only be called for streaming requests.

Args:

  • ttfb_s: Time to first token in seconds
  • model: Model identifier (e.g., "gpt-4", "llama2:7b")
  • provider: Provider name (e.g., "openai", "ollama", "watsonx")

FUNC classify_error ​

classify_error(exc: BaseException) -> str

Map an exception to a semantic error type string.

Checks OpenAI SDK exception types first (when openai is installed), then falls back to stdlib exceptions and name-based heuristics.

Args:

  • exc: The exception to classify.

Returns:

  • One of the ERROR_TYPE_* constants.

FUNC record_error ​

record_error(error_type: str, model: str, provider: str, exception_class: str) -> None

Record an LLM error metric.

This is a no-op when metrics are disabled, ensuring zero overhead.

Args:

  • error_type: Semantic error category (use ERROR_TYPE_* constants).
  • model: Model identifier (e.g. "gpt-4", "llama2:7b").
  • provider: Provider name (e.g. "openai", "ollama").
  • exception_class: Python exception class name (e.g. "RateLimitError").

FUNC record_cost ​

record_cost(cost: float, model: str, provider: str) -> None

Record estimated LLM request cost in USD.

This is a no-op when metrics are disabled, ensuring zero overhead. Only call this when pricing data is available (i.e., compute_cost returned a non-None value).

Args:

  • cost: Estimated request cost in US dollars.
  • model: Model identifier (e.g. "gpt-4o", "claude-sonnet-4-6").
  • provider: Provider name (e.g. "openai", "ollama").

FUNC record_sampling_attempt ​

record_sampling_attempt(strategy: str) -> None

Record one sampling attempt for the given strategy.

This is a no-op when metrics are disabled, ensuring zero overhead.

Args:

  • strategy: Sampling strategy class name (e.g. "RejectionSamplingStrategy").

FUNC record_sampling_outcome ​

record_sampling_outcome(strategy: str, success: bool) -> None

Record the final outcome (success or failure) of a sampling loop.

This is a no-op when metrics are disabled, ensuring zero overhead.

Args:

  • strategy: Sampling strategy class name (e.g. "RejectionSamplingStrategy").
  • success: True if at least one attempt passed all requirements.

FUNC record_requirement_check ​

record_requirement_check(requirement: str) -> None

Record one requirement validation check.

This is a no-op when metrics are disabled, ensuring zero overhead.

Args:

  • requirement: Requirement class name (e.g. "LLMaJRequirement").

FUNC record_requirement_failure ​

record_requirement_failure(requirement: str, reason: str) -> None

Record one requirement validation failure.

This is a no-op when metrics are disabled, ensuring zero overhead.

Args:

  • requirement: Requirement class name (e.g. "LLMaJRequirement").
  • reason: Human-readable failure reason from ValidationResult.reason.

FUNC record_tool_call ​

record_tool_call(tool: str, status: str) -> None

Record one tool invocation.

This is a no-op when metrics are disabled, ensuring zero overhead.

Args:

  • tool: Name of the tool that was invoked.
  • status: "success" if the tool executed without error, "failure" otherwise.