Skip to main content
Utilities for detecting and classifying the target inference server. Defines the _ServerType enum (LOCALHOST, OPENAI, REMOTE_VLLM, UNKNOWN) and _server_type, which classifies a URL by hostname. Also provides is_vllm_server_with_structured_output, which probes a server’s /version endpoint to determine whether it supports the structured_outputs parameter introduced in vLLM ≥ 0.12.0. Used by the OpenAI-compatible backend to choose between guided_json and structured_outputs request formats.

​
Functions

​
FUNC is_vllm_server_with_structured_output 

is_vllm_server_with_structured_output(base_url: str, headers: Mapping[str, Any]) -> bool
Attempts to determine if the backend is a vllm server with version >= v0.12.0. Defaults to false. v0.12.0 was the last version to support guided_json params. It’s now under structured_outputs. Args:
  • base_url: Base url for LLM API.
  • headers: Additional headers to pass to the request.
Returns:
  • True if the server is vLLM >= v0.12.0, False otherwise.