API Reference¶

This is the main orchestrator for auto-tuning the LLM components. As the central class of the Octuner optimization system, it that coordinates the entire parameter optimization workflow. It automatically discovers tunable parameters in complex component hierarchies, builds search spaces, and runs optimization algorithms to find the best configuration.

How it works:

1. Component Discovery: Automatically finds all tunable components in the object hierarchy using ComponentDiscovery, identifying parameters that can be optimized (temperature, model selection, provider choices, etc.)

2. Search Space Construction: Builds a multi-dimensional search space from discovered parameters, defining the optimization landscape with parameter types, ranges, and constraints.

3. Optimization Execution: Runs intelligent search algorithms (Pareto, constrained, or scalarized optimization) to explore the search space and find optimal parameter combinations.

4. Result Analysis: Provides comprehensive results including the best parameters, performance metrics, and detailed trial information for analysis and application.

Key Features:

• Multi-Objective Optimization: Supports Pareto optimization for balancing quality, cost, and latency objectives
• Constraint Handling: Supports hard constraints for real-world deployment requirements
• Scalarization: Converts multi-objective problems to single-objective optimization with custom weights
• Parallel Execution: Supports concurrent trial execution for faster optimization
• Flexible Filtering: Include/exclude specific parameters to focus optimization
• Reproducible Results: Seed support for consistent optimization runs

from octuner import AutoTuner, MultiProviderTunableLLM

# Create a tunable component
llm = MultiProviderTunableLLM(config_file="config.yaml")

# Define evaluation function and dataset
def evaluate(component, input_data):
    result = component.call(input_data["text"])
    return {"quality": compute_quality(result, input_data["target"])}

dataset = [{"text": "Hello", "target": "Hi there"}]

# Create and configure tuner
tuner = AutoTuner(
    component=llm,
    entrypoint=evaluate,
    dataset=dataset,
    metric=lambda output, target: output["quality"]
)

# Focus on specific parameters
tuner.include(["*.temperature", "*.provider_model"])

# Run optimization
result = tuner.search(max_trials=50, mode="pareto")

# Apply the best parameters
from octuner import apply_best
apply_best(llm, result.best_parameters)

Optimization Modes:

- **"pareto"**: Multi-objective optimization finding Pareto-optimal solutions
- **"constrained"**: Single-objective optimization with hard constraints
- **"scalarized"**: Multi-objective converted to single-objective with weights

Initialize the AutoTuner with a component and evaluation setup.

This constructor sets up the AutoTuner with all necessary components for parameter optimization. It validates the inputs and prepares the internal state for discovery and optimization.

Parameters:

# Create tuner using factory method
tuner = AutoTuner.from_component(
    component=my_llm,
    entrypoint=lambda comp, data: comp.call(data["text"]),
    dataset=test_dataset,
    metric=lambda output, target: compute_quality(output, target),
    max_workers=4
)

# Run optimization
result = tuner.search(max_trials=100)

# Focus on core LLM parameters
tuner.include(["*.temperature", "*.max_tokens", "*.provider_model"])

# Focus on specific components
tuner.include(["classifier_llm.*", "confidence_llm.*"])

# Chain with exclude for fine control
tuner.include(["*.temperature"]).exclude(["*.verbose"])

# Exclude debug parameters
tuner.exclude(["*.verbose", "*.debug", "*.log_level"])

# Exclude less important parameters
tuner.exclude(["*.frequency_penalty", "*.presence_penalty"])

# Chain with include for precise control
tuner.include(["*.temperature"]).exclude(["*.verbose"])

# Build search space manually
tuner.build_search_space()

# Inspect discovered parameters
summary = tuner.get_search_space_summary()
print(f"Found {summary['total_parameters']} tunable parameters")
print(f"Parameter types: {summary['parameter_types']}")

# View specific parameters
for param_path, param_def in tuner.search_space.items():
    print(f"{param_path}: {param_def}")

# Basic optimization
result = tuner.search()

# Advanced optimization with constraints
result = tuner.search(
    max_trials=200,
    mode="constrained",
    constraints={"max_cost": 0.01, "max_latency_ms": 500},
    replicates=3,
    timeout=3600,  # 1 hour timeout
    seed=42
)

# Multi-objective optimization with custom weights
result = tuner.search(
    mode="scalarized",
    scalarization_weights={"quality": 0.8, "cost": 0.2},
    max_trials=100
)

# Access results
print(f"Best quality: {result.best_trial.metrics.quality}")
print(f"Best parameters: {result.best_parameters}")

# Build search space and get summary
tuner.build_search_space()
summary = tuner.get_search_space_summary()

print(f"Total parameters: {summary['total_parameters']}")
print(f"Parameter types: {summary['parameter_types']}")
print(f"Components: {summary['components']}")

# Output might be:
# Total parameters: 12
# Parameter types: {'float': 6, 'choice': 4, 'int': 2}
# Components: {'llm': 8, 'classifier_llm': 4}

Bases: TunableMixin

A tunable LLM wrapper that optimizes provider, model, and parameter selection.

This class allows the optimization system to discover the best combination of: - LLM provider (OpenAI, Gemini, etc.) - Model within that provider - Model-specific parameters (temperature, max_tokens, etc.)

Configuration is defined explicitly via YAML files.

Initialize the tunable LLM with explicit configuration.

Parameters:

Mixin class for LLM components that can be auto-tuned.

Components can be made tunable by either: 1. Using instance methods like mark_as_tunable() (legacy approach) 2. Programmatic registration using register_tunable_class() (recommended)

    # Or programmatic registration (recommended)
    from octuner.tunable.registry import register_tunable_class
    register_tunable_class(
        self.__class__,
        params={
            "temperature": ("float", 0.0, 1.0),
            "max_tokens": ("int", 64, 4096),
        },
        call_method="send_prompt"
    )

Initialize the tunable mixin.

Bases: ABC

Abstract base class for implementing custom LLM providers in Octuner.

This class serves as the foundation for creating custom LLM provider implementations, enabling to integrate own self-hosted models, proprietary APIs, or any other LLM service.

Key Features: - Configuration-driven: Integrates with YAML-based configuration system - Parameter optimization: Supports automatic parameter tuning through the config loader - Type conversion: Automatic parameter type conversion based on configuration - Cost tracking: Built-in cost calculation and token usage tracking - Standardized responses: Returns consistent LLMResponse objects across all providers

To create a custom provider, you must:

1. Inherit from BaseLLMProvider and set the provider_name attribute
2. Implement abstract methods:
3. call(): Main interface for making LLM requests
4. _make_request(): Low-level API communication
5. _parse_response(): Convert raw API response to LLMResponse

6. _calculate_cost(): Calculate cost based on token usage

7. Create a configuration file (YAML) defining:

8. Available models and their parameters
9. Parameter types, ranges, and defaults
10. Pricing information for cost calculation
11. Provider-specific settings

Example Usage:

from octuner.providers.base import BaseLLMProvider, LLMResponse
from octuner.config.loader import ConfigLoader

class CustomProvider(BaseLLMProvider):
    def __init__(self, config_loader, **kwargs):
        super().__init__(config_loader, **kwargs)
        self.provider_name = "custom"
        # Initialize your API client here

    def call(self, prompt: str, system_prompt: Optional[str] = None, **kwargs) -> LLMResponse:
        # Implementation details...
        pass

    # Implement other abstract methods...

# Usage with configuration
config_loader = ConfigLoader("my_custom_config.yaml")
provider = CustomProvider(config_loader, api_key="your-key")
response = provider.call("Hello, world!")

Configuration File Structure:

providers:
  custom:
    default_model: "my-model-v1"
    available_models: ["my-model-v1", "my-model-v2"]
    pricing_usd_per_1m_tokens:
      my-model-v1: [0.5, 1.0]  # [input_cost, output_cost]
    model_capabilities:
      my-model-v1:
        supported_parameters: ["temperature", "max_tokens"]
        parameters:
          temperature:
            type: float
            range: [0.0, 2.0]
            default: 0.7
          max_tokens:
            type: int
            range: [1, 4000]
            default: 1000

This constructor sets up the provider with access to the configuration system and stores any provider-specific parameters. The config_loader is mandatory as it provides access to model capabilities, parameter definitions, and pricing.

Parameters:

def call(self, prompt: str, system_prompt: Optional[str] = None, **kwargs) -> LLMResponse:
    import time
    start_time = time.time()

    # Get model and parameters
    model = self._get_parameter("model", kwargs, "default-model")
    temperature = self._get_parameter("temperature", kwargs, model)
    max_tokens = self._get_parameter("max_tokens", kwargs, model)

    # Convert types
    temperature = self._convert_parameter_type("temperature", temperature, model)
    max_tokens = self._convert_parameter_type("max_tokens", max_tokens, model)

    # Prepare API request
    api_params = {
        "model": model,
        "prompt": prompt,
        "system_prompt": system_prompt,
        "temperature": temperature,
        "max_tokens": max_tokens
    }

    # Make API call
    response = self._make_request(**api_params)

    # Parse and return response
    result = self._parse_response(response)
    result.latency_ms = (time.time() - start_time) * 1000
    return result

Standard response format from all LLM providers.

Bases: BaseLLMProvider

OpenAI provider implementation for Octuner.

This module contains the OpenAI provider implementation using the Responses API.

Bases: BaseLLMProvider

Gemini provider implementation. Using the google-generativeai SDK.

Core optimization engine that uses Optuna to find optimal parameter configurations.

LLMOptimizer is the main optimization engine that coordinates the search for optimal parameter values using Optuna's sophisticated optimization algorithms. It integrates with DatasetExecutor to evaluate parameter configurations and uses optimization strategies to determine the best solutions.

The optimization process: 1. Parameter Suggestion: Uses Optuna to suggest parameter values from the search space 2. Trial Execution: Evaluates suggested parameters using DatasetExecutor 3. Objective Computation: Converts evaluation results to objective values using the strategy 4. Optimization: Uses Optuna's TPE sampler to intelligently explore the parameter space 5. Result Analysis: Extracts best parameters and trial results from completed optimization

Key features: - Supports multiple optimization strategies (Pareto, constrained, scalarized) - Intelligent parameter suggestion using TPE (Tree-structured Parzen Estimator) - Robust error handling with fallback objective values for failed trials - Comprehensive trial result tracking and analysis - Integration with Optuna's pruning mechanism for constraint handling

Initialize the LLMOptimizer with search space and optimization configuration.

Parameters:

Bases: ABC

Abstract base class for optimization strategies in Octuner.

This class defines the interface that all optimization strategies must implement to work with the LLMOptimizer. It provides a unified way to handle different optimization approaches while maintaining compatibility with Optuna.

Each strategy must implement methods for creating Optuna studies, computing objective values from metric results, and determining the best trial from completed studies.

DatasetExecutor is a core component of the optimization system that handles the execution of evaluation trials during parameter optimization. It manages the evaluation of components over datasets, collects performance metrics, and provides both sequential and parallel execution capabilities.

How it works:

1. Parameter Application: Applies trial parameters to the component using the parameter setter utilities, ensuring consistent parameter configuration across trials.

2. Dataset Evaluation: Executes the entrypoint function over each dataset item, collecting outputs and computing quality scores using the provided metric function.

3. Metrics Collection: Automatically collects comprehensive metrics including quality scores, execution costs, and latency measurements for each trial.

4. Parallel Execution: Supports concurrent evaluation of dataset items using ThreadPoolExecutor for faster trial execution when max_workers > 1.

5. Statistical Aggregation: Provides robust statistical aggregation across replicates and dataset items using median-based aggregation for stability.

Key Features:

• Parallel Execution: Multi-threaded evaluation for faster optimization
• Comprehensive Metrics: Quality, cost, and latency tracking
• Statistical Robustness: Median-based aggregation to handle outliers
• Error Handling: Graceful handling of individual item failures
• Replicate Support: Multiple trial runs for statistical significance
• Cost Tracking: Automatic cost collection from tunable components

Constructor

Parameters:

This is a core component of Octuner that automatically finds and catalogs all tunable parameters within an object hierarchy. It recursively traverses object attributes to identify components that implement the TunableMixin protocol, building a search space that can be optimized by the AutoTuner.

How it works: 1. Recursive Traversal: Starting from a root component, it recursively explores all object attributes using __dict__ introspection, avoiding circular references and method calls.

1. Tunable Detection: For each object found, it checks if it implements the TunableMixin protocol using is_llm_tunable(), which supports both: - Instance-based tunables (legacy): Objects with get_tunable_parameters() method - Registry-based tunables (recommended): Objects registered through register_tunable_class()

2. Parameter Extraction: For tunable objects, it extracts parameter definitions including type, range, and default values using get_tunable_parameters().

3. Path Mapping: Each discovered parameter is mapped to a dotted path (e.g., "classifier_llm.temperature") that uniquely identifies its location in the hierarchy.

4. Filtering: Optional include/exclude patterns can be used to focus the search space on specific parameters or components.

from octuner import ComponentDiscovery, MultiProviderTunableLLM

# Create a complex component hierarchy
analyzer = TunableSentimentAnalyzer(config_file)

# Discover all tunable parameters
discovery = ComponentDiscovery()
tunables = discovery.discover(analyzer)

# Result: {
#     "classifier_llm": {
#         "temperature": ("float", 0.0, 2.0),
#         "max_tokens": ("int", 64, 4096),
#         "provider_model": ("choice", ["openai:gpt-4", "gemini:gemini-pro"])
#     },
#     "confidence_llm": { ... },
#     "reasoning_llm": { ... }
# }

# Focus on specific parameters
focused_discovery = ComponentDiscovery(
    include_patterns=["*.temperature", "*.provider_model"],
    exclude_patterns=["*.verbose"]
)
focused_tunables = focused_discovery.discover(analyzer)

Attributes:

Initialize discovery with optional filters.

Parameters:

This class provides utilities to load YAML configuration files as described in config_templates/*.yaml.

It allows to get available providers, models, parameters, pricing, and capabilities. Those capabilities become available to the tuning algorithms to know what parameters can be optimized, their ranges, types, and default values.

IMPORTANT: Note that when instantiating this class, the configuration file is loaded immediately.

Initialize config loader with a specific file.

Parameters:

Result of a single metric evaluation.

Result of a single optimization trial.

Result of an optimization search.

Weights for scalarized optimization mode.