ModelSpeculatorConfiguration

class baseten.client.modelconfig.ModelSpeculatorConfiguration(*, speculative_decoding_mode='DRAFT_TOKENS_EXTERNAL', num_draft_tokens=None, checkpoint_repository=None, runtime=ModelTRTLLMRuntimeConfiguration(kv_cache_free_gpu_mem_fraction=0.9, kv_cache_host_memory_bytes=None, lora_cache_max_adapter_size=None, lora_cache_optimal_adapter_size=None, lora_cache_gpu_memory_fraction=None, lora_cache_host_memory_bytes=None, enable_chunked_context=True, batch_scheduler_policy='guaranteed_no_evict', request_default_max_tokens=None, served_model_name=None, total_token_limit=500000, webserver_default_route=None), build=None, lookahead_windows_size=None, lookahead_ngram_size=None, lookahead_verification_set_size=None, enable_b10_lookahead=False, **extra_data)

Bases: BaseModel

Parameters:

• speculative_decoding_mode (Literal['DRAFT_TOKENS_EXTERNAL', 'LOOKAHEAD_DECODING'] | str)

• num_draft_tokens (NumDraftTokens | None)

• checkpoint_repository (CheckpointRepository | None)

• runtime (ModelTRTLLMRuntimeConfiguration)

• build (ModelTRTLLMBuildConfiguration | None)

• lookahead_windows_size (LookaheadWindowsSize | None)

• lookahead_ngram_size (LookaheadNgramSize | None)

• lookahead_verification_set_size (LookaheadVerificationSetSize | None)

• enable_b10_lookahead (bool | None)

• extra_data (Any)

model_config = {'extra': 'allow'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].