ModelSpeculatorConfiguration

class baseten.client.modelconfig.ModelSpeculatorConfiguration(*, speculative_decoding_mode=ModelSpecDecMode.DRAFT_TOKENS_EXTERNAL, num_draft_tokens=None, checkpoint_repository=None, runtime=ModelTRTLLMRuntimeConfiguration(kv_cache_free_gpu_mem_fraction=0.9, kv_cache_host_memory_bytes=None, lora_cache_max_adapter_size=None, lora_cache_optimal_adapter_size=None, lora_cache_gpu_memory_fraction=None, lora_cache_host_memory_bytes=None, enable_chunked_context=True, batch_scheduler_policy=<ModelTRTLLMBatchSchedulerPolicy.guaranteed_no_evict: 'guaranteed_no_evict'>, request_default_max_tokens=None, served_model_name=None, total_token_limit=500000, webserver_default_route=None), build=None, lookahead_windows_size=None, lookahead_ngram_size=None, lookahead_verification_set_size=None, enable_b10_lookahead=False, **extra_data)

Bases: BaseModel

Parameters:

• speculative_decoding_mode (ModelSpecDecMode)

• num_draft_tokens (Annotated[NumDraftTokens | None, FieldInfo(annotation=NoneType, required=True, title='Num Draft Tokens')])

• checkpoint_repository (CheckpointRepository | None)

• runtime (Annotated[ModelTRTLLMRuntimeConfiguration, FieldInfo(annotation=NoneType, required=True, validate_default=True)])

• build (ModelTRTLLMBuildConfiguration | None)

• lookahead_windows_size (Annotated[LookaheadWindowsSize | None, FieldInfo(annotation=NoneType, required=True, title='Lookahead Windows Size')])

• lookahead_ngram_size (Annotated[LookaheadNgramSize | None, FieldInfo(annotation=NoneType, required=True, title='Lookahead Ngram Size')])

• lookahead_verification_set_size (Annotated[LookaheadVerificationSetSize | None, FieldInfo(annotation=NoneType, required=True, title='Lookahead Verification Set Size')])

• enable_b10_lookahead (Annotated[bool | None, FieldInfo(annotation=NoneType, required=True, title='Enable B10 Lookahead')])

• extra_data (Any)

model_config = {'extra': 'allow'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].