ModelSpeculatorConfiguration

class baseten.client.modelconfig.ModelSpeculatorConfiguration(*, speculative_decoding_mode=ModelSpecDecMode.DRAFT_TOKENS_EXTERNAL, num_draft_tokens=None, checkpoint_repository=None, runtime=<factory>, build=None, lookahead_windows_size=None, lookahead_ngram_size=None, lookahead_verification_set_size=None, enable_b10_lookahead=False, **extra_data)

Bases: BaseModel

Parameters:

• speculative_decoding_mode (ModelSpecDecMode | None)

• num_draft_tokens (NumDraftTokens | None)

• checkpoint_repository (CheckpointRepository | None)

• runtime (ModelTRTLLMRuntimeConfiguration | None)

• build (ModelTRTLLMBuildConfiguration | None)

• lookahead_windows_size (LookaheadWindowsSize | None)

• lookahead_ngram_size (LookaheadNgramSize | None)

• lookahead_verification_set_size (LookaheadVerificationSetSize | None)

• enable_b10_lookahead (bool | None)

• extra_data (Any)

model_config = {'extra': 'allow'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].