ModelTRTLLMBuildConfiguration

class baseten.client.modelconfig.ModelTRTLLMBuildConfiguration(*, base_model=ModelTRTLLMModel.decoder, max_seq_len=None, max_batch_size=256, max_num_tokens=8192, max_beam_width=1, max_prompt_embedding_table_size=0, checkpoint_repository=None, gather_all_token_logits=False, strongly_typed=False, quantization_type=ModelTRTLLMQuantizationType.no_quant, quantization_config=ModelTRTQuantizationConfiguration(calib_size=1024, calib_dataset='abisee/cnn_dailymail', calib_max_seq_length=1536), tensor_parallel_count=1, pipeline_parallel_count=1, moe_expert_parallel_option=-1, sequence_parallel_count=1, plugin_configuration=ModelTRTLLMPluginConfiguration(paged_kv_cache=True, use_paged_context_fmha=True, use_fp8_context_fmha=False), num_builder_gpus=None, speculator=None, lora_adapters=None, lora_configuration=None, skip_build_result=False, **extra_data)

Bases: BaseModel

Parameters:

• base_model (ModelTRTLLMModel)

• max_seq_len (MaxSeqLen | None)

• max_batch_size (Annotated[int, Ge(ge=1), Le(le=2048)])

• max_num_tokens (Annotated[int, Gt(gt=64), Le(le=1048576)])

• max_beam_width (Annotated[int, Ge(ge=1), Le(le=1)])

• max_prompt_embedding_table_size (int)

• checkpoint_repository (CheckpointRepository | None)

• gather_all_token_logits (bool)

• strongly_typed (bool)

• quantization_type (ModelTRTLLMQuantizationType)

• quantization_config (ModelTRTQuantizationConfiguration)

• tensor_parallel_count (Annotated[int, Ge(ge=1)])

• pipeline_parallel_count (int)

• moe_expert_parallel_option (int)

• sequence_parallel_count (int)

• plugin_configuration (ModelTRTLLMPluginConfiguration)

• num_builder_gpus (NumBuilderGpus | None)

• speculator (ModelSpeculatorConfiguration | None)

• lora_adapters (dict[Annotated[str, FieldInfo(annotation=NoneType, required=True, metadata=[_PydanticGeneralMetadata(pattern='^[a-zA-Z0-9_\\-\\.:]+$')])], CheckpointRepository] | None)

• lora_configuration (ModelTRTLLMLoraConfiguration | None)

• skip_build_result (bool)

• extra_data (Any)

model_config = {'extra': 'allow'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].