ModelTRTLLMRuntimeConfiguration

class baseten.client.modelconfig.ModelTRTLLMRuntimeConfiguration(*, kv_cache_free_gpu_mem_fraction=0.9, kv_cache_host_memory_bytes=None, lora_cache_max_adapter_size=None, lora_cache_optimal_adapter_size=None, lora_cache_gpu_memory_fraction=None, lora_cache_host_memory_bytes=None, enable_chunked_context=True, batch_scheduler_policy=ModelTRTLLMBatchSchedulerPolicy.guaranteed_no_evict, request_default_max_tokens=None, served_model_name=None, total_token_limit=500000, webserver_default_route=None, **extra_data)

Bases: BaseModel

Parameters:

• kv_cache_free_gpu_mem_fraction (float)

• kv_cache_host_memory_bytes (KvCacheHostMemoryBytes | None)

• lora_cache_max_adapter_size (LoraCacheMaxAdapterSize | None)

• lora_cache_optimal_adapter_size (LoraCacheOptimalAdapterSize | None)

• lora_cache_gpu_memory_fraction (LoraCacheGpuMemoryFraction | None)

• lora_cache_host_memory_bytes (LoraCacheHostMemoryBytes | None)

• enable_chunked_context (bool)

• batch_scheduler_policy (ModelTRTLLMBatchSchedulerPolicy)

• request_default_max_tokens (RequestDefaultMaxTokens | None)

• served_model_name (str | None)

• total_token_limit (int)

• webserver_default_route (WebserverDefaultRoute | None)

• extra_data (Any)

model_config = {'extra': 'allow'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].