Inference Service Model Property

The model property, specified in the service configuration file, holds the list of models that need to be loaded by the container. When the property is not provided, only pre-approved models that are included with the container will be available.

The property is a JSON Array of JSON objects, where each object describes a model. The properties of the model object in the configuration file determine the location of the model along with any metadata that should be used to load it.

The following model object properties are supported by the inference services:

Property	Default Value	Runtime Compatibility	Details
name	None	ONNX, vLLM, llama.cpp	This required property is used to provide the name of the model. This name must be unique across the other models in the configuration file. If the name is not unique, the container will fail with an error at startup.
path	None	ONNX, vLLM, llama.cpp	This require property provides the file location from which to load models. This can be a .zip file, a .onnx file (only for ONNX-format models), or a directory (only for LLMs and vLLM-compatible models). The file location is assumed to be relative to /privateai/models in the container, so a simple file name will suffice for loading models from that location. GGUF and remote HuggingFace refs, such as google/gemma-3-1b-it are also supported.
function	None	ONNX	Optionally provide the function for the model, such as EMBEDDING.
cache_on_startup	False	ONNX	An optional flag to indicate whether the model should be loaded and cached during the startup of the container. This is only valid for the ONNX runtime engine, as caching is handled differently for vLLM. When this flag is false, the model will be validated, but only loaded and cached upon the first request.
runtime	onnx	ONNX, vLLM, llama.cpp	Optionally provide the runtime engine that should be used to load and execute this model. The model must be compatible with this runtime or an error is thrown at startup. If not provided, the ONNX runtime engine is used by default.
properties	Values specified in the Environment property or an empty list	ONNX, vLLM, llama.cpp	Optionally provide properties specific to the runtime engine of the model. When present, these properties override any global property set in the Environment property.
chat_template	None	vLLM, llama.cpp	Optionally provide the location of a chat template file. vLLM and llama.cpp support jinja2 for the template format. This property is only used by the vLLM runtime engine and will be ignored by the ONNX runtime engine. Models typically include a chat template in their tokenizer_config.json file, which is part of the model directory, so this parameter is optional and only required by select models. Models that do not have a template in the tokenizer_config.json file and do not have the template property specified in config.json will cause an error to be thrown at startup.
capabilities	None	vLLM, llama.cpp	Because neither vLLM nor llama.cpp expose REST APIs for model metadata, this property must be provided (when using the vLLM or llama.cpp runtime engines) to specify the capabilities of a given model. Valid values can be found at Private Large Language Model Service Runtime Engines.
context_size	None	vLLM, llama.cpp	Optionally set a runtime-independent context length for the model. The service resolves this property to the appropriate runtime-specific argument. For vLLM, the resolved value maps to --max-model-len. For llama.cpp, the resolved value maps to --ctx-size or -c.
kv_cache	0.6	vLLM	Optionally provide a runtime fraction of backend memory that the vLLM runtime can use when budgeting memory for model execution and KV cache. The input is a numeric value between 0 and 1. For GPU backends, this is the fraction of GPU memory available to the vLLM worker. For CPU backends, this is the fraction of selected CPU NUMA-node memory available to the vLLM worker. The service resolves this property to the runtime argument --gpu-memory-utilization.
model_template	None	vLLM, llama.cpp	Optionally provide a model template that contains pre-filled model properties. These templates are useful for cases when specific models require specific runtime arguments to the runtime or in the case of complex setup.
runtime_arguments	None	vLLM, llama.cpp	Optionally provide a list of arguments that will be passed to the engine process at startup. Each argument must be specified as a separate entry in the array along with its optional value. In general, runtime_arguments will override configuration file properties if both are defined for the same arguments.
guardrails	None	vLLM, llama.cpp	Optionally provide an object that configures the built-in guardrail. This property is only available if the model is used with vLLM and llama.cpp and also has the TEXT_GENERATION capability.

The context_size property provides a unified way to configure model context length across runtime engines.

The Private Large Language Model Service supports the following input forms for context_size:

• Numeric value: a positive value interpreted as a token count
• Numeric string such as "32768": interpreted as a token count
• String with k suffix, such as "32k", "2m", or "1g": interpreted as thousands, millions, or billions of tokens, respectively
• "auto": preserves the runtime's native automatic behavior.

When context_size is omitted, the service leaves it unset so that the backend runtime can apply its own default behavior. For explicit numeric values, the service validates the value against model architecture metadata and then uses to the model-advertised maximum context length when needed.

The model_template property currently supports the following model templates. For each model, properties are pre-filled, but can be overridden if you want to define your own runtime arguments.

Name	Model	Capabilities
mxbai-rerank	mxbai-rerank-base-v2	TEXT_RERANK
Qwen3-Reranker	Qwen3-Reranker-0.6B	TEXT_RERANK
bge-reranker-v2-gemma	bge-reranker-v2-gemma	TEXT_RERANK
bge-m3	bge-m3	TEXT_EMBEDDINGS
Ministral-3-Reasoning	Ministral-3-3B-Reasoning-2512	TEXT_GENERATION