Inference Service Model Property

The model property, specified in the service configuration file, holds the list of models that need to be loaded by the container. When the property is not provided, only pre-approved models that are included with the container will be available.

The property is a JSON Array of JSON objects, where each object describes a model. The properties of the model object in the configuration file determine the location of the model along with any metadata that should be used to load it.

The following model object properties are supported by the inference services:

Property Default Value Runtime Compatibility Details
name None ONNX, vLLM, llama.cpp This required property is used to provide the name of the model.

This name must be unique across the other models in the configuration file. If the name is not unique, the container will fail with an error at startup.
path None ONNX, vLLM, llama.cpp This require property provides the file location from which to load models.

This can be a .zip file, a .onnx file (only for ONNX-format models), or a directory (only for LLMs and vLLM-compatible models). The file location is assumed to be relative to /privateai/models in the container, so a simple file name will suffice for loading models from that location.

GGUF and remote HuggingFace refs, such as google/gemma-3-1b-it are also supported.
function None ONNX Optionally provide the function for the model, such as EMBEDDING.
cache_on_startup False ONNX An optional flag to indicate whether the model should be loaded and cached during the startup of the container.

This is only valid for the ONNX runtime engine, as caching is handled differently for vLLM. When this flag is false, the model will be validated, but only loaded and cached upon the first request.
runtime onnx ONNX, vLLM, llama.cpp Optionally provide the runtime engine that should be used to load and execute this model.

The model must be compatible with this runtime or an error is thrown at startup. If not provided, the ONNX runtime engine is used by default.
properties Values specified in the Environment property or an empty list ONNX, vLLM, llama.cpp Optionally provide properties specific to the runtime engine of the model.

When present, these properties override any global property set in the Environment property.
chat_template None vLLM, llama.cpp Optionally provide the location of a chat template file.

vLLM and llama.cpp support jinja2 for the template format. This property is only used by the vLLM runtime engine and will be ignored by the ONNX runtime engine.

Models typically include a chat template in their tokenizer_config.json file, which is part of the model directory, so this parameter is optional and only required by select models. Models that do not have a template in the tokenizer_config.json file and do not have the template property specified in config.json will cause an error to be thrown at startup.
capabilities None vLLM, llama.cpp Because neither vLLM nor llama.cpp expose REST APIs for model metadata, this property must be provided (when using the vLLM or llama.cpp runtime engines) to specify the capabilities of a given model. Valid values can be found at Private Large Language Model Service Runtime Engines.
context_size None vLLM, llama.cpp Optionally set a runtime-independent context length for the model.

The service resolves this property to the appropriate runtime-specific argument. For vLLM, the resolved value maps to --max-model-len. For llama.cpp, the resolved value maps to --ctx-size or -c.
kv_cache 0.6 vLLM Optionally provide a runtime fraction of backend memory that the vLLM runtime can use when budgeting memory for model execution and KV cache. The input is a numeric value between 0 and 1.

For GPU backends, this is the fraction of GPU memory available to the vLLM worker. For CPU backends, this is the fraction of selected CPU NUMA-node memory available to the vLLM worker. The service resolves this property to the runtime argument --gpu-memory-utilization
model_template None vLLM, llama.cpp Optionally provide a model template that contains pre-filled model properties.

These templates are useful for cases when specific models require specific runtime arguments to the runtime or in the case of complex setup.
runtime_arguments None vLLM, llama.cpp Optionally provide a list of arguments that will be passed to the engine process at startup.

Each argument must be specified as a separate entry in the array along with its optional value. In general, runtime_arguments will override configuration file properties if both are defined for the same arguments.
guardrails None vLLM, llama.cpp Optionally provide an object that configures the built-in guardrail. This property is only available if the model is used with vLLM and llama.cpp and also has the TEXT_GENERATION capability.

The context_size property provides a unified way to configure model context length across runtime engines.

The Private Large Language Model Service supports the following input forms for context_size:

When context_size is omitted, the service leaves it unset so that the backend runtime can apply its own default behavior. For explicit numeric values, the service validates the value against model architecture metadata and then uses to the model-advertised maximum context length when needed.

The model_template property currently supports the following model templates. For each model, properties are pre-filled, but can be overridden if you want to define your own runtime arguments.

Name Model Capabilities
mxbai-rerank mxbai-rerank-base-v2 TEXT_RERANK
Qwen3-Reranker Qwen3-Reranker-0.6B TEXT_RERANK
bge-reranker-v2-gemma bge-reranker-v2-gemma TEXT_RERANK
bge-m3 bge-m3 TEXT_EMBEDDINGS
Ministral-3-Reasoning Ministral-3-3B-Reasoning-2512 TEXT_GENERATION