Private Large Language Model Service Runtime Engines

The Private Large Language Model Service employs runtime engines, which are responsible for loading model artifacts and exposing the model's supported capabilities for inference.

Models are associated with a runtime engine through the service configuration file, config.json by default.

ONNX Runtime Engine

Use the ONNX runtime engine for embedding generation, reranking, and text classification using ONNX-format models, which must be produced using the OML4Py 2.1.1 client tool. The tool packages the model with the required tokenizer and post-processing components expected by the service. If no runtime engine is specified in the configuration file, the ONNX runtime engine is used by default.

vLLM Runtime Engine

The vLLM Runtime Engine starts one dedicated vLLM server instance per configured vLLM-backed model. A single vLLM-backed model is supported per container instance.

Note:

The Private AI Services Container version 26.2.1.0.0 ships with vLLM 0.21.0.

You must provide metadata in the config.json file to indicate a model's capabilities. The property, capabilities, is used to specify the service that the model provides, for example TEXT_GENERATION or TEXT_EMBEDDING. For vLLM, exactly one capability must be specified in the format of a list. Additionally, the vLLM service derives a default --convert argument from the declared capability, which is passed to vLLM on startup.

The following are supported values for the capabilities property:

Capability	Goal	--convert Argument Value
TEXT_GENERATION	The model is capable of generating text. The model takes text as input and produces text as output. For example, LLMs.	auto
TEXT_RERANK	Rerank text related to a query. The model takes a single text input (query) and a list of texts (documents), and returns the rank for each document based on its relevance to the query. For example, reranker models.	classify
TEXT_EMBEDDINGS	The model is capable of generating embeddings for textual input.	embed
TEXT_CLASSICATION	The model is capable of classifying text. The model takes text as input and produces a sequence of labels and probabilities as output. For example, models that perform sentiment analysis.	classify

Llama.cpp Runtime Engine

The llama.cpp runtime engine starts one dedicated llama-server instance per configured llama.cpp-backed model. Llama.cpp models are loaded from GGUF files, which tend to be memory efficient. Still, a single llama.cpp-backed model is supported per container instance.

Note:

The Private AI Services Container version 26.2.1.0.0 ships with llama.cpp b9370.

Similar to the vLLM runtime engine, the llama.cpp runtime engine uses the capabilities property to specify what a given llama.cpp-backed model is capable of doing. The llama.cpp runtime engine allows only a single capability to be listed. The following are valid capabilities values:

Capability	Goal
TEXT_GENERATION	The model is capable of generating text. The model takes text as input and produces text as output. For example, LLMs.
TEXT_RERANK	Rerank text related to a query. The model takes a single text input (query) and a list of texts (documents), and returns the rank for each document based on its relevance to the query. For example, reranker models.
TEXT_EMBEDDINGS	The model is capable of generating embeddings for textual input.