6 Use the Private Large Language Model Service

The Private AI Services Container Private Large Language Model Service enables Chat, RAG, and agentic workloads with local LLMs.

The Private Large Language Model Service supports LLM inference, in addition to the existing ONNX-format model support provided by the Private AI embedding service. ONNX models are loaded and executed using the ONNX Runtime framework, while LLM inference is handled through either a vLLM server or llama.cpp server.

The vLLM Runtime Engine starts one dedicated vLLM server instance per configured vLLM-backed model. Each vLLM server can consume significant CPU or GPU memory resources, with a single vLLM-backed model supported per container instance.

The llama.cpp Runtime Engine starts one dedicated llama-server instance per configured llama.cpp-backed model. Llama.cpp models are loaded from GGUF files, which tend to be memory efficient. As with the vLLM runtime engine, one llama.cpp model is supported per container instance.

Llama.cpp is well suited for getting started and exploring the service’s capabilities, while vLLM is generally preferred for enterprise deployments because it handles concurrent requests more efficiently.

You can choose to set up either the vLLM or llama.cpp runtime in a container. They cannot both be run at the same time. The ONNX runtime engine can be used at the same time as vLLM or llama.cpp.

LLM guardrails are supported with ONNX-format models to reduce security risks, such as prompt injection. Guardrails are implemented using a prompt classification model, taking a user prompt as input and producing one or more class labels, which are then used to determine whether the prompt should be accepted or rejected. For more information, see Guardrails.

While vLLM and llama.cpp generally accept only text input, image inputs can be passed to chat/completions for vLLM with certain models, including ministral-3-3B-reasoning-2512, which is shipped with the container. For details about the chat/completions API endpoint, see Private AI Services Container API Reference.