Configure the Private Large Language Model Service

Models are associated with a runtime engine using the service configuration file, which is config.json by default. Included are examples of configuration files that are set up to use Llama.cpp or vLLM, along with an example chat completion request.

  • This is an example of a typical configuration for llama.cpp:

    { 
  "environment":{
    "OML_MAX_SCORE_PAYLOAD": 20000000,
    "OML_LLM_HTTP_READ_TIMEOUT": 600,
    "PRIVATE_AI_ALLOWED_MESSAGE_ROLES": ["user", "assistant", "tool"]
  },
  "models": [
    { 
      "name": "gemma-4-31B-it-GGUF",
      "path": "unsloth/gemma-4-31B-it-GGUF",
      "capabilities": ["TEXT_GENERATION"],
      "runtime": "llamacpp",
      "startup_timeout": 1200,
      "runtime_arguments": [
        "-c", "4096"
      ],
      "properties": {
        "primary_gguf": "gemma-4-31B-it-Q8_0.gguf"
      }
    }
  ]
}

    To use a different model, change the following attributes:

    • The name attribute is a string that should represent the model name.
    • The path attribute must be updated with the <provider>/<model_name> from Hugging Face.
    • The primary_gguf attribute is the path value copied from the Hugging Face "Files and Versions" section. The path may just be the file name, or a path/file name depending on the model in Hugging Face.
    • Add any model-specific llama.cpp runtime arguments as needed.

  • This is an example of a typical configuration for vLLM:

    {
  "models": [
    {
      "name": "ministral_vllm",
      "path": "Ministral-3-3B-Reasoning-2512",
      "runtime": "vllm",
      "capabilities": ["TEXT_GENERATION"],
      "model_template": "Ministral-3-Reasoning",
      "startup_timeout": 1200,
      "runtime_arguments": [
        "--max-model-len", "4096"
      ]
    }
  ]
}

    The Ministral-3-3B-Reasoning-2512 is shipped with the large inference container image for CPU hardware, making it available by default. It does not need to be separately downloaded in this case.

    To use a different model, change the following attributes:

    • The name attribute is a string that should represent the model name.
    • The path attribute must be updated with the <provider>/<model_name> from Hugging Face.
    • Add any model-specific vLLM runtime arguments as needed.

For LLMs not shipped with the container, for example gemma-4-31B-it-GGUF, the model can either be provided in your local file system, as a PAR link, or can be downloaded from Hugging Face. If the model is not found in the local file system or provided as a PAR link, the runtime engine will attempt to download the model from the Hugging Face repository based on the model info in the configuration file.

You can choose to explicitly download a model yourself from Hugging Face rather than have the container download the model for you:

wget https://huggingface.co/unsloth/gemma-3-1b-it-GGUF/resolve/main/gemma-3-1b-it-Q8_0.gguf
cp gemma-3-1b-it-Q8_0.gguf /home/opc/models

In this example, the configuration file would look like the following:

{
  "models": [
    {
      "name":"gemma-3-1b-it-Q8_0",
      "path":"gemma-3-1b-it-Q8_0.gguf",
      "runtime": "llamacpp",
      "context_size":4096,
      "capabilities":["TEXT_GENERATION"]
    }
  ]
}

The following example sends a test text prompt to the local LLM endpoint. A successful JSON response containing an assistant message confirms that the endpoint, TLS certificate, API key, and specified model are working together.

curl --noproxy '*' \
  -X POST \
  --header 'Content-Type: application/json' \
  --header "Authorization: Bearer $API_KEY" \
  --cacert "$SECRETS_DIR/cert.pem" \
  --data '{
    "model": "gpt-oss-20b-GGUF",
    "messages": [
      {
        "role": "user",
        "content": "What does the SELECT statement in SQL do?"
      }
    ],
    "max_tokens": 256,
    "temperature": 0.7,
    "stream": false
  }' https://localhost:8443/v1/chat/completions

The following example is similar, but includes text and image input and uses a different model:

BASE64_IMG=$(base64 -w 0 dog-kitten-small.jpg)
curl --noproxy '*' \
  -X POST \
  --header 'Content-Type: application/json' \
  --header "Authorization: Bearer $API_KEY" \
  --cacert "$SECRETS_DIR/cert.pem" \
  --data '{
      "model": "Ministral-3-3B-Reasoning-2512",
      "messages": [
        {
          "role": "user",
          "content": [
            { "type": "text", "text": "Describe the image in one short sentence." },
            { "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,'"${BASE64_IMG}"'" } }
      ]
    }
  ],
  "max_tokens": 64,
  "temperature": 0.7
}' https://localhost:9091/v1/chat/completions

For more information about configuring the Private AI Services Container, see Configure the Private AI Services Container.

Parent topic: Use the Private Large Language Model Service