Private AI Environment Variables

Environment variables can be used to set configuration properties, such as file locations, logging settings, and others. They can be passed to the container when running the container using the standard -e flag.

Typically, the JSON configuration file is the preferred method to specify most configuration settings, as it centralizes the settings and simplifies management. However, certain properties can only be set as environment variables using the -e flag and cannot be specified in the configuration file. This is because these environment variables are processed before the JSON file is loaded. The exceptions are listed in the following table:

Variable Name	Type	Default Value	Value Range	Description
PRIVATE_AI_CONFIG_FILE	String	None	None	Specifies the location of a config.json file (may have a different file name) that will be used as the configuration file. This variable is required if PRIVATE_AI_MODELFILE is not specified.
PRIVATE_AI_DRYRUN	Boolean	FALSE	TRUE FALSE	This optional variable specifies whether to run the container in dry run mode. Information about dry run mode can be found at the end of this section.
PRIVATE_AI_LOG_STDOUT_ENABLED	Boolean	FALSE	TRUE FALSE	This optional variable specifies whether to display logs in the console or write to the log directory.
PRIVATE_AI_API_KEY_SECRET_FILE_NAME	String	None	None	Optionally provide the location of a JSON file that contains API keys for multiple key configuration.

The following list of environment variables (all optional) can be set either in the JSON configuration file or using the -e flag when running the container.

Variable Name	Type	Default Value	Value Range	Description
PRIVATE_AI_ONNX_INTRA_OP_NUM_THREADS	Number	None	From 1 to the number of available processors on the host	The number of intra-op threads for the ONNX Runtime. This environment variable is used as the global default when using multiple models. The property can be set on individual models using the configuration file. This variable is only relevant to the ONNX runtime.
PRIVATE_AI_MODEL_SCORING_TIMEOUT	Number	120	1 to 3600	The maximum time (in seconds) that an ONNX model scoring thread may execute before an attempt is made to cancel it.
PRIVATE_AI_MODEL_CACHE_TTL	Number	5400	1 to 7200	The maximum idle time (in seconds) before a cache entry is expired and removed from the model cache.
PRIVATE_AI_MODEL_CACHE_MAXENTRIES	Number	16	1 to 64	The maximum number of entries that may exist in the model cache before the LRU entry is evicted.
PRIVATE_AI_MAX_FILE_SIZE	Number	800,000,000	1 or greater	The maximum model input byte length
PRIVATE_AI_MAX_IMG_SIZE	Number	20,000,000	1 or greater	The maximum image byte length for an ONNX model.
PRIVATE_AI_LOG_LEVEL	String	INFO	INFO, OFF, SEVERE, WARNING, FINE, FINER, FINEST, ALL	The logging level.
PRIVATE_AI_MAX_MODEL_COUNT	Number	64	1 or greater	The maximum number of models allowed in one configuration file.
PRIVATE_AI_MAX_SCORE_PAYLOAD	Number	20,000,000	1 or greater	The maximum payload for scoring requests.
PRIVATE_AI_MAX_BATCHSIZE	Number	256	1 or greater	The maximum batch size.
PRIVATE_AI_HTTP_READ_TIMEOUT	Number	120	>=1	Optionally provide an HTTP read timeout in seconds for service calls to the managed LLM runtime. This variable is only relevant to the vLLM and llama.cpp runtimes.
PRIVATE_AI_LLM_MAX_STREAM_BUFFER_SIZE	Number	1,000,000	>=1	Optionally specify the maximum stream buffer size used by LLM streaming responses. This variable is only relevant to the vLLM and llama.cpp runtimes.
PRIVATE_AI_LLM_MAX_COMPLETION_TOKENS	Number	None	>=1	Optional service-level cap for chat completion max_completion_tokens
PRIVATE_AI_ALLOWED_MESSAGE_ROLES	Array	["user"]	Valid chat role names: user system developer assistant	Optionally specify allowed chat message roles accepted by request validation. Certain models allow additional role names, so valid chat role names are not restricted to only those listed.
PRIVATE_AI_MAX_ACTIVE_API_KEYS	Number	1000	>=0	Optionally specify the maximum number of API keys that can be active at one time. A value of 0 indicates an unlimited number.
PRIVATE_AI_API_KEY_ID_MAX_LENGTH	Number	128	>=0	Optionally specify the maximum length of the API key string. A value of 0 indicates an unlimited length.

The PRIVATE_AI_DRYRUN environment variable enables dry run mode, which allows Kubernetes operators and automation tools to verify container readiness and environmental correctness without launching the full application server. Dry run mode is especially useful for validating dynamic secrets, environment variables, and configuration files prior to exposing application endpoints or initiating expensive processes.

In dry run mode, the container performs all validation and initialization steps normally performed during setup, except it does not start the Micronaut server. This approach helps detect misconfigurations early and integrates seamlessly into DevOps and Kubernetes workflows. The container process will exit immediately after successful validation or upon first failure.

The result of a dry run is indicated by one of two files in the container at /privateai/app/:

• dryrun.suc: indicates validation was successful.
• dryrun.dif: indicates validation failed; the file contains the reason(s) for the failure.

The PRIVATE_AI_API_KEY_SECRET_FILE_NAME environment variable supports the inference service's ability to use multiple API keys, which enables finer-grained access management and improved auditing and attribution of requests. For more information, see Support for Multiple API Keys.