Install the Private AI Services Container

2 Install the Private AI Services Container

Use the included steps to configure and install the container with best practices. You can either use setup bash scripts provided with the container image, or set up the container manually.

The Oracle Private AI Services Container is AI infrastructure designed to run on-premises, and optionally in air-gapped environments. With the vector embedding service, the container can create low latency, free vector embeddings in a secure manner within the privacy of your own realm. Using the vector index service, the container securely offloads the computational costs of vector index creation to an NVIDIA GPU. The Large Language Model (LLM) service enables Chat, RAG, and agentic workloads with local LLMs.

The container uses TLS 1.3 and API Keys for security. A PKCS12 keystore is used by default, but JKS is also supported. The container can also work with Security Enhanced Linux (SELinux) in enforcing mode.

The included scripts, provided for HTTP and HTTP/SSL, offer examples of how to use curl to create vector embeddings, list the loaded embedding models, check on the health of the container, and produce runtime metrics. For instructions on how to set up the container without using the setup scripts, see Oracle Container Registry.

Prerequisites

The Oracle Private AI Services Container uses Oracle Linux 8 within the container and works with the following host Linux x86_64 distributions:

Oracle Linux 8.6+
Oracle Linux 9
Oracle Linux 10

The AI Services Container uses TLS 1.3. This means that you need a version of OpenSSL that supports TLS 1.3 to create certificates, for example:

OpenSSL 1.1.1k+
OpenSSL 3.0+

The AI Services Container can be used with the following software:

Podman

The chosen software must be one of the following versions:

To use the vector embedding service:
- Podman 3.4.4+, 4.4+, 5+
- Kubernetes 1.31.1+
- Red Hat OpenShift 4.19+
To use the vector index service:
- Podman 3.4.4+, 4.4+, 5+

The included examples use Podman with Oracle Linux 8 as the host operating system on OCI. The example OCI VM where the container is installed is called privateaivm.

There are also some system requirements to be aware of when using the Private AI Services Container:

Vector Embedding Service:
- Memory: You must have at least 16 GB of free memory to effectively use the container to create vector embeddings. If you want to use large embedding models or create multiple different types of vectors at the same time, then you will need more memory. Insufficient memory can result in reduced performance.
- CPU cores: Although you can use a single CPU core with the ONNX Runtime to create vector embeddings, having more CPU cores will enhance performance. Multiple CPU cores enable either a single vector or many different vectors to be created at the same time using multi-threading.
- Disk space: You need at least 22 GB of disk space to effectively use the container. You will need additional disk space for each additional embedding model that you use. You will also need additional disk space depending on the log level for long running containers.
Private Large Language Model Service:
- Memory: At least 32 GB of memory is needed. If you use big LLMs, then more memory will be required. The model size, KV cache size, and level of concurrency determine the amount of memory that is necessary.
- CPU cores: A minimum of 8 CPU cores is required for the Private Large Language Model Service. Additional CPU cores will enable faster inference.
- Disk space: A minimum of 22 GB of disk space is needed, plus at least twice the model size.
Vector Index Service:
- Memory: At least 16 GB of free memory is required to create vector indexes, and it is recommended to have at least twice as much CPU memory as VRAM (GPU memory). Insufficient memory will cause index creation to fail.
- CPU cores: Oracle recommends a minimum of 4 CPU cores for vector index creation. Beyond 16 CPU cores, the performance benefits of additional cores diminishes.
- Disk space: Disk space is only needed for logging when using the vector index service. A minimum of 22 GB of disk space is suggested. More may be needed depending on the log level for long running containers.

Note that the container should not run on the same machine as the Oracle AI Database Server but on a Linux machine that is close to the database server. The container is designed to accelerate resource intensive tasks such as creating vectors or vector index offload of machines other than the Oracle AI Database. This enables the Oracle AI Database to have low latency and high throughput without being burdened with these resource-intensive AI infrastructure tasks.

The Vector Index Service and the Private Large Language Model Service of the Private AI Services Container have the following additional prerequisites:

NVIDIA GPUs only: Turing architecture and above. To see a current list of NVIDIA GPUs along with their compute capability, see https://developer.nvidia.com/cuda/gpus.
Compute capability 7.5+
NVIDIA driver version 580.65.06+
The NVIDIA Container Toolkit must be installed on the host to expose GPUs inside the container. Using the latest version of the Toolkit is suggested, for example 1.19.0 or later. Follow the steps provided in the NVIDIA installation guide and the CDI integration support documentation for Podman.

Use nvidia-smi to ensure your GPU resources meet the listed prerequisites and nvidia-ctk to verify that the CDI is working correctly. For example, the latter can be confirmed using the following command:

nvidia-ctk --debug cdi list

Install the container.
- If you are using Oracle Linux 8:
```
sudo dnf module install -y container-tools:ol8
```
- If you are using Oracle Linux 9 or 10:
```
sudo dnf install -y container-tools
```
Verify that Podman is installed correctly.
```
podman version
```
```
podman images
```
Sign in to Oracle Container Registry (OCR).

Go to Oracle Container Registry and click Sign In.

You will be asked to enter your SSO user name and password. If you are not yet registered with container-registry.oracle.com, you will be prompted to do so. Registering simply saves your SSO user name with container-registry.oracle.com as an SSO-allowed user.

After completing the one-time registration, clicking Sign In on future visits will prompt you to enter your SSO user name and password only.
Accept the License Agreement.

You must click Continue to read and accept the license agreement. This action only needs to be completed once.
Generate an auth token to pull the container image.

To use Podman to download the Private AI Services container image, you will need to use the Podman login with your SSO user name and a generated auth token. To generate an auth token, follow the steps at Generating an Oracle Container Registry Authentication Token in Podman User's Guide.

Note:
Be sure to copy the generated auth token to a secure location before navigating from the generation page. The token will not be shown again and you will have to delete any existing token before creating another.
Pull the container image.

Download (pull) the Private AI Services Container image to the host where you intend to run it.
1. Run the following command to log in to OCR.
```
podman login container-registry.oracle.com
```
  When prompted for a user name, enter your SSO user name. Enter the generated auth token from the previous step when prompted for a password.
2. Once you have successfully logged in, pull the image of your choice, depending on your use case and hardware.
  Use the following command, replacing <container-name> at the end of the statement with one of the listed images:
```
podman pull container-registry.oracle.com/database/private-ai:<container-name>
```
  - Vector index offloading on GPU:
```
gpu-index-26.1.0.0.0
```
  - Inference using CPU or GPU:
    - Small (~5GB) inference container on CPU.
      
      To be used with the ONNX runtime for text and image embeddings, reranking, and text classification. Five onnx embedding models are included.
      
      small-cpu-infer-26.2.1.0.0
    - Medium (~11GB) inference container on CPU.
      
      To be used with the ONNX runtime, vLLM, or llama.cpp for embeddings, text generation, reranking, and text classification. Five onnx embedding models are included.
      
      Note:
      The image that includes "avx512" is intended for use with CPU architectures that support avx512.
      
      medium-cpu-infer-26.2.1.0.0
      
      medium-cpu-avx512-infer-26.2.1.0.0
    - Large (~22GB) inference container on CPU.
      
      To be used with ONNX runtime, vLLM, or llama.cpp for embeddings, text generation, reranking, and text classification. Five onnx embedding models, as well as ministral-3-3B-reasoning-2512 (BF16 and Q8) are included.
      
      Note:
      The image that includes "avx512" is intended for use with CPU architectures that support avx512.
      
      large-cpu-infer-26.2.1.0.0
      
      large-cpu-avx512-infer-26.2.1.0.0
    - Medium (~24GB) inference container on GPU.
      
      To be used with vLLM for embeddings, text generation, reranking, and text classification.
      
      medium-gpu-infer-26.2.1.0.0
    - Large (~32GB) inference container on GPU.
      
      To be used with vLLM for embeddings, text generation, reranking, and text classification.
      
      Includes ministral-3-3B-reasoning-2512 (BF16).
      
      large-gpu-infer-26.2.1.0.0
Verify the image has been pulled.
Run the following command to show all images on your system, including, for example, container-registry.oracle.com/database/private-ai:small-cpu-infer-26.2.1.0.0.
```
podman images
```
(Optional) When using air-gapped environments, continue with the following steps so that you can access the Private AI Container Image offline.
1. Save the Private AI Container Image to a tar file.
  The following command saves the container image to a file in the current directory. The file name is dependent on your chosen image. For example, private-ai-gpu-index-26.1.0.0.0.tar or private-ai-medium-cpu-infer-26.2.1.0.0.tar.
  
  Replace <filename.tar> and <container-name> depending on your chosen container image:
```
podman save -o <filname.tar>  container-registry.oracle.com/database/private-ai:<container-name>
```
2. Load the tar file on the air-gapped machine.
  After copying the tar file to the air-gapped host, you can load it into the local podman image repository using the following command, replacing <filename.tar> with the same file name used in the previous step:
```
podman load -i <filename.tar>
```
  Once this load operation is successful, verify that the Private AI Container Image is accessible on the air-gapped host using podman images. This should list all podman images on the host.