Download Models from Hugging Face

The Private AI Services Container manages inference servers based on vLLM and llama.cpp. Both runtimes require that either model artifacts be present on local disk at startup, or that models be dynamically downloaded from Hugging Face.

There are three methods that can be used to provide a model; in the local file system, as a PAR link, or as a download from Hugging Face.

When a model path is provided that is not a local file or PAR link, the runtime engine (either vLLM or llama.cpp) will attempt to download the model from the Hugging Face repository.

In order to download a model from Hugging Face, you must provide configuration identifying the following information:

• The model to load, using the model name in Hugging Face (not the full URL)
• The target runtime; either vLLM or llama.cpp
• The intended capability (chat, embeddings, reranking, or classification)

Once the container validates the model repository, ensuring that no insecure model artifacts are present, the selected runtime is started and the service exposes the runtime endpoints for inference.

The following artifact formats are supported:

• .safetensors
• .gguf, including llama.cpp sharded files such as .gguf.00001-of-00005
• model directories containing the runtime-specific artifacts previously listed

Private or gated Hugging Face repositories require a Hugging Face token. The container looks for the token secret at the following path: /run/secrets/huggingface-token

Private repositories can be accessed by creating a secure file and then mounting it as a container secret. To create a local token file, use the following command:

printf '%s' '<hugging-face-token>' > /tmp/huggingface-token 
chmod 600 /tmp/huggingface-token

Then, create the podman secret:

podman secret create huggingface-token /tmp/huggingface-token

You can remove the temporary file after the secret is created:

rm /tmp/huggingface-token

Add the secret to the podman run command:

--secret huggingface-token

Models are stored in a cache directory that can be shared across multiple containers (for example, using a mounted volume) to reduce repeated downloads and increase startup speed.

The vLLM and llama.cpp runtimes use /privateai/hf_cache as the Hugging Face cache location inside the container. This directory is created automatically and can be mounted to a host directory.