Configure a Reranking Model

A container administrator can configure a reranking model in the container configuration file.

The following example configures a reranking model that uses a model template:

{
  "models": [
    {
      "name": "mxbai-rerank-base-v2",
      "path": "mixedbread-ai/mxbai-rerank-base-v2",
      "runtime": "vllm",
      "model_template": "mxbai-rerank"
    }
  ]
}

In this case, the model template supplies the runtime arguments and the TEXT_RERANK capability for the model.

The following examples provide sample configurations for reranking models using vLLM, llama.cpp, and ONNX runtimes, respectively:

• In this vLLM example, the reranker model is downloaded from Hugging Face. If using a zip file or PAR link, the path property can be updated accordingly.

  {
    "models": [
      {
        "name":"bge-reranker-v2",
        "path":"BAAI/bge-reranker-v2-m3",
        "runtime": "vllm",
        "capabilities":"TEXT_RERANK"
      }
    ]
  }

• This example specifies a llama.cpp runtime and downloads the reranker GGUF from Hugging Face.

  This particular Hugging Face repository contains multiple GGUF files for the model. In this case, the medium variant of the model is chosen by specifying Q3_K_M. Omitting the suffix will default to the more standard Q4_K_M variant.

  {
    "models": [
      {
        "name":"qwen3-reranker-4b",
        "path":"QuantFactory/Qwen3-Reranker-4B-GGUF:Q3_K_M",
        "runtime": "llamacpp",
        "capabilities":"TEXT_RERANK"
      }
    ]
  }

• In this example, the model is provided as a local ONNX format model file called "reranker.onnx".

  Note:

  The container will look for this file under /privateai/models in the container file system. When starting the container, it is important to mount the folder containing reranker.onnx to /privateai/models using the -v argument in podman.

  {
    "models": [
      {
        "name":"reranker-model",
        "path":"reranker.onnx",
        "function":"reranking"
      }
    ]
  }

The following example sends a reranking request to the /v1/rerank API endpoint. It asks the bge-reranker model to score and rank the three listed documents by how relevant they are to the given query.

curl --location 'https://localhost:9091/v1/rerank' \
  --header 'Content-Type: application/json' \
  --header "Authorization: Bearer $API_KEY" \
  --cacert "$SECRETS_DIR/cert.pem" \
  --data '{
    "model": "bge-reranker",
    "query": "What is the capital of France?",
    "documents": [
      "The capital of Brazil is Brasilia.",
      "The capital of France is Paris.",
      "Horses and cows are both animals."
    ]
}'

For more information about configuring the Private AI Services Container, see Configure the Private AI Services Container.