4 Use the Inference Service

The Private AI Services Container inference service supports a number of capabilities, including vector embedding generation, chat completion, reranking, and text classification.

The inference service provides REST APIs for running inference against models deployed in the container. Inference clients can use these APIs to list available models and submit requests for supported model capabilities. The following services make up the inference service:

  • Embedding Service

    Convert text or images into vector embeddings.

  • Private Large Language Model Service

    Host LLMs for Chat, RAG, and Agentic workloads.

  • Reranking Service

    Reorder a set of documents according to their relevance to a query.

  • Text Classification Service

    Classify one or more text inputs by assigning labels and probabilities to text inputs.

Each REST endpoint accepts only models that are configured or validated for the corresponding capability.

To accommodate different model types, distinct runtime engines are introduced, which are responsible for loading model artifacts and exposing a model's supported capabilities for execution. There are currently three available runtime engines:

  • ONNX Runtime Engine
  • vLLM Runtime Engine
  • Llama.cpp Runtime Engine

Models are associated with a runtime engine using the service configuration file, config.json by default. If a runtime engine is not specified for a model, the ONNX Runtime Engine is used. ONNX-format models must be produced using the OML4Py 2.1.1 client tool, which packages the model with the required tokenizer and post-processing components expected by the container.

ONNX-format models and llama.cpp-backed models can be deployed only on CPU hardware. With vLLM-backed models, you can choose CPU or GPU hardware for deployment.

The management of the Private AI Services Container is handled by certain logical roles, including the Container Admin, Model Creator, and Inference Client. These roles may be assumed by the same Linux user simultaneously.
  • Container Admin: Acting as container admin, a user can configure, stop, start, and perform general management of a container.

  • Model Creator: As model creator, a user performs actions that are related to the provision of model files. This includes tasks such as creating an ONNX pipeline and providing metadata and model specific configurations when necessary. When using the Private Large Language Model Service, the model creator is responsible for providing LLMs downloaded from HuggingFace. LLMs can be provided as a zip or as a directory.

  • Inference Client: As inference client, a user performs inference on data and queries available models using REST APIs.

Unless otherwise noted, the provided configurations are applicable to any of the inference services. For information about a particular inference service offering, see its dedicated chapter.