Considerations for the Private Large Language Model Service

• Private Large Language Model Service Runtime Engines
  The Private Large Language Model Service employs runtime engines, which are responsible for loading model artifacts and exposing the model's supported capabilities for inference.
• Models Available to the Private Large Language Model Service
  Included are models that are shipped with the container or available to be downloaded from Hugging Face and are known to work with the vLLM and llama.cpp runtimes.
• Download Models from Hugging Face
  The Private AI Services Container manages inference servers based on vLLM and llama.cpp. Both runtimes require that either model artifacts be present on local disk at startup, or that models be dynamically downloaded from Hugging Face.
• Guardrails
  Guardrails can be used with LLMs to determine whether user prompts should be accepted or rejected, guarding against prompt injection.