Rate Limiting

The Private AI Services Container provides a configurable method to control the number of requests per minute that the container can handle for different categories of endpoints. This improves scalability and helps prevent abuse, including denial-of-service (DOS) attacks.

There are two layers of limitation; a pre-authentication global limit for the whole container, and post-authentication category limits for validated API keys.

Every request first consumes from a shared, global bucket called GLOBAL. This happens before API key authentication, so requests with missing or invalid API keys are still rate limited. The global limit is configured using the following property:

"global_requests_per_min": 30000

This global value is per-container, per-pod limit and must be greater than 0. The default value is 30000.

Once authentication has succeeded, requests also consume from a category bucket tied to the validated API key identity.

The container distinguishes between two types of endpoints, each with an independently configurable rate limit:

• Monitor endpoints: These are used for operational monitoring and health checks, including /health and /metrics, and are controlled by monitor_requests_per_min (default value 60) in the configuration file.

• Service endpoints: These include all of the other API functions such as /v1/embeddings and /v1/models. These are controlled by service_requests_per_min (default value 3000) in the configuration file.

  Note: The /health endpoint uses only the global bucket.

This separation ensures that health and metrics scraping, as well as monitoring, does not interfere with the ability to serve regular API traffic from users or client applications.

If authentication is disabled, requests use only the global bucket. No monitor or service category bucket is created.

Container admins can start the container with rate limiting by adding a "ratelimiter" session in the configuration JSON file, as in the following:

{
  "ratelimiter": {
    "global_requests_per_min": 30000,
    "service_requests_per_min": 3000,
    "monitor_requests_per_min": 60
  }
}

This example configuration file would allow up to 30000 total requests per minute to the container, up to 3000 API (service) requests per minute per IP address, and up to 60 monitor (health, metrics) requests per minute per IP address.

Each IP address is tracked independently for both service and monitor groups and counters reset every minute. If a client exceeds its assigned quota within a 60-second window, the server responds with HTTP 429 (Too Many Requests). This logic protects the core APIs from being overwhelmed while ensuring observability endpoints remain responsive.

Several response headers are provided to help monitor the requests usage:

• Global limiter headers:

  • x-ratelimit-global-limit-requests

  • x-ratelimit-global-remaining-requests

  • x-ratelimit-global-reset-requests

• Category limiter headers:

  • x-ratelimit-limit-requests: This is the same as "service_requests_per_min" or "monitor_requests_per_min" defined in the configuration file, depending on the endpoints. It shows the maximum requests allowed per minute per IP address.

  • x-ratelimit-remaining-requests: This shows the number of requests remaining before the total request resets.

  • x-ratelimit-reset-requests: This indicates the number of seconds left until the total request is refilled.

If, for example, service_requests_per_min has a value of 500 and a client sends 400 scoring or inference requests in the first 25 seconds, the response will include: x-ratelimit-limit-requests = 500, x-ratelimit-remaining-requests = 100 (500 - 400), and x-ratelimit-reset-requests = 35s (60 seconds - 25 seconds).

With a monitor_requests_per_minute value of 120, if a client sends 60 health check requests in the first 15 seconds, the response will include: x-ratelimit-limit-requests = 120, x-ratelimit-remaining-requests = 60 (120-60), and x-ratelimit-reset-requests = 45s (60 seconds - 15 seconds).