Oracle Cloud Infrastructure Documentation

Scenario 1: Stochastic Length Benchmarks in Generative AI

This scenario mimics text generation use cases where the size of the prompt and response are unknown ahead of time. In this scenario, because of the unknown length of the prompt and response, we've used a stochastic approach where both the prompt and response length follow a normal distribution:

  • The prompt length follows a normal distribution with a mean of 480 tokens and a standard deviation of 240 tokens
  • The response length follows a normal distribution with a mean of 300 tokens and a standard deviation of 150 tokens.
Important

The performance (inference speed, throughput, latency) of a hosting dedicated AI cluster depends on the traffic scenarios going through the model that it's hosting. Traffic scenarios depend on:

  1. The number of concurrent requests.
  2. The number of tokens in the prompt.
  3. The number of tokens in the response.
  4. The variance of (2) and (3) across requests.

Review the terms used in the hosting dedicated AI cluster benchmarks. For a list of scenarios and their descriptions, see Chat and Text Generation Scenarios. The fusion scenario is performed in the following region.

Germany Central (Frankfurt)

Model: meta.llama-3-70b-instruct (Meta Llama 3) model hosted on one Large Generic unit of a dedicated AI cluster
Concurrency Token-level Inference Speed (token/second) Token-level Throughput (token/second) Request-level Latency (second) Request-level Throughput (Request per minute) (RPM)
1 49.76 49.58 6.42 9.33
2 48.04 95.38 6.80 17.53
4 46.09 181.21 6.99 33.60
8 44.19 330.46 7.43 60.67
16 40.56 591.52 8.40 104.42
32 31.35 869.36 9.68 168.46
64 23.87 1062.52 12.57 201.11
128 16.86 1,452.66 17.64 276.09
256 9.84 1,792.81 30.08 347.26
Model: cohere.command-r-16k v1.2 (Cohere Command R) model hosted on one Small Cohere V2 unit of a dedicated AI cluster
Concurrency Token-level Inference Speed (token/second) Token-level Throughput (token/second) Request-level Latency (second) Request-level Throughput (Request per minute) (RPM)
1 51.30 50.46 4.63 12.75
2 51.06 97.86 5.07 23.14
4 47.52 186.75 5.30 44.48
8 43.55 305.45 5.68 75.18
16 36.49 505.11 6.71 127.88
32 29.02 768.40 8.84 177.03
64 18.57 735.37 14.55 168.00
128 12.59 809.50 21.27 186.76
256 6.54 859.45 38.69 200.42

US Midwest (Chicago)

Model: meta.llama-3-70b-instruct (Meta Llama 3) model hosted on one Large Generic unit of a dedicated AI cluster
Concurrency Token-level Inference Speed (token/second) Token-level Throughput (token/second) Request-level Latency (second) Request-level Throughput (Request per minute) (RPM)
1 30.51 30.36 10.47 5.73
2 28.85 57.37 11.09 10.68
4 27.99 108.49 11.13 21.08
8 25.61 196.68 13.27 34.65
16 21.97 318.82 15.36 56.37
32 16.01 428.45 18.55 82.88
64 11.60 563.70 24.31 108.58
128 7.50 650.40 40.64 40.64
256 4.58 927.31 67.42 172.42
Model: cohere.command-r-16k v1.2 (Cohere Command R) model hosted on one Small Cohere V2 unit of a dedicated AI cluster
Concurrency Token-level Inference Speed (token/second) Token-level Throughput (token/second) Request-level Latency (second) Request-level Throughput (Request per minute) (RPM)
1 51.30 50.46 4.63 12.75
2 51.06 97.86 5.07 23.14
4 47.52 186.75 5.30 44.48
8 43.55 305.45 5.68 75.18
16 36.49 505.11 6.71 127.88
32 29.02 768.40 8.84 177.03
64 18.57 735.37 14.55 168.00
128 12.59 809.50 21.27 186.76
256 6.54 859.45 38.69 200.42
Model: cohere.command (Cohere Command 52 B) model hosted on one Large Cohere unit of a dedicated AI cluster
Concurrency Token-level Inference Speed (token/second) Token-level Throughput (token/second) Request-level Latency (second) Request-level Throughput (Request per minute) (RPM)
1 36.32 31.29 8.15 7.12
8 30.15 106.03 13.19 23.86
32 23.94 204.41 23.90 45.84
128 14.36 254.54 65.26 56.58
Model: cohere.command-light (Cohere Command Light 6 B) model hosted on one Small Cohere unit of a dedicated AI cluster
Concurrency Token-level Inference Speed (token/second) Token-level Throughput (token/second) Request-level Latency (second) Request-level Throughput (Request per minute) (RPM)
1 69.17 69.19 3.57 15.69
8 38.75 208.22 6.54 45.08
32 17.98 337.35 13.49 75.50
128 4.01 397.36 37.69 92.17
Model: meta.llama-2-70b-chat (Llama2 70 B) model hosted on one Llama2 70 unit of a dedicated AI cluster
Concurrency Token-level Inference Speed (token/second) Token-level Throughput (token/second) Request-level Latency (second) Request-level Throughput (Request per minute) (RPM)
1 17.86 17.18 13.60 4.32
8 14.48 68.62 16.63 16.58
32 9.82 174.40 20.78 44.58
128 3.89 319.34 43.87 85.33