CohereLlmInferenceRequest

class oci.generative_ai_inference.models.CohereLlmInferenceRequest(**kwargs)

Bases: LlmInferenceRequest

Details for the text generation request for Cohere models.

Attributes:

RETURN_LIKELIHOODS_ALL	A constant which can be used with the return_likelihoods property of a CohereLlmInferenceRequest.
RETURN_LIKELIHOODS_GENERATION	A constant which can be used with the return_likelihoods property of a CohereLlmInferenceRequest.
RETURN_LIKELIHOODS_NONE	A constant which can be used with the return_likelihoods property of a CohereLlmInferenceRequest.
RUNTIME_TYPE_COHERE	A constant which can be used with the runtime_type property of a LlmInferenceRequest.
RUNTIME_TYPE_LLAMA	A constant which can be used with the runtime_type property of a LlmInferenceRequest.
TRUNCATE_END	A constant which can be used with the truncate property of a CohereLlmInferenceRequest.
TRUNCATE_NONE	A constant which can be used with the truncate property of a CohereLlmInferenceRequest.
TRUNCATE_START	A constant which can be used with the truncate property of a CohereLlmInferenceRequest.
frequency_penalty	Gets the frequency_penalty of this CohereLlmInferenceRequest.
is_echo	Gets the is_echo of this CohereLlmInferenceRequest.
is_stream	Gets the is_stream of this CohereLlmInferenceRequest.
max_tokens	Gets the max_tokens of this CohereLlmInferenceRequest.
num_generations	Gets the num_generations of this CohereLlmInferenceRequest.
presence_penalty	Gets the presence_penalty of this CohereLlmInferenceRequest.
prompt	[Required] Gets the prompt of this CohereLlmInferenceRequest.
return_likelihoods	Gets the return_likelihoods of this CohereLlmInferenceRequest.
runtime_type	[Required] Gets the runtime_type of this LlmInferenceRequest.
stop_sequences	Gets the stop_sequences of this CohereLlmInferenceRequest.
temperature	Gets the temperature of this CohereLlmInferenceRequest.
top_k	Gets the top_k of this CohereLlmInferenceRequest.
top_p	Gets the top_p of this CohereLlmInferenceRequest.
truncate	Gets the truncate of this CohereLlmInferenceRequest.

Methods:

__init__(**kwargs)	Initializes a new CohereLlmInferenceRequest object with values from keyword arguments.
get_subtype(object_dictionary)	Given the hash representation of a subtype of this class, use the info in the hash to return the class of the subtype.

RETURN_LIKELIHOODS_ALL = 'ALL'

A constant which can be used with the return_likelihoods property of a CohereLlmInferenceRequest. This constant has a value of “ALL”

RETURN_LIKELIHOODS_GENERATION = 'GENERATION'

A constant which can be used with the return_likelihoods property of a CohereLlmInferenceRequest. This constant has a value of “GENERATION”

RETURN_LIKELIHOODS_NONE = 'NONE'

A constant which can be used with the return_likelihoods property of a CohereLlmInferenceRequest. This constant has a value of “NONE”

RUNTIME_TYPE_COHERE = 'COHERE'

A constant which can be used with the runtime_type property of a LlmInferenceRequest. This constant has a value of “COHERE”

RUNTIME_TYPE_LLAMA = 'LLAMA'

A constant which can be used with the runtime_type property of a LlmInferenceRequest. This constant has a value of “LLAMA”

TRUNCATE_END = 'END'

A constant which can be used with the truncate property of a CohereLlmInferenceRequest. This constant has a value of “END”

TRUNCATE_NONE = 'NONE'

A constant which can be used with the truncate property of a CohereLlmInferenceRequest. This constant has a value of “NONE”

TRUNCATE_START = 'START'

A constant which can be used with the truncate property of a CohereLlmInferenceRequest. This constant has a value of “START”

__init__(**kwargs)

Initializes a new CohereLlmInferenceRequest object with values from keyword arguments. The default value of the runtime_type attribute of this class is COHERE and it should not be changed. The following keyword arguments are supported (corresponding to the getters/setters of this class):

Parameters

• runtime_type (str) – The value to assign to the runtime_type property of this CohereLlmInferenceRequest. Allowed values for this property are: “COHERE”, “LLAMA”

• prompt (str) – The value to assign to the prompt property of this CohereLlmInferenceRequest.

• is_stream (bool) – The value to assign to the is_stream property of this CohereLlmInferenceRequest.

• num_generations (int) – The value to assign to the num_generations property of this CohereLlmInferenceRequest.

• is_echo (bool) – The value to assign to the is_echo property of this CohereLlmInferenceRequest.

• max_tokens (int) – The value to assign to the max_tokens property of this CohereLlmInferenceRequest.

• temperature (float) – The value to assign to the temperature property of this CohereLlmInferenceRequest.

• top_k (int) – The value to assign to the top_k property of this CohereLlmInferenceRequest.

• top_p (float) – The value to assign to the top_p property of this CohereLlmInferenceRequest.

• frequency_penalty (float) – The value to assign to the frequency_penalty property of this CohereLlmInferenceRequest.

• presence_penalty (float) – The value to assign to the presence_penalty property of this CohereLlmInferenceRequest.

• stop_sequences (list[str]) – The value to assign to the stop_sequences property of this CohereLlmInferenceRequest.

• return_likelihoods (str) – The value to assign to the return_likelihoods property of this CohereLlmInferenceRequest. Allowed values for this property are: “NONE”, “ALL”, “GENERATION”

• truncate (str) – The value to assign to the truncate property of this CohereLlmInferenceRequest. Allowed values for this property are: “NONE”, “START”, “END”

property frequency_penalty

Gets the frequency_penalty of this CohereLlmInferenceRequest. To reduce repetitiveness of generated tokens, this number penalizes new tokens based on their frequency in the generated text so far. Greater numbers encourage the model to use new tokens, while lower numbers encourage the model to repeat the tokens. Set to 0 to disable.

Returns

The frequency_penalty of this CohereLlmInferenceRequest.

Return type

float

static get_subtype(object_dictionary)

Given the hash representation of a subtype of this class, use the info in the hash to return the class of the subtype.

property is_echo

Gets the is_echo of this CohereLlmInferenceRequest. Whether or not to return the user prompt in the response. This option only applies to non-stream results.

Returns

The is_echo of this CohereLlmInferenceRequest.

Return type

bool

property is_stream

Gets the is_stream of this CohereLlmInferenceRequest. Whether to stream back partial progress. If set, tokens are sent as data-only server-sent events as they become available.

Returns

The is_stream of this CohereLlmInferenceRequest.

Return type

bool

property max_tokens

Gets the max_tokens of this CohereLlmInferenceRequest. The maximum number of tokens to predict for each response. Includes input plus output tokens.

Returns

The max_tokens of this CohereLlmInferenceRequest.

Return type

int

property num_generations

Gets the num_generations of this CohereLlmInferenceRequest. The number of generated texts that will be returned.

Returns

The num_generations of this CohereLlmInferenceRequest.

Return type

int

property presence_penalty

Gets the presence_penalty of this CohereLlmInferenceRequest. To reduce repetitiveness of generated tokens, this number penalizes new tokens based on whether they’ve appeared in the generated text so far. Greater numbers encourage the model to use new tokens, while lower numbers encourage the model to repeat the tokens.

Similar to frequency penalty, a penalty is applied to previously present tokens, except that this penalty is applied equally to all tokens that have already appeared, regardless of how many times they’ve appeared. Set to 0 to disable.

Returns

The presence_penalty of this CohereLlmInferenceRequest.

Return type

float

property prompt

[Required] Gets the prompt of this CohereLlmInferenceRequest. Represents the prompt to be completed. The trailing white spaces are trimmed before completion.

Returns

The prompt of this CohereLlmInferenceRequest.

Return type

str

property return_likelihoods

Gets the return_likelihoods of this CohereLlmInferenceRequest. Specifies how and if the token likelihoods are returned with the response.

Allowed values for this property are: “NONE”, “ALL”, “GENERATION”

Returns

The return_likelihoods of this CohereLlmInferenceRequest.

Return type

str

property runtime_type

[Required] Gets the runtime_type of this LlmInferenceRequest. The runtime of the provided model.

Allowed values for this property are: “COHERE”, “LLAMA”

Returns

The runtime_type of this LlmInferenceRequest.

Return type

str

property stop_sequences

Gets the stop_sequences of this CohereLlmInferenceRequest. The generated text is cut at the end of the earliest occurrence of this stop sequence. The generated text will include this stop sequence.

Returns

The stop_sequences of this CohereLlmInferenceRequest.

Return type

list[str]

property temperature

Gets the temperature of this CohereLlmInferenceRequest. A number that sets the randomness of the generated output. A lower temperature means a less random generations.

Use lower numbers for tasks with a correct answer such as question answering or summarizing. High temperatures can generate hallucinations or factually incorrect information. Start with temperatures lower than 1.0 and increase the temperature for more creative outputs, as you regenerate the prompts to refine the outputs.

Returns

The temperature of this CohereLlmInferenceRequest.

Return type

float

property top_k

Gets the top_k of this CohereLlmInferenceRequest. An integer that sets up the model to use only the top k most likely tokens in the generated output. A higher k introduces more randomness into the output making the output text sound more natural. Default value is 0 which disables this method and considers all tokens. To set a number for the likely tokens, choose an integer between 1 and 500.

If also using top p, then the model considers only the top tokens whose probabilities add up to p percent and ignores the rest of the k tokens. For example, if k is 20, but the probabilities of the top 10 add up to .75, then only the top 10 tokens are chosen.

Returns

The top_k of this CohereLlmInferenceRequest.

Return type

int

property top_p

Gets the top_p of this CohereLlmInferenceRequest. If set to a probability 0.0 < p < 1.0, it ensures that only the most likely tokens, with total probability mass of p, are considered for generation at each step.

To eliminate tokens with low likelihood, assign p a minimum percentage for the next token’s likelihood. For example, when p is set to 0.75, the model eliminates the bottom 25 percent for the next token. Set to 1.0 to consider all tokens and set to 0 to disable. If both k and p are enabled, p acts after k.

Returns

The top_p of this CohereLlmInferenceRequest.

Return type

float

property truncate

Gets the truncate of this CohereLlmInferenceRequest. For an input that’s longer than the maximum token length, specifies which part of the input text will be truncated.

Allowed values for this property are: “NONE”, “START”, “END”

Returns

The truncate of this CohereLlmInferenceRequest.

Return type

str