AsyncGenerativeAiInferenceClient

class oci.generative_ai_inference.AsyncGenerativeAiInferenceClient(config, **kwargs)

Asynchronous client for GenerativeAiInferenceClient.

OCI Generative AI is a fully managed service that provides a set of state-of-the-art, customizable large language models (LLMs) that cover a wide range of use cases for text generation, summarization, and text embeddings.

Use the Generative AI service inference API to access your custom model endpoints, or to try the out-of-the-box models to [chat](#/EN/generative-ai-inference/latest/ChatResult/Chat), [generate text](#/EN/generative-ai-inference/latest/GenerateTextResult/GenerateText), [summarize](#/EN/generative-ai-inference/latest/SummarizeTextResult/SummarizeText), and [create text embeddings](#/EN/generative-ai-inference/latest/EmbedTextResult/EmbedText).

To use a Generative AI custom model for inference, you must first create an endpoint for that model. Use the [Generative AI service management API](#/EN/generative-ai/latest/) to [create a custom model](#/EN/generative-ai/latest/Model/) by fine-tuning an out-of-the-box model, or a previous version of a custom model, using your own data. Fine-tune the custom model on a [fine-tuning dedicated AI cluster](#/EN/generative-ai/latest/DedicatedAiCluster/). Then, create a [hosting dedicated AI cluster](#/EN/generative-ai/latest/DedicatedAiCluster/) with an [endpoint](#/en/generative-ai/latest/Endpoint/) to host your custom model. For resource management in the Generative AI service, use the [Generative AI service management API](#/EN/generative-ai/latest/).

To learn more about the service, see the [Generative AI documentation](/iaas/Content/generative-ai/home.htm).

Important: The IP addresses behind each DNS endpoint might change over time. Always use the DNS hostname listed under the following API Endpoints section and avoid using hard-coded fixed IP addresses.

Methods

__init__(config, **kwargs) Creates a new asynchronous service client.
apply_guardrails(apply_guardrails_details, …) Applies guardrails to the input content, including content moderation, PII detection, and prompt injection protection.
chat(chat_details, **kwargs) Creates a response for the given conversation.
close() Closes the underlying asynchronous HTTP session.
embed_text(embed_text_details, **kwargs) Produces embeddings for the inputs.
generate_text(generate_text_details, **kwargs) Generates a text response based on the user prompt.
list_guardrail_versions(opc_compartment_id, …) List the available guardrail system versions.
rerank_text(rerank_text_details, **kwargs) Reranks the text responses based on the input documents and a prompt.
summarize_text(summarize_text_details, **kwargs) Summarizes the input text.
__init__(config, **kwargs)

Creates a new asynchronous service client.

circuit_breaker_strategy and circuit_breaker_callback use the same OCI circuit-breaker implementation and defaults as the synchronous client.

apply_guardrails(apply_guardrails_details, **kwargs)

Applies guardrails to the input content, including content moderation, PII detection, and prompt injection protection. Case 1: Use input when the customer wants simple single-text moderation. Existing customers can continue to use this field without changing their current integration. Case 2: Use multimodalInput when the customer wants moderation over text, image, or a combination of both. multimodalInput supports a single text item, an array of text items only, an array of images only, or a mixed ordered combination of text and image items. Clients may provide input, multimodalInput, or both. At least one of these fields must be provided. If both input and multimodalInput are provided, the service will process input and discard multimodalInput.

Parameters:
  • apply_guardrails_details (oci.generative_ai_inference.models.ApplyGuardrailsDetails) – (required) Details for applying guardrails to the input content.
  • opc_retry_token (str) – (optional) A token that uniquely identifies a request so it can be retried in case of a timeout or server error without risk of executing that same action again. Retry tokens expire after 24 hours, but can be invalidated before that, in case of conflicting operations. For example, if a resource is deleted and purged from the system, then a retry of the original creation request is rejected.
  • opc_request_id (str) – (optional) The client request ID for tracing.
  • retry_strategy (obj) – (optional) A retry strategy to apply to this specific operation/call. This will override any retry strategy set at the client-level.
  • allow_control_chars (bool) – (optional) allow_control_chars is a boolean to indicate whether or not this request should allow control characters in the response object.
  • enable_strict_url_encoding (bool) – (optional) enable_strict_url_encoding is a boolean to indicate whether or not this request should enable strict url encoding in path params of a request.
Returns:

A Response object with data of type ApplyGuardrailsResult
Return type:

Response
chat(chat_details, **kwargs)

Creates a response for the given conversation.

Parameters:
  • chat_details (oci.generative_ai_inference.models.ChatDetails) – (required) Details of the conversation for the model to respond.
  • opc_retry_token (str) – (optional) A token that uniquely identifies a request so it can be retried in case of a timeout or server error without risk of executing that same action again. Retry tokens expire after 24 hours, but can be invalidated before that, in case of conflicting operations. For example, if a resource is deleted and purged from the system, then a retry of the original creation request is rejected.
  • opc_request_id (str) – (optional) The client request ID for tracing.
  • retry_strategy (obj) – (optional) A retry strategy to apply to this specific operation/call. This will override any retry strategy set at the client-level.
  • allow_control_chars (bool) – (optional) allow_control_chars is a boolean to indicate whether or not this request should allow control characters in the response object.
  • enable_strict_url_encoding (bool) – (optional) enable_strict_url_encoding is a boolean to indicate whether or not this request should enable strict url encoding in path params of a request.
Returns:

A Response object with data of type ChatResult
Return type:

Response
close()

Closes the underlying asynchronous HTTP session.

embed_text(embed_text_details, **kwargs)

Produces embeddings for the inputs.

An embedding is numeric representation of a piece of text. This text can be a phrase, a sentence, or one or more paragraphs. The Generative AI embedding model transforms each phrase, sentence, or paragraph that you input, into an array with 1024 numbers. You can use these embeddings for finding similarity in your input text such as finding phrases that are similar in context or category. Embeddings are mostly used for semantic searches where the search function focuses on the meaning of the text that it’s searching through rather than finding results based on keywords.

Parameters:
  • embed_text_details (oci.generative_ai_inference.models.EmbedTextDetails) – (required) Details for generating the embed response.
  • opc_retry_token (str) – (optional) A token that uniquely identifies a request so it can be retried in case of a timeout or server error without risk of executing that same action again. Retry tokens expire after 24 hours, but can be invalidated before that, in case of conflicting operations. For example, if a resource is deleted and purged from the system, then a retry of the original creation request is rejected.
  • opc_request_id (str) – (optional) The client request ID for tracing.
  • retry_strategy (obj) – (optional) A retry strategy to apply to this specific operation/call. This will override any retry strategy set at the client-level.
  • allow_control_chars (bool) – (optional) allow_control_chars is a boolean to indicate whether or not this request should allow control characters in the response object.
  • enable_strict_url_encoding (bool) – (optional) enable_strict_url_encoding is a boolean to indicate whether or not this request should enable strict url encoding in path params of a request.
Returns:

A Response object with data of type EmbedTextResult
Return type:

Response
generate_text(generate_text_details, **kwargs)

Generates a text response based on the user prompt.

Parameters:
  • generate_text_details (oci.generative_ai_inference.models.GenerateTextDetails) – (required) Details for generating the text response.
  • opc_retry_token (str) – (optional) A token that uniquely identifies a request so it can be retried in case of a timeout or server error without risk of executing that same action again. Retry tokens expire after 24 hours, but can be invalidated before that, in case of conflicting operations. For example, if a resource is deleted and purged from the system, then a retry of the original creation request is rejected.
  • opc_request_id (str) – (optional) The client request ID for tracing.
  • retry_strategy (obj) – (optional) A retry strategy to apply to this specific operation/call. This will override any retry strategy set at the client-level.
  • allow_control_chars (bool) – (optional) allow_control_chars is a boolean to indicate whether or not this request should allow control characters in the response object.
  • enable_strict_url_encoding (bool) – (optional) enable_strict_url_encoding is a boolean to indicate whether or not this request should enable strict url encoding in path params of a request.
Returns:

A Response object with data of type GenerateTextResult
Return type:

Response
list_guardrail_versions(opc_compartment_id, **kwargs)

List the available guardrail system versions.

Parameters:
  • opc_compartment_id (str) – (required) The client compartment ID.
  • opc_request_id (str) – (optional) The client request ID for tracing.
  • state (str) –

    (optional) A filter to return only the guardrail versions whose state matches the given value.

    Allowed values are: “ACTIVE”, “PREVIEW”, “DEPRECATED”, “RETIRED”

  • limit (int) –

    (optional) For list pagination. The maximum number of results per page, or items to return in a paginated “List” call. For important details about how pagination works, see List Pagination.

  • page (str) –

    (optional) For list pagination. The value of the opc-next-page response header from the previous “List” call. For important details about how pagination works, see List Pagination.

  • retry_strategy (obj) – (optional) A retry strategy to apply to this specific operation/call. This will override any retry strategy set at the client-level.
  • allow_control_chars (bool) – (optional) allow_control_chars is a boolean to indicate whether or not this request should allow control characters in the response object.
  • enable_strict_url_encoding (bool) – (optional) enable_strict_url_encoding is a boolean to indicate whether or not this request should enable strict url encoding in path params of a request.
Returns:

A Response object with data of type GuardrailVersionCollection
Return type:

Response
rerank_text(rerank_text_details, **kwargs)

Reranks the text responses based on the input documents and a prompt.

Rerank assigns an index and a relevance score to each document, indicating which document is most related to the prompt.

Parameters:
  • rerank_text_details (oci.generative_ai_inference.models.RerankTextDetails) – (required) Details required for the rerank request.
  • opc_retry_token (str) – (optional) A token that uniquely identifies a request so it can be retried in case of a timeout or server error without risk of executing that same action again. Retry tokens expire after 24 hours, but can be invalidated before that, in case of conflicting operations. For example, if a resource is deleted and purged from the system, then a retry of the original creation request is rejected.
  • opc_request_id (str) – (optional) The client request ID for tracing.
  • retry_strategy (obj) – (optional) A retry strategy to apply to this specific operation/call. This will override any retry strategy set at the client-level.
  • allow_control_chars (bool) – (optional) allow_control_chars is a boolean to indicate whether or not this request should allow control characters in the response object.
  • enable_strict_url_encoding (bool) – (optional) enable_strict_url_encoding is a boolean to indicate whether or not this request should enable strict url encoding in path params of a request.
Returns:

A Response object with data of type RerankTextResult
Return type:

Response
summarize_text(summarize_text_details, **kwargs)

Summarizes the input text.

Parameters:
  • summarize_text_details (oci.generative_ai_inference.models.SummarizeTextDetails) – (required) Details for summarizing the text.
  • opc_retry_token (str) – (optional) A token that uniquely identifies a request so it can be retried in case of a timeout or server error without risk of executing that same action again. Retry tokens expire after 24 hours, but can be invalidated before that, in case of conflicting operations. For example, if a resource is deleted and purged from the system, then a retry of the original creation request is rejected.
  • opc_request_id (str) – (optional) The client request ID for tracing.
  • retry_strategy (obj) – (optional) A retry strategy to apply to this specific operation/call. This will override any retry strategy set at the client-level.
  • allow_control_chars (bool) – (optional) allow_control_chars is a boolean to indicate whether or not this request should allow control characters in the response object.
  • enable_strict_url_encoding (bool) – (optional) enable_strict_url_encoding is a boolean to indicate whether or not this request should enable strict url encoding in path params of a request.
Returns:

A Response object with data of type SummarizeTextResult
Return type:

Response