LLM Transformation Handlers
Each provider has its own format for the request, response, and error payloads. Because of this, the LLM provider and Oracle Digital Assistant can't communicate directly, so to facilitate the exchange between the skill and its LLM providers, you need to transform these payloads into Oracle Digital Assistant's Common LLM Interface and back again.
transformRequestPayload
,
transformResponsePayload
, and
transformErrorResponsePayload
methods execute the
payload transformations. These transformation methods have two
signatures:
event
: The properties used for this object depend on the event type (transformRequestPayload
,transformResponsePayload
,transformErrorResponsePayload
).context
: References theLlmTransformationContext
class, which provides access to convenience methods you can use to create your event handler logic.
Create an LLM Transformation Handler
- Click Components in the left navbar.
- Click +New Service.
- Complete the Create Service dialog:
- Name: Enter the service name.
- Service Type: Embedded Container
- Component Service Package Type: New Component
- Component Type: LLM Transformation
- Component Name: Enter an easily identifiable name for the entity event handler. You will reference this name when you create the LLM service for the skill.
- Template: We provide templates for
skills that call Cohere directly or via Oracle Generative AI service.
You don't have to edit these templates. If your skill calls a
non-Cohere/Oracle Generative AI model, such as Azure Open AI, you'll
need to add the appropriate code.
The templates for the Oracle Generative AI Cohere (text generation and summarization) and Llama (text summarization) are sorted under Generative AI in the Template list menu. The template for accessing Cohere directly is located under Other. To access the template that contains the starter code for other models, choose Custom (which is also located under Other).
- Click Create to generate the event handler code.
- After deployment completes, expand the service and then select the
transformation handler to open its properties page, which lists the three LLM
provider-CLMI transformation methods (
transformRequestPayload
,transformResponsePayload
, andtransformErrorResponsePayload
). - If you're creating a Cohere or Oracle Generative AI, service, the
handler code is complete.
If you're using the Custom template, click Edit to open the Edit Component and then update the following placeholder code with the provider-specific code:
Method Location in Editor (Custom Template) Placeholder Code (Custom Template) transformRequestPayload
Lines 23-25 transformRequestPayload: async (event, context) => { return event.payload; },
transformResponsePayload
Lines 33-35 transformResponsePayload: async (event, context) => { return event.payload; },
transformErrorResponsePayload
Lines 44-46 transformErrorResponsePayload: async (event, context) => { return event.payload; }
- Verify the sytnax of your updates by clicking Validate. Fix any validation errors (if any), then click Save. Then click Close.
LLM Provider Transformation Code Samples
Azure OpenAI
Method | Event Handler Transformation Code |
---|---|
Request |
|
Response (Non-streaming) | When
streaming is enabled, the response transformation event handler is
called in batches of 20 streamed messages. This batched array of
streamed responses is stored under the responseItems
key.
|
Error |
|
Oracle Generative AI Service – Cohere
Method | Event Handler Code |
---|---|
Request |
|
Response |
|
Error |
|
Oracle Generative AI - Llama
Method | Event Handler Code |
---|---|
Request |
|
Response |
|
Error |
|
Oracle Generative AI - Llama
Method | Event Handler Code |
---|---|
Request |
|
Response |
|
Error |
|
Cohere (Command Model) – Direct Access to Cohere
/generate
API and the associated
Cohere.command
model, not the /chat
API that's
used for the cohere.command.R
model. If you wish to use the
/chat
endpoint instead, then you will need to manually update the request and response
payloads in the generated code template.
Method | Event Handler Code |
---|---|
Request | This
handler manages the conversation history to maintain
the conversation context.
|
Response |
|
Error |
|
The Common LLM Interface
Each LLM provider has its own format for its request and response payloads. The Common LLM Interface, or CLMI, enables the invokeLLM component to handle these proprietary request and response payloads.
- A request body specification.
- A success response body specification, applicable when the LLM REST call returns an HTTP 200 status.
- An error response body specification, applicable when the LLM REST
call returns an HTTP status other than 200 but the invocation of the LLM service
was still successful.
Note:
For unsuccessful invocations, the invokeLLM component handles the 401 (not authorized) or 500 (internal server error) errors.
CLMI Request Body Specification
Property | Type | Default | Description | Required? |
---|---|---|---|---|
messages |
An array of message objects | N/A | A list of messages. The first message is the prompt
with the role property set to
system . If the LLM supports a multi-turn
conversation so that the LLM response can be refined or enhanced,
the subsequent messages will be pairs of messages from the
user and assistant roles. The
user message contains the follow-up instructions or question for the
LLM. The assistant message contains the LLM response to the user
message (the completion). If the LLM does not support multi-turn
conversations, then the messages array will only
contain a single system message holding the prompt.
|
Yes |
streamResponse |
boolean | false |
Determines whether the LLM's response will be
streamed back to the LLM component. Setting this property to
true enhances the user experience, because
streaming enables the LLM component to send partial response
messages back to users so that they don't have to wait for the LLM
to complete the response.
Set |
No |
maxTokens |
integer | 1024 |
The model generates tokens for the words in its results. Tokens can be thought of as pieces of words. 100 tokens equals about 75 words in English, for example. This property limits the size of the content generated by the model by setting the maximum number of tokens that it generates for the response. | No |
temperature |
number | 0 |
The model uses temperature to gauge the randomness –
and thus the creativity – of its responses. You set this as value
ranging from 0 (predictable results) to
1 (more randomized results). 0
means that the model will send the same results to a given prompt.
1 means that the model's results to a give
response can vary wildly. Use an event handler to apply a multiplier
if the LLM provider supports a range other than 0-1.
|
No |
user |
string | N/A | A unique identifier representing your end user, which can be used for monitoring and detecting abusive language. | No |
providerExtension |
object | N/A | Enables LLM provider-specific configuration options that are not defined as part of CLMI. | No |
The Message Object Structure
Property | Type | Description | Required? |
---|---|---|---|
role |
string | The message creator. The values are
system , user , and
assistant .
|
Yes |
content |
string | The message content | Yes |
turn |
integer | A number that indicates the current refinement turn of the chat messages exchange. When the first prompt is sent to the LLM, the turn is 1. | Yes |
retry |
boolean | A flag that indicates whether the message is sent to the LLM to correct an error in the response | No (defaults to false )
|
tag |
string | A custom tag that marks a specific prompt. If you're
improving the LLM response using Recursive Criticism and Improvement (RCI), you can
enable the custom logic in the
validateResponsePayload handler to detect the
current step of RCI process by setting the tag to
"criticize" or
"improve" .
|
No |
Success Response Body Specification
Property | Type | Description | Required? |
---|---|---|---|
candidates |
An array of candidate objects |
A list of candidate messages returned by the LLM | Yes |
Candidate Objects
Property | Type | Description | Required? |
---|---|---|---|
content |
string | The message content | Yes |
Error Response Body Specification
Property | Type | Description | Required? |
---|---|---|---|
errorCode |
String |
|
Yes |
errorMessage |
String | The LLM provider-specific error message. This might be a stringified JSON object. | Yes |