55 Chat API

The chat API enables interactive querying and contextual response generation by automatically augmenting LLM prompt with the results of a vector search.

POST /api/kb/<storeName>/chat

{
  "chatModel":    "OpenAI/gpt-4o-2024-08-06",
  "question":     "How can I implement PortableType in Java?",
  "maxResults":   10,
  "minScore":     0.7,
 "fullTextWeight": 0.5,
  "scoringModel": "-/ms-marco-TinyBERT-L-2-v2"
}

The chatModel attribute is optional. If not specified, the default/configured chat model for the store is used.

The scoringModel, minScore, and fullTextWeight attributes are also optional, and behave the same way as with the /search endpoint above.

text

To implement PortableType in Java, you need to follow several steps, including annotating your classes, configuring your build tool, and setting up the POF (Portable Object Format) context. Here's a step-by-step guide based on the information provided:

Step 1: Annotate Your Class with @PortableType:

You need to annotate your classes with @PortableType to indicate that they are portable types. Assign a unique id to each class.

@PortableType(id = 1)
   public class Pet {
       @Portable
       protected String name;

       // constructors, accessors, etc.
   }

   @PortableType(id = 2)
   public class Dog extends Pet {
       @Portable
       private String breed;

       // constructors, accessors, etc.
   }

Use @Portable for fields that should be serialized. There are other annotations like @PortableDate, @PortableArray, etc., for specific types.

Step 2: Instrument the Classes at Compile-time

Configure Maven or Gradle Plugin:

You need to use either the Maven or Gradle POF plugin to instrument these classes at compile time. This step is crucial because it generates the necessary serialization code.

Add the following configuration to your pom.xml:

<plugin>
     <groupId>com.oracle.coherence.ce</groupId>
     <artifactId>pof-maven-plugin</artifactId>
     <version>26.1.0-0-0</version>
     <executions>
       <execution>
         <id>instrument</id>
         <goals>
           <goal>instrument</goal>
         </goals>
       </execution>
       <execution>
         <id>instrument-tests</id>
         <goals>
           <goal>instrument-tests</goal>
         </goals>
       </execution>
     </executions>
   </plugin>

This plugin handles the instrumentation process during the build.

The response tokens are streamed to the client as soon as they are received from the chat model, and can be handled immediately (better for responsiveness), or processed as a whole after all the tokens are received by the client.

Chat API and AI

The chat model integrates document retrieval with conversational AI to provide context-aware responses. Unlike embedding and scoring models, the chat model only supports remote models, including OCI GenAI models, OpenAI, Cohere, Anthropic, AWS Bedrock, and other LangChain4J-supported providers. This ensures access to high-quality, state-of-the-art conversational AI capabilities.

The name of the chat model is specified using the same <provider>/<modelName> syntax described in Embedding Model.

In the example earlier in this topic we have used OpenAI/gpt-4o-2024-08-06. To use a different model, such as OCI/cohere.command-r-08-2024, for example, change the chatModel property in the request payload:

{
  "chatModel":    "OCI/cohere.command-r-08-2024",
  "question":     "How can I implement PortableType in Java?",
  "maxResults":   10,
  "minScore":     0.7,
  "scoringModel": "-/ms-marco-TinyBERT-L-2-v2"
}

Search & Chat Across Document Stores

Users can query multiple document stores simultaneously, enabling cross-repository search capabilities.

To do so, omit the store name from the search and chat endpoints described earlier. The request payload and the response are exactly the same.

POST /api/kb/search
POST /api/kb/chat

Chat context and protected data

Supply only the retrieved content the caller is authorized to access. Treat document text and model output as application inputs, including tool requests and generated links. Configure provider identity and secrets through deployment configuration, and establish request identity explicitly across asynchronous model calls.