Configure LLM

Large Language Models (LLMs) are used for tasks like text generation, summarization, question answering, and supporting MCP server tools.

Agent Factory supports multiple LLM connection types. Use the LLM Management screen to set up and manage both generative models and embedding models within the application. See LLM Preferences for the preferred order of choosing generative and embedding models.

Model availability changes by provider, region, tenancy, and endpoint. Treat model IDs in this page as examples or release-validated recommendations, not as a complete provider catalog. Always click Test connection before saving a model configuration and before using it in a workflow.

Supported Connection Types

Provider type Authentication modes Typical use
OCI Generative AI API key or instance principals Oracle Cloud-hosted generative and embedding models.
Anthropic API key Anthropic-hosted generative models.
Azure OpenAI API key Azure-hosted generative models.
OpenAI API key OpenAI-hosted generative models.
OpenAI Compatible Optional API key and endpoint URL OpenAI-compatible generative and embedding model endpoints, including OpenRouter.
Gemini API key or Google service account, with compatibility limits noted below Google-hosted generative and embedding models.
Private AI Services Optional API key, and optional Trusted CA file for HTTPS endpoints Oracle Private AI Services Container generative and embedding models.
vLLM Endpoint URL, port, optional API key depending on deployment Self-hosted OpenAI-compatible model endpoints.
Ollama Endpoint URL and port Local or self-hosted models for development and controlled environments.
Local embedding model No external provider credentials Bundled embedding model for local retrieval use cases.

Test Connection

The connection test validates that Agent Factory can reach the provider from the application container, authenticate, and run a small provider-specific request. A successful test does not guarantee that a model supports every advanced behavior, such as tool calling, structured output, or long context. Validate those behaviors in the target workflow.

To test a connection, go to the Model Management dashboard. Under the Actions column, click the horizontal ellipsis for the LLM you want to test, then select Test Connection.

Test LLM Connection

Description of the illustration llm-test-connection.png

Delete or Deactivate an LLM

Before deleting or disabling a generative model configuration, update every Knowledge Agent that uses it to select another configured generative model. A Knowledge Agent cannot generate responses while its associated model is unavailable. See Change the LLM for an Existing Knowledge Agent for instructions on changing the model association.

Note: If you delete or disable an embedding model, re-create any existing data sources that use that model.

To delete a generative model connection, go to the Model Management dashboard. Under the Actions column, click the horizontal ellipsis for the LLM you want to delete, then select Delete. If the model is used by a Knowledge Agent, Agent Factory warns you and lists the affected Knowledge Agents.

Delete LLM Warning

Description of the illustration delete-llm-warning.png

If no Knowledge Agents use the connection, the warning and affected-agent information are not displayed.

Delete LLM With No Warning

Generative Models

OCI Generative AI

Agent Factory supports On-demand and Dedicated serving modes for OCI Generative AI. See OCI GenAI Pre-trained Models for current provider availability.

On-demand

Use On-demand to configure a pre-trained OCI Generative AI model, such as xai.grok-4-fast-reasoning.

Prerequisites: Setup OCI Generative AI Service. See Getting Started with Generative AI.

Step 1: Click on Model Management on the left side navigation menu.

Click on Model Management

Step 2: Click on Add configuration button placed on the top-right corner.

Select Add Configuration

Step 3: A form will open, under Model type pick Generative model.

Choose Generative Model

Step 4: Give your LLM configuration a preferred name under Configuration name without whitespaces.

Configuration Name

Step 5: From list of LLM providers pick OCI GenAI.

LLM Providers

Step 6: For Serving mode, select On-demand.

Step 7: For Mode, choose API Key if you have a secret token and want to use it to access the LLM. See API Keys. Choose Instance principals otherwise.

With API Key

Step 8: In the Model ID field, enter xai.grok-4-fast-reasoning

Enter Model ID

Step 9: Fill Endpoint and Compartment ID.

Fill OCI credentials

Step 10: If you chose API Key mode in Step 7, fill User, Tenancy, Finger Print, and Region fields with your own credentials from OCI Generative AI Service. Upload your private API key file under Key File. You can request it from your tenancy owner or by logging into your tenancy with your user. See Set Up API Authentication for OCI.

Upload API Key File

Step 11: Click on Test connection to validate the credentials are correct.

Step 12: A success message Connection successful will appear on screen and the Save Configuration button will be enabled, click on it to finalize the process.

Save configuration

Dedicated

Use Dedicated to configure an OCI Generative AI dedicated cluster endpoint. Select Generative model, choose OCI GenAI as the LLM provider, select Dedicated as the serving mode, and enter a user-friendly configuration name.

For API Key authentication, enter the following values:

For Instance principals authentication, enter the model provider, dedicated endpoint ID, compartment ID, and OCI service endpoint, in the format https://inference.generativeai.<oci-region>.oci.oraclecloud.com.

Click Test connection and then Save Configuration after the connection succeeds.

Private AI Services

Use Private AI Services to connect Agent Factory to an Oracle Private AI Services Container endpoint. Choose this provider for Oracle Private AI Services Container deployments, even when the container exposes OpenAI-compatible inference APIs.

To configure a Private AI Services generative model, enter the model ID exposed by the container, the Base URL, and authentication details required by the endpoint. For a local HTTP container, the Base URL from the Agent Factory application container commonly uses http://host.containers.internal:<port>. For an HTTPS container, provide the endpoint API key and upload the Trusted CA file, such as the container-generated cert.pem.

Private AI Services as the Generative Model

Click Test connection before you save the configuration. Agent Factory validates that the endpoint is reachable, that the model ID is present in /v1/models, and that the model supports text generation.

Note: Private AI Services is separate from OpenAI Compatible. Use OpenAI Compatible for generic OpenAI-compatible APIs. Use Private AI Services for Oracle Private AI Services Container endpoints so Agent Factory can apply provider-specific validation and runtime handling.

OpenAI Compatible

Use OpenAI Compatible to configure generative models from compatible endpoints.

Generic

Use the Generic subtype for a generic OpenAI-compatible endpoint. You need a configuration name, model ID, and Base URL. Provide an API key when the endpoint requires authentication.

  1. Open LLM Management and click Add configuration.
  2. Select Generative model, then enter a user-friendly configuration name.
  3. Select OpenAI Compatible as the LLM provider.
  4. Select Generic, then enter the model ID, Base URL, and, if required, API key.
  5. Click Test connection. After the connection succeeds, click Save Configuration.

OpenRouter

Use the OpenRouter subtype to centralize access to available models and manage usage and costs through your OpenRouter account. You need an OpenRouter account, API key, and model ID.

  1. Open LLM Management and click Add configuration.
  2. Select Generative model, then enter a configuration name without spaces.
  3. Select OpenAI Compatible as the provider and OpenRouter as the subtype.
  4. Enter the API key and model ID.
  5. Click Test connection. After the connection succeeds, click Save Configuration.

Note: OpenRouter generative model configurations are not available for Knowledge Agents. Select another supported generative model for a Knowledge Agent.

You can select the saved configuration in an Agent node as the LLM source. OpenRouter model capabilities vary. Before using tools in a workflow, verify that the selected model supports tool calling.

Ollama

Agent Factory has support for Ollama so you can access the LLMs running locally on your machine.

Setup Ollama in Your Machine (Linux OCI VM)

Step 1: Open a terminal window in your system and start a new bash shell with root privileges.

sudo bash

Step 2 (Optional): If you are working behind a proxy please set the appropriate proxies.

Step 3: Download and install ollama as instructed by the official site.

Step 4: Pull the llama3.2 model to your local machine.

ollama pull llama3.2

Step 5: Edit the ollama service so Agent Factory’s container can access to it.

systemctl edit ollama

Step 6: Once the editor opens, paste the below two lines and close by using Ctrl + O, Enter, Ctrl + X

[Service]

Environment="OLLAMA_HOST=0.0.0.0:11434"

Step 7: Refresh systemd and restart ollama so the changes are visible.

systemctl daemon-reexec
systemctl daemon-reload
systemctl restart ollama

Step 8 (Optional): In a separate terminal verify ollama service is running by running the following command:

ollama run llama3.2

Adding Ollama Model to Agent Factory

Below are the steps to configure the locally hosted Llama 3.2 model from Ollama that was set up in the previous section.

Step 1: Click LLM management on the left side navigation menu.

Click on LLM Management

Step 2: Click Add configuration button placed on the top-right corner.

Select Add Configuration

Step 3: A form will open, under Model type choose Generative model.

Choose Generative Model

Step 4: Give your LLM configuration a preferred name under Configuration name, avoid whitespaces since they are not allowed.

Give your configuration a name

Step 5: From list of LLM providers pick Ollama.

Pick Ollama as provider

Step 6: Enter llama3.2 as Model ID.

Enter Model ID

Step 7: Enter http://host.containers.internal as URL.

Enter URL

Step 8: Enter 11434 as Port.

Enter URL

Step 9: Click on Test connection to validate the credentials are correct.

Test connection

Step 10: A success message “Connection successful” will appear on screen and the Save Configuration button will be enabled, click on it to finalize the process.

Save configuration

OpenAI

The following model IDs are commonly used examples for OpenAI configurations:

Adding OpenAI Model to Agent Factory

Step 1: Click on LLM management on the left side navigation menu.

Click on LLM Management

Step 2: Click on Add configuration button placed on the top-right corner.

Select Add Configuration

Step 3: A form will open, under Model type pick Generative model.

Choose Generative Model

Step 4: Give your LLM configuration a preferred name under Configuration name, avoid whitespaces since they are not allowed.

Add Configuration name

Step 5: From the list of LLM providers pick OpenAI.

Pick OpenAI

Step 6: Select gpt-4o as the Model ID

Select Model ID

Step 7: Enter your API Key

Enter your API key

Step 8: Click on Test connection to validate the credentials are correct.

Test the connection

Step 9: A success message “Connection successful” will appear on screen and the Save Configuration button will be enabled, click on it to finalize the process.

Save the configuration

Anthropic

Use Anthropic to configure an Anthropic-hosted generative model. You need an Anthropic API key and the model ID to configure the connection. For example, use claude-haiku-4-5 as the model ID when it is available in your Anthropic account.

  1. Open LLM Management and click Add configuration.
  2. Select Generative model, then enter a user-friendly configuration name.
  3. Select Anthropic as the LLM provider.

  4. Enter the following values:

    • Model ID: The Anthropic model identifier.
    • Base URL: Optional. The default is https://api.anthropic.com. Specify a different URL only for an Anthropic-compatible endpoint.
    • API key: The Anthropic API key for authenticating requests.

    Anthropic Configuration

    Description of the illustration anthropic-config.png

  5. Click Test connection to validate the endpoint, model ID, API key, and network access. After the connection succeeds, click Save Configuration.

Note: Anthropic models hosted through cloud providers, such as Amazon Bedrock, Microsoft Azure, or Google Vertex AI, are not supported. Configure an Anthropic-hosted model or an Anthropic-compatible endpoint instead.

Azure OpenAI

Use Azure OpenAI to configure an Azure-hosted generative model. You need an Azure OpenAI endpoint, deployment name, API key, and model ID. For example, use Phi-4-mini-instruct as the model ID when it is available in your Azure OpenAI deployment.

  1. Open LLM Management and click Add configuration.
  2. Select Generative model, then enter a user-friendly configuration name.
  3. Select Azure OpenAI as the LLM provider.

  4. Enter the following values:

    • Model ID: The logical identifier for the Azure-hosted model.
    • Azure endpoint: The Azure OpenAI-compatible endpoint, ending in /openai/v1.
    • Deployment name: The Azure deployment name that identifies the deployed model instance.
    • API key: The Azure API key for authenticating requests.

    Important: Agent Factory sends the deployment name as the model value in Azure OpenAI requests. Enter the Azure deployment name, not only the base model name.

    Azure OpenAI Configuration

    Description of the illustration azure-config.png

  5. Click Test connection to validate the endpoint, deployment name, model ID, API key, and network access. After the connection succeeds, click Save Configuration.

vLLM

You can connect to any self-hosted model endpoint.

These are the required fields you need to configure a vLLM:

  1. Model ID: The model identifier/path that the vLLM server is serving (often a filesystem path or a registry-style name).
  2. URL: Host/DNS clients use to reach the server.
  3. Port: The port where the HTTP service is exposed.

Gemini

The following Gemini model IDs are examples commonly used with Agent Factory when available in your Google project and region:

Adding Gemini Model to Agent Factory

You can configure Gemini LLM in Agent Factory using the following authentication methods:

Authentication Compatibility Notice: Knowledge Agent supports both Gemini authentication methods: Service Account (JSON) and API key authentication. However, Agent Builder and other prebuilt agents currently support only Gemini API key authentication. If you plan to use Agent Builder or any prebuilt agents, please ensure your Gemini configuration is created using the API key authentication method.

Configuring Gemini LLM (Google Service Account Authentication)

Prerequisites:

Before creating a Gemini configuration ensure:

See Google Cloud Service Account Setup Guide for detailed instructions on how to create and configure a Service Account.

Steps to Create a Gemini LLM Configuration Using Service Account Authentication:

Step 1: Go to Model Management from the left navigation menu.

Step 2: Click on Add configuration button placed on the top-right corner.

Add Configuration

Step 3: A form will open, under Model type pick Generative model.

Step 4: Give your LLM configuration a preferred name under Configuration name, avoid whitespaces since they are not allowed.

Step 5: From list of LLM providers pick Gemini.

Gemini Configuration

Step 6: Select from authentication method the Service account option.

Step 7: Enter the Model ID.

Recommended commonly used Gemini LLM models:

Step 8: Enter the Region.

Most commonly used regions:

Step 9: Upload the Google Service Account JSON file.

Upload the JSON file by:

Note: The Service Account must belong to a project where Vertex AI (Gemini) is enabled.

Gemini Service Account

Step 10: Click on Test connection to validate the credentials are correct and connection is possible. Wait a few seconds while a validation request is sent to the Gemini LLM.

Gemini Connection

Step 11: A success message Connection successful will appear on screen and the Save Configuration button will be enabled; click on it to finalize the process.

Once completed, the new configuration will appear in the LLM Configurations table.

Gemini Configuration

Configuring Gemini LLM (API Key Authentication)

Prerequisites:

Before configuring Gemini using API key authentication:

Ensure the API key is not restricted in a way that blocks requests from Agent Factory.

Steps to Create a Gemini LLM Configuration Using API Key Authentication:

Step 1: Go to Model Management from the left navigation menu.

Step 2: Click on Add configuration button placed on the top-right corner.

Step 3: A form will open, under Model type pick Generative model.

Step 4: Give your LLM configuration a preferred name under Configuration name, avoid whitespaces since they are not allowed.

Step 5: From list of LLM providers pick Gemini.

Gemini Configuration

Step 6: Select from authentication method the API Key option.

Step 7: Enter the Model ID
Recommended commonly used Gemini LLM models:

Step 8: Paste your Gemini API key into the field provided.

Gemini API Key

Step 9: Click on Test connection to validate the credentials are correct and connection is possible.
Wait a few seconds while a validation request is sent to the Gemini LLM.

Gemini Connection

Step 10: A success message Connection successful will appear on screen and the Save Configuration button will be enabled, click on it to finalize the process.
Once completed, the new configuration will appear in the LLM Configurations table.

Embedding Models

Agent Factory supports the following embedding models. Use these models for transforming text into numerical vectors, enabling semantic search, and retrieval-augmented generation (RAG). Agent Factory include out-of-the-box support for several high-performing embedding models, while also allowing you to bring your preferred models hosted on the OCI Generative AI service, or served via Ollama or vLLM endpoints.

Local Models

The following pre-trained sentence transformer model is bundled with the application and run locally.

Configure a Local Embedding Model

Below are the steps to configure a local embedding model multilingual-e5-base, which is available in Agent Factory out of the box.

Note: If you plan to use local embedding models, ensure the machine running the application has access to GPUs. Otherwise, embedding-related processes—such as Knowledge Agent ingestion—may take a significant amount of time.

Step 1: Click on LLM management on the left side navigation menu.

Click on LLM Management

Step 2: Click on Add configuration button placed on the top-right corner.

Select Add Configuration

Step 3: A form will open, under Model type choose Embedding model.

Pick Embedding Model

Step 4: Give your configuration a name, avoid using whitespaces since they are not allowed.

Give your config a name

Step 5: For Embedding provider pick Local.

Pick local

Step 6: Click on Model ID and pick multilingual-e5-base from the list.

Select a Model ID

Step 7: Verify the connection by clicking on Test connection.

Select a Model ID

Step 8: Save the new configuration.

Select a Model ID

OCI Generative AI

The following Cohere embedding model IDs from OCI Generative AI are commonly used examples:

Configure an OCI Generative AI embedding model

Below are the steps to configure a local embedding model cohere.embed-v4.0 which is available through OCI Generative AI service using a Fingerprint based authentication.

Step 1: Click on LLM management on the left side navigation menu.

Click on LLM Management

Step 2: Click on Add configuration button placed on the top-right corner.

Select Add Configuration

Step 3: A form will open, under Model type pick Embedding model.

Pick Model Type

Step 4: Give your configuration a name, avoid using whitespaces since they are not allowed.

Enter Configuration Name

Step 5: Pick OCI GenAI as Embedding provider.

Step 6: For Mode, choose API Key if you have a secret token and want to use it to access the LLM. See API Keys. Choose Instance principals otherwise.

Choose API Key

Step 7: Enter cohere.embed-v4.0 as Model ID.

Enter Model ID

Step 8: Fill in Endpoint and Compartment ID.

Fill OCI credentials

Step 9: If you chose API Key mode in Step 6, fill User, Tenancy, Finger Print, and Region fields with your own credentials from OCI Generative AI Service. Upload your private API key file under Key File. You can request it from your tenancy owner or by logging into your tenancy with your user. See Set Up API Authentication for OCI.

Upload API Key File

Step 10: Click on Test connection to validate the credentials are correct.

Step 11: A success message “Connection successful” will appear on screen and the Save Configuration button will be enabled, click on it to finalize the process.

Save configuration

Private AI Services

Use Private AI Services to configure an embedding model served by an Oracle Private AI Services Container. The selected container model must expose an embedding capability.

To configure a Private AI Services embedding model, enter the model ID exposed by the container, the Base URL, and authentication details required by the endpoint. For HTTPS endpoints, provide the endpoint API key and upload the Trusted CA file. For local unauthenticated HTTP containers, the API key and Trusted CA file can be blank.

Private AI Services as the Embedding Model

Click Test connection before you save the configuration. Agent Factory validates that the endpoint is reachable, that the model ID is present in /v1/models, and that the selected model is embedding-capable. If the selected model supports only text generation, the connection test fails.

OpenAI Compatible

Use OpenAI Compatible to configure embedding models from compatible endpoints.

Generic

Use the Generic subtype for a generic OpenAI-compatible endpoint. You need a configuration name, model ID, and Base URL. Provide an API key when the endpoint requires authentication.

  1. Open LLM Management and click Add configuration.
  2. Select Embedding model, then enter a user-friendly configuration name.
  3. Select OpenAI Compatible as the embedding provider.
  4. Select Generic, then enter the model ID, Base URL, and, if required, API key.
  5. Click Test connection. After the connection succeeds, click Save Configuration.

The connection test confirms that the endpoint is reachable and that the selected model supports embeddings.

OpenRouter

Use the OpenRouter subtype to configure an OpenRouter embedding model. You need an OpenRouter account, API key, and embedding-capable model ID.

  1. Open LLM Management and click Add configuration.
  2. Select Embedding model, then enter a configuration name without spaces.
  3. Select OpenAI Compatible as the embedding provider and OpenRouter as the subtype.
  4. Enter the API key and embedding-capable model ID.
  5. Click Test connection. After the connection succeeds, click Save Configuration.

vLLM/Ollama

You can connect to any self-hosted model endpoint.

These are the required fields you need to configure a vLLM:

  1. Model ID: The model identifier/path that the vLLM server is serving (often a filesystem path or a registry-style name).
  2. URL: Host/DNS clients use to reach the server.
  3. Port: The port where the HTTP service is exposed.

Gemini

The following Gemini embedding model IDs are examples commonly used with Agent Factory when available in your Google project and region:

Adding Gemini Embedding Model to Agent Factory

You can configure Gemini Embedding Model in Agent Factory using the following authentication methods:

Authentication Compatibility Notice: Knowledge Agent supports both Gemini authentication methods: Service Account (JSON) and API key authentication. However, Agent Builder and other prebuilt agents currently support only Gemini API key authentication. If you plan to use Agent Builder or any prebuilt agents, please ensure your Gemini configuration is created using the API key authentication method.

Configuring Gemini Embedding Model (Google Service Account Authentication)

Prerequisites:

Before creating a Gemini configuration ensure:

See Google Cloud Service Account Setup Guide for detailed instructions on how to create and configure a Service Account.

Steps to Create a Gemini Embedding Configuration Using Service Account Authentication:

Step 1: Go to Model Management from the left navigation menu.

Step 2: Click on Add configuration button placed on the top-right corner.

Step 3: A form will open, under Model type pick Embedding model.

Step 4: Give your LLM configuration a preferred name under Configuration name, avoid whitespaces since they are not allowed.

Step 5: From list of LLM providers pick Gemini.

Gemini Embedding Model Configuration

Step 6: Select from authentication method the Service account option.

Step 7: Enter the Model ID.

Recommended Gemini embedding models:

Step 8: Enter the Region.

Most commonly used regions:

Step 9: Upload the Google Service Account JSON file.

Upload the JSON file by:

Note: The Service Account must belong to a project where Vertex AI (Gemini) is enabled.

Gemini Embedding Service Account

Step 10: Click on Test connection to validate the credentials are correct and connection is possible. Wait a few seconds while a validation request is sent to the Gemini LLM.

Gemini Embedding Model Configuration

Step 11: A success message Connection successful will appear on screen and the Save Configuration button will be enabled; click on it to finalize the process.

Gemini Embedding Model Table

Once completed, the new configuration will appear in the LLM Configurations table.

Configuring Gemini Embedding Model (API Key Authentication)

Prerequisites:

Before configuring Gemini using API key authentication:

Ensure the API key is not restricted in a way that blocks requests from Agent Factory.

Steps to Create a Gemini Embedding Configuration Using API Key Authentication:

Step 1: Go to Model Management from the left navigation menu.

Step 2: Click on Add configuration button placed on the top-right corner.

Step 3: A form will open, under Model type pick Embedding model.

Step 4: Give your LLM configuration a preferred name under Configuration name, avoid whitespaces since they are not allowed.

Step 5: From list of LLM providers pick Gemini.

Gemini Embedding Model Configuration

Step 6: Select from authentication method the API Key option.

Step 7: Enter the Model ID
Recommended Gemini embedding models:

Step 8: Paste your Gemini API key into the field provided.

Gemini Embedding Model Configuration

Step 9: Click on Test connection to validate the credentials are correct and request runs embedding.
Wait a few seconds while embedding validation request runs.

Gemini Embedding Model Configuration

Step 10: A success message Connection successful will appear on screen and the Save Configuration button will be enabled, click on it to finalize the process.
Once completed, the new configuration will appear in the Embedding Configurations table.

Gemini Embedding Model Table

Share a Model Configuration

You can share a generative or embedding model configuration that you own or for which you have the Can manage access level.

  1. Open Model Management.
  2. To share a generative model, select Share from the Actions menu for the model. To share an embedding model, open the Embedding models tab and select Share from its Actions menu.
  3. In the Share dialog, review the users and groups that have access and their current access levels.
  4. Search for and select users or groups to add, change an existing access level, or remove access. Click Save changes to apply the changes.

The Share dialog identifies the resource owner as Owner. You cannot change or remove the owner’s access. See Manage Identity and Access for access-level definitions.