Oracle Cloud Infrastructure Documentation

OCI Responses API Compaction Tutorial

Use the OpenAI Python SDK with the OCI Responses API to compact conversation history explicitly or automatically, then continue the conversation with the returned context.

Overview

As a conversation grows, an application sends more history with each request. Compaction reduces that history to a smaller representation that the model can use in later turns. Use it to manage context in long-running conversations and agent workflows.

In this tutorial, you describe a fictional deployment configuration, compact the conversation, and ask a follow-up question to check which details the model retained.

You use the OpenAI Python SDK with an OCI Generative AI API key and a regional OCI endpoint. The SDK provides the client interface. Requests authenticate with OCI and are sent to OCI Generative AI.

All models supported by the OCI Responses API, including imported models, support both explicit and automatic compaction. Compaction is available in Enterprise AI regions and follows each model's region and hosting availability. Both modes are supported for on-demand inference and dedicated hosting.

Compaction Modes
ModeOperationWhat to Send Next
Explicit (standalone) Call client.responses.compact(...), which sends POST /responses/compact, when an application is ready to compact history. Pass the complete returned output array, followed by the next user message.
Automatic (server-managed) Call client.responses.create(...) with context_management and a token threshold. OCI compacts context during POST /responses when the threshold is reached. With store=False, pass the latest compaction item and all the next output items, followed by the next user message. If no compaction item is returned, retain the existing history and append the new output.

Compaction items contain opaque, encrypted context. Preserve their serialized fields and order. Don't read, edit, or replace their contents with a text summary.

Note

Compaction consumes model tokens and can lose details during summarization. Compact before reaching the model's context limit, and leave room for the response and later turns. Explicit compaction is synchronous and doesn't support storage, background execution, or streaming.

The examples manage history in the application and set store=False on response-creation requests. For the separate workflow that uses OCI-managed conversation state, see Use Agent Memory with Responses API.

Before You Begin

To complete this tutorial, you need:

The examples use the OpenAI Python SDK with OCI credentials and an OCI base URL. They don't require an OpenAI API key, the OCI Python SDK, or the OCI CLI.

These examples use OCI Generative AI API key authentication. Compaction also supports OCI IAM authentication. For that setup, see Generative AI IAM-Based Authentication. An OCI Generative AI project OCID is required with either authentication method.

1. Prepare the Python Environment and OCI Access

  1. In a working directory, create and activate a virtual environment, then install the OpenAI Python SDK: 
    python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade 'openai>=2.45.0'

    If .venv already exists, activate it and run the installation command.

  2. Set an OCI base URL, model ID, and project OCID: 
    export OCI_BASE_URL='https://inference.generativeai.<REGION>.oci.oraclecloud.com/openai/v1'
export OCI_MODEL='<MODEL_ID>'
export OCI_OPENAI_PROJECT='<PROJECT_OCID>'

    Replace the placeholders with their values:

    • <REGION>: A supported OCI region identifier, such as us-chicago-1. The base URL must end in /openai/v1. Don't append /responses.
    • <MODEL_ID>: The ID of an on-demand model that supports the OCI Responses API and is available to your tenancy in the selected region. Use the same model throughout each example. To use a model hosted on a dedicated AI cluster, see Use a Model Hosted on a Dedicated AI Cluster.
    • <PROJECT_OCID>: Your OCI Generative AI project OCID. For guidance, see the Quick Start Guide.

    Each script passes the project OCID through OpenAI(..., project=...). The SDK adds the OpenAI-Project request header automatically. This value identifies an OCI project.

  3. Have your OCI Generative AI API key ready.
    Each script reads OCI_GENAI_API_KEY if it's set by your secret manager or another secure mechanism. Otherwise, it prompts for the key without displaying it. Don't add the key to the scripts or commit it to a repository.

2. Compact a Conversation Explicitly

This example creates a response, adds the complete response output to the history, and calls responses.compact. It then replaces the history with the complete compacted output and asks about the deployment's backup retention and maintenance window.

The short conversation makes the sequence easy to follow. In an application, call compaction when the accumulated history needs to be reduced and still fits within the model's context limit.

  1. Save the following code as standalone_compaction.py: 
    """Compact a conversation with the OCI Responses API."""

import os
from getpass import getpass

from openai import OpenAI


def main() -> None:
    model = os.environ["OCI_MODEL"]
    api_key = os.environ.get("OCI_GENAI_API_KEY") or getpass(
        "OCI Generative AI API key: "
    )
    client = OpenAI(
        api_key=api_key,
        base_url=os.environ["OCI_BASE_URL"],
        project=os.environ["OCI_OPENAI_PROJECT"],
    )
    history = [{
        "role": "user",
        "content": (
            "Remember our deployment configuration: Frankfurt region, three "
            "application instances, PostgreSQL, daily backups with 30-day "
            "retention, and maintenance on Sunday at 02:00 UTC. "
            "Summarize the configuration."
        ),
    }]

    response = client.responses.create(model=model, input=history, store=False)
    print(response.output_text)
    history.extend(
        item.model_dump(mode="json", exclude_none=True)
        for item in response.output
    )

    compacted = client.responses.compact(model=model, input=history)
    print("Compaction observed:", any(
        item.type == "compaction" for item in compacted.output
    ))

    # Keep the complete compacted output, including any retained messages.
    history = [
        item.model_dump(mode="json", exclude_none=True)
        for item in compacted.output
    ]
    history.append({
        "role": "user",
        "content": "What are our backup retention and maintenance window?",
    })
    response = client.responses.create(model=model, input=history, store=False)
    print(response.output_text)
    client.close()


if __name__ == "__main__":
    main()
  2. With the virtual environment activated, run the script: 
    python standalone_compaction.py
  3. Review the output and the follow-up answer.

    Compaction observed: True indicates that a compaction item was returned. Check that the follow-up answer retains the 30-day backup retention and Sunday 02:00 UTC maintenance window. The exact wording can vary.

    The script sends all items in compacted.output to the next request. Don't keep only the item whose type is compaction: the returned array can contain other context needed for continuation.

3. Enable Automatic Compaction

For automatic compaction, include context_management in a response-creation request with type: "compaction" and an explicit compact_threshold of at least 1000 tokens. Select a threshold that's less than the model's context limit and leave room for output and later turns.

This example creates 40 synthetic status exchanges locally to build a longer history. It sends that history in one request with a 1,000-token threshold, checks for compaction, and sends a follow-up request. The 40 exchanges don't make separate API calls.

  1. Save the following code as server_compaction.py: 
    """Use automatic compaction with the OCI Responses API."""

import os
from getpass import getpass

from openai import OpenAI


def main() -> None:
    model = os.environ["OCI_MODEL"]
    api_key = os.environ.get("OCI_GENAI_API_KEY") or getpass(
        "OCI Generative AI API key: "
    )
    client = OpenAI(
        api_key=api_key,
        base_url=os.environ["OCI_BASE_URL"],
        project=os.environ["OCI_OPENAI_PROJECT"],
    )
    context_management = [{"type": "compaction", "compact_threshold": 1000}]
    history = [{
        "role": "user",
        "content": (
            "Remember our deployment configuration: Frankfurt region, three "
            "application instances, PostgreSQL, daily backups with 30-day "
            "retention, and maintenance on Sunday at 02:00 UTC."
        ),
    }]

    # Build synthetic history locally. These exchanges don't call the API.
    for day in range(1, 41):
        history.extend([
            {
                "role": "user",
                "content": (
                    f"Day {day}: health checks passed, database connections "
                    "are stable, and the daily backup completed. "
                    "Keep the deployment configuration unchanged."
                ),
            },
            {
                "role": "assistant",
                "content": [{
                    "type": "output_text",
                    "text": (
                        f"Day {day} status recorded. "
                        "The deployment configuration remains unchanged."
                    ),
                }],
            },
        ])
    history.append({
        "role": "user",
        "content": "Summarize our deployment configuration.",
    })

    response = client.responses.create(
        model=model,
        input=history,
        store=False,
        extra_body={"context_management": context_management},
    )
    print(response.output_text)
    output = [
        item.model_dump(mode="json", exclude_none=True)
        for item in response.output
    ]
    boundaries = [
        index for index, item in enumerate(output)
        if item["type"] == "compaction"
    ]
    print("Compaction observed:", bool(boundaries))

    # Keep the latest compaction item and everything after it.
    # Without a new compaction item, retain history and append all output.
    history = output[boundaries[-1]:] if boundaries else history + output
    history.append({
        "role": "user",
        "content": "What are our backup retention and maintenance window?",
    })
    response = client.responses.create(
        model=model,
        input=history,
        store=False,
        extra_body={"context_management": context_management},
    )
    print(response.output_text)
    client.close()


if __name__ == "__main__":
    main()
  2. Run the script: 
    python server_compaction.py
  3. Check whether compaction occurred and review the follow-up answer.

    The script prints Compaction observed: True when the first response includes a compaction item. Check that the follow-up retains the backup retention and maintenance window.

    A value of False means that no compaction item was returned. The script preserves the existing history and appends the full output so it can continue without dropping context. See Troubleshooting if you expected compaction.

    The extra_body argument adds context_management to the JSON request. With store=False, repeat the same history-handling logic after each response when extending this example to more turns.

4. Use a Model Hosted on a Dedicated AI Cluster (Optional)

You can use models that support the OCI Responses API for inference on demand or host them on dedicated AI clusters, according to each model's hosting options. Both compaction modes work with either hosting option. The earlier examples use an on-demand model ID. When you host a model on a dedicated AI cluster, you create a model endpoint on that cluster to call the model. Pass that endpoint's OCID as the request's model value.

The hosting cluster provides the Compute resources for inference. You can create several endpoints for the same hosted model, for example, to give different applications, teams, or cost centers their own access points. Each endpoint has its own OCID, and the endpoints share the cluster's Compute resources. For more information, see Managing Model Endpoints.

  1. Select a model that supports the OCI Responses API and is available for dedicated hosting in the region that you want to work in.

    Check Supported Models and Regions for API compatibility, then check the model's hosting options:

    • Foundational models: OpenAI gpt-oss-20b and OpenAI gpt-oss-120b support both on-demand inference and dedicated hosting in US Midwest (Chicago), Germany Central (Frankfurt), and Japan Central (Osaka). For availability in other regions, see Models by Region.
    • Imported models: Supported imported models include Qwen/Qwen3.6-35B-A3B and google/gemma-4-31B-it. These models require Model Import and an endpoint on a hosting dedicated AI cluster. They aren't available on demand. Follow the model links for supported hardware shapes and deployment requirements.

    Both explicit and automatic compaction are supported by these foundational and imported models. Use either example with the model endpoint that you create.

  2. Create or reuse a model endpoint on a compatible hosting dedicated AI cluster.

    If you need a hosting cluster, follow Creating a Dedicated AI Cluster for Hosting Models. After the cluster becomes active, follow Creating an Endpoint to select the model and hosting cluster. Wait for the model endpoint to become active.

  3. Copy the model endpoint's OCID from its details page.
    If the cluster has several endpoints, select the endpoint that you want this application to use. See Getting an Endpoint's Details.
  4. Set the OCI base URL of the endpoint's region, and update the model and project variables: 
    export OCI_BASE_URL='https://inference.generativeai.<REGION>.oci.oraclecloud.com/openai/v1'
export OCI_MODEL='<INFERENCE_ENDPOINT_OCID>'
export OCI_OPENAI_PROJECT='<PROJECT_OCID>'

    Replace <INFERENCE_ENDPOINT_OCID> with the model endpoint OCID that you copied. The script passes this value as model. Set <REGION> to the endpoint's region and <PROJECT_OCID> to the OCI Generative AI project OCID. The client continues to use the regional OCI Responses API base URL. Use an OCI Generative AI API key authorized to access the endpoint and project.

  5. Run the explicit or automatic compaction example:

    For explicit compaction, run:

    python standalone_compaction.py

    For automatic compaction, run:

    python server_compaction.py

    Both scripts use the inference endpoint OCID as the model for all requests. Review the returned compaction indicator and the follow-up answer as described in the explicit compaction or automatic compaction example.

Check the Results

A returned compaction item confirms that compaction occurred. It doesn't establish that every detail was preserved. Review the follow-up answer against the original configuration.

To evaluate compaction for an application, use representative conversations with facts and instructions that matter to the workflow. Compare the same follow-up question with the full history, with explicit compaction, and with automatic compaction. Check retained facts, instruction following, output format, and any tool calls the workflow requires.

Run several examples and adjust the compaction threshold based on what you observe. Compaction itself uses model tokens, so a smaller context doesn't guarantee less total token usage or faster responses for every request.

Troubleshooting

SymptomWhat to Check
A script reports a missing environment variable.Set OCI_BASE_URL, OCI_MODEL, and OCI_OPENAI_PROJECT in the shell where you run the script.
401 or 403Confirm that you're using an OCI Generative AI API key and the correct regional OCI endpoint. Check the key's validity, project permissions, and access to the model or inference endpoint.
404 for /responses/compactConfirm that the base URL ends in /openai/v1. Check that the model ID or inference endpoint OCID is correct and available in the selected region.
A request reports missing project context.Set OCI_OPENAI_PROJECT to the OCI Generative AI project OCID and pass it through project= when creating the SDK client. Project context is required for every model.
Automatic compaction doesn't return a compaction item.Confirm that the request includes context_management with type: "compaction" and a threshold of at least 1000 tokens. Use a longer history or reduce a larger threshold while keeping it at or more than 1,000.
The SDK has no responses.compact method.Activate the virtual environment and install openai>=2.45.0. Check the installed version with python -m pip show openai.
A follow-up answer loses expected details.For explicit compaction, pass the complete returned output array. For automatic compaction, keep the latest compaction item and all later output items. If no compaction item is returned, retain the existing history and new output. Compaction can lose details, so also evaluate the model and threshold with representative conversations.