S3-Compatible Object Storage Data Source

Connect S3-compatible object storage to Agent Factory so Knowledge Agents can retrieve and ground answers in bucket content.

S3-compatible object storage data sources let Agent Factory ingest documents from AWS S3, OCI Object Storage, Google Cloud Storage, and other providers with an S3-compatible API.

Configure an S3-Compatible Object Storage Data Source

Select Data Sources in the navigation menu.
Select Create new data source.
Select S3-compatible object storage source as the source type.
Enter a Source name and Description.
Select the storage Provider:
- AWS for Amazon S3.
- OCI for OCI Object Storage.
- GCS for Google Cloud Storage.
- Other for another S3-compatible provider.
Enter the required storage connection details:
- For AWS, enter the Bucket name, Region, Access key, and Secret access key.
- For OCI, enter the Bucket name, Region, Namespace, Access key, and Secret access key.
- For GCS, enter the Bucket name, Access key, and Secret access key.
- For Other, enter the Bucket name, Endpoint URL, Access key, and Secret access key. The endpoint URL is required for other providers.
Important: Use long-lived access key credentials that have permission to list the bucket and read the objects that you want to ingest. For Amazon S3, the IAM policy must allow s3:ListBucket and s3:GetObject on the required bucket.
To limit the crawl scope, select Add filter, then configure include or exclude URL filters. You can also exclude supported file extensions from ingestion.
Set Crawl depth and Crawl frequency to control the crawl scope and how often Agent Factory revisits the bucket.
Enter a Proxy URL if access to the storage service requires a network proxy. See Configure Proxy to verify whether the administrator has enabled proxy usage.
Select Test connection and confirm that Agent Factory can reach the bucket.

Note: If the connection test fails, verify the bucket name, provider details, credentials, endpoint URL when required, and proxy configuration.
Select Create data source.

After the data source is created, monitor the ingestion status on the Data Sources page. When the status is ingested, you can select the S3-compatible object storage data source while creating or editing a Knowledge Agent. See Create Knowledge Agent.

See Data Sources Troubleshooting for information about data source-related statuses and errors.

Security Note:

Keep access keys and secret access keys confidential. Do not include them in source names, descriptions, filters, screenshots, logs, or support requests. Treat pre-authenticated object URLs and other object-access metadata as sensitive.

Grant credentials only the least privileges required to list the selected bucket and read the objects or paths that you intend to ingest. Use only trusted and approved proxy endpoints because bucket content can transit through the proxy.

Review the content before ingestion and make sure it meets your organization’s privacy, security, and data-governance requirements. Administrators must protect downloaded content in local storage and apply appropriate retention and cleanup practices.

Control Access to Ingested Content

S3 bucket permissions control Agent Factory’s access while it crawls the bucket. After Agent Factory ingests the content, Knowledge Agent access controls determine which users can access it. A user who can access a Knowledge Agent can access content from its S3-compatible object storage data sources even when that user does not have direct permission to the source bucket.

Before you add a data source, confirm that its content is appropriate for all intended Knowledge Agent users. Administrators must review and limit Knowledge Agent access to authorized users and groups. See Manage Identity and Access.

Minimum Amazon S3 IAM Policy

For Amazon S3, use credentials with the minimum required permissions. The exact policy depends on your bucket and prefix strategy, but the credentials must be able to list the selected bucket and read selected objects.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::<bucket-name>"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": "arn:aws:s3:::<bucket-name>/<prefix-or-path>*"
    }
  ]
}

Troubleshoot S3-Compatible Object Storage Sources

Symptom	Check
Test connection fails with an authentication error	Verify the access key, secret access key, provider details, and access policy.
Test connection fails for another S3-compatible provider	Verify the endpoint URL and credentials.
Bucket is reachable but objects are missing	Verify the object prefix, include and exclude filters, file extension exclusions, and read permissions.
Connection times out	Verify outbound internet, NAT, or proxy access from the Agent Factory container.
Ingestion is too broad	Use include and exclude filters and file extension exclusions to narrow the crawl.