Working with Chunking

Chunking enables the system to divide large text into smaller segments before generating vector embedding for Semantic Search. When enabled, the search engine processes the text in chunks and generates embedding for those chunks during indexing.

This improves semantic search coverage for indexes that contain large text values, such as long descriptions, discussions, notes, or other detailed text fields. However, chunking can increase indexing time, CPU and memory usage, and index storage because the system generates and stores additional vector embedding.

When chunking is disabled, the system generates embedding without dividing the text into chunks. If the indexed text exceeds the supported processing limit, only part of the text may be used for embedding generation.

By default, chunking is disabled. Enable chunking only for indexes that contain large text fields selected for vector embedding.

Note:

For a deployed index, the Enable Chunking option is disabled. To modify this setting, you must update the value and redeploy the search definition.

You use the Advanced Index Settings page to enable chunking for individual indexes.

Complete these steps to enable chunking for an index:

1. Navigate to PeopleTools, and then Search Framework, and then Search Admin Activity Guide and from Administration in the left panel, select Deploy/Delete Object.
2. On the Deploy Search Definition page, select the Index Settings link for the desired index.
3. On the Advanced Index Settings page, select the Enable Chunking check box.
4. Select OK to save your setting.

This example illustrates the Advanced Index Settings page where you enable chunking for an index.