HNSW Distribution on Oracle RAC

Learn how Hierarchical Navigable Small World (HNSW) indexes can be distributed across multiple Oracle Real Application Clusters (Oracle RAC) instances.

A distributed HNSW index on Oracle RAC enables scalable similarity search across multiple instances in a RAC (Real Application Clusters) environment. In a distributed HNSW index:

• The vector data is segmented and distributed across RAC instances in a memory-efficient manner.

• Each instance builds and manages an HNSW slice graph on its assigned subset of the vector data. The complete HNSW index is the logical union of these HNSW slice graphs.

• During query execution, each instance searches its HNSW slice graph, and the partial results are merged to produce the final query result.

Why use a distributed HNSW index?

The following section summarize the advantages of using a distributed HNSW index.

• Horizontal Scalability

  Distributing vector data across multiple RAC instances in a memory-efficient manner enables the total available memory to scale with cluster size. HNSW indexes are no longer constrained by the memory or processing resources of a single instance. Each RAC instance independently builds and maintains its HNSW slice graph, allowing a single distributed HNSW index to scale with increasing data volume and vector dimensionality.

• Parallelization

  Enables parallel index operations and query execution across multiple RAC instances. Each RAC instance handles its own HNSW index maintenance and query processing, preventing a single instance from becoming a bottleneck, resulting in improved query throughput.

• Efficiency

  Eliminates redundant graph storage and build-time computations across instances. This significantly improves memory utilization and reduces overall index build and maintenance costs.

• Resilience and Flexibility

  Automatically adapts to cluster changes by redistributing the index data as needed. This improves fault isolation and supports cluster re-balancing when instances fail or are removed from the cluster. For more information, refer to About HNSW Index Availability and Performance During Cluster Node Failure

Usage Notes

• No Support for Parallel DML Operations

  Distributed HNSW index does not support parallel DML operations.

• No Support for Snapshots (Full Repopulation Required)

  Distributed HNSW indexes do not support snapshots. Full repopulation is the only method used to refresh a distributed HNSW index. See HNSW Index Architecture: Transaction Support and Persistence for more information about why and when HNSW index full repopulation operation is triggered.

• No Support for Covering Columns

  Including covering columns (extra columns) in the HNSW index is not supported. If your query needs columns other than the indexed vector column, Oracle AI Database automatically uses the base table while processing the query.

• Index Rebuild on Certain Cluster Changes

  If the cluster size decreases such that there are not enough available instances to host all HNSW slice graphs, Oracle rebuilds the index.

• No Support for Online Index Build

  Online creation of distributed HNSW indexes with concurrent DML operations is not supported.

• Index Unavailability During Cluster Reconfiguration

  When an instance fails or is removed, affected HNSW slice graphs are temporarily disabled. The index remains unavailable for queries until graphs are redistributed or rebuilt.

• Scalar Quantization Support

  Scalar quantization is supported with distributed HNSW indexes and can be specified during index creation. For more information, see Scalar Quantized HNSW Indexes.

Distributed HNSW Index: High-Level Workflow

• Index Creation with the DISTRIBUTE Clause

  You create a distributed HNSW index by specifying the DISTRIBUTE clause in the CREATE VECTOR INDEX statement. You can choose one of the following distribution strategies:

  • BY ROWID RANGE

  • BY PARTITION

  • BY SUBPARTITION

  If no distribution strategy is specified along with the DISTRIBUTE clause, the database defaults to DISTRIBUTE AUTO. In DISTRIBUTE AUTO mode, non-partitioned tables use ROWID RANGE distribution and partitioned tables use [SUB] PARTITION distribution.

  Distributed HNSW index creation syntax:

  CREATE VECTOR INDEX index_name ON table_name(column_name)
  ORGANIZATION INMEMORY NEIGHBOR GRAPH
  WITH <distance_and_target_metrics>
  [QUANTIZATION SCALAR COMPRESSION RATIO {2|4|8}]
  [PARAMETERS (TYPE HNSW ,
               {NEIGHBORS max_closest_vectors_connected | M max_closest_vectors_connected},
               EFCONSTRUCTION max_candidates_to_consider,
               RESCORE FACTOR rescore_factor,
               ALGORITHM UNIFORM_QUANTIZATION
              )]
  DISTRIBUTE [AUTO | BY ROWID RANGE | BY PARTITION | BY SUBPARTITION ];

  See Also: CREATE VECTOR INDEX

• RAC Instance Memory Assessment

  The coordinating instance initiates a memory check (a cross-instance coordination call) to estimate available vector pool memory across all participating RAC instances. Each participating instance reports its projected available memory. This information is used to assign vector data proportionally and efficiently across instances.

• Partitioning the Vector Dataset

  Oracle AI Database divides the vector dataset into logical units called vector distribution units using the selected distribution strategy. The mapping between distribution units and HNSW slice graphs is recorded in auxiliary metadata structures for maintenance and recovery.

• Initiation of Distributed Graph Build

  The coordinating instance triggers all participating instances (including itself) to start building their HNSW slice graphs for their assigned vector data. Graph construction is centrally coordinated and monitored to ensure all instances complete their HNSW slice graph construction successfully.

• Graph Construction on RAC

  Each RAC instance reads its assigned vector data from the base table and builds its HNSW slice graph. It then creates a checkpoint of the completed graph.

  The HNSW slice graph constructed on assigned vector data is categorized according to the distribution strategy used to partition the vector dataset:

  • HNSW rowid-range-slice graph: An HNSW graph built over a subset of vector data partitioned using the ROWID RANGE distribution strategy. In this context, “slice” refers to the subset of vector data, while “rowid-range” specifies the distribution strategy.

  • HNSW partition-slice graph: An HNSW graph built over a subset of vector data partitioned using the PARTITION or SUBPARTITION distribution strategy. In this context, “slice” refers to the subset of vector data, while “partition” specifies the distribution strategy.

• Index Finalization

  Once all HNSW slice graphs of the distributed HNSW index are built and checkpointed, the coordinating instance marks the index as usable. The index is now available for distributed, parallel similarity search queries, each routed efficiently to the relevant instances.

• Distributed Query Execution

  When a vector search query is issued, the database identifies all instances that host an HNSW slice graph. The query is executed in parallel across these instances.

  Each query performs an approximate nearest neighbor (ANN) search on its HNSW slice graph using the configured search parameters such as ef_search and k (number of neighbors).

  To ensure transactional consistency, each instance scans the shared journal for unindexed changes (committed inserts and deletes that are not yet indexed) and includes them in the similarity search.

  The query optimizer selects distributed index plans only when sufficient parallelism is available (at least one PQ process per index-hosting instance). If not, the database falls back to a no-index plan.

  For information on optimizer plans, refer to Optimizer Plans for HNSW Vector Indexes.

• Index Maintenance and Persistence

  • HNSW multiple checkpoints: If checkpoint creation is enabled, multiple checkpoints are created for faster recovery after instance failures. One checkpoint for each HNSW slice graph hosted by each RAC instance. For more information refer to Checkpointing .

  • Cluster reconfiguration: A leader instance monitors cluster changes such as instance failures, or removals to maintain index consistency by dynamically adapting to these changes. For more information refer to About HNSW Index Availability and Performance During Cluster instance Failure

HNSW RAC Duplication vs. HNSW Index Distribution

Duplicated vs. Distributed

HNSW RAC Duplication	HNSW Index Distribution
Data access Each instance reads the entire vector dataset to build the full HNSW graph.	Data access The vector dataset is split into vector distribution units (by ROWID RANGE, by [SUB]PARTITION). Each instance only reads a subset of vector data assigned to it.
Index structure Each RAC instance has an identical, full copy of the HNSW graph for the entire vector dataset.	Index structure Each instance holds only a portion (HNSW slice graph) of the full HNSW index.
Memory usage High: Every instance must hold the entire index in memory.	Memory usage Efficient: Memory load is split across instances, and each instance holds only a portion (HNSW slice graph) of the full HNSW index.
Index build process The coordinating RAC instance creates a disk checkpoint of the HNSW graph that it created and then all the other participating RAC instances use that disk checkpoint to load the graph. Every instance independently loads the entire HNSW graph using the full vector dataset.	Index build process Each instance builds an HNSW slice graph for a subset of vector data assigned to the instance.
Query execution Queries execute on a single instance (no parallelism across RAC instances).	Query execution Queries are distributed and parallelized across all instances holding HNSW slice graphs.
Resource utilization Poor: Redundant storage and build compute on all instances.	Resource utilization Good: Storage is not redundant and workload is balanced across all instances.
Scalability Limited by the memory and processing power of a single instance.	Scalability Scales with cluster size; can handle much larger datasets.
Cluster reconfiguration All instances must reload or rebuild full HNSW index if cluster changes.	Cluster reconfiguration Only affected HNSW slice graphs are redistributed or rebuilt.
Best use case Small datasets	Best use case Large or growing datasets.

• About Distributed HNSW Index Availability and Performance During Cluster instance Failure