HNSW Distribution on Oracle RAC

A distributed HNSW index on an Oracle RAC enables scalable similarity search across multiple instances in a RAC (Real Application Clusters) environment. In a distributed HNSW index:

• The vector data is segmented and distributed across RAC instances in a memory-efficient manner.
• Each instance builds and manages an instance local graph on its assigned subset of the vector data. The complete HNSW index is the logical union of these localized graphs.
• During query execution, each instance searches its instance local graph, and the partial results are merged to produce the final query result.

Why use a distributed HNSW index?

The following section summarize the advantages of using a distributed HNSW index.

• Horizontal Scalability

  Distributing vector data across multiple RAC instances in a memory-efficient manner enables the total available memory to scale with cluster size. HNSW indexes are no longer constrained by the memory or processing resources of a single instance. Each RAC instance independently builds and maintains its instance local graphs, allowing a single distributed HNSW index to scale with increasing data volume and vector dimensionality.

• Parallelization

  Enables parallel index operations and query execution across multiple RAC instances. Each RAC instance handles its own HNSW index maintenance and query processing, preventing a single instance from becoming a bottleneck, resulting in improved query throughput.

• Efficiency

  Eliminates redundant graph storage and build-time computations across instances. This significantly improves memory utilization and reduces overall index build and maintenance costs.

• Resilience and Flexibility

  Automatically adapts to cluster changes by redistributing the index data as needed. This improves fault isolation and supports cluster re-balancing when instances fail or are removed from the cluster. For more information, refer to About HNSW Index Availability and Performance During Cluster Node Failure​

Usage Notes

• No Support for Parallel DML Operations

  Distributed HNSW index does not support parallel DML operations.

• No Support for Snapshots

  Distributed HNSW indexes do not support snapshots. Full repopulation is the only method used to refresh a distributed HNSW index. See HNSW Index Architecture: Transaction Support and Persistence for more information about why and when HNSW index full repopulation operation is triggered.

• Index Rebuild on Certain Cluster Changes

  If the cluster size decreases such that there are not enough available instances to host all instance local graphs, Oracle rebuilds the index.

• No Support for Online Index Build

  Online creation of distributed HNSW indexes with concurrent DML operations is not supported.

• Index Unavailability During Cluster Reconfiguration

  When an instance fails or is removed, affected instance local graphs are temporarily disabled. The index remains unavailable for queries until graphs are redistributed and rebuilt.

Distributed HNSW Index: High-Level Workflow

• Index Creation with the DISTRIBUTE Clause

  You create a distributed HNSW index by specifying the DISTRIBUTE clause in the CREATE VECTOR INDEX statement. You can choose one of the following distribution strategies:

  • BY ROWID RANGE
  • BY PARTITION
  • BY SUBPARTITION

  If no distribution strategy is specified along with the DISTRIBUTE clause, the Oracle defaults to DISTRIBUTE AUTO. In DISTRIBUTE AUTO mode, non-partitioned tables use ROWID RANGE distribution and partitioned tables use [SUB] PARTITION distribution.

  Distributed HNSW index creation syntax:

  CREATE VECTOR INDEX index_name ON table_name(column_name)
  ORGANIZATION INMEMORY NEIGHBOR GRAPH
  WITH <distance_and_target_metrics>
  DISTRIBUTE [AUTO | BY ROWID RANGE | BY PARTITION | BY SUBPARTITION | BY SIMILARITY];

  See Also:

  CREATE VECTOR INDEX

• RAC Instance Memory Assessment

  The coordinating instance initiates a memory check (a cross-instance coordination call) to estimate available vector pool memory across all participating RAC instances. Each participating instance reports its projected available memory. This information is used to assign vector data proportionally and efficiently across instances.

• Partitioning the Vector Dataset

  Oracle divides the vector dataset into logical units called vector distribution units using the selected distribution strategy. The mapping between distribution units, RAC instances, and HNSW graphs are recorded in auxiliary metadata structures for maintenance and recovery.

• Initiation of Distributed Graph Build

  The coordinating instance triggers all participating instances (including itself) to start building their instance local graphs for their assigned vector data. Graph construction is centrally coordinated and monitored to ensure all instances complete their instance local graph construction successfully.

• Graph Construction on RAC

  Each RAC instance reads its assigned vector data from the base table and builds its instance local graph. It then creates a checkpoint of the completed graph.

• Index Finalization

  Once all localized graphs are built and checkpointed, the coordinating instance marks the distributed HNSW index as usable. The index is now available for distributed, parallel similarity search queries, each routed efficiently to the relevant instances.

• Distributed Query Execution

  When a vector search query is issued, the Oracle identifies all instances that host an instance local graph. The query is executed in parallel across these instances.

  Each performs an approximate nearest neighbor (ANN) search on its localized graph using the configured search parameters such as ef_search and k (number of neighbors).

  To ensure transactional consistency, each instance scans the shared journal for unindexed changes (committed inserts and deletes that are not yet indexed) and includes them in the similarity search.

  The query optimizer selects distributed index plans only when sufficient parallelism is available (at least one PQ process per index-hosting instance). If not, Oracle falls back to a no-index plan.

  For information on optimizer plans, refer to Optimizer Plans for HNSW Vector Indexes.

• Index Maintenance and Persistence

  • HNSW multiple checkpoints: If checkpoint creation is enabled, multiple checkpoints are created for faster recovery after instance failures. One checkpoint for each instance local graph hosted by each RAC instance. For more information refer to Checkpointing .
  • Cluster reconfiguration: A leader instance monitors cluster changes such as instance failures, or removals to maintain index consistency by dynamically adapting to these changes. For more information refer to About HNSW Index Availability and Performance During Cluster instance Failure

HNSW RAC Duplication vs. HNSW Index Distribution

Table 6-1 Duplicated vs. Distributed

HNSW RAC Duplication	HNSW Index Distribution
Data access Each instance reads the entire vector dataset to build the full HNSW graph.	Data access The vector dataset is split into vector distribution units (by ROWID RANGE, by [SUB]PARTITION). Each instance only reads a subset of vector data assigned to it.
Index structure Each RAC instance has an identical, full copy of the HNSW graph for the entire vector dataset.	Index structure Memory load is split across instances. Each instance holds only a portion (localized graph) of the full HNSW index.
Memory usage High: Every instance must hold the entire index in memory.	Memory usage Efficient: Memory load is split across instances, and holds only a portion (localized graph) of the full HNSW index.
Index build process The coordinating RAC instance creates a disk checkpoint of the HNSW graph that it created and then all the other participating RAC instances use that disk checkpoint to load the graph. Every instance independently loads the entire HNSW graph using the full vector dataset.	Index build process Each instance build a HNSW graph for a subset of vector data assigned to the instance.
Query execution Queries execute on a single instance (no parallelism across RAC instances).	Query execution Queries are distributed and parallelized across all instances holding instance local graphs.
Resource utilization Poor: Redundant storage and build compute on all instances.	Resource utilization Good: Storage is not redundant and workload is balanced across all instances.
Scalability Limited by the memory and processing power of a single instance.	Scalability Scales with cluster size; can handle much larger datasets.
Cluster reconfiguration All instances must reload or rebuild full HNSW index if cluster changes.	Cluster reconfiguration Only affected instance local graphs are redistributed or rebuilt.
Best use case Small datasets	Best use case Large or growing datasets

• About Distributed HNSW Index Availability and Performance During Cluster instance Failure