Resiliency and Congestion Management for AI Workloads Using the OCCL-NET Plugin

Before using the plugin, the following configuration is required:

• Each participating host needs the same compatible plugin build, an NCCL runtime that supports the exported NET-plugin ABI, and access to the InfiniBand or RoCE devices selected for the job. The plugin exports ABI versions 6 through 12.
• Use an absolute path to the library. Ensure the process can load the plugin and its runtime dependencies; for example, install it in a system library path or add its directory to LD_LIBRARY_PATH.
• The library embeds an occl-net-ib version string. Release tooling sets it with OCCL_NET_IB_PLUGIN_VERSION which follows the Major.Minor.Patch format.
• Record that version with every production deployment and incident report.
• Basic port/link failover and recovery requires NCCL 2.20.3 or later.
• Full NIC failover and recovery requires NCCL 2.24.3 or later.

To extract the embedded version from a plugin, run:

readelf -p .occl_net_ib_version "/path/to/liboccl-net-occl-ib.so"

If readelf is unavailable, use strings:

strings "/path/to/liboccl-net-occl-ib.so" | grep -F 'occl-net-ib version '

Replace the example path with the plugin library you will load. Look for occl-net-ib version <version> in the output. This is the OCCL-NET plugin version, not the application's NCCL runtime version.

Set NCCL_NET_PLUGIN to the complete path of the shared object. Use NCCL_NET=IB when the job must select the IB network transport explicitly.

export NCCL_NET_PLUGIN=/opt/occl-net/lib/liboccl-net-occl-ib.so
export NCCL_NET=IB
                            
mpirun -np 16 ./your_application

For a launcher that propagates environment variables explicitly:

mpirun -np 16 \
  -x NCCL_NET_PLUGIN=/opt/occl-net/lib/liboccl-net-occl-ib.so \
  -x NCCL_NET=IB \
  ./your_application

Set the same values on every rank. Do not mix plugin builds or incompatible NCCL runtimes within one job.

Enable NET and INIT information for one diagnostic run:

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,NET

The NCCL host runtime, not the shared object by itself, prints the authoritative plugin-load records. With the settings above, expect these records, with the configured library path substituted for <path>:

NCCL_NET_PLUGIN set by environment to <path>
Successfully loaded external network plugin <path>
Initialized NET plugin IB
Assigned NET plugin IB to comm

The second record proves that the runtime opened the library and found a supported exported NET-plugin ABI. Initialized proves that plugin initialization and device enumeration succeeded; Assigned proves that the IB plugin was selected for that communicator. The plugin then prints one or more device records in this form:

NET/IB: [<index>] <device>:<port>/<link-layer> provider=<provider>
    speed=<speed> context=<pointer> pciPath=<path> ar=<0|1>

Treat all four runtime records plus at least one NET/IB device record as a successful load-and-selection confirmation. A useful minimum capture is stdout/stderr from every rank plus the plugin path, NCCL version, host name, and selected NIC list.

For per-request and completion tracing, build the plugin with TRACE=1, then use:

export NCCL_DEBUG=TRACE
export NCCL_DEBUG_SUBSYS=NET

For per-request and completion tracing, the plugin needs to be built with TRACE=1, use liboccl-net-occl-ib-trace.so:

export NCCL_DEBUG=TRACE
export NCCL_DEBUG_SUBSYS=NET

NCCL_DEBUG=TRACE alone does not restore normal TRACE(...) statements if the plugin was built without ENABLE_TRACE.

The following variables are implemented by OCCL-NET's portable NET/IB code. Start with defaults unless a workload or troubleshooting procedure calls for a change.

Values are process environment values. Prefer a job launcher, scheduler environment, or a controlled NCCL configuration file over shell profiles so a run is reproducible.

Default resilient/steered deployment:

export NCCL_NET_PLUGIN=/opt/occl-net/lib/liboccl-net-occl-ib.so
export NCCL_NET=IB
export NCCL_IB_RESILIENCY_PORT_FAILOVER=1
export OCI_FRM_ENABLE=1

Simple compatibility profile with OCCL-NET still loaded:

export NCCL_NET_PLUGIN=/opt/occl-net/lib/liboccl-net-occl-ib.so
export NCCL_NET=IB
export OCI_FRM_ENABLE=0
export NCCL_IB_RESILIENCY_PORT_FAILOVER=0
export NCCL_IB_QPS_PER_CONNECTION=1
export NCCL_IB_SPLIT_DATA_ON_QPS=0
export NCCL_IB_MERGE_NICS=0
unset NCCL_NET_FORCE_MERGE

See the following Troubleshooting section for an ordered isolation plan.

Full NIC Backup uses NCCL_NET_FORCE_MERGE to place multiple physical devices in a backup group. OCCL-NET exposes GPU-local aliases for normal traffic and keeps the group's other members as warm standby paths. With NCCL_IB_RESILIENCY_PORT_FAILOVER=1, failed requests can move to those paths; the existing probe/rejoin mechanism can return traffic to a recovered primary.

Use NCCL 2.24.3 or newer and the same updated OCCL-NET build on every rank. NCCL 2.24.3–2.29 supports up to four members per group through NET ABI v9–v11; NCCL 2.30+ supports up to eight through v12. A backup group needs at least two members, all using the same link layer (InfiniBand or RoCE). Choose members with working peer connectivity and include a separate physical NIC so a whole-NIC failure leaves a usable path.

For a four-device group, start with:

export NCCL_NET_PLUGIN=/opt/occl-net/lib/liboccl-net-occl-ib.so
export NCCL_NET=IB
export NCCL_NET_FORCE_MERGE="mlx5_0,mlx5_1,mlx5_2,mlx5_3"
export NCCL_IB_RESILIENCY_PORT_FAILOVER=1
export NCCL_IB_QPS_PER_CONNECTION=1
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,NET

Replace the example names with devices visible on each host and propagate the configuration to every rank. NCCL_IB_HCA, if set, must allow the intended members.

The host runtime parses comma-separated members and semicolon-separated groups, for example NCCL_NET_FORCE_MERGE="mlx5_0,mlx5_1;mlx5_2,mlx5_3" creates two separate backup groups. Keep the quotes so the shell does not interpret the semicolon. Devices omitted from the list can still be selected or automatically merged; the variable is not a device filter. See NVIDIA's NIC Fusion syntax. Avoid duplicate members and overlapping groups.

The supported failure model is sequential/one-sided N-1: a group can retain a usable path as members fail, provided the corresponding peer paths remain reachable. Arbitrary simultaneous failures leaving unrelated survivors at opposite endpoints are outside this contract. Standby paths consume QP and registration resources; cross-NUMA backup traffic can reduce performance. In other words, Full NIC Backup is designed to handle successive device failures on one host, down to one surviving device in the group, while the peer’s corresponding network paths remain available. Recovery is not guaranteed when both hosts lose devices and their surviving devices lack a configured path between them.

Capture this information:

• Plugin path and embedded OCCL-NET version.
• NCCL version, application version, host names, rank-to-host mapping, and launcher command.
• NCCL_*, OCI_FRM_*, and FRM_* variables used by the job.
• Selected IB devices/ports, link state, link layer, and MTU.
• All rank logs with NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=INIT,NET.

Do not begin with TRACE on a large production workload. It can make a failure harder to read and can materially increase output volume.

The first successful step identifies the feature boundary crossed by the previous step. Keep all earlier changes in place while progressing.

Step 8 is diagnostic only. Do not use it to mask a known device, CQ, QP, or GID health event in production.

Start with the plugin explicitly enabled:

export NCCL_NET_PLUGIN=/opt/occl-net/lib/liboccl-net-occl-ib.so
                        export NCCL_NET=IB
                        export NCCL_DEBUG=INFO
                        export NCCL_DEBUG_SUBSYS=INIT,NET

Disable steering:

export OCI_FRM_ENABLE=0

Disable failover/rejoin after that:

export NCCL_IB_RESILIENCY_PORT_FAILOVER=0

Reduce to one non-merged, non-split QP configuration:

export NCCL_IB_QPS_PER_CONNECTION=1
                        export NCCL_IB_SPLIT_DATA_ON_QPS=0
                        unset NCCL_NET_FORCE_MERGE
                        export NCCL_IB_MERGE_NICS=0

Finally, stop loading the plugin:

export NCCL_NET_PLUGIN=none
                        unset NCCL_NET

If the problem persists after Step 9, it is outside OCCL-NET's loaded plugin path. Continue with the application's NCCL, launcher, host networking, and hardware support procedures.

Start with the Full NIC Backup profile and collect NCCL_DEBUG=INFO, NCCL_DEBUG_SUBSYS=INIT,NET logs from every rank. Confirm the plugin loaded before diagnosing backup behavior. Record the exact forced groups, HCA filter, QPs per connection, split-data setting, failover setting, plugin build, and NCCL runtime on both endpoints.

For four members arranged as two pairs with matching normalized PCI paths within each pair, expect alias creation like:

NET/IB : Made 2 GPU-local aliases for forced merge beginning at virtual device <index>

A connection between paired aliases with QPs per connection set to one reports:

NET/IB: Force-merge negotiated activeWidth=2 Q=1 logicalQps=2 physicalQps=8 localPrivateDevices=4 peerPrivateDevices=4

activeWidth is the number of active ports. logicalQps describes the traffic lanes; physicalQps includes their standby paths. localPrivateDevices and peerPrivateDevices show the backup membership at each endpoint. Alias counts and widths depend on PCI topology, so a single-port alias is not itself an error. The alias's name and speed do not list or sum its backup members.

To isolate forced grouping while retaining failover, first unset NCCL_NET_FORCE_MERGE on every rank and rerun with the other settings unchanged. To remove merging entirely, also set NCCL_IB_MERGE_NICS=0. Remove persistent forced-merge entries from launcher/configuration files too. Setting the forced list to 0 is not an off switch. Restore the backup profile before evaluating failover; the progressive isolation sequence disables failover at step 2.

Keep configuration and log comparisons separate from hardware fault injection. For a backup incident, add the failure/recovery timeline and surviving devices on both endpoints to the escalation bundle. The supported model is sequential/one-sided N-1 with reachable peer paths, not arbitrary simultaneous bilateral NIC loss.

For a reproducible issue, include the smallest reproducer, complete logs from all ranks, the environment before and after each isolation step, plugin version, NCCL version, ibv_devinfo/port-state output, and whether the issue follows a specific HCA, port, host, rank, or message size.