rdma_cm - RDMA communication manager.
Librdmacm Programmer's Manual RDMA_CM(7) NAME rdma_cm - RDMA communication manager. SYNOPSIS #include <rdma/rdma_cma.h> DESCRIPTION Used to establish communication over RDMA transports. ATTRIBUTES See attributes(7) for descriptions of the following attributes: +---------------+-----------------------+ |ATTRIBUTE TYPE | ATTRIBUTE VALUE | +---------------+-----------------------+ |Availability | network/open-fabrics | +---------------+-----------------------+ |Stability | Pass-through volatile | +---------------+-----------------------+ NOTES The RDMA CM is a communication manager used to setup reliable, con- nected and unreliable datagram data transfers. It provides an RDMA transport neutral interface for establishing connections. The API con- cepts are based on sockets, but adapted for queue pair (QP) based semantics: communication must be over a specific RDMA device, and data transfers are message based. The RDMA CM can control both the QP and communication management (con- nection setup / teardown) portions of an RDMA API, or only the communi- cation management piece. It works in conjunction with the verbs API defined by the libibverbs library. The libibverbs library provides the underlying interfaces needed to send and receive data. The RDMA CM can operate asynchronously. The mode of operation is con- trolled by the user through the use of the rdma_cm event channel param- eter in specific calls. If an event channel is provided, an rdma_cm identifier will report its event data (results of connecting, for exam- ple), on that channel. If a channel is not provided, then all rdma_cm operations for the selected rdma_cm identifier will block until they complete. Source code for open source software components in Oracle Solaris can be found at https://www.oracle.com/downloads/opensource/solaris-source- code-downloads.html. This software was built from source available at https://github.com/oracle/solaris-userland. The original community source was downloaded from ['https://www.openfabrics.org/down- loads/ibutils/ibutils-1.5.7-0.2.gbd7e502.tar.gz', 'https://www.openfab- rics.org/downloads/libibverbs/libibverbs-1.1.8.tar.gz', 'https://www.openfabrics.org/downloads/libmlx4/libmlx4-1.0.6.tar.gz', 'https://www.openfabrics.org/downloads/libsdp/lib- sdp-1.1.108-0.15.gd7fdb72.tar.gz', 'https://www.openfabrics.org/down- loads/management/infiniband-diags-1.6.5.tar.gz', 'https://www.openfab- rics.org/downloads/management/libibmad-1.3.12.tar.gz', 'https://www.openfabrics.org/downloads/management/libibu- mad-22.214.171.124.tar.gz', 'https://www.openfabrics.org/downloads/manage- ment/opensm-3.3.19.tar.gz', 'https://www.openfabrics.org/down- loads/perftest/perftest-1.3.0-0.42.gf350d3d.tar.gz', 'https://www.open- fabrics.org/downloads/qperf/qperf-0.4.9.tar.gz', 'https://www.openfab- rics.org/downloads/rdmacm/librdmacm-1.0.21.tar.gz', 'https://www.open- fabrics.org/downloads/rds-tools/rds-tools-2.0.4.tar.gz']. Further information about this software can be found on the open source community website at http://www.openfabrics.org/. RDMA VERBS The rdma_cm supports the full range of verbs available through the libibverbs library and interfaces. However, it also provides wrapper functions for some of the more commonly used verbs funcationality. The full set of abstracted verb calls are: CLIENT OPERATION This section provides a general overview of the basic operation for the active, or client, side of communication. This flow assume asynchro- nous operation with low level call details shown. Users may also refer to the example applications for code samples. A general connection flow would be: rdma_getaddrinfo retrieve address information of the destination rdma_create_event_channel create channel to receive events rdma_create_id allocate an rdma_cm_id, this is conceptually similar to a socket rdma_resolve_addr obtain a local RDMA device to reach the remote address rdma_get_cm_event wait for RDMA_CM_EVENT_ADDR_RESOLVED event rdma_ack_cm_event ack event rdma_create_qp allocate a QP for the communication rdma_resolve_route determine the route to the remote address rdma_get_cm_event wait for RDMA_CM_EVENT_ROUTE_RESOLVED event rdma_ack_cm_event ack event rdma_connect connect to the remote server rdma_get_cm_event wait for RDMA_CM_EVENT_ESTABLISHED event rdma_ack_cm_event ack event Perform data transfers over connection rdma_disconnect tear-down connection rdma_get_cm_event wait for RDMA_CM_EVENT_DISCONNECTED event rdma_ack_cm_event ack event rdma_destroy_qp destroy the QP rdma_destroy_id release the rdma_cm_id rdma_destroy_event_channel release the event channel An almost identical process is used to setup unreliable datagram (UD) communication between nodes. No actual connection is formed between QPs however, so disconnection is not needed. Although this example shows the client initiating the disconnect, either side of a connection may initiate the disconnect. SERVER OPERATION This section provides a general overview of the basic operation for the passive, or server, side of communication. A general connection flow would be: rdma_create_event_channel create channel to receive events rdma_create_id allocate an rdma_cm_id, this is conceptually similar to a socket rdma_bind_addr set the local port number to listen on rdma_listen begin listening for connection requests rdma_get_cm_event wait for RDMA_CM_EVENT_CONNECT_REQUEST event with a new rdma_cm_id rdma_create_qp allocate a QP for the communication on the new rdma_cm_id rdma_accept accept the connection request rdma_ack_cm_event ack event rdma_get_cm_event wait for RDMA_CM_EVENT_ESTABLISHED event rdma_ack_cm_event ack event Perform data transfers over connection rdma_get_cm_event wait for RDMA_CM_EVENT_DISCONNECTED event rdma_ack_cm_event ack event rdma_disconnect tear-down connection rdma_destroy_qp destroy the QP rdma_destroy_id release the connected rdma_cm_id rdma_destroy_id release the listening rdma_cm_id rdma_destroy_event_channel release the event channel RETURN CODES = 0 success = -1 error - see errno for more details Most librdmacm functions return 0 to indicate success, and a -1 return value to indicate failure. If a function operates asynchronously, a return value of 0 means that the operation was successfully started. The operation could still complete in error; users should check the status of the related event. If the return value is -1, then errno will contain additional information regarding the reason for the fail- ure. Prior versions of the library would return -errno and not set errno for some cases related to ENOMEM, ENODEV, ENODATA, EINVAL, and EADDRNO- TAVAIL codes. Applications that want to check these codes and have com- patability with prior library versions must manually set errno to the negative of the return code if it is < -1. SEE ALSO rdma_accept(3), rdma_ack_cm_event(3), rdma_bind_addr(3), rdma_con- nect(3), rdma_create_event_channel(3), rdma_create_id(3), rdma_cre- ate_qp(3), rdma_destroy_event_channel(3), rdma_destroy_id(3), rdma_destroy_qp(3), rdma_disconnect(3), rdma_event_str(3), rdma_free_devices(3), rdma_getaddrinfo(3), rdma_get_cm_event(3), rdma_get_devices(3), rdma_get_dst_port(3), rdma_get_local_addr(3), rdma_get_peer_addr(3), rdma_get_src_port(3), rdma_join_multicast(3), rdma_leave_multicast(3), rdma_listen(3), rdma_notify(3), rdma_reject(3), rdma_resolve_addr(3), rdma_resolve_route(3), rdma_set_option(3) mckey(1), rdma_client(1), rdma_server(1), rping(1), ucmatose(1), udaddy(1) librdmacm 2010-07-19 RDMA_CM(7)