Programming Interfaces Guide

Export-Side Memory Segment Operations

When exporting a memory segment, the application begins by allocating memory in its virtual address space through the normal operating system interfaces such as the System V Shared Memory Interface, mmap, or valloc. After allocating memory, the application calls the RSMAPI library interfaces to create and label a segment. After labelling the segment, the RSMAPI library interfaces bind physical pages to the allocated virtual range. After binding the physical pages, the RSMAPI library interfaces publish the segment for access by importing processes.

Note –

If virtual address space is obtained by using mmap, the mapping must be MAP_PRIVATE.

Export side memory segment operations include:

Memory segment creation and destruction
Memory segment publishing and unpublishing
Rebinding backing store for a memory segment

Memory Segment Creation and Destruction

Establishing a new memory segment with rsm_memseg_export_create enables the association of physical memory with the segment at creation time. The operation returns an export-side memory segment handle to the new memory segment. The segment exists for the lifetime of the creating process or until destroyed with rsm_memseg_export_destroy.

Note –

If destroy operation is performed before an import side disconnect, the disconnect is forced.

Segment Creation

int rsm_memseg_export_create(rsmapi_controller_handle_t controller, rsm_memseg_export_handle_t *memseg, void *vaddr, size_t size, uint_t flags);

This function creates a segment handle. After the segment handle is created, the segment handle is bound to the specified virtual address range [vaddr..vaddr+size]. The range must be valid and aligned on the controller's alignment property. The flags argument is a bitmask, which enables:

Unbinding on the segment
Rebinding on the segment
Passing RSM_ALLOW_REBIND to flags
Support of lock operations
Passing RSM_LOCK_OPS to flags

Note –

The RSM_LOCK_OPS flag is not included in the initial release of RSMAPI.

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_CTLR_HNDL: Invalid controller handle
RSMERR_CTLR_NOT_PRESENT: Controller not present
RSMERR_BAD_SEG_HNDL: Invalid segment handle
RSMERR_BAD_LENGTH: Length zero or length exceeds controller limits
RSMERR_BAD_ADDR: Invalid address
RSMERR_PERM_DENIED: Permission denied
RSMERR_INSUFFICIENT_MEM: Insufficient memory
RSMERR_INSUFFICIENT_RESOURCES: Insufficient resources
RSMERR_BAD_MEM_ALIGNMENT: Address not aligned on page boundary
RSMERR_INTERRUPTED: Operation interrupted by signal

Segment Destruction

int rsm_memseg_export_destroy(rsm_memseg_export_handle_t memseg);

This function deallocates segment and its free resources. All importing processes are forcibly disconnected.

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_SEG_HNDL: Invalid segment handle
RSMERR_POLLFD_IN_USE: pollfd in use

Memory Segment Publish, Republish, and Unpublish

The publish operation enables the importing of a memory segment by other nodes on the interconnect. An export segment might be published on multiple interconnect adapters.

The segment ID might be specified from within authorized ranges or specified as zero, in which case a valid segment ID is generated by the RSMAPI framework and is passed back.

The segment access control list is composed of pairs of node ID and access permissions. For each node ID specified in the list, the associated read/write permissions are provided by three octal digits for owner, group and other, as with Solaris file permissions. In the access control list, each octal digit can have the following values:

2: Write access.
4: Read only access.
6: Read and write access.

An access permission value of 0624 specifies the following kind of access:

An importer with the same uid as the exporter has both read and write access.
An importer with the same gid as the exporter has write access only.
All other importers have read access only.

When an access control list is provided, nodes not included in the list cannot import the segment. However, if the access list is null, any node can import the segment. The access permissions on all nodes equal the owner-group-other file creation permissions of the exporting process.

Note –

Node applications have the responsibility of managing the assignment of segment identifiers to ensure uniqueness on the exporting node.

Publish Segment

int rsm_memseg_export_publish(rsm_memseg_export_handle_t memseg, rsm_memseg_id_t *segment_id, rsmapi_access_entry_t ACCESS_list[], uint_t access_list_length);

typedef struct {
    rsm_node_id_t       ae_node;    /* remote node id allowed to access resource */
    rsm_permission_t    ae_permissions;    /* mode of access allowed */
} rsmapi_access_entry_t;.

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_SEG_HNDL: Invalid segment handle
RSMERR_SEG_ALREADY_PUBLISHED: Segment already published
RSMERR_BAD_ACL: Invalid access control list
RSMERR_BAD_SEGID: Invalid segment identifier
RSMERR_SEGID_IN_USE: Segment identifier in use
RSMERR_RESERVED_SEGID: Segment identifier reserved
RSMERR_NOT_CREATOR: Not creator of segment
RSMERR_BAD_ADDR: Bad address
RSMERR_INSUFFICIENT_MEM: Insufficient memory
RSMERR_INSUFFICIENT_RESOURCES: Insufficient resources

Authorized Segment ID Ranges:

#define RSM_DRIVER_PRIVATE_ID_BASE: 0
#define RSM_DRIVER_PRIVATE_ID_END: 0x0FFFFF
#define RSM_CLUSTER_TRANSPORT_ID_BASE: 0x100000
#define RSM_CLUSTER_TRANSPORT_ID_END: 0x1FFFFF
#define RSM_RSMLIB_ID_BASE: 0x200000
#define RSM_RSMLIB_ID_END: 0x2FFFFF
#define RSM_DLPI_ID_BASE: 0x300000
#define RSM_DLPI_ID_END: 0x3FFFFF
#define RSM_HPC_ID_BASE: 0x400000
#define RSM_HPC_ID_END: 0x4FFFFF

The following range is reserved for allocation by the system when the publish value is zero.

#define RSM_USER_APP_ID_BASE: 0x80000000
#define RSM_USER_APP_ID_END: 0xFFFFFFF

Republish Segment

int rsm_memseg_export_republish(rsm_memseg_export_handle_t memseg, rsmapi_access_entry_t access_list[], uint_t access_list_length);

This function establishes a new node access list and segment access mode. These changes only affect future import calls and do not revoke already granted import requests.

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_SEG_HNDL: Invalid segment handle
RSMERR_SEG_NOT_PUBLISHED: Segment not published
RSMERR_BAD_ACL: Invalid access control list
RSMERR_NOT_CREATOR: Not creator of segment
RSMERR_INSUFFICIENT_MEMF: Insufficient memory
RSMERR_INSUFFICIENT_RESOURCES: Insufficient resources
RSMERR_INTERRUPTED: Operation interrupted by signal

Unpublish Segment

int rsm_memseg_export_unpublish(rsm_memseg_export_handle_t memseg);

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_SEG_HNDL: Invalid segment handle
RSMERR_SEG_NOT_PUBLISHED: Segment not published
RSMERR_NOT_CREATOR: Not creator of segment
RSMERR_INTERRUPTED: Operation interrupted by signal

Memory Segment Rebind

The rebind operation releases the current backing store for an export segment. After releasing the current backing store for an export segment, the rebind operation allocates a new backing store. The application must first obtain a new virtual memory allocation for the segment. This operation is transparent to importers of the segment.

Note –

The application has the responsibility of preventing access to segment data until the rebind operation is complete. Retrieving data from a segment during rebinding does not cause a system failure, but the results of such an operation are undefined.

Rebind Segment

int rsm_memseg_export_rebind(rsm_memseg_export_handle_t memseg, void *vaddr, offset_t off, size_t size);

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_SEG_HNDL: Invalid segment handle
RSMERR_BAD_LENGTH: Invalid length
RSMERR_BAD_ADDR: Invalid address
RSMERR_REBIND_NOT_ALLOWED: Rebind not allowed
RSMERR_NOT_CREATOR: Not creator of segment
RSMERR_PERM_DENIED: Permission denied
RSMERR_INSUFFICIENT_MEM: Insufficient memory
RSMERR_INSUFFICIENT_RESOURCES: Insufficient resources
RSMERR_INTERRUPTED: Operation interrupted by signal