An RSM segment represents a set of (generally) non-contiguous physical memory pages mapped to a contiguous virtual address range. RSM segment export and segment import operations enable the sharing of regions of physical memory among systems on an interconnect. A process of the node on which the physical pages reside is referred to as the exporter of the memory. An exported segment that is published for remote access will have a segment identifier that is unique for the given node. The segment ID might be specified by the exporter or assigned by the RSMAPI framework.
Processes of nodes on the interconnect obtain access to exported memory by creating an RSM import segment. The RSM import segment has a connection with an exported segment, rather than local physical pages. When the interconnect supports memory mapping, importers can read and write the exported memory by using the local memory-mapped addresses of the import segment. When the interconnect does not support memory mapping, the importing process uses memory access primitives.
When exporting a memory segment, the application begins by allocating memory in its virtual address space through the normal operating system interfaces such as the System V Shared Memory Interface, mmap, or valloc. After allocating memory, the application calls the RSMAPI library interfaces to create and label a segment. After labelling the segment, the RSMAPI library interfaces bind physical pages to the allocated virtual range. After binding the physical pages, the RSMAPI library interfaces publish the segment for access by importing processes.
If virtual address space is obtained by using mmap, the mapping must be MAP_PRIVATE.
Export side memory segment operations include:
Memory segment creation and destruction
Memory segment publishing and unpublishing
Rebinding backing store for a memory segment
Establishing a new memory segment with rsm_memseg_export_create enables the association of physical memory with the segment at creation time. The operation returns an export-side memory segment handle to the new memory segment. The segment exists for the lifetime of the creating process or until destroyed with rsm_memseg_export_destroy.
If destroy operation is performed before an import side disconnect, the disconnect is forced.
This function creates a segment handle. After the segment handle is created, the segment handle is bound to the specified virtual address range [vaddr..vaddr+size]. The range must be valid and aligned on the controller's alignment property. The flags argument is a bitmask, which enables:
Unbinding on the segment
Rebinding on the segment
Passing RSM_ALLOW_REBIND to flags
Support of lock operations
Passing RSM_LOCK_OPS to flags
The RSM_LOCK_OPS flag is not included in the initial release of RSMAPI.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid controller handle
Controller not present
Invalid segment handle
Length zero or length exceeds controller limits
Invalid address
Permission denied
Insufficient memory
Insufficient resources
Address not aligned on page boundary
Operation interrupted by signal
This function deallocates segment and its free resources. All importing processes are forcibly disconnected.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
pollfd in use
The publish operation enables the importing of a memory segment by other nodes on the interconnect. An export segment might be published on multiple interconnect adapters.
The segment ID might be specified from within authorized ranges or specified as zero, in which case a valid segment ID is generated by the RSMAPI framework and is passed back.
The segment access control list is composed of pairs of node ID and access permissions. For each node ID specified in the list, the associated read/write permissions are provided by three octal digits for owner, group and other, as with Solaris file permissions. In the access control list, each octal digit can have the following values:
Write access.
Read only access.
Read and write access.
An access permission value of 0624 specifies the following kind of access:
An importer with the same uid as the exporter has both read and write access.
An importer with the same gid as the exporter has write access only.
All other importers have read access only.
When an access control list is provided, nodes not included in the list cannot import the segment. However, if the access list is null, any node can import the segment. The access permissions on all nodes equal the owner-group-other file creation permissions of the exporting process.
Node applications have the responsibility of managing the assignment of segment identifiers to ensure uniqueness on the exporting node.
typedef struct { rsm_node_id_t ae_node; /* remote node id allowed to access resource */ rsm_permission_t ae_permissions; /* mode of access allowed */ }rsmapi_access_entry_t;.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Segment already published
Invalid access control list
Invalid segment identifier
Segment identifier in use
Segment identifier reserved
Not creator of segment
Bad address
Insufficient memory
Insufficient resources
Authorized Segment ID Ranges:
0
0x0FFFFF
0x100000
0x1FFFFF
0x200000
0x2FFFFF
0x300000
0x3FFFFF
0x400000
0x4FFFFF
The following range is reserved for allocation by the system when the publish value is zero.
0x80000000
0xFFFFFFF
This function establishes a new node access list and segment access mode. These changes only affect future import calls and do not revoke already granted import requests.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Segment not published
Invalid access control list
Not creator of segment
Insufficient memory
Insufficient resources
Operation interrupted by signal
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Segment not published
Not creator of segment
Operation interrupted by signal
The rebind operation releases the current backing store for an export segment. After releasing the current backing store for an export segment, the rebind operation allocates a new backing store. The application must first obtain a new virtual memory allocation for the segment. This operation is transparent to importers of the segment.
The application has the responsibility of preventing access to segment data until the rebind operation is complete. Retrieving data from a segment during rebinding does not cause a system failure, but the results of such an operation are undefined.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Invalid length
Invalid address
Rebind not allowed
Not creator of segment
Permission denied
Insufficient memory
Insufficient resources
Operation interrupted by signal
The following list describes Import-side operations:
Memory segment connection and disconnection
Access to imported segment memory
Barrier operations used to impose order on data access operations and for access error detection
The connect operation is used to create an RSM import segment and form a logical connection with an exported segment.
Access to imported segment memory is provided by three interface categories:
Segment access.
Data transfer.
Segment memory mapping.
This function connects to segment segment_id on remote node node_id by using the specified permission perm. The function returns a segment handle after connecting to the segment.
The argument perm specifies the access mode requested by the importer for this connection. To establish the connection, the access permissions specified by the exporter are compared to the access mode, user ID, and group ID used by the importer. If the request mode is not valid, the connection request is denied. The perm argument is limited to the following octal values:
Read mode
Write mode
Read/write mode
The specified controller must have a physical connection to the controller that is used in the export of the segment.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid controller handle
Controller not present
Invalid segment handle
Permission denied
Segment not published to node
No such segment published
Remote node not reachable
Connection interrupted
Insufficient memory
Insufficient resources
Bad address
This function disconnects a segment. This function frees a segment's resources after disconnecting a segment. All existing mappings to the disconnected segment are removed. The handle im_memseg is freed.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Segment still mapped
pollfd in use
The following interfaces provide a mechanism for transferring between 8 bits and 64 bits of data. The get interfaces use a repeat count (rep_cnt) to indicate the number of data items of a given size the process will read from successive locations. The locations begin at byte offset offset in the imported segment. The data is written to successive locations that begin at datap. The put interfaces use a repeat count (rep_cnt). The count indicates the number of data items the process will read from successive locations. The locations begin at datap. The data is then written to the imported segment at successive locations. The locations begin at the byte offset specified by the offset argument.
These interfaces also provide byte swapping in case the source and destination have incompatible endian characteristics.
Function Prototypes:
int rsm_memseg_import_get8(rsm_memseg_import_handle_t im_memseg, off_t offset, uint8_t *datap, ulong_t rep_cnt);The following interfaces are intended for data transfers that are larger than the ones supported by the segment access operations.
int rsm_memseg_import_put(rsm_memseg_import_handle_t im_memseg, off_t offset, void *src_addr, size_t length);This function copies data from local memory, specified by the src_addr and length, to the corresponding imported segment locations specified by the handle and offset.
int rsm_memseg_import_get(rsm_memseg_import_handle_t im_memseg, off_t offset, void *dst_addr, size_t length);This function is similar to rsm_memseg_import_put(), but data flows from the imported segment into local regions defined by the dest_vec argument
The put and get routines write or read the specified quantity of data from the byte offset location specified by the argument offset. The routines begin at the base of the segment. The offset must align at the appropriate boundary. For example, rsm_memseg_import_get64() requires that offset and datap align at a double-word boundary, while rsm_memseg_import_put32() requires an offset that is aligned at a word boundary.
By default, the barrier mode attribute of a segment is implicit. Implicit barrier mode means that the caller assumes the data transfer has completed or has failed upon return from the operation. Because the default barrier mode is implicit, the application must initialize the barrier. The application initializes the barrier by using the rsm_memseg_import_init_barrier() function before calling put or get routines when using the default mode. To use the explicit operation mode, the caller must use a barrier operation to force the completion of a transfer. After forcing the completion of the transfer, the caller must determine if any errors have occurred as a result of the forced completion.
An import segment can be partially mapped by passing an offset in the rsm_memseg_import_map() routine. If the import segment is partially mapped, the offset argument in the put or get routines is from the base of the segment. The user must make sure that the correct byte offset is passed to put and get routines.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Bad address
Invalid memory alignment
Invalid offset
Invalid length
Permission denied
Barrier not initialized
I/O completion error
Connection aborted
Insufficient resources
The rsm_memseg_import_putv() and rsm_memseg_import_getv() functions allow the use of a list of I/O requests instead of a single source and single destination address.
Function Prototypes:
int rsm_memseg_import_putv(rsm_scat_gath_t *sg_io);The I/O vector component of the scatter-gather list (sg_io) enables the specification of local virtual addresses or local_memory_handles. Handles are an efficient way to repeatedly use a local address range. Allocated system resources, such as locked down local memory, are maintained until the handle is freed. The supporting functions for handles are rsm_create_localmemory_handle() and rsm_free_localmemory_handle().
You can gather virtual addresses or handles into the vector in order to write to a single remote segment. You can also scatter the results of reading from a single remote segment to the vector of virtual addresses or handles.
I/O for the entire vector is initiated before returning. The barrier mode attribute of the import segment determines whether the I/O has completed before the function returns. Setting the barrier mode attribute to implicit guarantees that data transfer is completed in the order entered in the vector. An implicit barrier open and close surrounds each list entry. If an error is detected, I/O for the vector is terminated and the function returns immediately. The residual count indicates the number of entries for which the I/O either did not complete or was not initiated.
You can specify that a notification event be sent to the target segment when a putv or getv operation is successful. To specify the delivery of a notification event, specify the RSM_IMPLICIT_SIGPOST value in the flags entry of the rsm_scat_gath_t structure. The flags entry can also contain the value RSM_SIGPOST_NO_ACCUMULATE, which is passed on to the signal post operation if RSM_IMPLICIT_SIGPOST is set.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid scatter-gather structure pointer
Invalid segment handle
Invalid controller handle
Bad address
Invalid offset
Invalid length
Permission denied
I/O completion error
Connection aborted
Insufficient resources
Operation interrupted by signal
This function obtains a local handle for use in the I/O vector for subsequent calls to putv or getv. Freeing the handle as soon as possible conserves system resources, notably the memory spanned by the local handle, which might be locked down.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid controller handle
Invalid local memory handle
Invalid length
Invalid address
Insufficient memory
This function releases the system resources associated with the local handle. While all handles that belong to a process are freed when the process exits, calling this function conserves system resources.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid controller handle
Invalid local memory handle
The following example demonstrates the definition of primary data structures.
typedef void *rsm_localmemory_handle_t typedef struct { ulong_t io_request_count; number of rsm_iovec_t entries ulong_t io_residual_count; rsm_iovec_t entries not completed in flags; rsm_memseg_import_handle_t remote_handle; opaque handle for import segment rsm_iovec_t *iovec; pointer to array of io_vec_t } rsm_scat_gath_t; typedef struct { int io_type; HANDLE or VA_IMMEDIATE union { rsm_localmemory_handle_t handle; used with HANDLE caddr_t virtual_addr; used with VA_IMMEDIATE } local; size_t local_offset; offset from handle base vaddr size_t import_segment_offset; offset from segment base vaddr size_t transfer_length; } rsm_iovec_t;
Mapping operations are only available for native architecture interconnects such as Dolphin-SCI or NewLink. Mapping a segment grants CPU memory operations access to that segment, saving the overhead of calling memory access primitives.
int rsm_memseg_import_map(rsm_memseg_import_handle_t im_memseg, void **address, rsm_attribute_t attr, rsm_permission_t perm, off_t offset, size_t length);This function maps an imported segment into the caller address space. If the attribute RSM_MAP_FIXED is specified, the function maps the segment at the value specified in **address.
typedef enum { RSM_MAP_NONE = 0x0, /* system will choose available virtual address */ RSM_MAP_FIXED = 0x1, /* map segment at specified virtual address */ } rsm_map_attr_t;
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Invalid address
Invalid length
Invalid offset
Invalid permissions
Segment already mapped
Segment not connected
Connection aborted
Error during mapping
Address not aligned on page boundary
Unmap segment
int rsm_memseg_import_unmap(rsm_memseg_import_handle_t im_memseg);This function unmaps an imported segment from user virtual address space.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Use Barrier operations to resolve order-of-write-access memory model issues. Barrier operations also provide remote memory access error detection.
The barrier mechanism is made up of the following operations:
Initialization
Open
Close
Order
Successfully performing a close operation guarantees the successful completion of covered access operations, which take place between the barrier open and the barrier close. After a barrier open operation, failures of individual data access operations, both reads and writes, are not reported until the barrier close operation.
To impose a specific order of write completion within a barrier's scope, use an explicit barrier-order operation. A write operation that is issued before the barrier-order operation finishes before operations that are issued after the barrier-order operation. Write operations within a given barrier scope are ordered with respect to another barrier scope.
Initialize Barrier
int rsm_memseg_import_init_barrier(rsm_memseg_import_handle_t im_memseg, rsm_barrier_type_t type, rsmapi_barrier_t *barrier);At present, RSM_BAR_DEFAULT is the only supported type.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Invalid barrier pointer
Insufficient memory
Open Barrier
int rsm_memseg_import_open_barrier(rsmapi_barrier_t *barrier);Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Invalid barrier pointer
Close Barrier
int rsm_memseg_import_close_barrier(rsmapi_barrier_t *barrier);This function closes the barrier and flushes all store buffers. This call assumes that the calling process will retry all remote memory operations since the last rsm_memseg_import_open_barrier call if the call to rsm_memseg_import_close_barrier() fails.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Invalid barrier pointer
Barrier not initialized
Barrier not opened
Memory access error
Connection aborted
Order Barrier
int rsm_memseg_import_order_barrier(rsmapi_barrier_t *barrier);This function flushes all store buffers.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Invalid barrier pointer
Barrier not initialized
Barrier not opened
Memory access error
Connection aborted
Destroy Barrier
int rsm_memseg_import_destroy_barrier(rsmapi_barrier_t *barrier);This function deallocates all barrier resources.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Invalid barrier pointer
Set Mode
int rsm_memseg_import_set_mode(rsm_memseg_import_handle_t im_memseg, rsm_barrier_mode_t mode);This function supports the optional explicit barrier scoping that is available in the put routines. The two valid barrier modes are RSM_BARRIER_MODE_EXPLICIT and RSM_BARRIER_MODE_IMPLICIT. The default value of the barrier mode is RSM_BARRIER_MODE_IMPLICIT. While in implicit mode, an implicit barrier open and barrier close is applied to each put operation. Before setting the barrier mode value to RSM_BARRIER_MODE_EXPLICIT, use the rsm_memseg_import_init_barrier routine to initialize a barrier for the imported segment im_memseg.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Get Mode
int rsm_memseg_import_get_mode(rsm_memseg_import_handle_t im_memseg, rsm_barrier_mode_t *mode);This function obtains the current mode value for barrier scoping in the put routines.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle.
Event operations enable processes synchronization on memory access events. If a process cannot use the rsm_intr_signal_wait() function, it can multiplex event waiting by obtaining a poll descriptor with rsm_memseg_get_pollfd() and using the poll system call.
Using the rsm_intr_signal_post() and rsm_intr_signal_wait() operations incurs the need to process of ioctl calls to the kernel.
Post Signal
int rsm_intr_signal_post(void *memseg, uint_t flags);The void pointer *memseg can be type cast to either an import segment handle or an export segment handle. If *memseg refers to an import handle, this function sends a signal the exporting process. If *memseg refers to an export handle, this function sends a signal to all importers of that segment. Setting the flags argument to RSM_SIGPOST_NO_ACCUMULATE discards this event if an event is already pending for the target segment.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Remote node not reachable
Wait for Signal
int rsm_intr_signal_wait(void * memseg, int timeout);The void pointer *memseg can be type cast to either an import segment handle or an export segment handle. The process blocks for up to timeout milliseconds or until an event occurs. If the value is -1, the process blocks until an event occurs or until interrupted.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Timer expired
Wait interrupted
Get pollfd
int rsm_memseg_get_pollfd(void *memseg, struct pollfd *pollfd);This function initializes the specified pollfd structure with a descriptor for the specified segment and the singular fixed event generated by rsm_intr_signal_post(). Use the pollfd structure with the poll system call to wait for the event signalled by rsm_intr_signal_post. If the memory segment is not currently published, the poll system call does not return a valid pollfd. Each successful call increments a pollfd reference count for the specified segment.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Release pollfd
int rsm_memseg_release_pollfd(oid *memseg);This call decrements the pollfd reference count for the specified segment. If the reference count is nonzero, operations that unpublish, destroy, or unmap the segment fail.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle