Import-Side Memory Segment Operations (Programming Interfaces Guide)

Programming Interfaces Guide

Import-Side Memory Segment Operations

The following list describes Import-side operations:

Memory segment connection and disconnection
Access to imported segment memory
Barrier operations used to impose order on data access operations and for access error detection

The connect operation is used to create an RSM import segment and form a logical connection with an exported segment.

Access to imported segment memory is provided by three interface categories:

Segment access.
Data transfer.
Segment memory mapping.

Memory Segment Connection and Disconnection

Connect to Segment

int rsm_memseg_import_connect(rsmapi_controller_handle_t controller, rsm_node_id_t node_id, rsm_memseg_id_t segment_id, rsm_permission_t perm, rsm_memseg_import_handle_t *im_memseg);

This function connects to segment segment_id on remote node node_id by using the specified permission perm. The function returns a segment handle after connecting to the segment.

The argument perm specifies the access mode requested by the importer for this connection. To establish the connection, the access permissions specified by the exporter are compared to the access mode, user ID, and group ID used by the importer. If the request mode is not valid, the connection request is denied. The perm argument is limited to the following octal values:

0400: Read mode
0200: Write mode
0600: Read/write mode

The specified controller must have a physical connection to the controller that is used in the export of the segment.

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_CTLR_HNDL: Invalid controller handle
RSMERR_CTLR_NOT_PRESENT: Controller not present
RSMERR_BAD_SEG_HNDL: Invalid segment handle
RSMERR_PERM_DENIED: Permission denied
RSMERR_SEG_NOT_PUBLISHED_TO_NODE: Segment not published to node
RSMERR_SEG_NOT_PUBLISHED: No such segment published
RSMERR_REMOTE_NODE_UNREACHABLE: Remote node not reachable
RSMERR_INTERRUPTED: Connection interrupted
RSMERR_INSUFFICIENT_MEM: Insufficient memory
RSMERR_INSUFFICIENT_RESOURCES: Insufficient resources
RSMERR_BAD_ADDR: Bad address

Disconnect from Segment

int rsm_memseg_import_disconnect(rsm_memseg_import_handle_t im_memseg);

This function disconnects a segment. This function frees a segment's resources after disconnecting a segment. All existing mappings to the disconnected segment are removed. The handle im_memseg is freed.

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_SEG_HNDL: Invalid segment handle
RSMERR_SEG_STILL_MAPPED: Segment still mapped
RSMERR_POLLFD_IN_USE: pollfd in use

Memory Access Primitives

The following interfaces provide a mechanism for transferring between 8 bits and 64 bits of data. The get interfaces use a repeat count (rep_cnt) to indicate the number of data items of a given size the process will read from successive locations. The locations begin at byte offset offset in the imported segment. The data is written to successive locations that begin at datap. The put interfaces use a repeat count (rep_cnt). The count indicates the number of data items the process will read from successive locations. The locations begin at datap. The data is then written to the imported segment at successive locations. The locations begin at the byte offset specified by the offset argument.

These interfaces also provide byte swapping in case the source and destination have incompatible endian characteristics.

Function Prototypes:

int rsm_memseg_import_get8(rsm_memseg_import_handle_t im_memseg, off_t offset, uint8_t *datap, ulong_t rep_cnt);
int rsm_memseg_import_get16(rsm_memseg_import_handle_t im_memseg, off_t offset, uint16_t *datap, ulong_t rep_cnt);
int rsm_memseg_import_get32(rsm_memseg_import_handle_t im_memseg, off_t offset, uint32_t *datap, ulong_t rep_cnt);
int rsm_memseg_import_get64(rsm_memseg_import_handle_t im_memseg, off_t offset, uint64_t *datap, ulong_t rep_cnt);
int rsm_memseg_import_put8(rsm_memseg_import_handle_t im_memseg, off_t offset, uint8_t *datap, ulong_t rep_cnt);
int rsm_memseg_import_put16(rsm_memseg_import_handle_t im_memseg, off_t offset, uint16_t *datap, ulong_t rep_cnt);
int rsm_memseg_import_put32(rsm_memseg_import_handle_t im_memseg, off_t offset, uint32_t *datap, ulong_t rep_cnt);
int rsm_memseg_import_put64(rsm_memseg_import_handle_t im_memseg, off_t offset, uint64_t *datap, ulong_t rep_cnt);

The following interfaces are intended for data transfers that are larger than the ones supported by the segment access operations.

Segment Put

int rsm_memseg_import_put(rsm_memseg_import_handle_t im_memseg, off_t offset, void *src_addr, size_t length);

This function copies data from local memory, specified by the src_addr and length, to the corresponding imported segment locations specified by the handle and offset.

Segment Get

int rsm_memseg_import_get(rsm_memseg_import_handle_t im_memseg, off_t offset, void *dst_addr, size_t length);

This function is similar to rsm_memseg_import_put(), but data flows from the imported segment into local regions defined by the dest_vec argument

The put and get routines write or read the specified quantity of data from the byte offset location specified by the argument offset. The routines begin at the base of the segment. The offset must align at the appropriate boundary. For example, rsm_memseg_import_get64() requires that offset and datap align at a double-word boundary, while rsm_memseg_import_put32() requires an offset that is aligned at a word boundary.

By default, the barrier mode attribute of a segment is implicit. Implicit barrier mode means that the caller assumes the data transfer has completed or has failed upon return from the operation. Because the default barrier mode is implicit, the application must initialize the barrier. The application initializes the barrier by using the rsm_memseg_import_init_barrier() function before calling put or get routines when using the default mode. To use the explicit operation mode, the caller must use a barrier operation to force the completion of a transfer. After forcing the completion of the transfer, the caller must determine if any errors have occurred as a result of the forced completion.

Note –

An import segment can be partially mapped by passing an offset in the rsm_memseg_import_map() routine. If the import segment is partially mapped, the offset argument in the put or get routines is from the base of the segment. The user must make sure that the correct byte offset is passed to put and get routines.

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_SEG_HNDL: Invalid segment handle
RSMERR_BAD_ADDR: Bad address
RSMERR_BAD_MEM_ALIGNMENT: Invalid memory alignment
RSMERR_BAD_OFFSET: Invalid offset
RSMERR_BAD_LENGTH: Invalid length
RSMERR_PERM_DENIED: Permission denied
RSMERR_BARRIER_UNINITIALIZED: Barrier not initialized
RSMERR_BARRIER_FAILURE: I/O completion error
RSMERR_CONN_ABORTED: Connection aborted
RSMERR_INSUFFICIENT_RESOURCES: Insufficient resources

Scatter-Gather Access

The rsm_memseg_import_putv() and rsm_memseg_import_getv() functions allow the use of a list of I/O requests instead of a single source and single destination address.

Function Prototypes:

int rsm_memseg_import_putv(rsm_scat_gath_t *sg_io);
int rsm_memseg_import_getv(rsm_scat_gath_t *sg_io);

The I/O vector component of the scatter-gather list (sg_io) enables the specification of local virtual addresses or local_memory_handles. Handles are an efficient way to repeatedly use a local address range. Allocated system resources, such as locked down local memory, are maintained until the handle is freed. The supporting functions for handles are rsm_create_localmemory_handle() and rsm_free_localmemory_handle().

You can gather virtual addresses or handles into the vector in order to write to a single remote segment. You can also scatter the results of reading from a single remote segment to the vector of virtual addresses or handles.

I/O for the entire vector is initiated before returning. The barrier mode attribute of the import segment determines whether the I/O has completed before the function returns. Setting the barrier mode attribute to implicit guarantees that data transfer is completed in the order entered in the vector. An implicit barrier open and close surrounds each list entry. If an error is detected, I/O for the vector is terminated and the function returns immediately. The residual count indicates the number of entries for which the I/O either did not complete or was not initiated.

You can specify that a notification event be sent to the target segment when a putv or getv operation is successful. To specify the delivery of a notification event, specify the RSM_IMPLICIT_SIGPOST value in the flags entry of the rsm_scat_gath_t structure. The flags entry can also contain the value RSM_SIGPOST_NO_ACCUMULATE, which is passed on to the signal post operation if RSM_IMPLICIT_SIGPOST is set.

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_SGIO: Invalid scatter-gather structure pointer
RSMERR_BAD_SEG_HNDL: Invalid segment handle
RSMERR_BAD_CTLR_HNDL: Invalid controller handle
RSMERR_BAD_ADDR: Bad address
RSMERR_BAD_OFFSET: Invalid offset
RSMERR_BAD_LENGTH: Invalid length
RSMERR_PERM_DENIED: Permission denied
RSMERR_BARRIER_FAILURE: I/O completion error
RSMERR_CONN_ABORTED: Connection aborted
RSMERR_INSUFFICIENT_RESOURCES: Insufficient resources
RSMERR_INTERRUPTED: Operation interrupted by signal

Get Local Handle

int rsm_create_localmemory_handle(rsmapi_controller_handle_t cntrl_handle, rsm_localmemory_handle_t *local_handle, caddr_t local_vaddr, size_t length);

This function obtains a local handle for use in the I/O vector for subsequent calls to putv or getv. Freeing the handle as soon as possible conserves system resources, notably the memory spanned by the local handle, which might be locked down.

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_CTLR_HNDL: Invalid controller handle
RSMERR_BAD_LOCALMEM_HNDL: Invalid local memory handle
RSMERR_BAD_LENGTH: Invalid length
RSMERR_BAD_ADDR: Invalid address
RSMERR_INSUFFICIENT_MEM: Insufficient memory

Free Local Handle

rsm_free_localmemory_handle(rsmapi_controller_handle_t cntrl_handle, rsm_localmemory_handle_t handle);

This function releases the system resources associated with the local handle. While all handles that belong to a process are freed when the process exits, calling this function conserves system resources.

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_CTLR_HNDL: Invalid controller handle
RSMERR_BAD_LOCALMEM_HNDL: Invalid local memory handle

The following example demonstrates the definition of primary data structures.

Example 2–1 Primary Data Structures

typedef void *rsm_localmemory_handle_t
typedef struct {
   ulong_t	io_request_count;	number of rsm_iovec_t entries
   ulong_t	io_residual_count;	rsm_iovec_t entries not completed 

   in	flags;      
   rsm_memseg_import_handle_t		remote_handle; opaque handle for
                                                    import segment
   rsm_iovec_t							*iovec;		  pointer to
                                                    array of io_vec_t
} rsm_scat_gath_t;

typedef struct {
   int	io_type;		HANDLE or VA_IMMEDIATE
   union {
        rsm_localmemory_handle_t	handle;			used with HANDLE
        caddr_t						virtual_addr;		used with
                                                         VA_IMMEDIATE
    } local;
   size_t          local_offset;		       offset from handle base vaddr
   size_t          import_segment_offset;    offset from segment base vaddr
   size_t          transfer_length; 
} rsm_iovec_t;

Segment Mapping

Mapping operations are only available for native architecture interconnects such as Dolphin-SCI or NewLink. Mapping a segment grants CPU memory operations access to that segment, saving the overhead of calling memory access primitives.

Imported Segment Map

int rsm_memseg_import_map(rsm_memseg_import_handle_t im_memseg, void **address, rsm_attribute_t attr, rsm_permission_t perm, off_t offset, size_t length);

This function maps an imported segment into the caller address space. If the attribute RSM_MAP_FIXED is specified, the function maps the segment at the value specified in **address.

typedef enum {
RSM_MAP_NONE = 0x0,   /* system will choose available virtual address */
RSM_MAP_FIXED = 0x1,  /* map segment at specified virtual address */
} rsm_map_attr_t;

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_SEG_HNDL: Invalid segment handle
RSMERR_BAD_ADDR: Invalid address
RSMERR_BAD_LENGTH: Invalid length
RSMERR_BAD_OFFSET: Invalid offset
RSMERR_BAD_PERMS: Invalid permissions
RSMERR_SEG_ALREADY_MAPPED: Segment already mapped
RSMERR_SEG_NOT_CONNECTED: Segment not connected
RSMERR_CONN_ABORTED: Connection aborted
RSMERR_MAP_FAILED: Error during mapping
RSMERR_BAD_MEM_ALIGNMENT: Address not aligned on page boundary

Unmap segment

int rsm_memseg_import_unmap(rsm_memseg_import_handle_t im_memseg);

This function unmaps an imported segment from user virtual address space.

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_SEG_HNDL: Invalid segment handle

Barrier Operations

Use Barrier operations to resolve order-of-write-access memory model issues. Barrier operations also provide remote memory access error detection.

The barrier mechanism is made up of the following operations:

Initialization
Open
Close
Order

The open and close operations define a span-of-time interval for error detection and ordering. The initialization operation enables barrier creation for each imported segment, as well as barrier type specification. The only barrier type currently supported has a span-of-time scope per segment. Use a type argument value of RSM_BAR_DEFAULT.

Successfully performing a close operation guarantees the successful completion of covered access operations, which take place between the barrier open and the barrier close. After a barrier open operation, failures of individual data access operations, both reads and writes, are not reported until the barrier close operation.

To impose a specific order of write completion within a barrier's scope, use an explicit barrier-order operation. A write operation that is issued before the barrier-order operation finishes before operations that are issued after the barrier-order operation. Write operations within a given barrier scope are ordered with respect to another barrier scope.

Initialize Barrier

int rsm_memseg_import_init_barrier(rsm_memseg_import_handle_t im_memseg, rsm_barrier_type_t type, rsmapi_barrier_t *barrier);

Note –

At present, RSM_BAR_DEFAULT is the only supported type.

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_SEG_HNDL: Invalid segment handle
RSMERR_BAD_BARRIER_PTR: Invalid barrier pointer
RSMERR_INSUFFICIENT_MEM: Insufficient memory

Open Barrier

int rsm_memseg_import_open_barrier(rsmapi_barrier_t *barrier);

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_SEG_HNDL: Invalid segment handle
RSMERR_BAD_BARRIER_PTR: Invalid barrier pointer

Close Barrier

int rsm_memseg_import_close_barrier(rsmapi_barrier_t *barrier);

This function closes the barrier and flushes all store buffers. This call assumes that the calling process will retry all remote memory operations since the last rsm_memseg_import_open_barrier call if the call to rsm_memseg_import_close_barrier() fails.

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_SEG_HNDL: Invalid segment handle
RSMERR_BAD_BARRIER_PTR: Invalid barrier pointer
RSMERR_BARRIER_UNINITIALIZED: Barrier not initialized
RSMERR_BARRIER_NOT_OPENED: Barrier not opened
RSMERR_BARRIER_FAILURE: Memory access error
RSMERR_CONN_ABORTED: Connection aborted

Order Barrier

int rsm_memseg_import_order_barrier(rsmapi_barrier_t *barrier);

This function flushes all store buffers.

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_SEG_HNDL: Invalid segment handle
RSMERR_BAD_BARRIER_PTR: Invalid barrier pointer
RSMERR_BARRIER_UNINITIALIZED: Barrier not initialized
RSMERR_BARRIER_NOT_OPENED: Barrier not opened
RSMERR_BARRIER_FAILURE: Memory access error
RSMERR_CONN_ABORTED: Connection aborted

Destroy Barrier

int rsm_memseg_import_destroy_barrier(rsmapi_barrier_t *barrier);

This function deallocates all barrier resources.

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_SEG_HNDL: Invalid segment handle
RSMERR_BAD_BARRIER_PTR: Invalid barrier pointer

Set Mode

int rsm_memseg_import_set_mode(rsm_memseg_import_handle_t im_memseg, rsm_barrier_mode_t mode);

This function supports the optional explicit barrier scoping that is available in the put routines. The two valid barrier modes are RSM_BARRIER_MODE_EXPLICIT and RSM_BARRIER_MODE_IMPLICIT. The default value of the barrier mode is RSM_BARRIER_MODE_IMPLICIT. While in implicit mode, an implicit barrier open and barrier close is applied to each put operation. Before setting the barrier mode value to RSM_BARRIER_MODE_EXPLICIT, use the rsm_memseg_import_init_barrier routine to initialize a barrier for the imported segment im_memseg.

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_SEG_HNDL: Invalid segment handle

Get Mode

int rsm_memseg_import_get_mode(rsm_memseg_import_handle_t im_memseg, rsm_barrier_mode_t *mode);

This function obtains the current mode value for barrier scoping in the put routines.

Return Values: Returns 0 if successful. Returns an error value otherwise.

RSMERR_BAD_SEG_HNDL: Invalid segment handle.