Solaris Cluster OS™ systems can be configured with a memory-based interconnect such as Dolphin-SCI and layered system software components. These components implement a mechanism for user-level inter-node messaging that is based on direct access to memory residing on remote nodes. This mechanism is referred to as Remote Shared Memory (RSM). This chapter defines the RSM Application Programming Interface (RSMAPI).
API Framework describes the RSM API framework.
API Library Functions covers RSM API library functions.
RSMAPI Usage Example shows an example of use.
In the shared memory model, an application process creates an RSM export segment from the process's local address space. One or more remote application processes create an RSM import segment with a virtual connection between export and import segments across the interconnect. All processes make memory references for the shared segment with addresses local to their specific address space.
An application process creates an RSM export segment by allocating locally addressable memory to the export segment. This allocation is done by using one of the standard Solaris interfaces, such as System V Shared Memory, mmap(2), or valloc(3C). The process then calls on the RSMAPI for the creation of a segment, which provides a reference handle for the allocated memory. The RSM segment is published through one or more interconnect controllers. A published segment is remotely accessible. A list of access privileges for the nodes that are permitted to import the segment is also published.
A segment ID is assigned to the exported segment. This segment ID, along with the cluster node ID of the creating process, allows an importing process to uniquely specify an export segment. Successfully creating an export segment returns a segment handle to the process for use in subsequent segment operations.
An application process obtains access to a published segment by using the RSMAPI to create an import segment. After creating the import segment, the application process forms a virtual connection across the interconnect. Successfully creating this import segment returns an RSM import segment handle to the application process for use in subsequent segment import operations. After establishing the virtual connection, the application might request RSMAPI to provide a memory map for local access, if supported by the interconnect. If memory mapping is not supported, the application can use memory access primitives provided by RSMAPI.
The RSMAPI provides a mechanism to support remote access error detection and to resolve write-order memory model issues. This mechanism is called a barrier.
RSMAPI provides a notification mechanism to synchronize local and remote accesses. An export process can call a function to block while an import process finishes a data write operation. When the import process finishes writing, the process unblocks the export process by calling a signal function. Once unblocked, the export process processes the data.
The RSM application support components are delivered in software packages as follows:
A shared library (/usr/lib/librsm.so) that exports the RSMAPI functions.
A Kernel Agent (KA) pseudo device driver (/usr/kernel/drv/rsm) that interfaces with the memory interconnect driver through the RSMAPI interface on behalf of the user library.
A cluster interface module for obtaining interconnect topology.
Interconnect driver service module (/kernel/misc/rsmops).
Header files providing API function and data structure prototypes (/opt/SUNWrsmdk/include).
An optional extension to librsm.so that provides RSM support for the specific interconnect is configured in the system. The extension is provided in the form of a library, librsminterconnect.so.
The API library functions support the following operations:
Interconnect controller operations
Cluster topology operations
Memory segment operations, including segment management and data access
Barrier operations
Event operations
The controller operations provide mechanisms for obtaining access to a controller. Controller operations can also determine the characteristics of the underlying interconnect. The following list contains information on controller operations:
Get controller
Get controller attributes
Release controller
The rsm_get_controller operation acquires a controller handle for the given controller instance, such as sci0 or loopback. The returned controller handle is used for subsequent RSM library calls.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid controller handle
Controller not present
Insufficient memory
Invalid library version
Bad address
This function releases the controller associated with the given controller handle. Each call to rsm_release_controller must have a matching rsm_get_controller. When all the controller handles associated with a controller are released, the system resources associated with the controller are freed. Attempting to access a controller handle, or attempting to access import or export segments on a released controller handle, is not legal. The results of such an attempt are undefined.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid controller handle
This function retrieves attributes for the specified controller handle. The following list describes the currently defined attributes for this function:
typedef struct { uint_t attr_direct_access_sizes; uint_t attr_atomic_sizes; size_t attr_page_size; size_t attr_max_export_segment_size; size_t attr_tot_export_segment_size; ulong_t attr_max_export_segments; size_t attr_max_import_map_size; size_t attr_tot_import_map_size; ulong_t attr_max_import_segments; } rsmapi_controller_attr_t;
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid controller handle
Bad address
The key interconnect data required for export operations and import operations are:
Export cluster node ID
Import cluster node ID
Controller name
An application component that exports memory can use the data provided by the interface to find the set of existing local controllers. The data provided by the interface can also be used to correctly assign controllers for the creation and publishing of segments. Application components can efficiently distribute exported segments over the set of controllers that is consistent with the hardware interconnect and with the application software distribution.
An application component that is importing memory must be informed of the segment IDs and controllers used in the memory export. This information is typically conveyed by a predefined segment and controller pair. The importing component can use the topology data to determine the appropriate controllers for the segment import operations.
This function returns a pointer to the topology data in a location specified by an application pointer. The topology data structure is defined next.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid topology pointer
Insufficient memory
Insufficient memory
The rsm_free_interconnect_topology operation frees the memory allocated by rsm_get_interconnect_topology.
Return Values: None.
The pointer returned from rsm_get_topology_data references a rsm_topology_t structure. This structure provides the local node ID and an array of pointers to a connections_t structure for each local controller.
typedef struct rsm_topology { rsm_nodeid_t local_nodeid; uint_t local_cntrl_count; connections_t *connections[1]; } rsm_topology_t;
RSM segment IDs can be specified by the application or generated by the system using the rsm_memseg_export_publish() function. Applications that specify segment IDs require a reserved range of segment IDs to use. To reserve a range of segment IDs, use rsm_get_segmentid_range and define the reserved range of segment IDs in the segment ID configuration file /etc/rsm/rsm.segmentid. The rsm_get_segmentid_range function can be used by applications to obtain the segment ID range that is reserved for the applications. This function reads the segment ID range defined in the /etc/rsm/rsm.segmentid file for a given application ID.
An application ID is a null-terminated string that identifies the application. The application can use any value equal to or greater than baseid and less than baseid+length. If baseid or length are modified, the segment ID returned to the application might be outside the reserved range. To avoid this problem, use an offset within the range of reserved segment IDs to obtain a segment ID.
Entries in the /etc/rsm/rsm.segmentid file are of the form:
#keyword appid baseid length reserve SUNWfoo 0x600000 100 |
The entries are composed of strings, which can be separated by tabs or blanks. The first string is the keyword reserve, followed by the application identifier, which is a string without spaces. Following the application identifier is the baseid, which is the starting segment ID of the reserved range in hexadecimal. Following the baseid is the length, which is the number of segment IDs that are reserved. Comment lines have a # in the first column. The file should not contain blank or empty lines. Segment IDs that are reserved for the system are defined in the /usr/include/rsm/rsm_common.h header file. The segment IDs that are reserved for the system cannot be used by the applications.
The rsm_get_segmentid_range function returns 0 to indicate success. If the function fails, the function returns one of the following error values:
The address that is passed is invalid
Application ID not defined in the/etc/rsm/rsm.segmentid file
The configuration file /etc/rsm/rsm.segmentid is not present or not readable. The file format's configuration is incorrect
An RSM segment represents a set of (generally) non-contiguous physical memory pages mapped to a contiguous virtual address range. RSM segment export and segment import operations enable the sharing of regions of physical memory among systems on an interconnect. A process of the node on which the physical pages reside is referred to as the exporter of the memory. An exported segment that is published for remote access will have a segment identifier that is unique for the given node. The segment ID might be specified by the exporter or assigned by the RSMAPI framework.
Processes of nodes on the interconnect obtain access to exported memory by creating an RSM import segment. The RSM import segment has a connection with an exported segment, rather than local physical pages. When the interconnect supports memory mapping, importers can read and write the exported memory by using the local memory-mapped addresses of the import segment. When the interconnect does not support memory mapping, the importing process uses memory access primitives.
When exporting a memory segment, the application begins by allocating memory in its virtual address space through the normal operating system interfaces such as the System V Shared Memory Interface, mmap, or valloc. After allocating memory, the application calls the RSMAPI library interfaces to create and label a segment. After labelling the segment, the RSMAPI library interfaces bind physical pages to the allocated virtual range. After binding the physical pages, the RSMAPI library interfaces publish the segment for access by importing processes.
If virtual address space is obtained by using mmap, the mapping must be MAP_PRIVATE.
Export side memory segment operations include:
Memory segment creation and destruction
Memory segment publishing and unpublishing
Rebinding backing store for a memory segment
Establishing a new memory segment with rsm_memseg_export_create enables the association of physical memory with the segment at creation time. The operation returns an export-side memory segment handle to the new memory segment. The segment exists for the lifetime of the creating process or until destroyed with rsm_memseg_export_destroy.
If destroy operation is performed before an import side disconnect, the disconnect is forced.
This function creates a segment handle. After the segment handle is created, the segment handle is bound to the specified virtual address range [vaddr..vaddr+size]. The range must be valid and aligned on the controller's alignment property. The flags argument is a bitmask, which enables:
Unbinding on the segment
Rebinding on the segment
Passing RSM_ALLOW_REBIND to flags
Support of lock operations
Passing RSM_LOCK_OPS to flags
The RSM_LOCK_OPS flag is not included in the initial release of RSMAPI.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid controller handle
Controller not present
Invalid segment handle
Length zero or length exceeds controller limits
Invalid address
Permission denied
Insufficient memory
Insufficient resources
Address not aligned on page boundary
Operation interrupted by signal
This function deallocates segment and its free resources. All importing processes are forcibly disconnected.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
pollfd in use
The publish operation enables the importing of a memory segment by other nodes on the interconnect. An export segment might be published on multiple interconnect adapters.
The segment ID might be specified from within authorized ranges or specified as zero, in which case a valid segment ID is generated by the RSMAPI framework and is passed back.
The segment access control list is composed of pairs of node ID and access permissions. For each node ID specified in the list, the associated read/write permissions are provided by three octal digits for owner, group and other, as with Solaris file permissions. In the access control list, each octal digit can have the following values:
Write access.
Read only access.
Read and write access.
An access permission value of 0624 specifies the following kind of access:
An importer with the same uid as the exporter has both read and write access.
An importer with the same gid as the exporter has write access only.
All other importers have read access only.
When an access control list is provided, nodes not included in the list cannot import the segment. However, if the access list is null, any node can import the segment. The access permissions on all nodes equal the owner-group-other file creation permissions of the exporting process.
Node applications have the responsibility of managing the assignment of segment identifiers to ensure uniqueness on the exporting node.
typedef struct { rsm_node_id_t ae_node; /* remote node id allowed to access resource */ rsm_permission_t ae_permissions; /* mode of access allowed */ }rsmapi_access_entry_t;.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Segment already published
Invalid access control list
Invalid segment identifier
Segment identifier in use
Segment identifier reserved
Not creator of segment
Bad address
Insufficient memory
Insufficient resources
Authorized Segment ID Ranges:
0
0x0FFFFF
0x100000
0x1FFFFF
0x200000
0x2FFFFF
0x300000
0x3FFFFF
0x400000
0x4FFFFF
The following range is reserved for allocation by the system when the publish value is zero.
0x80000000
0xFFFFFFF
This function establishes a new node access list and segment access mode. These changes only affect future import calls and do not revoke already granted import requests.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Segment not published
Invalid access control list
Not creator of segment
Insufficient memory
Insufficient resources
Operation interrupted by signal
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Segment not published
Not creator of segment
Operation interrupted by signal
The rebind operation releases the current backing store for an export segment. After releasing the current backing store for an export segment, the rebind operation allocates a new backing store. The application must first obtain a new virtual memory allocation for the segment. This operation is transparent to importers of the segment.
The application has the responsibility of preventing access to segment data until the rebind operation is complete. Retrieving data from a segment during rebinding does not cause a system failure, but the results of such an operation are undefined.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Invalid length
Invalid address
Rebind not allowed
Not creator of segment
Permission denied
Insufficient memory
Insufficient resources
Operation interrupted by signal
The following list describes Import-side operations:
Memory segment connection and disconnection
Access to imported segment memory
Barrier operations used to impose order on data access operations and for access error detection
The connect operation is used to create an RSM import segment and form a logical connection with an exported segment.
Access to imported segment memory is provided by three interface categories:
Segment access.
Data transfer.
Segment memory mapping.
This function connects to segment segment_id on remote node node_id by using the specified permission perm. The function returns a segment handle after connecting to the segment.
The argument perm specifies the access mode requested by the importer for this connection. To establish the connection, the access permissions specified by the exporter are compared to the access mode, user ID, and group ID used by the importer. If the request mode is not valid, the connection request is denied. The perm argument is limited to the following octal values:
Read mode
Write mode
Read/write mode
The specified controller must have a physical connection to the controller that is used in the export of the segment.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid controller handle
Controller not present
Invalid segment handle
Permission denied
Segment not published to node
No such segment published
Remote node not reachable
Connection interrupted
Insufficient memory
Insufficient resources
Bad address
This function disconnects a segment. This function frees a segment's resources after disconnecting a segment. All existing mappings to the disconnected segment are removed. The handle im_memseg is freed.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Segment still mapped
pollfd in use
The following interfaces provide a mechanism for transferring between 8 bits and 64 bits of data. The get interfaces use a repeat count (rep_cnt) to indicate the number of data items of a given size the process will read from successive locations. The locations begin at byte offset offset in the imported segment. The data is written to successive locations that begin at datap. The put interfaces use a repeat count (rep_cnt). The count indicates the number of data items the process will read from successive locations. The locations begin at datap. The data is then written to the imported segment at successive locations. The locations begin at the byte offset specified by the offset argument.
These interfaces also provide byte swapping in case the source and destination have incompatible endian characteristics.
Function Prototypes:
int rsm_memseg_import_get8(rsm_memseg_import_handle_t im_memseg, off_t offset, uint8_t *datap, ulong_t rep_cnt);The following interfaces are intended for data transfers that are larger than the ones supported by the segment access operations.
int rsm_memseg_import_put(rsm_memseg_import_handle_t im_memseg, off_t offset, void *src_addr, size_t length);This function copies data from local memory, specified by the src_addr and length, to the corresponding imported segment locations specified by the handle and offset.
int rsm_memseg_import_get(rsm_memseg_import_handle_t im_memseg, off_t offset, void *dst_addr, size_t length);This function is similar to rsm_memseg_import_put(), but data flows from the imported segment into local regions defined by the dest_vec argument
The put and get routines write or read the specified quantity of data from the byte offset location specified by the argument offset. The routines begin at the base of the segment. The offset must align at the appropriate boundary. For example, rsm_memseg_import_get64() requires that offset and datap align at a double-word boundary, while rsm_memseg_import_put32() requires an offset that is aligned at a word boundary.
By default, the barrier mode attribute of a segment is implicit. Implicit barrier mode means that the caller assumes the data transfer has completed or has failed upon return from the operation. Because the default barrier mode is implicit, the application must initialize the barrier. The application initializes the barrier by using the rsm_memseg_import_init_barrier() function before calling put or get routines when using the default mode. To use the explicit operation mode, the caller must use a barrier operation to force the completion of a transfer. After forcing the completion of the transfer, the caller must determine if any errors have occurred as a result of the forced completion.
An import segment can be partially mapped by passing an offset in the rsm_memseg_import_map() routine. If the import segment is partially mapped, the offset argument in the put or get routines is from the base of the segment. The user must make sure that the correct byte offset is passed to put and get routines.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Bad address
Invalid memory alignment
Invalid offset
Invalid length
Permission denied
Barrier not initialized
I/O completion error
Connection aborted
Insufficient resources
The rsm_memseg_import_putv() and rsm_memseg_import_getv() functions allow the use of a list of I/O requests instead of a single source and single destination address.
Function Prototypes:
int rsm_memseg_import_putv(rsm_scat_gath_t *sg_io);The I/O vector component of the scatter-gather list (sg_io) enables the specification of local virtual addresses or local_memory_handles. Handles are an efficient way to repeatedly use a local address range. Allocated system resources, such as locked down local memory, are maintained until the handle is freed. The supporting functions for handles are rsm_create_localmemory_handle() and rsm_free_localmemory_handle().
You can gather virtual addresses or handles into the vector in order to write to a single remote segment. You can also scatter the results of reading from a single remote segment to the vector of virtual addresses or handles.
I/O for the entire vector is initiated before returning. The barrier mode attribute of the import segment determines whether the I/O has completed before the function returns. Setting the barrier mode attribute to implicit guarantees that data transfer is completed in the order entered in the vector. An implicit barrier open and close surrounds each list entry. If an error is detected, I/O for the vector is terminated and the function returns immediately. The residual count indicates the number of entries for which the I/O either did not complete or was not initiated.
You can specify that a notification event be sent to the target segment when a putv or getv operation is successful. To specify the delivery of a notification event, specify the RSM_IMPLICIT_SIGPOST value in the flags entry of the rsm_scat_gath_t structure. The flags entry can also contain the value RSM_SIGPOST_NO_ACCUMULATE, which is passed on to the signal post operation if RSM_IMPLICIT_SIGPOST is set.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid scatter-gather structure pointer
Invalid segment handle
Invalid controller handle
Bad address
Invalid offset
Invalid length
Permission denied
I/O completion error
Connection aborted
Insufficient resources
Operation interrupted by signal
This function obtains a local handle for use in the I/O vector for subsequent calls to putv or getv. Freeing the handle as soon as possible conserves system resources, notably the memory spanned by the local handle, which might be locked down.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid controller handle
Invalid local memory handle
Invalid length
Invalid address
Insufficient memory
This function releases the system resources associated with the local handle. While all handles that belong to a process are freed when the process exits, calling this function conserves system resources.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid controller handle
Invalid local memory handle
The following example demonstrates the definition of primary data structures.
typedef void *rsm_localmemory_handle_t typedef struct { ulong_t io_request_count; number of rsm_iovec_t entries ulong_t io_residual_count; rsm_iovec_t entries not completed in flags; rsm_memseg_import_handle_t remote_handle; opaque handle for import segment rsm_iovec_t *iovec; pointer to array of io_vec_t } rsm_scat_gath_t; typedef struct { int io_type; HANDLE or VA_IMMEDIATE union { rsm_localmemory_handle_t handle; used with HANDLE caddr_t virtual_addr; used with VA_IMMEDIATE } local; size_t local_offset; offset from handle base vaddr size_t import_segment_offset; offset from segment base vaddr size_t transfer_length; } rsm_iovec_t;
Mapping operations are only available for native architecture interconnects such as Dolphin-SCI or NewLink. Mapping a segment grants CPU memory operations access to that segment, saving the overhead of calling memory access primitives.
int rsm_memseg_import_map(rsm_memseg_import_handle_t im_memseg, void **address, rsm_attribute_t attr, rsm_permission_t perm, off_t offset, size_t length);This function maps an imported segment into the caller address space. If the attribute RSM_MAP_FIXED is specified, the function maps the segment at the value specified in **address.
typedef enum { RSM_MAP_NONE = 0x0, /* system will choose available virtual address */ RSM_MAP_FIXED = 0x1, /* map segment at specified virtual address */ } rsm_map_attr_t;
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Invalid address
Invalid length
Invalid offset
Invalid permissions
Segment already mapped
Segment not connected
Connection aborted
Error during mapping
Address not aligned on page boundary
Unmap segment
int rsm_memseg_import_unmap(rsm_memseg_import_handle_t im_memseg);This function unmaps an imported segment from user virtual address space.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Use Barrier operations to resolve order-of-write-access memory model issues. Barrier operations also provide remote memory access error detection.
The barrier mechanism is made up of the following operations:
Initialization
Open
Close
Order
Successfully performing a close operation guarantees the successful completion of covered access operations, which take place between the barrier open and the barrier close. After a barrier open operation, failures of individual data access operations, both reads and writes, are not reported until the barrier close operation.
To impose a specific order of write completion within a barrier's scope, use an explicit barrier-order operation. A write operation that is issued before the barrier-order operation finishes before operations that are issued after the barrier-order operation. Write operations within a given barrier scope are ordered with respect to another barrier scope.
Initialize Barrier
int rsm_memseg_import_init_barrier(rsm_memseg_import_handle_t im_memseg, rsm_barrier_type_t type, rsmapi_barrier_t *barrier);At present, RSM_BAR_DEFAULT is the only supported type.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Invalid barrier pointer
Insufficient memory
Open Barrier
int rsm_memseg_import_open_barrier(rsmapi_barrier_t *barrier);Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Invalid barrier pointer
Close Barrier
int rsm_memseg_import_close_barrier(rsmapi_barrier_t *barrier);This function closes the barrier and flushes all store buffers. This call assumes that the calling process will retry all remote memory operations since the last rsm_memseg_import_open_barrier call if the call to rsm_memseg_import_close_barrier() fails.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Invalid barrier pointer
Barrier not initialized
Barrier not opened
Memory access error
Connection aborted
Order Barrier
int rsm_memseg_import_order_barrier(rsmapi_barrier_t *barrier);This function flushes all store buffers.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Invalid barrier pointer
Barrier not initialized
Barrier not opened
Memory access error
Connection aborted
Destroy Barrier
int rsm_memseg_import_destroy_barrier(rsmapi_barrier_t *barrier);This function deallocates all barrier resources.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Invalid barrier pointer
Set Mode
int rsm_memseg_import_set_mode(rsm_memseg_import_handle_t im_memseg, rsm_barrier_mode_t mode);This function supports the optional explicit barrier scoping that is available in the put routines. The two valid barrier modes are RSM_BARRIER_MODE_EXPLICIT and RSM_BARRIER_MODE_IMPLICIT. The default value of the barrier mode is RSM_BARRIER_MODE_IMPLICIT. While in implicit mode, an implicit barrier open and barrier close is applied to each put operation. Before setting the barrier mode value to RSM_BARRIER_MODE_EXPLICIT, use the rsm_memseg_import_init_barrier routine to initialize a barrier for the imported segment im_memseg.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Get Mode
int rsm_memseg_import_get_mode(rsm_memseg_import_handle_t im_memseg, rsm_barrier_mode_t *mode);This function obtains the current mode value for barrier scoping in the put routines.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle.
Event operations enable processes synchronization on memory access events. If a process cannot use the rsm_intr_signal_wait() function, it can multiplex event waiting by obtaining a poll descriptor with rsm_memseg_get_pollfd() and using the poll system call.
Using the rsm_intr_signal_post() and rsm_intr_signal_wait() operations incurs the need to process of ioctl calls to the kernel.
Post Signal
int rsm_intr_signal_post(void *memseg, uint_t flags);The void pointer *memseg can be type cast to either an import segment handle or an export segment handle. If *memseg refers to an import handle, this function sends a signal the exporting process. If *memseg refers to an export handle, this function sends a signal to all importers of that segment. Setting the flags argument to RSM_SIGPOST_NO_ACCUMULATE discards this event if an event is already pending for the target segment.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Remote node not reachable
Wait for Signal
int rsm_intr_signal_wait(void * memseg, int timeout);The void pointer *memseg can be type cast to either an import segment handle or an export segment handle. The process blocks for up to timeout milliseconds or until an event occurs. If the value is -1, the process blocks until an event occurs or until interrupted.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Timer expired
Wait interrupted
Get pollfd
int rsm_memseg_get_pollfd(void *memseg, struct pollfd *pollfd);This function initializes the specified pollfd structure with a descriptor for the specified segment and the singular fixed event generated by rsm_intr_signal_post(). Use the pollfd structure with the poll system call to wait for the event signalled by rsm_intr_signal_post. If the memory segment is not currently published, the poll system call does not return a valid pollfd. Each successful call increments a pollfd reference count for the specified segment.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
Release pollfd
int rsm_memseg_release_pollfd(oid *memseg);This call decrements the pollfd reference count for the specified segment. If the reference count is nonzero, operations that unpublish, destroy, or unmap the segment fail.
Return Values: Returns 0 if successful. Returns an error value otherwise.
Invalid segment handle
These usage notes describe general considerations for the export and import sides of a shared-memory operation. These usage notes also contain general information regarding segments, file descriptors, and RSM configurable parameters.
The system allocates a file descriptor, which is inaccessible to the application importing or exporting memory, for each export operation or import operation. The default limit on file descriptor allocation for each process is 256. The importing or exporting application must adjust the allocation limit appropriately. If the application increases the file descriptor limit beyond 256, the values of the file descriptors that are allocated for export segments and import segments starts at 256. These file descriptor values are chosen to avoid interfering with normal file descriptor allocation by the application. This behavior accommodates the use of certain libc functions in 32-bit applications that only work with file descriptor values lower than 256.
The application must prevent access to segment data until the rebind operation is complete. Segment data access during rebind does not cause a system failure, but data content results are undefined. The virtual address space must be currently mapped and valid.
The controller that is specified for a segment import must have a physical connection with the controller that is used in the export of the segment.
The SUNWrsm software package includes an rsm.conf file. This file is located in /usr/kernel/drv. This file is a configuration file for RSM. The rsm.conf file can be used to specify values for certain configurable RSM properties. The configurable properties currently defined in rsm.conf include max-exported-memory and enable-dynamic-reconfiguration.
This property specifies an upper limit on the amount of exportable memory. The upper limit is expressed as a percentage of total available memory. Giving this property a value of zero indicates that the amount of exportable memory is unlimited.
The value of this property indicates whether dynamic reconfiguration is enabled. A value of zero indicates dynamic reconfiguration is disabled. A value of one enables dynamic reconfiguration support. The default value for this property is one.
This section provides a simple program to illustrate the usage of the RSMAPI. The program runs on two nodes: an exporter node and an importer node. The exporter node creates and publishes a memory segment, then waits for a message to be written in the segment. The importer node connects to the exported segment, writes a message, and then signals the exporter.
/* * Copyright (c) 1998 by Sun Microsystems, Inc. * All rights reserved. */ #include <stdio.h> #include <rsm/rsmpai.h> #include <errno.h> /* To run this program do the following: First node(assuming node id = 1): rsmtest -e -n 2 Second node(assuming node id = 2): rsmtest -i -n 1 The program will prompt the importer for a message at the console. Enter any message and hit return. The message will be displayed on the export console. */ typedef struct { char out; char in; char data[1]; }msg_t; #define SEG_ID 0x400000 #define EXPORT 0 #define IMPORT 1 #define BUFSIZE (1024 * 8) #define DEFAULT_SEGSZ BUFSIZE #define RSM_PERM_READ 0400 #define RSM_PERM_WRITE 0200 #define RSM_PERM_RDWR (RSM_PERM_READ|RSM_PERM_WRITE) #define RSM_ACCESS_TRUSTED 0666 rsm_topology_t *tp; int iterations = 10; int mode = EXPORT; int test = 0; char *buf; int buflen = BUFSIZE; int offset = 0; volatile char *iva; int status; rsm_memseg_id_t segid; rsmapi_controller_handle_t ctrl; rsmapi_controller_attr_t attr; rsm_memseg_export_handle_t seg; rsm_memseg_import_handle_t imseg; rsm_access_entry_t list[2]; rsm_node_id_t dest; extern void *valloc(size_t); extern void exit(); extern void sleep(); extern int atoi(const char *); /* The following function exports a segment and publishes it. */ static int export() { int i; /* allocate and clear memory */ buf = (char *)valloc(buflen); if (!buf) { (void) fprintf(stderr, "Unable to allocate memory\n"); exit (1); } for (i = 0; i < buflen; i++) buf[i] = 0; /* Create an export memory segment */ status = rsm_memseg_export_create(ctrl, &seg, (void *)buf, buflen); if (status != 0) { (void) fprintf(stderr, "unable to create an exported segment %d\n", status); exit(1); } /* Set up access list for publishing to nodes 1 and 2 */ list[0].ae_node = tp->topology_hdr.local_nodeid ; /* Allow read and write permissions */ list[0].ae_permission = RSM_ACCESS_TRUSTED; list[1].ae_node = tp->topology_hdr.local_nodeid + 1; /* Allow read and write permissions */ list[1].ae_permission = RSM_ACCESS_TRUSTED; /* Publish the created export segment */ status = rsm_memseg_export_publish(seg, &segid, list, 0); if (status != 0) { (void) fprintf(stderr, "unable to pub segment %d\n", status); exit(1); } return (0); } /* The following function is used to connect to an exported memory segment. */ static void import() { /* Connect to exported segment and set up mapping for * access through local virtual addresses. */ again: status = rsm_memseg_import_connect(ctrl, dest, segid, RSM_PERM_RDWR, &imseg); if (status != 0) { (void) fprintf(stderr, "unable to conect to segment %x err %x\n", segid, status); sleep(1); goto again; } iva = NULL; status = rsm_memseg_import_map(imseg, (void **)&iva, RSM_MAP_NONE, RSM_PERM_RDWR, 0, buflen); if (status != 0) { (void) fprintf(stderr, "unable to mmap segment %d\n", status); exit(1); } } /* Unpublish and destroy the export segment */ static void export_close() { again: status = rsm_memseg_export_unpublish(seg); if (status != 0) { (void) fprintf(stderr, "unable to create an unpub segment %d\n", status); sleep(10); goto again; } status = rsm_memseg_export_destroy(seg); if (status != 0) { (void) fprintf(stderr, "unable to destroy segment %d\n", status); exit(1); } } /* Unmap the virtual address mapping and disconnect the segment */ static void import_close() { status = rsm_memseg_import_unmap(imseg); if (status != 0) { (void) fprintf(stderr, "unable to unmap segment %d\n", status); exit(1); } status = rsm_memseg_import_disconnect(imseg); if (status != 0) { (void) fprintf(stderr, "unable to disconnect segment %d\n", status); exit(1); } } static void test0() { volatile msg_t *mbuf; /* Barrier to report error */ rsmapi_barrier_t bar; int i; if (mode == EXPORT) { (void) export(); mbuf = (msg_t *)(buf + offset); mbuf->in = mbuf->out = 0; } else { import(); mbuf = (msg_t *)(iva + offset); rsm_memseg_import_init_barrier(imseg, RSM_BARRIER_NODE, &bar); } (void) printf("Mbuf is %x\n", (uint_t)mbuf); while (iterations-- > 0) { int e; switch (mode) { case EXPORT: while (mbuf->out == mbuf->in) { (void) rsm_intr_signal_wait(seg, 1000); } (void) printf("msg [0x%x %d %d] ", (uint_t)mbuf, (int)mbuf->out, mbuf->in); for (i = 0; mbuf->data[i] != '\0' && i < buflen; i++) { (void) putchar(mbuf->data[i]); mbuf->data[i] = '?'; } (void) putchar('\n'); mbuf->out++; break; case IMPORT: (void) printf("Enter msg [0x%x %d]: ", (uint_t)mbuf, mbuf->out, mbuf->in); retry: e = rsm_memseg_import_open_barrier(&bar); if (e != 0) { (void) printf("Barrier open failed %x\n", e); exit(1); } for (i = 0; (mbuf->data[i] = getchar()) != '\n'; i++) ; mbuf->data[i] = '\0'; rsm_memseg_import_order_barrier(&bar); mbuf->in++; e = rsm_memseg_import_close_barrier(&bar); if (e != 0) { (void) printf("Barrier close failed, %d\n", e); goto retry; } (void)rsm_intr_signal_post(imseg); break; } } if (mode == IMPORT) { import_close(); } else { export_close(); } } void main(int argc, char *argv[]) { int unit = 0; char *device = "sci0"; int i; segid = SEG_ID; buflen = DEFAULT_SEGSZ; while ((i = getopt(argc, argv, "OCGeid:b:sl:n:k:t:c:u:v")) != -1) { switch (i) { case 'e': mode = EXPORT; break; case 'i': mode = IMPORT; break; case 'n': dest = atoi(optarg); if ((int)dest < 0) dest = 0; break; default: (void) fprintf(stderr, "Usage: %s -ei -n dest\n", argv[0]); exit(1); } } status = rsm_get_controller(device, &ctrl); if (status != 0) { (void) fprintf(stderr, "Unable to get controller\n"); exit(1); } status = rsm_get_controller_attr(ctrl, &attr); status = rsm_get_interconnect_topology(&tp); if (status != 0) { (void) fprintf(stderr, "Unable to get topology\n"); exit(1); } else { (void) printf("Local node id = %d\n", tp->topology_hdr.local_nodeid); } if (dest == 0) { dest = tp->topology_hdr.local_nodeid; (void) printf("Dest is adjusted to %d\n", dest); } switch (test) { case 0: test0(); break; default: (void) printf("No test executed\n"); break; } }