Go to main content

man pages section 3: Library Interfaces and Headers

Exit Print View

Updated: Wednesday, July 27, 2022
 
 

seg_exp(3LIB)

Name

libcmi, cmi_ini, atm_cas, cmi_get_error, attr_get, cflush, close_fb, cmi_ctl, cmi_enb, evt_get, evt_ret, fini, flush_fb, ini_th, mb_fn, open_fb, rmb_fn, rseg_del, seg_at, seg_ctl, seg_dt, seg_exp, seg_get, seg_imp, tok_del, tok_new, wmb_fn - Coherent Memory Interface

Synopsis

#include <cmi.h>

cmi_ctxt *cmi_ini (uint16_t verno, cmi_cbs *callback)

cmi_error cmi_get_error (cmi_ctxt *ctxt)

int atm_cas(cmi_ctxt *ctxt, uint64_t *addr, uint64_t
   cmpval, uint64_t swpval, uint64_t *rval)

int attr_get(cmi_ctxt *ctxt, cmi_seg seg, int32_t cmd, void
   *optval, size_t *optlen)

int cflush(cmi_ctxt *ctxt, void *vaddr[], int32_t addrcnt)

int close_fb(cmi_ctxt *ctxt, cmi_fb epoch)

int cmi_ctl(cmi_ctxt *ctxt, int32_t cmd, cmi_ctl_cfg *cfg)

int cmi_enb(cmi_ctxt *ctxt, int enable)

cmi_event* evt_get(cmi_ctxt *ctxt)

void evt_ret(cmi_event *event, cmi_event_ret status)

int fini(cmi_ctxt *ctxt)

int flush_fb(cmi_ctxt *ctxt, cmi_fb epoch)

int ini_th(cmi_ctxt *ctxt)

int mb_fn(cmi_ctxt *ctxt);

cmi_fb open_fb(cmi_ctxt *ctxt)

int rmb_fn(cmi_ctxt *ctxt);

int rseg_del(cmi_ctxt *ctxt, cmi_rseg *rseg)

void* seg_at(cmi_ctxt *ctxt, cmi_seg seg, void *addr,
   int32_t flags)

int seg_ctl(cmi_ctxt *ctxt, cmi_seg seg, int32_t cmd,
   cmi_ds *ds)

int seg_dt(cmi_ctxt *ctxt, cmi_seg seg, void *addr)

cmi_rseg* seg_exp(cmi_ctxt *ctxt, cmi_seg seg, int32_t
   attrib)

cmi_seg seg_get(cmi_ctxt *ctxt, size_t size, int32_t flags)

cmi_seg seg_imp(cmi_ctxt *ctxt, cmi_rseg *rseg)

int tok_del(cmi_ctxt *ctxt, cmi_token *tok)

cmi_token* tok_new(cmi_ctxt *ctxt, cmi_seg seg, cmi_naddr
   *naddr, cmi_acc flags)

cmi_seg seg_getva(cmi_ctxt *ctxt, void * va, size_t size,
   int32_t flags)

cmi_token* tok_newva(cmi_ctxt *ctxt, cmi_seg seg, void *va,
   size_t size, cmi_naddr *naddr, cmi_acc flags)

int wmb_fn(cmi_ctxt *ctxt);

Description

Coherent Memory Interface (CMI) exposes a Distributed Shared Memory abstraction. In order to support a diverse range of networking technologies, vendors, and approaches to CMI (hardware vs emulated 'Soft CMI' for instance) a portable interface is required. CMI clients (and possibly compiler backends) interact with the CMI using the Application Programming Interface (API) outlined in this document.

       Only cmi_ini() and cmi_get_error() are available externally.
       The rest of the functions are referenced by the function pointers
       defined in cmi_ctxt, which is returned by cmi_ini().

       To reference the rest of the functions, use the macro below, which is
       defined in /usr/include/cmi.h.

           CMIFN(ctxt,ver,fn)

       Here is an example of how to call seg_get() for API version 10.

           ctxt = cmi_ini(10, NULL);
           seg = CMIFN(ctxt, 10, seg_get)(ctxt, size, flags);


       Platform specific CMI interfaces are defined in cmi_impl.h, which is a
       platform specific header file located at /usr/include/cmi/<vendor name>
       These interfaces are platform specific implementations, which are
       optimized for the particular platform and/or support the platform
       specific features.
       To utilize such interfaces, the platform specific cmi_impl.h need to be
       included.

       All exceptions occurred on CMI segments will generate SIGSEGV with
       SEGV_CMI in si_code. CMI clients identify the type of CMI exception by
       si_errno and locate the faulty segment id by si_id.
       si_addr and si_pc are assigned the virtual address and the PC where the
       exception occurred.
       For more detail, see CMI Exceptions in cmi(5).

   cmi_ctxt* cmi_ini (uint16_t verno, cmi_cbs *callback)

       Initialize the CMI library

       Parameters:
	   verno Client requested API version number
	   callback CMI client callbacks.

       Returns:
	   ctxt CMI context for process. NULL on failure.

	   CMI_ERR_NOTSUPP Requested API version is not supported

	   CMI_ERR_NOMEM Out of resources allocating context (Such as the
	   maximum number of contexts are already open)

	   CMI_ERR_BOUND Calling thread is already associated with context

       This routine is called once per process to initialize the CMI library.
       This will be the first CMI routine invoked by a process. A successful
       call to cmi_ini() implicitly associates the calling thread with the
       context as if the thread had called ini_th(ctxt). A matching call to
       fini() is required to deallocate any thread specific resources
       allocated by the library.

       The version number field passed while initializing the library is the
       requested API version the client supports. The CMI library should set
       the API version supported by the library in the cmi_ctxt handle. The
       returned version should be the minimum of the client requested and
       library supported version. CMI will maintain binary ABI compatibility
       between minor releases. Major version releases may break binary
       compatibility. In such cases it is acceptable for the initialization of
       the library to fail and return a NULL handle. cmi_get_error() shall
       return CMI_ERR_NOTSUPP as the error code.

       Various CMI objects (context and segment handles, tokens etc.) are
       allocated by the library however in certain environments CMI clients
       may want to utilize special memory heaps to allocate CMI objects.
       Clients may provide memory management callbacks that the library must
       utilize for all memory allocation. CMI vendor library can assume that
       the specified memory allocator is thread safe. If no callbacks are
       specified the library can use any suitable mechanism (such as OS
       provided allocation routines) for allocation. Memory allocation failures
       from client callbacks should be handled similarly and return
       CMI_ERR_NOMEM. The callback vector need to be valid during the duration
       of this call. The library will cache the callback vector internally in
       the CMI context handle after the call is done.

       CMI libraries provide support for basic diagnosability to aid in
       debugging issues in the fields. Clients can request CMI library to
       trace one or more facilities (sub-components in CMI specification)
       along with level of tracing ranging from CMI_TRACE_LVL_ERROR (error
       conditions only) to CMI_TRACE_LVL_HIGHEST (verbose tracing). The
       facilities to trace are specified in cmi_cbs.trace_facilities. The
       following facilities are currently defined:

       o CMI_TRACE_FAC_INI: Trace library/platform initialization calls such
	 as cmi_ini()/cmi_ini_th() and cmi_fini().

       o CMI_TRACE_FAC_CTRL: Trace control operations in the library
	 (cmi_ctl(), seg_ctl(), attr_get()).

       o CMI_TRACE_FAC_SEG: Trace segment operations such as seg_get(),
	 seg_exp(), seg_imp(), seg_at(), seg_dt(), rseg_del().

       o CMI_TRACE_FAC_TOK: Trace all access token operations such as
	 tok_new(), and tok_del().

       o CMI_TRACE_FAC_EVT: Trace all event related operations such as
	 evt_get() and evt_ret().

       o CMI_TRACE_FAC_MEM: Trace all memory related operations such as flush
	 barriers (cmi_enb(), open_fb(), close_fb(), flush_fb()) and memory
	 barriers (cmi_rmb(), cmi_wmb() and cmi_mb()) along with cache flush
	 operations (cflush()) and atm_cas().

       Note:
         All operations traced by the library utilize the alert and log
	 callbacks provided by the client. If no such callback is provided
	 by the client then the library still trace the requested
	 facilities at the specified level to a platform specific location.


   cmi_error cmi_get_error (cmi_ctxt *ctxt)

       Return last error set for calling thread

       Parameters:
	   ctxt CMI Context associated with thread. Can be NULL if the
	   cmi_ini() or ini_th() calls failed. In this case return value
	   should be the reason the allocation/association failed.

       Returns:
	   cmi_error Last error set for thread.

       Clients should obtain error code immediately following a failed
       operation to avoid losing the error status as intervening operations
       that succeed may clear the error.


   int atm_cas(cmi_ctxt *ctxt, uint64_t *addr, uint64_t
     cmpval, uint64_t swpval, uint64_t *rval)

     Perform atomic Compare and Swap

     Parameters:
         ctxt CMI context
         addr Address to perform Compare and Swap
         cmpval Value to compare against
         swpval Value to swap if compare succeeds
         rval Address of location to store result

     Returns:
         0 on success. -1 on failure.

         CMI_ERR_INIT Calling thread is not associated with
         context

         CMI_ERR_INVAL Provided context is invalid

     Perform atomic compare and swap on provided address. If the
     current value of *addr is cmpval then write swpval intp
     *addr. The content of *addr before the operation is returned
     in rval. The address atm_cas() is being performed on must be
     within a segment. This segment may either be a remote CMI
     segment (accessed after importing segment via seg_imp) or a
     'local' CMI segment hosted on the same node and directly
     attached to via seg_at(). If the atm_cas() operation is performed
     on a remote segment (addr resides on a remote segment) the
     atm_cas() operation cannot utilize cached values.

     The result of the operation must be reflected on the
     home node. If the segment is un-accessible or the calling
     thread does not have access (access token revoked for
     instance) then the operation can fail with
     an access error/exception. The location of the result value
     (rval parameter) should be in local memory.

     Note:
       This atomic operation is a store memory barrier i.e. all stores
       preceding the atm_cas() are completed (globally visible) before
       returning from this call.

   int attr_get(cmi_ctxt *ctxt, cmi_seg seg, int32_t
     cmd, void *optval, size_t *optlen)

     Query CMI context/segment attributes

     Parameters:
         ctxt CMI context
         seg CMI segment to get attribute of
         cmd CMI context/segment attribute being queried
         optval Buffer to receive attribute query result
         optlen Size of the optval buffer

     Returns:
         0 on success. -1 on failure.

         CMI_ERR_INIT Calling thread is not associated with
         context

         CMI_ERR_INVAL Provided context, segment or command is
         invalid

         CMI_ERR_NOMEM Optval buffer too small to hold result

     The optval buffer is allocated by the client and used by the
     library to return the attribute query result. The size of
     the optval buffer on entry is set by the client in the
     optlen parameter. If the size of the optval buffer is not
     sufficient the query is failed with CMI_ERR_NOMEM and
     optlen is updated to the required size to hold the result.
     On successful return the size of the result is in the optlen
     value result parameter.

     The following attributes are currently defined:

     o CMI_ATTR_RSEG_SIZE  - Query the size of a remote segment handle.
       Optval buffer should be of size_t size.

     o CMI_ATTR_TOKEN_SIZE - Query the size of an access token.
       Optval buffer should be of size_t size.

     o CMI_ATTR_NODEADDR_SIZE - Query the size of a CMI node address.
       Optval buffer should be of size_t size.

   int cflush(cmi_ctxt *ctxt, void *vaddr[], int32_t addrcnt)

     Flush specified addresses to home node

     Parameters:
         ctxt CMI context
         vaddr Array of virtual addresses that need to be flushed
         addrcnt Number of elements in vaddr array

     Returns:
         0 on success. -1 on failure.

         CMI_ERR_INIT Calling thread is not associated with
         context

         CMI_ERR_INVAL Provided context is invalid

         CMI_ERR_HW Hardware error prevented flushing stores to
         home node

     CMI clients can request the CMI system to flush cache lines
     containing specified addresses to the home node using
     cflush(). This is a synchronous operation and on return all
     dirty cache lines for the addresses specified should be
     flushed to their home node. If an access error is
     encountered during the flush operation an exception is
     thrown. The number of cache lines/addresses to flush is
     specified in the addrcnt parameter. The cflush operation can
     be initiated even if the calling thread does not have a valid
     flush epoch. It is also legal to flush cache lines to home
     node while a flush epoch is open. To optimize performance a CMI
     system can avoid flushing cache lines on a flush epoch
     that have already been flushed using cflush() as long as no
     additional stores have occurred that modify the cache line.


   int close_fb(cmi_ctxt *ctxt, cmi_fb epoch)

     Close flush epoch for a thread

     Parameters:
         ctxt CMI context
         epoch CMI epoch to close

     Returns:
         0 on success. -1 on failure.

         CMI_ERR_INIT Calling thread is not associated with context

         CMI_ERR_INVAL Provided context or epoch is invalid

         CMI_ERR_HW Hardware error prevented flushing stores to home node

     Closing the flush epoch for the thread implicitly flushes
     all pending changes back to the home node as if the thread
     had invoked flush_fb(). The CMI system does not need to
     track any further stores for the thread till a new flush
     epoch is started.

   int cmi_ctl(cmi_ctxt *ctxt, int32_t cmd, cmi_ctl_cfg *cfg)

     Query/configure CMI system attributes

     Parameters:
         ctxt CMI context for calling thread
         cmd Command operation being requested
         cfg CMI Control structure for operation

     Returns:
         0 on success. -1 on failure.

         CMI_ERR_INIT Calling thread is not associated with context

         CMI_ERR_INVAL Provided context or command is invalid

         CMI_ERR_PERM Insufficient permissions for command execution

         CMI_ERR_NOMEM Insufficient resources for memory reservation request

	 CMI_ERR_NOTSUPP Requested command/operation is not supported

     CMI clients can query/configure various attributes of the
     underlying CMI system. The cfg parameter is a value result
     parameter. For commands that configure/set certain
     attributes of underlying CMI implementation the client will
     set the values being modified in the cfg parameter. For
     commands that query CMI state/attributes the cfg parameter
     will contain the result of the operation.

     The following CMI control commands are defined:

     o CMI_CTL_INFO: Obtain currently configured parameter values
       and limits for the specified CMI context. Following
       information is available:

       o max_mem_avail: Maximum size of CMI memory currently
         available. This is a snapshot of currently available
         memory and can dynamically change as segments are
         (de)allocated.

       o max_mem_cfg: This is size of CMI memory configured for
         the system.

       o max_seg_sz: The maximum size of a CMI segment that can
         be created. Similar to SHMMAX for shared memory segment.

       o max_exp_segs : Maximum number of segments that can be
         exported.

       o max_imp_segs : Maximum number of segments that can be
         imported.

       o max_write_through_segs: Maximum number of write through
         segments. Can be 0 if CMI implementation does not
         support write through segments.

       o max_acc_toks : Maximum number of access tokens that can
         be active.

       o max_acc_stok : Maximum number of access tokens per
         segment.

       o cur_exp_segs : Current number of segments that are
         exported.

       o cur_imp_segs : Current number of segments that are
         imported.

       o cache_line_sz: Cache line size used by CMI
         implementation.

       o max_reco_segsz: Maximum segment size that can be passed
         to a single call to seg_ctl() when recovering segment
         (CMI_SEG_CHECK | CMI_SEG_RECO). Must be a multiple of
         cache_line_sz.

       o prot_units: This is the granularity of access protection provided
	 by the platform in bytes. Clients must ensure that segment
         allocation sizes are a multiple of this size in seg_get(), otherwise
         segment allocation will fail.

       o seg_alloc_units: Clients may benefit by allocating
         segments in multiple of seg_alloc_unit sizes to minimize
         internal fragmentation as platform allocates segments
         that are multiple of this size internally. Platform can
         set this value to 1 if memory fragmentation is not an
         issue.

       o seg_alignment: Alignment restrictions on platforms for normal
         segments i.e. those segments not created with CMI_SEG_LARGE_PAGES,
         where segment can be attached to. This is usually determined by any
         OS specific restrictions based on underlying page size used for the
         segment. Clients attempting to attach a normal segment at an
         address that is not a multiple of this alignment size will fail
         with error CMI_ERR_INVAL. On platforms that have no restrictions
         this value can be set to 1.

       o seg_lrgpg_alignment: Similar to seg_alignment for normal segment
         this is the alignment restriction for segments created using
         CMI_SEG_LARGE_PAGES. Clients must ensure that segments backed by
         large pages are attached to an address that is multiple of this
         size.

     o CMI_CTL_CACHE_RSEG_SET: Some CMI systems may implement a
       local cache to back remotely attached segments. As pages
       are evicted from the cache they are synchronized with the
       home node for the segment. This command allows a client to
       set the size of the local cache to backup a given segment
       (or potentially all segments). Only the process that
       created or imported the segment can update size of local
       cache.

       o seg_hndl: Segment handle for which the local cache size
         is being set. Can specify a valid segment handle
         imported via seg_imp() or CMI_SEG_ALL for all remote
         segments. CMI_SEG_ALL indicates that the mpool_sz be
         applied to ALL segments exported AND imported by the
         calling process.

       o mpool_sz: Size of the backing cache in bytes for all (or
         a specific segment).

     o CMI_CTL_CACHE_RSEG_GET: Query the currently configured
       size of the local cache backing remote segments.

       o seg_hndl: Segment handle for which the local cache size
         is being set. CMI_SEG_ALL cannot be used to query cache
         size.

       o mpool_sz: Size of the backing cache in bytes for
         segment.

     o CMI_CTL_RECONF_TOUT: Specify the maximum network
       reconfiguration timeout after which an access exception
       must be thrown for an operation.

       o rcfg_tout: Reconfiguration timeout in milliseconds

     o CMI_CTL_CLIENT_CONSIST: CMI client ensures data
       consistency guarantees across network/node failures.
       Client is not required to call seg_ctl() to perform
       recovery of data. Client specific information, for e.g.
       recovery logs, will be used to detect and recovery data
       consistency across network/node failures.

     o CMI_CTL_NODE_CMAP_GET: Retrieve the connectivity map for
       all nodes that either have imported/exported segments to
       this context. Since determining the connectivity map may
       take some finite amount of time, the CMI platform must
       deliver asynchronous notifications via cmi_einfo_cmap
       event. A single notification per CMI_CTL_NODE_CMAP_GET
       command must be delivered. Client must specify a non-zero
       ctl_cfg_cmap_reqid for a request. The cmi_einfo_cmap event
       contains the client provided request ID in the event.

     o CMI_CTL_MEM_RESERVE: Request reservation of CMI memory for
       context. If the context is not associated with any memory
       reservation then a new memory reservation of specified
       size is requested, or else this is a request to modify the
       reservation associated with the calling context. The
       cmi_ctl_cfg mreq structure contains the amount of memory
       reservation requested for the calling context. For new
       reservations the key is returned in rsv_key and the
       context is implicitly associated with it. Clients can
       modify a previously associated reservation by
       increasing/decreasing the reservation amount. If the
       currently allocated memory associated with reservation is
       greater than the reduced reservation request then the
       request will fail with CMI_ERR_INVAL. Similarly if the
       requested memory reservation exceeds the available CMI
       memory then the request will fail with CMI_ERR_NOMEM. In
       all cases on return from cmi_ctl() the currently allocated
       memory for reservation should be returned in
       ctl_cfg_mem_allocd. See Memory Reservation in cmi(5)
       for more information.

     o CMI_CTL_MEM_RESERVE_DEL: Delete memory reservation
       associated with the context. The context must have a
       memory reservation associated with it previously either by
       calling CMI_CTL_MEM_RESERVE_SET or allocating a context
       using CMI_CTL_MEM_RESERVE. If no memory reservation is
       associated with context then this call shall fail with
       CMI_ERR_INVAL. See Memory Reservation in cmi(5) for more
       information.

     o CMI_CTL_MEM_RESERVE_SET: Associate a memory reservation
       with context. All subsequent segment allocations will
       utilize the specified memory reservation. Specifying a
       reservation key of 0 results in the context being dis-
       associated with any reservations. Subsequent segment
       allocations may fail if out of resource. If the calling
       context is already associated with a reservation then the
       association is replaced with the new reservation being
       set. Attempt to set a previously deleted or invalid
       reservation will fail with CMI_ERR_INVAL and the current
       reservation association for context is not changed.

     o CMI_CTL_NODE_ADDR_CMP: Compare two or more node addresses.
       Clients provide the node address to compare in
       ctl_cfg_ncmp_naddr. Multiple node addresses to compare
       against are provided in ctl_cfg_ncmp_caddr. The size of
       the ctl_cfg_ncmp_caddr array on input is specified in
       ctl_cfg_ncmp_addrs. On return from this routine the
       ctl_cfg_ncmp_midx array, which is allocated by the caller
       and is of size ctl_cfg_ncmp_addrs is populated with the
       indexes of nodes in the ctl_cfg_ncmp_caddr that match the
       node being compared. The number of matched node addresses
       on return from this routine is specified in
       ctl_cfg_ncmp_addrs. If any of the provided node addresses
       are invalid then this routine shall fail with
       CMI_ERR_INVAL.

  int cmi_enb(cmi_ctxt *ctxt, int enable)

     Enable/Disable CMI Segment access for calling thread

     Parameters:
         ctxt CMI context
         enable Enable/Disable access to CMI segments

     Returns:
         0 on success. -1 on failure

         CMI_ERR_INIT Calling thread is not associated with
         context

         CMI_ERR_NOMEM Out of resources enabling CMI access for
         thread

     Access to remote segments can be controlled on a per thread
     basis using cmi_enb(). Enabling remote segment access for a
     thread is different than setting the access token for a
     segment on a remote node. The remote access token for a
     segment is configured using seg_ctl() by the process that
     imports the segment (seg_imp()). This access tokens for a
     segment must be configured as a pre-requisite for any
     process or thread on the remote node to access the segment.
     Once the access token is configured by the importing process
     all CMI processes and threads on the node can utilize the
     token to access the segment. cmi_enb() on the other hand
     enables/disables access to ANY CMI segments for the calling
     thread. Each thread must explicitly enable access to CMI
     segments before access ANY CMI remote segments.


     Note:
         A thread opens a CMI access epoch by invoking
         cmi_enb(TRUE) and closes the access epoch by invoking
         cmi_enb(FALSE). Access to CMI remote segments by a thread are
         only permitted when the CMI access is open. Any access
         to CMI segments (loads or stores) outside the epoch
         result in an access violation exception (except
         if the CMI implementation does not support per thread
         enable/disable).

         Explicitly enabling/disabling CMI segment access
         provides an additional layer of security to guard
         against errant memory accesses by a thread. CMI
         applications may have well defined regions of code that
         operate on CMI segments. Access to CMI can then be
         enabled only for the required duration. If an
         application does not wish to protect against errant
         accesses to CMI segments then it can enable CMI segment
         on startup and disable it on exit for all threads.

         Access to CMI remote segments is controlled on a
         per thread basis using cmi_enb(). Even though a segment
         may be attached and a valid access token configured for it,
         access to the segment is only allowed if the thread
         explicitly enables it via call to cmi_enb().

  cmi_event* evt_get(cmi_ctxt *ctxt)

     Retrieve a notification event for context

     Parameters:
         ctxt CMI context

     Returns:
         Notification event or NULL if no notification events to
         process

     Retrieve a CMI notification event for a context. Certain
     error conditions require the cooperation of the client and
     the CMI system to recover the system and program state.
     Event notifications for various error conditions are defined
     in section CMI Events in cmi(5). Only CMI processes that create or
     import segments on a node are required to handle event
     notifications. Once an event has been processed/handled by
     the process it is returned back to the library via
     evt_ret().

     Some event types require processing the event successfully.
     Any status other than CMI_EVENT_RET_DONE for these events is
     fatal and the calling process can terminate when returning
     the event.

     Note:
         The event token is allocated by the CMI library and
         ownership is transferred back to the library (and may be
         de-allocated) when it is returned. The client must not
         reference a returned event.

  void evt_ret(cmi_event *event, cmi_event_ret status)

     Return a notification event back to CMI library

     Parameters:
         event CMI event being returned
         status CMI event completion status

     CMI asynchronous notification events obtained via evt_get()
     are returned back to the library with the specified
     completion status. Some event types require processing the
     event successfully. Any status other than CMI_EVENT_RET_DONE
     for these events is fatal and the calling process can
     terminate when returning the event.

     Note:
         The event token is allocated by the CMI library and
         ownership is transferred to the CMI client. The event
         token should remain valid till the client returns it by
         invoking evt_ret().

  int fini(cmi_ctxt *ctxt)

     Disassociate the calling thread with a CMI context

     Parameters:
         ctxt CMI context to dis-associate with calling thread

     Returns:
         0 on success. -1 on failure.

         CMI_ERR_INVAL Provided context is invalid

         CMI_ERR_INIT Calling thread is not associated with
         context

     Disassociate calling thread with provided CMI context. The
     library can de-allocate any thread specific resources. When
     the last thread is disassociated with the context the
     context can be deallocated. It is illegal for the calling
     thread to invoke any CMI calls after a successful invocation
     of fini(). A thread can re-associate itself with a CMI
     context by invoking ini_th() and perform CMI transfers.

     Note:
       If the CMI context is de-allocated and was used to import segments
       from remote contexts then the CMI platform can deliver a
       CMI_EVENT_RCTXT_DOWN event to the remote contexts (if supported).

       If the CMI context is de-allocated and exported segments that were
       imported by some remote contexts then the CMI platform can deliver
       a CMI_EVENT_HCTXT_DOWN event to the remote contexts (if supported).

       Thread Safety: This call is multi thread safe. Multiple
       threads may dis-associate themselves with a context concurrently.

  int flush_fb(cmi_ctxt *ctxt, cmi_fb epoch)

     Flush all stores since beginning of epoch to home node for
     thread

     Parameters:
         ctxt CMI context
         epoch CMI epoch to flush

     Returns:
         0 on success. -1 on failure.

         CMI_ERR_INIT Calling thread is not associated with
         context

         CMI_ERR_INVAL Provided context or epoch is invalid

         CMI_ERR_STORE An error occurred while flushing stores to
         home node

         CMI_ERR_HW Hardware error prevented flushing stores to
         home node

     Forces flushing (writing) of all stores back to the home
     node for calling thread since last flush_fb or start of
     epoch. Flushing stores is a synchronous operation. On return
     from this call all stores between previous open_fb() or
     flush_fb() have been reflected on the home node for the
     segment.

     If the platform encounters an un-recoverable error (such as
     network failure) that results in possible data
     corruption/loss then CMI_ERR_STORE error will be returned
     to indicate failure to calling thread. An asynchronous event
     may also be generated to the process that imported the
     affected segment on the node. The event type to indicate
     store failure should be CMI_EVENT_STORE_FAILURE.

     Store failures due to loss of network connectivity or remote
     node being down are distinguished from local hardware
     failures by the CMI_ERR_HW error code. CMI clients may
     perform different recovery operations based on the network
     or remote node down vs. local hardware being faulty (for
     instance shutdown the local node instead if evict the remote
     node on store failures). An asynchronous event may also be
     generated to the process that imported the affected segment
     on the node. The event type to indicate local hardware
     failure should be CMI_EVENT_HW with the segment and
     addresses that failed the flush as event data.

  int ini_th(cmi_ctxt *ctxt)

     Associate a CMI context with the calling thread

     Parameters:
         ctxt CMI context to associate with calling thread

     Returns:
         0 on success. -1 on failure. Invoke cmi_get_error() to
         retrieve error codes listed below.

         CMI_ERR_NOMEM Out of resources (such as maximum number
         of threads already associated for context)

         CMI_ERR_INVAL Provided context is invalid

         CMI_ERR_BOUND Calling thread is already associated with
         context

     This routine is invoked by all threads to
     associate themselves with a previously allocated CMI context
     (except the thread that initialized the CMI context itself
     which automatically implies association). The vendor library
     can allocate and associate any internal state with the
     calling thread.

     All threads in a process must associate themselves with a
     CMI context successfully before any CMI calls can be made.
     The only exception is a thread can call cmi_get_error() to
     determine the reason why the associate call failed. For
     every call to ini_th() a corresponding call to fini() will
     be made by the thread to dis-associate itself from a
     context.

     Note:
       Thread Safety:  This call is multi thread safe.
       Multiple threads may associate themselves with a context
       concurrently.

  int mb_fn(cmi_ctxt *ctxt)

     Perform a full (Read/Write) memory barrier

     Parameters:
         ctxt CMI context

     Returns:
         0 on success. -1 on failure.

	 CMI_ERR_INIT Calling thread is not associated with context

	 CMI_ERR_INVAL Provided context is invalid

     Perform a 'Memory Barrier' that orders both loads and stores.
     Loads and stores preceding the memory barrier are committed to memory
     before any loads and stores following the memory barrier.

     On platforms that perform natural ordering of loads and stores,
     the implementation of this function can be a no-op or even a NULL
     function pointer to indicate to client that full memory barriers are not
     required on the platform.

     Note:
       A CMI platform also provides an implementation of the
       full barrier using the defined cmi_mb() macro. Platform specific
       implementations are made available in cmi_impl.h header file.
       Clients may invoke memory barriers via function pointers or inline
       the memory barrier macros with their code.

  cmi_fb open_fb(cmi_ctxt *ctxt)

     Open a flush epoch to start tracking stores by thread

     Parameters:
         ctxt CMI context

     Returns:
         log Handle to flush epoch or NULL on failure

         CMI_ERR_INIT Calling thread is not associated with
         context

         CMI_ERR_INVAL Provided context is invalid

         CMI_ERR_NOMEM Out of resources allocating epoch

         CMI_ERR_BOUND Thread already has a flush epoch open

         CMI_ERR_HW Underlying hardware detected on node

     In order to aid in maintaining consistency in presence of
     failures it is necessary to update home node memory at well
     defined points. The CMI system needs to track stores on a
     per thread basis. Tracking changes is meaningful only on
     remote nodes. Starting to log changes on a home node can be
     a no-op as hardware provided coherence on the node is
     sufficient to guarantee consistency. It is illegal for a
     thread to have more than one flush epoch. Epoch creation in
     this case can fail (return NULL) and set the error to
     CMI_ERR_BOUND. However multiple threads can be concurrently
     operating on a segment with a separate epoch tracking
     changes for each thread.


  int rmb_fn(cmi_ctxt *ctxt)

     Perform a read memory barrier

     Parameters:
     ctxt CMI context

     Returns:
     0 on success. -1 on failure.

     CMI_ERR_INIT Calling thread is not associated with context

     CMI_ERR_INVAL Provided context is invalid

     Perform a 'Read Memory Barrier' that only orders loads i.e. loads
     preceding the memory barrier are completed before any loads following
     the memory barrier (in program order). Stores can be re-ordered across
     the barrier if underlying implementation supports it.

     On platforms that perform natural ordering of loads the implementation
     of this function can be a no-op or even a NULL function pointer to
     indicate to client that read memory barriers are not required on the
     platform.

     Note:
       A CMI platform also provides an implementation of the
       read barrier using the defined cmi_rmb() macro. Platform specific
       implementations are made available in cmi_impl.h header file.
       Clients may invoke memory barriers via function pointers or inline
       the memory barrier macros with their code.

  int rseg_del(cmi_ctxt *ctxt, cmi_rseg *rseg)

     Delete a previously allocated remote segment handle

     Parameters:
         ctxt CMI context
         rseg CMI remote segment handle to deallocate

     Returns:
         0 on success. -1 on failure.

         CMI_ERR_INIT Calling thread is not associated with
         context

         CMI_ERR_INVAL Provided context, or remote segment handle
         is invalid

     Delete a remote segment handle previously obtained via call
     to seg_exp(). This routine does not disable access to
     the remote segment - to disable access to the segment the
     access tokens associated with the segment should be revoked.
     Similarly this routine does not deallocate the underlying
     CMI segment on the node - for that the caller should use
     CMI_SEG_RM command to seg_ctl(). This routine deallocates
     any resources used by the calling process when exporting
     the segment. If the exported handle was transmitted over the
     network to remote nodes it is still valid till the access
     token or the segment itself is deallocated.

  void* seg_at(cmi_ctxt *ctxt, cmi_seg seg, void *addr, int32_t flags)

     Attach specified CMI segment to address space of calling
     process

     Parameters:
         ctxt CMI context
         seg CMI segment to attach to
         addr Address to attach segment to
         flags Attach modifier flags

     Returns:
         Address of the attached segment

         CMI_ERR_INIT Calling thread is not associated with
         context

         CMI_ERR_INVAL Provided context or segment is invalid or
         page is not properly aligned

         CMI_ERR_NOMEM Out of resources to attach segment

         CMI_ERR_PERM Insufficient permissions to attach segment

         CMI_ERR_RECONFIG If segment is currently being
         reconfigured

         CMI_ERR_NOTSUPP Attach operation attributes not
         supported

     Attach the specified CMI segment to address space of a calling process.
     The segment handle is obtained from a previous call to seg_get() or
     imported using a call to seg_imp(). The address to attach in the
     calling process is determined by the addr argument. The CMI library can
     select a suitable unused address to map the segment to for an address
     value of NULL. If the provided address is not NULL then it must be
     naturally aligned to the alignment size for the segment as returned in
     cmi_ctl() for CMI_CTL_INFO command. Attempt to attach to an address
     that is not a multiple of the alignment size will fail with
     CMI_ERR_INVAL error. Attach flags can be used to modify segment
     attributes. Following attributes are currently defined:

     o CMI_SEG_READ - Segment is attached for reading. Any stores
       will generate an access error/exception.

     Multiple segments can be attached within a process.
     Additionally a given segment can be mapped into different
     addresses within the process possibly using different
     attributes (READ, READ|WRITE etc.). Any thread associated
     with a context can request a segment attach. The attached
     segment is visible to all threads within a process.

     Note:
          Thread Safety:  This call is multi thread safe.
         Multiple threads may request a segment to be attached
         concurrently. CMI library may serialize segment attach
         internally as this operation is expected to be
         infrequent.

  int seg_ctl(cmi_ctxt *ctxt, cmi_seg seg, int32_t cmd, cmi_ds *ds)

     Perform control operations on a CMI segment

     Parameters:
         ctxt CMI context
         seg CMI segment to perform operation on
         cmd Operation to perform
         ds Segment operand structure

     Returns:
         0 on success. -1 on failure.

         CMI_ERR_INIT Calling thread is not associated with
         context

         CMI_ERR_INVAL Provided segment is invalid

         CMI_ERR_PERM Insufficient permissions for requested
         operation

         CMI_ERR_NOMEM Out of resources increasing segment break
         point for extensible segments

     Perform the specified operation on the CMI segment. The
     cmi_ds operand is a value result structure allocated by the
     client and filled in by the library for some of the
     operations. Currently the following operations are defined:

     o CMI_SEG_RM: Mark the specified segment for deletion. This
       operation can be performed in the context of the process
       that created the segment via seg_get() on the home node or
       a process that imports a segment via seg_imp() on remote
       node. Once the last process detaches from the segment the
       backing resources can be reclaimed. See Segment Lifetime
       in cmi(5) for more details.

     o CMI_SEG_TOKEN: Configure the remote access token for
       provided segment. Remote access tokens are configured on
       nodes for remote segments imported via calls to seg_imp().
       Access tokens for remote segments are configured by the
       process that imported the segment. Access tokens for a
       segment can be configured multiple times while the segment
       is active (for instance during cluster reconfiguration
       where new access tokens are generated). Configuring an
       access token over-writes the previously configured access
       token for the segment. A CMI_ERR_PERM error is generated
       if attempting to configure a remote access token on a node
       it was not generated for. This is only possible for node
       specific tokens.

     o CMI_SEG_STATUS: Fill in the segment operand structure for
       the segment.

     o CMI_SEG_CHECK: Check if specified segment address is in
       need of recovery. This is invoked on the home node in the
       context of the process that allocated the segment via
       seg_get() on notification of death of a remote context
       (CMI_EVENT_RCTXT_DOWN). Clients that already provide
       guarantees of consistent data (CMI_CTL_CLIENT_CONSIST set
       via cmi_ctl()) do not need to check for segments requiring
       recovery as data consistency is guaranteed by client
       provided protocols. Segment addresses must be checked in
       multiples of CMI cache line size obtained via cmi_ctl()
       using command CMI_CTL_INFO. If any addresses within the
       range being checked require recovery then on return the ds
       operands op.reco struct is filled with the address and
       size of region requiring recovery. The client can then
       re-issue the CMI_SEG_RECO command for entire/subset of the
       affected region.

     o CMI_SEG_RECO: Recovery specified segment address that may
       be in an undetermined state across error boundaries (such
       as when a remote node accessing the segment dies). This is
       invoked on the home node in the context of the process
       that allocated the segment via seg_get() on notification
       of death of a remote context (CMI_EVENT_RCTXT_DOWN). Clients
       that already provide guarantees of consistent data
       (CMI_CTL_CLIENT_CONSIST set via cmi_ctl()) do not need to
       explicitly recover segment ranges that may be in flux.
       Segment addresses must be recovered in multiples of CMI
       cache line size obtained via cmi_ctl() using command
       CMI_CTL_INFO.

     o CMI_SEG_BRK: Set the segment break point to the value
       specified in brk.seg_brksz. Attempt to set the segment break
       point beyond the maximal size of the segment specified
       during creation in seg_get() shall return CMI_ERR_INVAL.
       The configured break point size must be a multiple of
       protection unit size on the platform, or else the resize
       request will fail with CMI_ERR_INVAL.
       If sufficient memory is not available to grow the segment
       break point then CMI_ERR_NOMEM shall be returned. Memory
       allocations for extensible segments should utilize any
       configured memory reservation for the context. See Memory
       Reservation in cmi(5) for more details.
       Segment break point can only be changed for extensible
       segments in the context of the process that created the
       segment. Any attempt to modify segment break point for non-
       extensible segments shall return CMI_ERR_INVAL.
       CMI_ERR_PERM is generated if any process other than the
       creator attempts to modify the segment break point. Segment
       break point can be reduced i.e. memory freed up if the new
       segment break point is smaller than the currently set break
       point. A segment with 0 break point has no backing memory
       allocated to it. See Extensible Segments in cmi(5) for more
       details.

     o CMI_SEG_INFO: Retrieve segment attributes into cmi_ds.info. This
       command can be run on the home node for a locally allocated segment
       as well as a remote node for an imported segment i.e. seg is an
       imported segment obtained via seg_imp(). The attributes of the
       exported and imported segments match. Some platforms may have
       restrictions on address a segment can be attached at (due to page
       size used for segment). The cmi_ds.info.seg_align field indicates the
       alignment requirement for this segment and must match the alignment
       restriction returned via cmi_ctl() for CMI_CTL_INFO command for this
       segment type i.e. normal or large segment. Clients must specify an
       address that is a multiple of this alignment constraint during
       seg_at(). The size of the segment is returned in cmi_ds.info.seg_sz
       along with the segment creation flags in cmi_ds.info.seg_flags.
       cmi_ds.info.seg_prot_align and seg_prot_sz are new fields, seg_prot_align
       indicates a minimum alignment that va for tok_newva() must meet.
       seg_prot_sz indicates the granularity of token size created by tok_newva().
       The size specified by tok_newva() must be a multiple of seg_prot_sz.

  int seg_dt(cmi_ctxt *ctxt, cmi_seg seg, void *addr)

     Detach specified segment from address space of the calling
     process

     Parameters:
         ctxt CMI context
         seg CMI segment to detach
         addr Address to detach

     Returns:
         0 on success. -1 on failure.

         CMI_ERR_INIT Calling thread is not associated with
         context

         CMI_ERR_INVAL Provided context or segment is invalid

     Detach specified segment from address space of the calling
     process. The specified segment and address should have been
     obtained from a previous call to seg_at(). Any thread
     associated with a CMI context can request a segment detach.
     On a successful return from seg_dt() the segment handle and
     address space in the calling process are undefined.
     Subsequent access by any thread in the process will result
     in a segmentation fault or unpredictable operations. It's
     the CMI clients responsibility to ensure that no threads are
     executing or referencing objects within the detached
     segment.

     Note:
         Thread Safety:  This call is multi thread safe.
         Multiple threads may request a segment to be detached
         concurrently.

  cmi_rseg* seg_exp(cmi_ctxt *ctxt, cmi_seg seg,
     int32_t attrib)

     Export a locally allocated segment

     Parameters:
         ctxt CMI context
         seg CMI segment to export
         attrib Attributes of exported segments

     Returns:
         Remote segment handle to CMI segment on success.
	 NULL on failure.

         CMI_ERR_INIT Calling thread is not associated with
         context

         CMI_ERR_INVAL Provided context or segment is invalid

         CMI_ERR_NOMEM Out of resources to export segment

         CMI_ERR_PERM Insufficient permissions to export segment

         CMI_ERR_NOTSUPP Segment attributes not supported

     Export a segment so it can be accessed over the network. A
     segment can only be exported by the process that allocates
     it. The remote segment handle uniquely identifies the
     segment on the cluster. Remote segment handles can be
     distributed over the network. The size of the handle is
     vendor specific and can be obtained using CMI_ATTR_RSEG_SIZE
     attribute to attr_get(). The size of the remote segment is
     at least sizeof(cmi_rseg) but can include additional opaque
     elements that are library specific. The exported segment can
     be accessed over the network via an access token. The
     context that exports the segment can allocate access tokens
     via tok_new(). Exported segments need to be imported on
     remote nodes (via seg_imp()) before they can be accessed.

     The segment parameter should be a locally allocated segment
     obtained by a previous call to seg_get(). It is illegal to
     obtain a remote segment handle for a segment obtained via
     seg_imp(). Any attempt to do so can fail with error
     CMI_ERR_INVAL. A segment can be exported multiple times with
     potentially differing attributes however some attributes are
     mutually exclusive. A process attempting to import a segment
     that conflicts with an ongoing import will fail with
     CMI_ERR_INVAL during seg_imp().

     Clients can specify certain attributes for the exported
     segment. These attributes can modify the behavior of remote
     nodes in how they access the segment. Following attributes
     are currently defined:

     o CMI_SEG_WRITE_THROUGH - Segment is configured for write
       through policy. All stores are un-cached and synchronous
       and reflected on home node. Write through segments are
       mutually exclusive to cacheable segments i.e. a segment
       can't be imported both as a cacheable and un-cacheable
       segment by same or different process. It is still
       acceptable to export a segment as cacheable and un-
       cacheable however at any given time it can only be
       imported as either a cacheable or un-cacheable segment.

     Note:
       The size of the remote segment handle can vary depending on the
       size (and attributes) of the segment being exported.

       Attach Attributes:  Some CMI system may not support all
       segment attributes - for example if write through
       segments are not supported by the underlying CMI
       implementation then attempting to exporting segment with
       this attribute can fail with error CMI_ERR_NOTSUPP.

  cmi_seg seg_getva(cmi_ctxt *ctxt, void * va, size_t size,
  	int32_t flags)

     Create a CMI Segment of specified size by association with an OSM
     segment which base virtual address is equal to the value specified
     by va and the size is equal to the specified size.

     Parameters:
         ctxt CMI context
         va   base address of an OSM segment the new CMI segment will
	 	be associated with
	 size Size of the OSM segment
	 flags Segment association flags

     Returns:
         segment Handle to CMI segment on success.
         CMI_SEG_INVALID on failure.

         CMI_ERR_INIT Calling thread is not associated with
         context

	 CMI_ERR_INVAL
	   - Provided context/va/size are invalid
	   - A object that is mapped to the given va range is not an OSM.
	   - The given va range doesn't match a OSM's entire va range or
	     covers multiple OSM segments.

         CMI_ERR_NOTSUPP The given flags are not supported.

         CMI_ERR_NOMEM Out of resources to allocate segment

         CMI_ERR_PERM Insufficient permissions to access the OSM segment

     This call is similar to the seg_get() call. It creates a CMI segment
     associated with an OSM segment provided that the specified virtual
     address and the size are equal to the OSM base virtual address and
     the OSM size respectively. For each CMI context only a single CMI
     segment can be associated with any particular OSM segment.

     All flags that can be used for seg_get() are accepted, but only
     CMI_SEG_CLIENT_CONSIST, CMI_SEG_LARGE_PAGES, and CMI_SEG_IMPORT_ANY
     are supported by seg_getva().

     Creating a client non-consist mode CMI segment by seg_getva will
     fail. (User needs to explicitly specify CMI_SEG_CLIENT_CONSIST in
     the "flags" or needs to specify 0 in the "flags" and the context's
     default mode must be Client Consist (it can be set through cmi_ctl()).

     Only one CMI segment is allowed per shmid.  Subsequent call to the
     same OSM will fail with CMI_ERR_INVAL.

     No CMI specific signal will be generated for the segment on the home
     node (because the segment is never attached as CMI segment on the
     home node).

     CMI specific signals generated for imported segments are the same
     as for any CMI segment created by the seg_get() API. The only difference
     is a case when a GRANULE for the accessed VA doesn't exist.
     In this particular case, SIGSEGV(CMI_ERROR_NOPAGE) will be generated.

     The OSM segment specified by the va/size must indicate a valid OSM
     segment and must match its entire range (Otherwise, the it will fail).
     As the result of seg_getva, the returned CMI segment id is associated
     with *the underlying OSM segment* (not the va/size).  The given va/size
     are used just to identify the OSM segment.  Even if the OSM segment is
     detached from the va and a different OSM segment is attached to the va,
     the CMI segment ID is still associated with the original OSM segment,
     which was attached to the va when seg_getva() was called.

     Removing the OSM segment by shmctl(2) or ipcrm(1) will not affect
     the associated CMI segment and the CMI segment stays valid until
     it is removed by seg_ctl() with command CMI_SEG_RM.  The CMI segment
     might be still accessible if the OSM segment is still accessible.

     cmi_seg seg_get(cmi_ctxt *ctxt, size_t size, int32_t flags)

        Allocate a CMI Segment of specified size

        Parameters:
             ctxt CMI context
             size Size of segment to allocate
             flags Segment allocation flags

        Returns:
             segment Handle to CMI segment on success.
             CMI_SEG_INVALID on failure.

             CMI_ERR_INIT Calling thread is not associated with
             context

             CMI_ERR_INVAL Provided context/operands are invalid

             CMI_ERR_NOTSUPP Requested operation not supported

             CMI_ERR_NOMEM Out of resources to allocate segment

             CMI_ERR_PERM Insufficient permissions to allocate
             segment

     Allocate a CMI segment of specified size. The size of the
     segment must be a multiple of the access protection unit
     size on the platform. The size of the protection unit can
     be queried via cmi_ctl() using CMI_CTL_INFO command.
     Attempting to allocate a segment that is not a multiple of
     the protection unit size will fail with CMI_ERR_INVAL.
     Multiple segments can be created by a process.

     Following flags are currently supported.

     o CMI_SEG_NORESERVE: Do not reserve swap space for segment.

     o CMI_SEG_LOCK: Prevent swapping of segment. Caller must
       fault in pages that are to be locked explicitly.

     o CMI_SEG_LARGE_PAGES: Use large pages to allocate segment. If
       sufficient large pages are not available to satisfy the request then
       the segment allocate operation should fail with error CMI_ERR_NOMEM.
       Clients must ensure this segment is attached to an address that is a
       multiple of ctl_cfg_seg_lrgpg_alignment size returned in
       CMI_CTL_INFO.

     o CMI_SEG_EXTENSIBLE: Create as extensible segment. Support for
       extensible segments are optional. Platforms indicate support for
       extensible segments by setting the CMI_CAP_EXTENSIBLE_SEGMENTS
       capability flag. Any attempt to create an extensible segment on a
       platform that does not support it shall fail with CMI_ERR_NOTSUPP.
       For extensible segments the segment size is the maximum size of the
       segment. On creation extensible segments have a size of 0 and can be
       extended only by the creating context using seg_ctl() with command
       CMI_SEG_BRK. See Extensible Segments in cmi(5) for more information.

     o CMI_SEG_CLIENT_CONSIST: Create a segment in client consist mode
       whereby the client is using application specific protocols to
       guarantee consistency across failure boundaries. By default all
       segments are created in client inconsistent mode (except if client
       explicitly enabled client consist mode via cmi_ctl(). Clients can
       request specific segments to operate in client consist mode if the
       default is not changed. Platforms that do not support mixed mode
       operation i.e. both consist and in-consistent segments together can
       fail the operation with CMI_ERR_NOTSUPP error. See Consistent data
       section in cmi(5) for more information.

     o CMI_SEG_CLIENT_INCONSIST: Create a segment in client inconsistent
       mode. This is the default mode of operation if the client has not
       explicitly enabled client consist mode via cmi_ctl(). Data residing
       in this segment is not guaranteed to be consistent across failure
       boundaries by any application specific protocols. Any attempt to
       access memory that was in a modified/cached state on a remote node
       that died or is un-reachable will result in an access exception.
       Clients must explicitly perform recovery of the affected regions
       using seg_ctl() with command CMI_SEG_CHECK and CMI_SEG_RECO. See
       Consistent data section in cmi(5) for more information.

     Note:
       If the client does not specify the consistency mode when creating a
       segment (CMI_SEG_CLIENT_CONSIST or CMI_SEG_CLIENT_INCONSIST) then
       the segment is created with the currently configured consistency
       mode. The default consistency mode for CMI is in-consist mode.
       Clients can change the default consistency mode via cmi_ctl() using
       CMI_CTL_CLIENT_CONSIST mode macro. Once a client has changed the
       default consistency mode to CONSIST it cannot be changed back to
       in-consist however clients can still create segments with in-
       consist mode by specifying the CMI_SEG_CLIENT_INCONSIST flag during
       segment creation. If platforms do not support mixed mode segments
       then all segment allocations with conflicting modes can fail with
       CMI_ERR_NOTSUPP error.

       CMI error handling requires that processes that create
       segments be capable of processing asynchronous event
       notifications using evt_get(). CMI library may generate
       notification events targeted to the context to help in
       recovery of CMI and program state for allocated segment
       under some error conditions. See Error Handling in cmi(5) for more
       information.

       Some CMI systems may lock all pages backing a CMI
       segment even if the client does not explicitly specify
       CMI_SEG_LOCK. This mode of operation is acceptable
       however it should not be a requirement that a client
       specify the CMI_SEG_LOCK for correct operation i.e. the
       segment locking can be done implicitly if required by
       the underlying implementation.

       Thread Safety: This call is multi thread safe.
       Multiple threads may request a segment to be allocated
       concurrently.

  cmi_seg seg_imp(cmi_ctxt *ctxt, cmi_rseg *rseg)

     Import a remotely allocated segment

     Parameters:
         ctxt CMI context
         rseg CMI segment to import

     Returns:
         segment Handle to CMI segment on success.
         CMI_SEG_INVALID on failure

         CMI_ERR_INIT Calling thread is not associated with
         context

         CMI_ERR_INVAL Provided context or remote segment is
         invalid

         CMI_ERR_NOMEM Out of resources to imprt segment

         CMI_ERR_PERM Insufficient permissions to allocate
         segment

     Import a remote segment so it can be manipulated by the
     process (such as an attach or control operations). A remote
     segment must be imported before it can be attached to. The
     library allocates a segment handle to represent the remote
     segment. This operation is only required on remote nodes as
     it operates on remote segment handles. If operation is
     executed on the home node where the segment identified by
     rseg resides it should return a new 'remote' segment for the
     import even though the exporting segment resides on the same
     node. Access via this imported segment requires standard
     access control mechanisms.

     Note:
         CMI error handling requires that processes that import
         segments be capable of processing asynchronous event
         notifications using evt_get(). CMI library may generate
         notification events targeted to the context to help in
         recovery of CMI and program state for imported segment
         under some error conditions. See Error Handling in cmi(5)
	 for more information.


  int tok_del(cmi_ctxt *ctxt, cmi_token *tok)

     Delete/revoke an access token for a segment

     Parameters:
         ctxt CMI context
         token Access token to delete/revoke

     Returns:
         0 on success. -1 on failure.

         CMI_ERR_INIT Calling thread is not associated with
         context

         CMI_ERR_INVAL Provided context, or token is invalid

         CMI_ERR_PERM Insufficient permissions to delete access
         token

     Remote access to segments can be revoked by deleting access
     tokens for the segment. All subsequent accesses to the
     segment with a revoked segment should result in access
     error/exception on the requesting node. Access tokens
     control access to segment from remote nodes. Standard OS
     specific access control primitives are employed for local
     access to the segment. This routine is only called on the
     home node for the segment in the context of the process that
     allocated the token using a call to tok_new().

     For CMI implementations that utilize node specific tokens
     this call only revokes the specified token associated with
     the node specified during token generation. The remaining
     tokens for the segment continue to be valid. A token must be
     deleted for a node before a new one can be generated using
     tok_new() for a given segment.

  cmi_token* tok_new(cmi_ctxt *ctxt, cmi_seg seg,
     cmi_naddr *naddr, cmi_acc flags)

     Generate an access token for a segment

     Parameters:
         ctxt CMI context
         seg CMI segment to generate access token
         naddr CMI node address to generate access token for
         flags Permissions to grant with access token

     Returns:
         Token on success. NULL on failure.

         CMI_ERR_INIT Calling thread is not associated with
         context

         CMI_ERR_INVAL Provided context, segment or naddr is
         invalid

         CMI_ERR_PERM Insufficient permissions to generate access
         token

         CMI_ERR_NOMEM Out of resources allocating token

         CMI_ERR_BOUND Access token already generated for remote
         node

     Access to segments from remote nodes are controlled via
     access tokens. Remote nodes performing operations on the
     segment must provide an access token (set on remote node
     using seg_ctl()). Access (both read and write) is only
     allowed if the requesting access token matches the token
     associated with the segment. Additionally the type of access
     being performed should match the privileges granted for the
     token (for example CMI_ACC_ATOMIC is required to issue a
     atm_cas() operation for a remote segment). All access using
     a mismatched token or privileges results in access
     error/exception on the requesting node.

     Access tokens are semi-opaque objects generated by the CMI
     library. CMI tokens are generated on the home node for the
     segment and can only be called by the process that created
     the segment via call to seg_get(). Multiple access tokens
     can be created for a segment to provide fine grained access
     control to the segment across a number of nodes. Access
     tokens are deleted/revoked via tok_del(). The following
     access privileges are defined which must be specified when
     creating an access token:

     o CMI_ACC_WRITE: Token can be used to perform writes/stores
       to the segment

     o CMI_ACC_READ: Token can be used to perform loads to the
       segment

     o CMI_ACC_ATOMIC: Token can be used to perform atomic
       operations on the segment.

     Access tokens can be generated for a specific CMI node by
     providing a CMI node address. The CMI node address for
     remote node is exchanged using some out of bound mechanism
     and uniquely identifies a CMI node on the fabric. A node
     specific access token can only be utilized by the node it is
     exported to. Attempting to set a node specific token on a
     remote node to which it was not exported will result in a
     CMI_ERR_PERM error. Only one access token per CMI segment
     can be created for a remote node. Attempting to generate
     multiple access tokens for a remote node before deallocating
     a previous token for the segment should fail with error
     CMI_ERR_BOUND. The CMI node address is available as part of
     cmi_ctxt handle and exchanged between nodes using some out
     of band mechanism.

     Some CMI implementations may not support exporting access
     tokens at node specific granularity i.e.
     CMI_CAP_NODE_SPECIFIC_TOKEN is NOT set. CMI clients can
     only  generate node agnostic access tokens using tok_new()
     by providing CMI_NADDR_ANY as the node address fields. Node
     agnostic access tokens should never  generate CMI_ERR_BOUND
     error during token generation. On CMI implementations that
     require node specific tokens attempting to generate node
     agnostic tokens will fail with error CMI_ERR_INVAL.

     cmi_token* tok_newva(cmi_ctxt *ctxt, cmi_seg seg, void *va,
         size_t size, cmi_naddr *naddr, cmi_acc flags)

     Parameters:
         ctxt CMI context
         seg  a CMI segment associated with an OSM segment

         va   a base address of an area of virtual addresses inside the
              associated OSM segment for which the token grants access

         size the size of the area of virtual addresses inside the
              associated OSM segment for which the token grants access

         naddr CMI node address to generate access token for
         flags Permissions to grant with access token

     Returns:
         Token on success. NULL on failure.

      CMI_ERR_INIT:
        - Calling thread is not associated with context
      CMI_ERR_INVAL:
        - Provided context, segment or naddr is invalid
      CMI_ERR_PERM:
        - Insufficient permissions to generate access token
      CMI_ERR_NOMEM:
        - Out of resources allocating token
      CMI_ERR_BOUND:
        - Access token for the given address are already generated
          for remote node and the given address
      CMI_ERR_NOTSUPP:
        - Legacy CMI segment (returned by seg_get()) is specified.

     Creates a token for the given range va/len of the cmi segment. The
     cmi segment must be the one created by seg_getva().  The given "va"
     is treated differently whether or not the given CMI segment is
     created with CMI_SEG_TOKEN_OFFSET.

     If the segment was created with CMI_SEG_TOKEN_OFFSET, the given "va"
     is treated as an offset inside the CMI segment.  Otherwise, the
     specified va/len must be the entire or part of the range given to
     seg_getva().  Even if va/len is inside the alias mapping of the same
     OSM's segment, the API will fail.

     Note:
         For "flags", the same flags that is available for tok_new()
         can be specified.
         The "va" value must be aligned to the seg_prot_align value that is
         returned by cmi_ctl(CMI_CTL_INFO).
  	 The "size" value must be aligned to the seg_prot_sz value that is
         returned by cmi_ctl(CMI_CTL_INFO).
         Multiple tokens can be created for the same segment but
         overlapped ranges are prohibited for the same naddr (CMI_ERR_BOUND).

  int wmb_fn(cmi_ctxt *ctxt)

     Perform a write memory barrier

     Parameters:
     ctxt CMI context

     Returns:
     0 on success. -1 on failure.

     CMI_ERR_INIT Calling thread is not associated with context

     CMI_ERR_INVAL Provided context is invalid

     Perform a 'Write Memory Barrier' that only orders stores i.e. stores
     preceding the memory barrier are committed to memory before any stores
     following the memory barrier (in program order).

     On platforms that perform natural ordering of stores (such as TSO
     architectures) the implementation of this function can be a no-op or
     even a NULL function pointer to indicate to client that write memory
     barriers are not required on the platform.

     Note:
       A CMI platform also provides an implementation of the
       write barrier using the defined cmi_wmb() macro. Platform specific
       implementations are made available in cmi_impl.h header file.
       Clients may invoke memory barriers via function pointers or inline
       the memory barrier macros with their code.

Attributes

See attributes(7) for descriptions of the following attributes:

ATTRIBUTE TYPE
ATTRIBUTE VALUE
Availability
system/cmi
Interface Stability
Uncommitted
MT-Level
MT-Safe

See Also

cmi(7), cmiadm(8)