Sun MPI 4.0 Programming and Reference Guide

TNF Probes for MPI

Each MPI routine is associated with two TNF probes: one ending in _start, and one ending in _end. Probes are also included for some specific arguments, most of which are defined in the MPI standard and described in the man pages included with Sun MPI 4.0. Four of the arguments, however, are not mentioned in the standard or man pages:

bytes - The number of bytes sent or received by an MPI process. See the next section for more information about the bytes argument.

ctxt - The context id is a number assigned to a particular communicator. The processes in a given communicator may be associated with only one context id. You can determine the context id associated with a communicator using either the MPI_Comm_set_name_end or the MPI_Comm_get_name_end probe.

newctxt - The context id associated with a communicator that is returned as a newcomm or comm_out argument.

request - An integer that uniquely refers to a request object. For Fortran calls, this integer is equal to the request-handle argument.

The `bytes` Argument

The meaning of the bytes argument varies slightly depending on the situation. Here is some general information for different types of sends and receives, followed by some examples of how the byte argument works with them.

point-to-point blocking sends - The start probes for routines that initiate point-to-point blocking sends report the number of bytes to be sent. These routines are:
- MPI_Bsend
- MPI_Rsend
- MPI_Send
- MPI_Ssend
- MPI_Sendrecv
- MPI_Sendrecv_replace

point-to-point nonblocking sends - The end probes for routines that initiate point-to-point nonblocking sends report the number of bytes to be sent. These routines are:
- MPI_Bsend_init
- MPI_Ibsend
- MPI_Irsend
- MPI_Isend
- MPI_Issend
- MPI_Rsend_init
- MPI_Send_init
- MPI_Ssend_init

point-to-point receives - The end probes for routines that terminate or could terminate point-to-point nonblocking sends report the number of bytes actually received. These routines are:
- MPI_Iprobe
- MPI_Probe
- MPI_Recv
- MPI_Test
- MPI_Testall
- MPI_Testany
- MPI_Testsome
- MPI_Wait
- MPI_Waitall
- MPI_Waitany
- MPI_Waitsome
- MPI_Sendrecv
- MPI_Sendrecv_replace

collectives - The start probes for collective routines report the number of bytes to be sent from an MPI process and the number to be received at the process. Such byte counts are independent of the algorithm used. For example, the number of bytes sent from the root in a broadcast is given as the number of bytes in the broadcast message, regardless of whether the root sends this message multiple times as part of a binary-tree fan-out. These collective routines are:
- MPI_Allgather, MPI_Allgatherv

sendbytes - Number of bytes to be sent from this process.

recvbytes - Total number of bytes to be received at any process from all processes.

MPI_Allreduce, MPI_Reduce, MPI_Reduce_scatter

bytes - Number of bytes on any process to be reduced.

MPI_Alltoall, MPI_Alltoallv

sendbytes - Total number of bytes to be sent from this process to all processes.

recvbytes - Total number of bytes to be received at this process from all processes.

MPI_Bcast

bytes - Number of bytes to be broadcast.

MPI_Gather, MPI_Gatherv

sendbytes - Root reports total number of bytes to be sent; other processes report 0.

recvbytes - Root reports total number of bytes to be received; other processes report 0.

MPI_Scan

bytes - Number of bytes contributed by any process.

MPI_Scatter, MPI_Scatterv

sendbytes - Root reports total number of bytes to be sent; other processes report 0.

recvbytes - Number of bytes to be received at this process from the root.

pack and unpack - The start probes for these routines report the number of bytes packed or unpaced. These routines are:
- MPI_Pack
- MPI_Unpack

Examples:

MPI_Send

call MPI_Send(x,m,MPI_REAL8,...)

Probe mpi_send_start reports that 8*m bytes are to be sent.

MPI_Recv

call MPI_Recv(x,n,MPI_REAL8,...)

Probe mpi_recv_end reports the number of bytes that were actually received, which must be at most 8*n.

MPI_Sendrecv

call MPI_Sendrecv(x,m,MPI_REAL8,...,y,n,MPI_REAL8,...)

Probe mpi_sendrecv_start reports that 8*m bytes are to be sent, and probe mpi_sendrecv_end reports the number of bytes that were actually received, which must be at most 8*n.

MPI_Irecv, MPI_Wait

integer req
call MPI_Irecv(x,n,MPI_REAL8,...,req,...)
call MPI_Wait(req,...)

Probe mpi_wait_end reports the number of bytes that were actually received, which must be at most 8*n.

MPI_Isend, MPI_Irecv, MPI_Wait

integer reqs(2)
call MPI_Isend(x,m,MPI_REAL8,...,reqs(1),...)
call MPI_Irecv(Y,N,MPI_REAL8,...,reqs(2),...)
call MPI_Waitany(2,reqs,...)
call MPI_Waitany(2,reqs,...)

Probe mpi_isend_start reports that 8*m bytes are to be sent. The MPI_Waitany call that completes the receive will show the number of bytes that were actually received, which must be at most 8*n, in its mpi_waitany_end probe. The other MPI_Waitany call, which completes the send, will report 0 bytes received.

MPI_Waitall

integer reqs(8)
call MPI_Isend(x1,m,MPI_REAL8,...,reqs(1),...)
call MPI_Isend(x2,m,MPI_REAL8,...,reqs(2),...)
call MPI_Isend(x3,m,MPI_REAL8,...,reqs(3),...)
call MPI_Isend(x4,m,MPI_REAL8,...,reqs(4),...)
call MPI_Irecv(x5,n,MPI_REAL8,...,reqs(5),...)
call MPI_Irecv(x6,n,MPI_REAL8,...,reqs(6),...)
call MPI_Irecv(x7,n,MPI_REAL8,...,reqs(7),...)
call MPI_Irecv(x8,n,MPI_REAL8,...,reqs(8),...)
call MPI_Waitall(8,reqs,..)

Probe mpi_isend_start reports that 8*m bytes are to be sent in each of the four MPI_Isend cases. Probe mpi_waitall_end reports the number of bytes that were actually received, which must be at most 4*8*n.

Groups

Every TNF probe for MPI is associated with the mpi_api group, so choosing that group allows Prism to probe all the MPI routines for which probes exist (including the I/O routines). Additional groups exist to probe subsets of the MPI routines, as well. Some routines are associated with more than one group. The ten groups for MPI routine probes are these:

mpi_api - All the TNF probes for MPI routines

mpi_blkp2p - Probes for blocking point-to-point routines

mpi_coll - Probes for collective routines

mpi_comm - Probes for communicator-related routines

mpi_datatypes - Probes for data type-related routines

mpi_nblkp2p - Probes for nonblocking point-to-point routines

mpi_procmgmt - Probes for process-management routines

mpi_pt2pt - Probes for all point-to-point routines (blocking and nonblocking)

mpi_request - Probes for functions producing or acting on request(s)

mpi_topo - Probes for topology-related routines

Probes for MPI (Non-I/O Routines)

Table C-1 TNF Probes and Associated Arguments and Groups for MPI Calls


Probe	Argument(s)	Group(s), in Addition to `mpi_api`
`MPI_Address_start`
`MPI_Address_end`
`MPI_Allgather_start`	`sendbytes recvbytes ctxt`	`mpi_coll`
`MPI_Allgather_end`		`mpi_coll`
`MPI_Allgatherv_start`	`sendbytes recvbytes ctxt`	`mpi_coll`
`MPI_Allgatherv_end`		`mpi_coll`
`MPI_Allreduce_start`	`bytes ctxt`	`mpi_coll`
`MPI_Allreduce_end`		`mpi_coll`
`MPI_Alltoall_start`	`sendbytes recvbytes ctxt`	`mpi_coll`
`MPI_Alltoall_end`		`mpi_coll`
`MPI_Alltoallv_start`	`sendbytes recvbytes ctxt`	`mpi_coll`
`MPI_Alltoallv_end`		`mpi_coll`
`MPI_Attr_delete_start`
`MPI_Attr_delete_end`
`MPI_Attr_get_start`
`MPI_Attr_get_end`
`MPI_Attr_put_start`
`MPI_Attr_put_end`
`MPI_Barrier_start`	`ctxt`	`mpi_coll`
`MPI_Barrier_end`		`mpi_coll`
`MPI_Bcast_start`	`bytes root ctxt`	`mpi_coll`
`MPI_Bcast_end`		`mpi_coll`
`MPI_Bsend_start`	`bytes dest tag`	`mpi_pt2pt mpi_blkp2p`
`MPI_Bsend_end`		`mpi_pt2pt mpi_blkp2p`
`MPI_Bsend_init_start`		`mpi_pt2pt mpi_request`
`MPI_Bsend_init_end`	`bytes dest tag request`	`mpi_pt2pt mpi_request`
`MPI_Buffer_attach_start`	`buffer size`
`MPI_Buffer_attach_end`	`buffer size`
`MPI_Buffer_detach_start`	`buffer size`
`MPI_Buffer_detach_end`
`MPI_Cancel_start`	`request`	`mpi_request`
`MPI_Cancel_end`		`mpi_request`
`MPI_Cart_coords_start`		`mpi_topo`
`MPI_Cart_coords_end`		`mpi_topo`
`MPI_Cart_create_start`		`mpi_topo`
`MPI_Cart_create_end`		`mpi_topo`
`MPI_Cartdim_get_start`		`mpi_topo`
`MPI_Cartdim_get_end`		`mpi_topo`
`MPI_Cart_get_start`		`mpi_topo`
`MPI_Cart_get_end`		`mpi_topo`
`MPI_Cart_map_start`		`mpi_topo`
`MPI_Cart_map_end`		`mpi_topo`
`MPI_Cart_rank_start`		`mpi_topo`
`MPI_Cart_rank_end`		`mpi_topo`
`MPI_Cart_shift_start`		`mpi_topo`
`MPI_Cart_shift_end`		`mpi_topo`
`MPI_Cart_sub_start`		`mpi_topo`
`MPI_Cart_sub_end`		`mpi_topo`
`MPI_Close_port_start`	`port_name`	`mpi_procmgmt`
`MPI_Close_port_end`		`mpi_procmgmt`
`MPI_Comm_accept_start`	`port_name root ctxt`	`mpi_procmgmt`
`MPI_Comm_accept_end`	`port_name root ctxt newctxt`	`mpi_procmgmt`
`MPI_Comm_compare_start`		`mpi_comm`
`MPI_Comm_compare_end`		`mpi_comm`
`MPI_Comm_connect_start`	`port_name root ctxt`	`mpi_procmgmt`
`MPI_Comm_connect_end`	`port_name root ctxt newctxt`	`mpi_procmgmt`
`MPI_Comm_create_start`	`ctxt group`	`mpi_comm`
`MPI_Comm_create_end`	`ctxt newctxt`	`mpi_comm`
`MPI_Comm_create_errhandler_start`
`MPI_Comm_create_errhandler_end`
`MPI_Comm_create_keyval_start`
`MPI_Comm_create_keyval_end`
`MPI_Comm_delete_attr_start`
`MPI_Comm_delete_attr_end`
`MPI_Comm_disconnect_start`	`ctxt`	`mpi_procmgmt`
`MPI_Comm_disconnect_end`		`mpi_procmgmt`
`MPI_Comm_dup_start`	`ctxt`	`mpi_comm`
`MPI_Comm_dup_end`	`ctxt newctxt`	`mpi_comm`
`MPI_Comm_free_start`	`ctxt`	`mpi_comm`
`MPI_Comm_free_end`		`mpi_comm`
`MPI_Comm_free_keyval_start`
`MPI_Comm_free_keyval_end`
`MPI_Comm_get_attr_start`
`MPI_Comm_get_attr_end`
`MPI_Comm_get_errhandler_start`
`MPI_Comm_get_errhandler_end`
`MPI_Comm_get_name_start`
`MPI_Comm_get_name_end`	ctxt comm_name	`mpi_comm`
`MPI_Comm_group_start`
`MPI_Comm_group_end`
`MPI_Comm_remote_group_start`
`MPI_Comm_remote_group_end`
`MPI_Comm_set_attr_start`
`MPI_Comm_set_attr_end`
`MPI_Comm_set_errhandler_start`
`MPI_Comm_set_errhandler_end`
`MPI_Comm_set_name_start`
`MPI_Comm_set_name_end`	`ctxt comm_name`	`mpi_comm`
`MPI_Comm_split_start`	`ctxt`	`mpi_comm`
`MPI_Comm_split_end`	`ctxt newctxt`	`mpi_comm`
`MPI_Comm_test_inter_start`
`MPI_Comm_test_inter_end`
`MPI_Dims_create_start`		`mpi_topo`
`MPI_Dims_create_end`		`mpi_topo`
`MPI_Errhandler_create_start`
`MPI_Errhandler_create_end`
`MPI_Errhandler_free_start`
`MPI_Errhandler_free_end`
`MPI_Errhandler_get_start`
`MPI_Errhandler_get_end`
`MPI_Errhandler_set_start`
`MPI_Errhandler_set_end`
`MPI_Error_class_start`
`MPI_Error_class_end`
`MPI_Error_string_start`
`MPI_Error_string_end`
`MPI_Finalize_start`
`MPI_Finalize_end`
`MPI_Gather_start`	`sendbytes recvbytes root ctxt`	`mpi_coll`
`MPI_Gather_end`		`mpi_coll`
`MPI_Gatherv_start`	`sendbytes recvbytes root ctxt`	`mpi_coll`
`MPI_Gatherv_end`		`mpi_coll`
`MPI_Get_address_start`
`MPI_Get_address_end`
`MPI_Get_count_start`
`MPI_Get_count_end`
`MPI_Get_elements_start`
`MPI_Get_elements_end`
`MPI_Get_processor_name_start`
`MPI_Get_processor_name_end`
`MPI_Get_version_start`
`MPI_Get_version_end`
`MPI_Graph_create_start`		`mpi_topo`
`MPI_Graph_create_end`		`mpi_topo`
`MPI_Graphdims_get_start`		`mpi_topo`
`MPI_Graphdims_get_end`		`mpi_topo`
`MPI_Graph_get_start`		`mpi_topo`
`MPI_Graph_get_end`		`mpi_topo`
`MPI_Graph_map_start`		`mpi_topo`
`MPI_Graph_map_end`		`mpi_topo`
`MPI_Graph_neighbors_start`		`mpi_topo`
`MPI_Graph_neighbors_end`		`mpi_topo`
`MPI_Graph_neighbors_count_start`		`mpi_topo`
`MPI_Graph_neighbors_count_end`		`mpi_topo`
`MPI_Grequest_complete_start`	`request`	`mpi_request`
`MPI_Grequest_complete_end`		`mpi_request`
`MPI_Grequest_start_start`	`request`	`mpi_request`
`MPI_Grequest_start_end`		`mpi_request`
`MPI_Group_compare_start`
`MPI_Group_compare_end`
`MPI_Group_difference_start`
`MPI_Group_difference_end`
`MPI_Group_excl_start`
`MPI_Group_excl_end`
`MPI_Group_free_start`
`MPI_Group_free_end`
`MPI_Group_incl_start`
`MPI_Group_incl_end`
`MPI_Group_intersection_start`
`MPI_Group_intersection_end`
`MPI_Group_range_excl_start`
`MPI_Group_range_excl_end`
`MPI_Group_range_incl_start`
`MPI_Group_range_incl_end`
`MPI_Group_translate_ranks_start`
`MPI_Group_translate_ranks_end`
`MPI_Group_union_start`
`MPI_Group_union_end`
`MPI_Ibsend_start`		`mpi_pt2pt mpi_nblkp2p mpi_request`
`MPI_Ibsend_end`	`bytes dest tag done request`	`mpi_pt2pt mpi_nblkp2p mpi_request`
`MPI_Info_create_start`
`MPI_Info_create_end`
`MPI_Info_delete_start`
`MPI_Info_delete_end`
`MPI_Info_dup_start`
`MPI_Info_dup_end`
`MPI_Info_free_start`
`MPI_Info_free_end`
`MPI_Info_get_start`
`MPI_Info_get_end`
`MPI_Info_get_nkeys_start`
`MPI_Info_get_nkeys_end`
`MPI_Info_get_nthkey_start`
`MPI_Info_get_nthkey_end`
`MPI_Info_get_valuelen_start`
`MPI_Info_get_valuelen_end`
`MPI_Info_set_start`
`MPI_Info_set_end`
`MPI_Intercomm_create_start`		`mpi_comm`
`MPI_Intercomm_create_end`		`mpi_comm`
`MPI_Intercomm_merge_start`		`mpi_comm`
`MPI_Intercomm_merge_end`		`mpi_comm`
`MPI_Iprobe_start`	`source tag ctxt`
`MPI_Iprobe_end`	`source tag ctxt flag`
`MPI_Irecv_start`		`mpi_pt2pt mpi_nblkp2p mpi_request`
`MPI_Irecv_end`	`done request`	`mpi_pt2pt mpi_nblkp2p mpi_request`
`MPI_Irsend_start`		`mpi_pt2pt mpi_nblkp2p mpi_request`
`MPI_Irsend_end`	`bytes dest tag done request`	`mpi_pt2pt mpi_nblkp2p mpi_request`
`MPI_Isend_start`		`mpi_pt2pt mpi_nblkp2p mpi_request`
`MPI_Isend_end`	`bytes dest tag done request`	`mpi_pt2pt mpi_nblkp2p mpi_request`
`MPI_Issend_start`		`mpi_pt2pt mpi_nblkp2p mpi_request`
`MPI_Issend_end`	`bytes dest tag done request`	`mpi_pt2pt mpi_nblkp2p mpi_request`
`MPI_Keyval_create_start`
`MPI_Keyval_create_end`
`MPI_Keyval_free_start`
`MPI_Keyval_free_end`
`MPI_Op_create_start`
`MPI_Op_create_end`
`MPI_Open_port_start`	`port_name`	`mpi_procmgmt`
`MPI_Open_port_end`	`port_name`	`mpi_procmgmt`
`MPI_Op_free_start`
`MPI_Op_free_end`
`MPI_Pack_start`	`bytes`	`mpi_datatypes`
`MPI_Pack_end`		`mpi_datatypes`
`MPI_Pack_size_start`	`count datatype`	`mpi_datatypes`
`MPI_Pack_size_end`	`count datatype size`	`mpi_datatypes`
`MPI_Pcontrol_start`
`MPI_Pcontrol_end`
`MPI_Probe_start`	`source tag ctxt`
`MPI_Probe_end`	`source tag ctxt`
`MPI_Query_thread_start`
`MPI_Query_thread_end`
`MPI_Recv_start`		`mpi_pt2pt mpi_blkp2p`
`MPI_Recv_end`	`bytes source tag`	`mpi_pt2pt mpi_blkp2p`
`MPI_Recv_init_start`		`mpi_pt2pt mpi_request`
`MPI_Recv_init_end`	`request`	`mpi_pt2pt mpi_request`
`MPI_Reduce_start`	`bytes root ctxt`	`mpi_coll`
`MPI_Reduce_end`		`mpi_coll`
`MPI_Reduce_scatter_start`	`bytes ctxt`	`mpi_coll`
`MPI_Reduce_scatter_end`		`mpi_coll`
`MPI_Request_free_start`	`request`	`mpi_request`
`MPI_Request_free_end`		`mpi_request`
`MPI_Rsend_start`	`bytes dest tag`	`mpi_pt2pt mpi_blkp2p`
`MPI_Rsend_end`		`mpi_pt2pt mpi_blkp2p`
`MPI_Rsend_init_start`		`mpi_pt2pt mpi_request`
`MPI_Rsend_init_end`	`bytes dest tag request`	`mpi_pt2pt mpi_request`
`MPI_Scan_start`	`bytes ctxt`	`mpi_coll`
`MPI_Scan_end`		`mpi_coll`
`MPI_Scatter_start`	`sendbytes recvbytes root ctxt`	`mpi_coll`
`MPI_Scatter_end`		`mpi_coll`
`MPI_Scatterv_start`	`sendbytes recvbytes root ctxt`	`mpi_coll`
`MPI_Scatterv_end`		`mpi_coll`
`MPI_Send_start`	`bytes dest tag`	`mpi_pt2pt mpi_blkp2p`
`MPI_Send_end`		`mpi_pt2pt mpi_blkp2p`
`MPI_Send_init_start`		`mpi_pt2pt mpi_request`
`MPI_Send_init_end`	`bytes dest tag request`	`mpi_pt2pt mpi_request`
`MPI_Sendrecv_start`	`bytes dest sendtag`	`mpi_pt2pt mpi_blkp2p`
`MPI_Sendrecv_end`	`bytes source recvtag`	`mpi_pt2pt mpi_blkp2p`
`MPI_Sendrecv_replace_start`	`bytes dest sendtag`	`mpi_pt2pt mpi_blkp2p`
`MPI_Sendrecv_replace_end`	`bytes source recvtag`	`mpi_pt2pt mpi_blkp2p`
`MPI_Ssend_start`	`bytes dest tag`	`mpi_pt2pt mpi_blkp2p`
`MPI_Ssend_end`		`mpi_pt2pt mpi_blkp2p`
`MPI_Ssend_init_start`		`mpi_pt2pt mpi_request`
`MPI_Ssend_init_end`	`bytes dest tag request`	`mpi_pt2pt mpi_request`
`MPI_Start_start`	`request`	`mpi_pt2pt mpi_request`
`MPI_Start_end`		`mpi_pt2pt mpi_request`
`MPI_Startall_start`	`count`	`mpi_pt2pt mpi_request`
`MPI_Startall_end`		`mpi_pt2pt mpi_request`
`MPI_Status_set_cancelled_start`
`MPI_Status_set_cancelled_end`
`MPI_Status_set_elements_start`
`MPI_Status_set_elements_end`
`MPI_Test_start`	`request`	`mpi_request`
`MPI_Test_end`	`recvbytes source recvtag flag request`	`mpi_request`
`MPI_Testall_start`	`count`	`mpi_request`
`MPI_Testall_end`	`bytes count flag`	`mpi_request`
`MPI_Testany_start`	`count`	`mpi_request`
`MPI_Testany_end`	`bytes index flag`	`mpi_request`
`MPI_Test_cancelled_start`		`mpi_request`
`MPI_Test_cancelled_end`	`flag`	`mpi_request`
`MPI_Testsome_start`	`incount`	`mpi_request`
`MPI_Testsome_end`	`bytes outcount`	`mpi_request`
`MPI_Topo_test_start`		`mpi_topo`
`MPI_Topo_test_end`		`mpi_topo`
`MPI_Type_contiguous_start`		`mpi_datatypes`
`MPI_Type_contiguous_end`		`mpi_datatypes`
`MPI_Type_create_hindexed_start`		`mpi_datatypes`
`MPI_Type_create_hindexed_end`		`mpi_datatypes`
`MPI_Type_create_f90_integer_start`		`mpi_datatypes`
`MPI_Type_create_f90_integer_end`		`mpi_datatypes`
`MPI_Type_create_keyval_start`		`mpi_datatypes`
`MPI_Type_create_keyval_end`		`mpi_datatypes`
`MPI_Type_create_struct_start`		`mpi_datatypes`
`MPI_Type_create_struct_end`		`mpi_datatypes`
`MPI_Type_delete_attr_start`		`mpi_datatypes`
`MPI_Type_delete_attr_end`		`mpi_datatypes`
`MPI_Type_dup_start`		`mpi_datatypes`
`MPI_Type_dup_end`		`mpi_datatypes`
`MPI_Type_extent_start`		`mpi_datatypes`
`MPI_Type_extent_end`		`mpi_datatypes`
`MPI_Type_free_start`		`mpi_datatypes`
`MPI_Type_free_end`		`mpi_datatypes`
`MPI_Type_free_keyval_start`		`mpi_datatypes`
`MPI_Type_free_keyval_end`		`mpi_datatypes`
`MPI_Type_get_attr_start`		`mpi_datatypes`
`MPI_Type_get_attr_end`		`mpi_datatypes`
`MPI_Type_get_contents_start`		`mpi_datatypes`
`MPI_Type_get_contents_end`		`mpi_datatypes`
`MPI_Type_get_envelope_start`		`mpi_datatypes`
`MPI_Type_get_envelope_end`		`mpi_datatypes`
`MPI_Type_get_extent_start`		`mpi_datatypes`
`MPI_Type_get_extent_end`		`mpi_datatypes`
`MPI_Type_get_name_start`		`mpi_datatypes`
`MPI_Type_get_name_end`		`mpi_datatypes`
`MPI_Type_get_true_extent_start`		`mpi_datatypes`
`MPI_Type_get_true_extent_end`		`mpi_datatypes`
`MPI_Type_hindexed_start`		`mpi_datatypes`
`MPI_Type_hindexed_end`		`mpi_datatypes`
`MPI_Type_indexed_start`		`mpi_datatypes`
`MPI_Type_indexed_end`		`mpi_datatypes`
`MPI_Type_create_indexed_block_start`		`mpi_datatypes`
`MPI_Type_create_indexed_block_end`		`mpi_datatypes`
`MPI_Type_lb_start`		`mpi_datatypes`
`MPI_Type_lb_end`		`mpi_datatypes`
`MPI_Type_create_resized_start`		`mpi_datatypes`
`MPI_Type_create_resized_end`		`mpi_datatypes`
`MPI_Type_set_attr_start`		`mpi_datatypes`
`MPI_Type_set_attr_end`		`mpi_datatypes`
`MPI_Type_set_name_start`		`mpi_datatypes`
`MPI_Type_set_name_end`		`mpi_datatypes`
`MPI_Type_size_start`		`mpi_datatypes`
`MPI_Type_size_end`		`mpi_datatypes`
`MPI_Type_struct_start`		`mpi_datatypes`
`MPI_Type_struct_end`		`mpi_datatypes`
`MPI_Type_ub_start`		`mpi_datatypes`
`MPI_Type_ub_end`		`mpi_datatypes`
`MPI_Unpack_start`	`bytes`	`mpi_datatypes`
`MPI_Unpack_end`		`mpi_datatypes`
`MPI_Wait_start`	request	`mpi_request`
`MPI_Wait_end`	`recvbytes source recvtag request`	`mpi_request`
`MPI_Waitall_start`	`count`	`mpi_request`
`MPI_Waitall_end`	`bytes count`	`mpi_request`
`MPI_Waitany_start`	`count`	`mpi_request`
`MPI_Waitany_end`	`bytes index`	`mpi_request`
`MPI_Waitsome_start`	`incount`	`mpi_request`
`MPI_Waitsome_end`	`bytes outcount`	`mpi_request`

TNF Probes for MPI

The bytes Argument

Examples:

Groups

Probes for MPI (Non-I/O Routines)

The `bytes` Argument