Finding Hotspots.

Clicking on the Plot tab, we return to the view seen in Figure 7-3. Again, most of the MPI_Wait calls are seen to take relatively little time, but some calls stretch out as long as 40 ms, or so. The pattern of these timings is basically the same from one iteration to the next. Clicking on any one of the high-latency points in the scatter plot shifts the timeline view onto the interval in question, as in Figure 7-5. We may investigate events that surround the hotspot by pulling down the Navigate by... field and choosing current tid to restrict navigation to a particular thread id (MPI process). We can navigate backward or forward and add events to the Event Table. It happens that the most time-consuming pattern is:

MPI_Isend
MPI_Irecv
(no MPI activity)
MPI_Wait (on the MPI_Isend)
MPI_Wait (on the MPI_Irecv) 

Notably, the program spends the most time in the MPI_Wait calls, generally in the first of the pair.

Figure 7-5 Timeline View Centered About a High-Latency MPI_Wait Call.

Figure 7-5 shows the start and end events defining the interval, as well as other neighboring events in the Event Table at the bottom of the tnfview window.