RGM 不會直接呼叫 PROBE 方法,而會呼叫 Monitor_start 方法以在資源於節點上啟動後啟動監視器。xfnts_monitor_start 方法啟動 PMF 控制下的故障監視器。xfnts_monitor_stop 方法停止故障監視器。
SUNW.xfnts 故障監視器執行下列作業︰
透過使用特別為用於檢查簡單基於 TCP 的服務 (例如,xfs) 設計的公用程式,來定期監視 xfs 伺服器常駐程式的運作狀態。
追蹤在一定時間間隔中應用程式遇到的問題 (使用 Retry_count 和 Retry_interval 特性),並決定如果應用程式完全失敗,是否重新啟動資料服務或容錯移轉資料服務。scds_fm_action() 與 scds_fm_sleep() 函數為此追蹤及決定機制提供內建支援。
透過使用 scds_fm_action() 來實作容錯移轉或重新啟動決定。
更新資源狀態並使資源狀態可用於管理工具和 GUI。
xfonts_probe 方法將實作迴路。實作迴路之前,xfonts_probe 將執行以下作業︰
擷取 xfnts 資源的網路位址資源,如下所示︰
/* Get the ip addresses available for this resource */ if (scds_get_netaddr_list(scds_handle, &netaddr)) { scds_syslog(LOG_ERR, "No network address resource in resource group."); scds_close(&scds_handle); return (1); } /* Return an error if there are no network resources */ if (netaddr == NULL || netaddr->num_netaddrs == 0) { scds_syslog(LOG_ERR, "No network address resource in resource group."); return (1); }
呼叫 scds_fm_sleep() 並將 Thorough_probe_interval 的值作為逾時值傳送。探測由於探測之間的 Thorough_probe_interval 的值暫停,如下所示︰
timeout = scds_get_ext_probe_timeout(scds_handle); for (;;) { /* * sleep for a duration of thorough_probe_interval between * successive probes. */ (void) scds_fm_sleep(scds_handle, scds_get_rs_thorough_probe_interval(scds_handle));
xfnts_probe 方法將實作以下迴路︰
for (ip = 0; ip < netaddr->num_netaddrs; ip++) { /* * Grab the hostname and port on which the * health has to be monitored. */ hostname = netaddr->netaddrs[ip].hostname; port = netaddr->netaddrs[ip].port_proto.port; /* * HA-XFS supports only one port and * hence obtain the port value from the * first entry in the array of ports. */ ht1 = gethrtime(); /* Latch probe start time */ scds_syslog(LOG_INFO, "Probing the service on port: %d.", port); probe_result = svc_probe(scds_handle, hostname, port, timeout); /* * Update service probe history, * take action if necessary. * Latch probe end time. */ ht2 = gethrtime(); /* Convert to milliseconds */ dt = (ulong_t)((ht2 - ht1) / 1e6); /* * Compute failure history and take * action if needed */ (void) scds_fm_action(scds_handle, probe_result, (long)dt); } /* Each net resource */ } /* Keep probing forever */
svc_probe() 函數實施探測邏輯。svc_probe() 的傳回值傳送至 scds_fm_action(),可決定是否重新啟動應用程式、容錯移轉資源群組或不執行任何作業。
svc_probe() 函數透過呼叫 scds_fm_tcp_connect() 與將簡單通訊端連接至指定的連接埠。如果連接失敗,svc_probe() 將傳回一個為 100 的值,表示完全失敗。如果連接成功,但是中斷連接失敗,則 svc_probe() 將傳回一個為 50 的值,表示部分失敗。如果連接和斷開連接均成功,則 svc_probe() 將傳回一個為 0 的值,表示成功。
svc_probe() 的程式碼如下所示︰
int svc_probe(scds_handle_t scds_handle, char *hostname, int port, int timeout) { int rc; hrtime_t t1, t2; int sock; char testcmd[2048]; int time_used, time_remaining; time_t connect_timeout; /* * probe the data service by doing a socket connection to the port * specified in the port_list property to the host that is * serving the XFS data service. If the XFS service which is configured * to listen on the specified port, replies to the connection, then * the probe is successful. Else we will wait for a time period set * in probe_timeout property before concluding that the probe failed. */ /* * Use the SVC_CONNECT_TIMEOUT_PCT percentage of timeout * to connect to the port */ connect_timeout = (SVC_CONNECT_TIMEOUT_PCT * timeout)/100; t1 = (hrtime_t)(gethrtime()/1E9); /* * the probe makes a connection to the specified hostname and port. * The connection is timed for 95% of the actual probe_timeout. */ rc = scds_fm_tcp_connect(scds_handle, &sock, hostname, port, connect_timeout); if (rc) { scds_syslog(LOG_ERR, "Failed to connect to port <%d> of resource <%s>.", port, scds_get_resource_name(scds_handle)); /* this is a complete failure */ return (SCDS_PROBE_COMPLETE_FAILURE); } t2 = (hrtime_t)(gethrtime()/1E9); /* * Compute the actual time it took to connect. This should be less than * or equal to connect_timeout, the time allocated to connect. * If the connect uses all the time that is allocated for it, * then the remaining value from the probe_timeout that is passed to * this function will be used as disconnect timeout. Otherwise, the * the remaining time from the connect call will also be added to * the disconnect timeout. * */ time_used = (int)(t2 - t1); /* * Use the remaining time(timeout - time_took_to_connect) to disconnect */ time_remaining = timeout - (int)time_used; /* * If all the time is used up, use a small hardcoded timeout * to still try to disconnect. This will avoid the fd leak. */ if (time_remaining <= 0) { scds_syslog_debug(DBG_LEVEL_LOW, "svc_probe used entire timeout of " "%d seconds during connect operation and exceeded the " "timeout by %d seconds. Attempting disconnect with timeout" " %d ", connect_timeout, abs(time_used), SVC_DISCONNECT_TIMEOUT_SECONDS); time_remaining = SVC_DISCONNECT_TIMEOUT_SECONDS; } /* * Return partial failure in case of disconnection failure. * Reason: The connect call is successful, which means * the application is alive. A disconnection failure * could happen due to a hung application or heavy load. * If it is the later case, don't declare the application * as dead by returning complete failure. Instead, declare * it as partial failure. If this situation persists, the * disconnect call will fail again and the application will be * restarted. */ rc = scds_fm_tcp_disconnect(scds_handle, sock, time_remaining); if (rc != SCHA_ERR_NOERR) { scds_syslog(LOG_ERR, "Failed to disconnect to port %d of resource %s.", port, scds_get_resource_name(scds_handle)); /* this is a partial failure */ return (SCDS_PROBE_COMPLETE_FAILURE/2); } t2 = (hrtime_t)(gethrtime()/1E9); time_used = (int)(t2 - t1); time_remaining = timeout - time_used; /* * If there is no time left, don't do the full test with * fsinfo. Return SCDS_PROBE_COMPLETE_FAILURE/2 * instead. This will make sure that if this timeout * persists, server will be restarted. */ if (time_remaining <= 0) { scds_syslog(LOG_ERR, "Probe timed out."); return (SCDS_PROBE_COMPLETE_FAILURE/2); } /* * The connection and disconnection to port is successful, * Run the fsinfo command to perform a full check of * server health. * Redirect stdout, otherwise the output from fsinfo * ends up on the console. */ (void) sprintf(testcmd, "/usr/openwin/bin/fsinfo -server %s:%d > /dev/null", hostname, port); scds_syslog_debug(DBG_LEVEL_HIGH, "Checking the server status with %s.", testcmd); if (scds_timerun(scds_handle, testcmd, time_remaining, SIGKILL, &rc) != SCHA_ERR_NOERR || rc != 0) { scds_syslog(LOG_ERR, "Failed to check server status with command <%s>", testcmd); return (SCDS_PROBE_COMPLETE_FAILURE/2); } return (0); }
完成後,svc_probe() 將傳回一個值,表示成功 (0)、部分成功 (50) 或完全失敗 (100)。xfnts_probe 方法將此值傳送至 scds_fm_action()。
xfnts_probe方法將呼叫 scds_fm_action() 以決定要執行的動作。 scds_fm_action() 的邏輯如下所示︰
在 Retry_interval 特性值中保持累計的失敗歷程記錄。
如果累計的失敗達到 100 (完全失敗),則重新啟動資料服務。如果超過 Retry_interval,則重設歷史。
如果重新啟動的數目超出 Retry_count 特性的值,則在 Retry_interval 指定的時間內容錯移轉資料服務。
例如,假定探測可以連接至 xfs 伺服器,但無法中斷連接。這指示該伺服器正在執行,但可能掛起或恰好在進行暫時載入。中斷連接失敗會將部分失敗 (50) 傳送至 scds_fm_action()。此值低於重新啟動資料服務的臨界值,但該值保留在故障歷史中。
如果在下一個探測期間,伺服器中斷連接再次失敗,則一個為 50 的值會增加至 scds_fm_action() 保持的失敗歷程記錄中。現在,累計的失敗值為 100,因此,scds_fm_action() 將重新啟動資料服務。