SUNW.xfnts 故障監視器

在一個節點上啟動資源後，RGM 將不直接呼叫 PROBE 方法，而是呼叫 Monitor_start 方法來啟動監視器。 xfnts_monitor_start 方法啟動 PMF 控制下的故障監視器。 xfnts_monitor_stop 方法停止該故障監視器。

SUNW.xfnts 故障監視器執行下列作業︰

• 使用專門用來檢查基於 TCP 之簡單服務 (如 xfs) 的公用程式，定期監視 xfs 伺服器常駐程式的運作狀況。

• 在時間視窗內追蹤應用程式遇到的問題 (使用 Retry_count 與 Retry_interval 屬性)，並決定當應用程式發生完全故障時，重新啟動資料服務還是對其進行故障轉移。 scds_fm_action() 與 scds_fm_sleep() 函式為此追蹤及決定機制提供內建支援。

• 使用 scds_fm_action() 來實施故障轉移或重新啟動的決定。

• 更新資源狀態，並使其可用於管理工具及圖形使用者介面。

xfonts_probe 主迴圈

xfonts_probe 方法實施迴圈。 在實施迴圈之前，xfonts_probe

• 將為 xfnts 資源擷取網路位址資源，如下所示。

  ---

  /* Get the ip addresses available for this resource */
     if (scds_get_netaddr_list(scds_handle, &netaddr)) {
        scds_syslog(LOG_ERR,
            "No network address resource in resource group.");
        scds_close(&scds_handle);
        return (1);
     }
  
     /* Return an error if there are no network resources */
     if (netaddr == NULL || netaddr->num_netaddrs == 0) {
        scds_syslog(LOG_ERR,
            "No network address resource in resource group.");
        return (1);
     }

  ---

• 呼叫 scds_fm_sleep()，並將 Thorough_probe_interval 的值作為逾時值傳送。 對於探測之間的 Thorough_probe_interval 值，探測將靜止。

  ---

  timeout = scds_get_ext_probe_timeout(scds_handle);
  
     for (;;) {
        /*
         * sleep for a duration of thorough_probe_interval between
         *  successive probes.
         */
        (void) scds_fm_sleep(scds_handle,
            scds_get_rs_thorough_probe_interval(scds_handle));

  ---

xfnts_probe 方法實施迴圈，如下所示。

for (ip = 0; ip < netaddr->num_netaddrs; ip++) {
         /*
          * Grab the hostname and port on which the
          * health has to be monitored.
          */
         hostname = netaddr->netaddrs[ip].hostname;
         port = netaddr->netaddrs[ip].port_proto.port;
         /*
          * HA-XFS supports only one port and
          * hence obtain the port value from the
          * first entry in the array of ports.
          */
         ht1 = gethrtime(); /* Latch probe start time */
         scds_syslog(LOG_INFO, "Probing the service on port: %d.", port);

         probe_result =
         svc_probe(scds_handle, hostname, port, timeout);

         /*
          * Update service probe history,
          * take action if necessary.
          * Latch probe end time.
          */
         ht2 = gethrtime();

         /* Convert to milliseconds */
         dt = (ulong_t)((ht2 - ht1) / 1e6);

         /*
          * Compute failure history and take
          * action if needed
          */
         (void) scds_fm_action(scds_handle,
             probe_result, (long)dt);
      }   /* Each net resource */
   }    /* Keep probing forever */

svc_probe() 函式實施探測邏輯。 來自 svc_probe() 的傳回值被傳送至 scds_fm_action()，該值將決定是否重新啟動應用程式，對資源群組進行故障轉移，或者不執行任何作業。

svc_probe() 函式

svc_probe() 函式透過呼叫scds_fm_tcp_connect()，建立與指定通訊埠的簡單套接字連接。 如果連接失敗，svc_probe() 將傳回值 100，指示發生完全故障。 如果連接成功但斷開連接失敗，svc_probe() 將傳回值 50，指示發生部分故障。 如果連接與斷開連接均成功，svc_probe() 將傳回值 0，指示成功。

svc_probe() 的程式碼如下所示。

int svc_probe(scds_handle_t scds_handle, 
char *hostname, int port, int timeout)
{
   int  rc;
   hrtime_t   t1, t2;
   int    sock;
   char   testcmd[2048];
   int    time_used, time_remaining;
   time_t      connect_timeout;


   /*
    * probe the data service by doing a socket connection to the port */
    * specified in the port_list property to the host that is
    * serving the XFS data service. If the XFS service which is configured
    * to listen on the specified port, replies to the connection, then
    * the probe is successful. Else we will wait for a time period set
    * in probe_timeout property before concluding that the probe failed.
    */

   /*
    * Use the SVC_CONNECT_TIMEOUT_PCT percentage of timeout
    * to connect to the port
    */
   connect_timeout = (SVC_CONNECT_TIMEOUT_PCT * timeout)/100;
   t1 = (hrtime_t)(gethrtime()/1E9);

   /*
    * the probe makes a connection to the specified hostname and port.
    * The connection is timed for 95% of the actual probe_timeout.
    */
   rc = scds_fm_tcp_connect(scds_handle, &sock, hostname, port,
       connect_timeout);
   if (rc) {
      scds_syslog(LOG_ERR,
          "Failed to connect to port <%d> of resource <%s>.",
          port, scds_get_resource_name(scds_handle));
      /* this is a complete failure */
      return (SCDS_PROBE_COMPLETE_FAILURE);
   }

   t2 = (hrtime_t)(gethrtime()/1E9);

   /*
    * Compute the actual time it took to connect. This should be less than
    * or equal to connect_timeout, the time allocated to connect.
    * If the connect uses all the time that is allocated for it,
    * then the remaining value from the probe_timeout that is passed to
    * this function will be used as disconnect timeout. Otherwise, the
    * the remaining time from the connect call will also be added to
    * the disconnect timeout.
    *
    */

   time_used = (int)(t2 - t1);

   /*
    * Use the remaining time(timeout - time_took_to_connect) to disconnect
    */

   time_remaining = timeout - (int)time_used;

   /*
    * If all the time is used up, use a small hardcoded timeout
    * to still try to disconnect. This will avoid the fd leak.
    */
   if (time_remaining <= 0) {
      scds_syslog_debug(DBG_LEVEL_LOW,
          "svc_probe used entire timeout of "
          "%d seconds during connect operation and exceeded the "
          "timeout by %d seconds. Attempting disconnect with timeout"
          " %d ",
          connect_timeout,
          abs(time_used),
          SVC_DISCONNECT_TIMEOUT_SECONDS);

      time_remaining = SVC_DISCONNECT_TIMEOUT_SECONDS;
   }

   /*
    * Return partial failure in case of disconnection failure.
    * Reason: The connect call is successful, which means
    * the application is alive. A disconnection failure
    * could happen due to a hung application or heavy load.
    * If it is the later case, don't declare the application
    * as dead by returning complete failure. Instead, declare
    * it as partial failure. If this situation persists, the
    * disconnect call will fail again and the application will be
    * restarted.
    */
   rc = scds_fm_tcp_disconnect(scds_handle, sock, time_remaining);
   if (rc != SCHA_ERR_NOERR) {
      scds_syslog(LOG_ERR,
          "Failed to disconnect to port %d of resource %s.",
          port, scds_get_resource_name(scds_handle));
      /* this is a partial failure */
      return (SCDS_PROBE_COMPLETE_FAILURE/2);
   }

   t2 = (hrtime_t)(gethrtime()/1E9);
   time_used = (int)(t2 - t1);
   time_remaining = timeout - time_used;

   /*
    * If there is no time left, don't do the full test with
    * fsinfo. Return SCDS_PROBE_COMPLETE_FAILURE/2
    * instead. This will make sure that if this timeout
    * persists, server will be restarted.
    */
   if (time_remaining <= 0) {
      scds_syslog(LOG_ERR, "Probe timed out.");
      return (SCDS_PROBE_COMPLETE_FAILURE/2);
   }

   /*
    * The connection and disconnection to port is successful,
    * Run the fsinfo command to perform a full check of
    * server health.
    * Redirect stdout, otherwise the output from fsinfo
    * ends up on the console.
    */
   (void) sprintf(testcmd,
       "/usr/openwin/bin/fsinfo -server %s:%d > /dev/null",
       hostname, port);
   scds_syslog_debug(DBG_LEVEL_HIGH,
       "Checking the server status with %s.", testcmd);
   if (scds_timerun(scds_handle, testcmd, time_remaining,
      SIGKILL, &rc) != SCHA_ERR_NOERR || rc != 0) {

      scds_syslog(LOG_ERR,
         "Failed to check server status with command <%s>",
         testcmd);
      return (SCDS_PROBE_COMPLETE_FAILURE/2);
   }
   return (0);
}

完成後，svc_probe() 將傳回成功值 (0)、部分故障值 (50) 或完全故障值 (100)。 xfnts_probe 方法將此值傳送至 scds_fm_action()。

決定故障監視器的動作

xfnts_probe 方法呼叫 scds_fm_action() 來決定要執行的動作。 scds_fm_action() 的邏輯如下所示︰

• 在 Retry_interval 屬性值範圍內，保留累計故障歷史。

• 如果累計故障達到 100 (完全故障)，則重新啟動資料服務。 如果超過 Retry_interval，則重設歷史。

• 如果在 Retry_interval 指定的時間內，重新啟動的次數超過 Retry_count 屬性值，則對資料服務進行故障轉移。

例如，假定探測連接至 xfs 伺服器，但無法取消連接。 這指示該伺服器正在執行，但可能掛起或恰好在進行暫時載入。 斷開連接的故障將部分 (50) 故障發送至 scds_fm_action()。 此值低於重新啟動資料服務的臨界值，但該值保留在故障歷史中。

如果在下一次探測期間，伺服器仍無法結束連線，則值 50 將被新增至 scds_fm_action() 維護的故障歷史中。 累計故障值現在為 100，因此 scds_fm_action() 將重新啟動資料服務。