Chapter 7   Understanding Log Files


You can monitor your server's activity using several different methods. You can view the server's status in real time by using the Hypertext Transfer Protocol (HTTP). This chapter discusses how to monitor your server by recording and viewing log files or by using the performance monitoring tools provided with your operating system.

This chapter contains the following sections:

• About Log Files

• Viewing an Access Log File

• Monitoring the Server Using HTTP

• Archiving Log Files

• Setting Log Preferences

• Flushing the Log Buffer

• Running the Log Analyzer

• Viewing Events (Windows NT)

About Log Files

---

Server log files record your server's activity. You can use these logs to monitor your server and to help you when troubleshooting. The error log file, located in https-servername/logs/errors in the server root directory, lists all the errors the server has encountered. The access log, located in
https-servername/logs/access in the server root directory, records information about requests to the server and the responses from the server. You can configure the information recorded in the iPlanet Web Server access log file. You use the log analyzer to generate server statistics. You can back up server error and access log files by archiving them.

Viewing an Access Log File

---

You can view the server's active and archived access log files.

To view the Administration Server's access log from the Administration Server, choose the Preferences tab, and then choose The View Access Log Page.

To view an access log from the Server Manager, choose the Status tab, and then choose The View Access Log Page.

The following is an example of an access log in the Common Logfile Format (you specify the format in the Log Preferences window; see Setting Log Preferences for more information):

wiley.a.com - - [16/Feb/1999:21:18:26 -0800] "GET / HTTP/1.0" 200 751 wiley.a.com - - [17/Feb/1999:1:04:38 -0800] "GET /docs/grafx/icon.gif HTTP/1.0" 204 342 wiley.a.com - - [20/Feb/1999:4:36:53 -0800] "GET /help HTTP/1.0" 401 571 arrow.a.com - john [29/Mar/1999:4:36:53 -0800] "GET /help HTTP/1.0" 401 571

Table 7-1 describes the last line of this sample access log.

Table 7-1 The fields in the last line of the sample access log file 

Access Log Field	Example
Hostname or IP address of client	arrow.a.com. (In this case, the hostname is shown because the web server's setting for DNS lookups is enabled; if DNS lookups were disabled, the client's IP address would appear.
RFC 931 information	- (RFC 931 identity not implemented)
Username	john (username entered by the client for authentication)
Date/time of request	29/Mar/1999:4:36:53 -0800
Request	GET /help
Protocol	HTTP/1.0
Status code	401
Bytes transferred	571

The following is an example of an access log using the flexible logging format (you specify the format in the Log Preferences page; see Setting Log Preferences for more information):

wiley.a.com - - [25/Mar/1999:12:55:26 -0800] "GET /index.htm HTTP/1.0" "GET" "/?-" "HTTP/   1.0" 304 0 - Mozilla/2.0 (WinNT; I) wiley.a.com - - [25/Mar/1999:12:55:26 -0800] "GET / HTTP/1.0" "GET" "/?-" "HTTP/1.0" 304 0    - Mozilla/2.0 (WinNT; I) wiley.a.com - - [25/Mar/1999:12:55:26 -0800] "GET / HTTP/1.0" "GET" "/?-" "HTTP/1.0" 304 0    - Mozilla/2.0 (X11; I; IRIX 5.3 IP22)

Viewing the Error Log File

---

The error log file contains errors the server has encountered since the log file was created; it also contains informational messages about the server, such as when the server was started. Unsuccessful user authentication is also recorded in the error log. Use the error log to find broken URL paths or missing files.

To view the Administration Server's error log file, from the Administration Server, choose the Preferences tab, and choose The View Error Log Page (Administration Server).

To view a server's error log file, from the Server Manager, choose the Status tab, and choose The View Error Log Page.

The following is an example of an error log for Unix/Linux:

[13/Feb/1999:16:56:51] info: successful server startup [20/Mar/1999 19:08:52] warning: for host wiley.a.com trying to GET /report.html,     append-trailer reports: error opening /usr/netscape/server4/docs/report.html (No such file or directory)

The following is an example of an error log for Windows NT:

[13/Feb/1996:16:56:51] info: successful server startup [20/Mar/1996 19:08:52] warning: for host wiley.a.com trying to GET /report.html,     append-trailer reports: error opening C:/Netscape/Server4/docs/report.html     (ERROR_FILE_NOT_FOUND)

In these examples, the first line is an informational message—the server started up successfully. The second log entry shows that the client wiley.a.com requested the file report.html, but the file wasn't in the primary document directory on the server.

Monitoring the Server Using HTTP

---

You can monitor your server's usage with the interactive server monitor. You can see how many requests your server is handling and how well it is handling these requests. If the interactive server monitor reports that the server is handling a great number of requests, you may need to adjust the server configuration or the system's network kernel to accommodate the requests. The interactive server monitor is shown in Figure 7-1.

For a description of the various server statistics for which the interactive server monitor reports the totals, see The Monitor Current Activity Page in the online help.

To monitor your server, use launch the monitoring program from the Monitor Current Activity page.

Figure 7-1    The Interactive Server Monitor

Archiving Log Files

---

You can set up your log files to be automatically archived. At a certain time, or after a specified interval, your logs will be rotated. iPlanet Web Server saves the old log files and stamps the saved file with a name that includes the date and time they were saved. For example, you can set up your access log files to rotate every hour, and iPlanet Web Server saves and names the file "access.199907152400," where name of the log file, year, month, day, and 24-hour time is concatenated together into a single character string. The exact format of the access log archive file varies depending upon which type of log rotation you set up.

iPlanet Web Server offers the two types of log rotation for archiving files: Internal-daemon log rotation and Cron-based log rotation.

Internal-daemon Log Rotation

This type of log rotation happens within the HTTP daemon, and can only be configured at startup time. Internal daemon log rotation allows the server to rotate logs internally without requiring a server restart. Logs rotated using this method are saved in the following format:

access.<4 digit year><2 digit month><2 digit day><4 digit 24-hour time>

You can specify the time used as a basis to rotate log files and start a new log file. For example, if the rotation start time is 12:00 a.m., and the rotation interval is 1440 minutes (one day), a new log file will be created immediately upon save regardless of the present time and collect information until the rotation start time. The log file will rotate every day at 12:00 a.m., and the access log will be stamped at 12:00 a.m. and saved as access.199907152400. Likewise, if you set the interval at 240 minutes (4 hours), the 4 hour intervals begin at 12:00 a.m. such that the access log files will contain information gathered from 12:00 a.m. to 4:00 a.m., from 4:00 a.m. to 8:00 a.m., and so forth.

If access log rotation is enabled, log file rotation starts at server startup. The first access log file to be rotated gathers information from the current time until the next rotation time. Using the previous example, if you set your start time at 12:00 a.m. and your rotation interval at 240 minutes, and the current time is 6:00 a.m., the first log file to be rotated will contain the information gathered from 6:00 a.m. to 8:00 a.m, and the next log file will contain information from 8:00 a.m. to 12:00 p.m. (noon), and so forth.

Cron-based Log Rotation

This type of log rotation is based on the time stored in the cron.conf file in the server_root/https-admserv/config/ directory. This method allows you to archive log files immediately or have the server archive log files at a specific time on specific days. The server's cron configuration options are stored in ns-cron.conf in the server_root/https-admserv/config/ directory. Logs rotated using the cron based method are saved as the original filename followed by the date and time the file was rotated. For example, access might become access.24Apr-0430PM when it is rotates at 4:30 p.m. For more information about cron controls, see Using Cron Controls (Unix/Linux).

Log rotation is initialized at server startup. If rotation is turned on, iPlanet Web Server creates a time-stamped access log file and rotation starts at server startup.

Once the rotation starts, iPlanet Web Server creates a new time stamped log file when there is a request that needs to be logged to the access log file and it occurs after the prior-scheduled "next rotate time".

---

Note	You should archive the server logs before running the log analyzer.

---

To archive log files and to specify whether to use the Internal daemon method or the cron based method, use The Archive Log Files Page in the Server Manager.

Setting Log Preferences

---

During installation, an access log file named access was created for the server. You can customize access logging for any resource by specifying whether to log accesses, what format to use for logging, and whether the server should spend time looking up the domain names of clients when they access a resource.

Server access logs can be in Common Logfile Format or flexible log formator your own customizable format. The Common Logfile Format is a commonly supported format that provides a fixed amount of information about the server. The flexible log format allows you to choose (from iPlanet Web Server) what to log. A customizable format uses parameter blocks that you specify to control what gets logged. For a list of customizable format parameters, see the NSAPI Programmer's Guide for iPlanet Web Server.

Once an access log for a resource has been created, you cannot change its format unless you archive it or create a new access log file for the resource.

You can specify logging preferences using The Log Preferences Page in the Server Manager, or you can manually configure the following directives in the obj.conf file. In obj.conf, the server calls the function flex-init to initializes the flexible logging system and the function flex-log to record request-specific data in a flexible log format. To log requests using the common log file format, the server calls init-clf to initialize the Common Log subsystem which is used if obj.conf, and common-log to record request-specific data in the common log format (used by most HTTP servers).

For more information on the NSAPI logging functions, including valid directives and parameters, see the NSAPI Programmer's Guide for iPlanet Web Server.

Easy Cookie Logging

In previous versions of iPlanet Web Server, if you want to log the value of a specific cookie, you had to write a plugin API that extracted the cookie's value out of the "Cookie" header sent by the client, insert this value as a new variable to the request's pblock, and log that new variable.

iPlanet Web Server has an easy way to log a specific cookie using the flexlog facility. Add "Req->headers.cookie.cookie_name" to the line that initializes the felxlog subsystem in the configuration file obj.conf. This logs the value of the cookie variable cookie_name if the cookie variable is present in the request's headers, and logs "-" if it is not present.

Relaxed Logging

There is an unpleasant side effect to logging a variable other than the following standard variables: Status, Content-Length, Client-Host, Full-Request, Method, Protocol, Query-String, URI, Referer, User-Agent, Authorization, and Auth-User. Because other variables cannot be provided by the static file accelerator cache, the accelerator cache will not be used at all. Therefore performance numbers will decrease significantly for requests that would typically benefit from the accelerator, such as static files and images.

iPlanet Web Server eases the requirements of the log subsystem. Adding "relaxed.logname=true" to the "flex-init" line in obj.conf allows you to log variables outside of the standard set and still use the accelerator cache. If the accelerator is used, unavailable variables are logged as "-". The server does not use the accelerator for dynamic content such as CGI scrips or SHTML pages, so all the variables are always logged correctly for these requests.

Flushing the Log Buffer

---

You can flush the log buffer instantaneously or at a schedule other than the default time by setting buffer-flush in the logbufInit function in obj.conf. The value should be in milliseconds and greater that 0. For example:

int logbuf_init(pblock *args) {    char err[MAGNUS_ERROR_LEN];    char *t;

logbuf_size = 0;    logbuf_flush = DEFAULT_LOGBUF_FLUSH;    if((t = pblock_findval("buffer-size", args)) ) {       logbuf_size = atoi(t);       if(logbuf_size < 0) {          util_snprintf(err, MAGNUS_ERROR_LEN, "buffer size of %d is invalid", logbuf_size);          pblock_nvinsert("error", err, args);          return REQ_ABORTED;       }       if(logbuf_size < (REQ_MAX_LINE * 2))          logbuf_size = REQ_MAX_LINE * 2;       }       if( (t = ("buffer-flush", args)) ) {          logbuf_flush = atoi(t) * 1000;          if(logbuf_flush < 0) {             util_snprintf(err, MAGNUS_ERROR_LEN,"flush rate of %d seconds is invalid", logbuf_flush);             pblock_nvinsert("error", err, args);             return REQ_ABORTED;          }       }    }    return REQ_PROCEED; }

Running the Log Analyzer

---

The server-install/extras/log_anly directory contains the log analysis tool that runs through the Server Manager. This log analyzer analyzes files in common log format only. The HTML document in the log_anly directory that explains the tool's parameters. The server-install/extras/flex_anlg directory contains the command-line log analyzer for the flexible log file format. However, Server Manager defaults to using the flexible log file reporting tool, regardless of whether you've selected common or flexible log file format.

Use the log analyzer to generate statistics about your server, such as a summary of activity, most commonly accessed URLs, times during the day when the server is accessed most frequently, and so on. You can run the log analyzer from iPlanet Web Server or the command line.

---

Note	Before running the log analyzer, you should archive the server logs. For more information about archiving server logs, see Archiving Log Files.

---

The following section contains instructions for running the log analyzer from the command line. To run the log analyzer from the Server Manager, see The Generate Report Page in the online help.

To analyze access log files from the command line, run the tool, flexanlg, which is in the directory server-install/extras/flex_anlg.

To run flexanlg, type the following command and options at the command prompt:

flexanlg [ -P ] [-n name] [-x] [-r] [-p order] [-i file]* [ -m metafile ]* [ o file][ c opts] [-t opts] [-l opts]

The following describes the syntax. (You can get this information online by typing Using Performance Monitor (Windows NT)

flexanlg -h.): -P: proxy log format Default: no -n servername: The name of the server -x : Output in HTML Default: no -r : Resolve IP addresses to hostnames Default: no -p [c,t,l]: Output order (counts, time stats, lists) Default: ctl -i filename: Input log file(s) Default: none -o filename: Output log file Default: stdout -m filename: Meta file(s) Default: none -c [h,n,r,f,e,u,o,k,c,z]: Count these item(s) - Default: hnreuokc     h: total hits     n: 304 Not Modified status codes (Use Local Copy)     r: 302 Found status codes (Redirects)     f: 404 Not Found status codes (Document Not Found)     e: 500 Server Error status codes (Misconfiguration)     u: total unique URL's     o: total unique hosts     k: total kilobytes transferred     c: total kilobytes saved by caches     z: Do not count any items. -t [sx,mx,hx, xx,z]: Find general stats - Default:s5m5h24x10     s(number): Find top (number) seconds of log     m(number): Find top (number) minutes of log     h(number): Find top (number) hours of log     u(number): Find top (number) users of log     a(number): Find top (number) user agents of log     r(number): Find top (number) referers of log     x(number): Find top (number) for miscellaneous keywords     z: Do not find any general stats. -l [cx,hx]: Make a list of - Default: c+3h5     c(x,+x): Most commonly accessed URLs              (x: Only list x entries)              (+x: Only list if accessed more than x times)     h(x,+x): Hosts (or IP addresses) most often accessing your server              (x: Only list x entries)              (+x: Only list if accessed more than x times)     z: Do not make any lists

You can also monitor your server by using the Windows NT Performance Monitor, which graphically shows information about your computer's performance. Use Performance Monitor to see performance data about iPlanet Web Server.

To monitor iPlanet Web Server performance using Performance Monitor:

1. From the Start menu, select Programs and then Administrative Tools. Choose Performance Monitor in the Administrative Tools program group.

2. Choose Add to Chart from the Edit menu.

   The Add to Chart window appears.

3. If the iPlanet Web Server you want to monitor is on a remote system, type its name in the Computer field.

4. Choose Netscape Server from the Object pull-down menu.

5. Choose the instance you want to monitor.

   If you have multiple servers installed, you can choose multiple instances.

6. Choose the counters you want to see in your chart.

   The following counters are available:

   • Server Conn/sec—Rate of incoming connections per second.

   • Server Throughput (Kb/sec)—Rate of outgoing data from the server.

   • Server Total Bytes—Total bytes sent by the server.

   • Server Total Errors—Number of errors requests handled by the server.

   • Server Total Requests—Total requests handled by the server.

   • Status: 403 Forbidden—Number of "Forbidden" requests.

   • Status: 200 level—Number of 200-level status requests handled by the server.

   • Status: 200 OK—Number of "OK" requests.

   • Status: 300 level—Number of 300-level status requests handled by the server.

   • Status: 302 Moved Temporarily—Number of "Moved Temporarily" requests.

   • Status: 304 Not Modified—Number of "Not modified" requests.

   • Status: 400 level—Number of 400-level status requests handled by the server.

   • Status: 401 Unauthorized—Number of "Unauthorized" requests.

   • Status: 500 level—Number of 500-level status requests handled by the server.

   To see the counter definition online, click Explain.

7. Click Add.

8. To monitor other computers or objects, repeat steps 1 through 7 for each item you want to monitor.

9. Click Done.

   The Performance Monitor displays a chart with your selected items. A legend at the bottom of the page displays your choices.

For more information about Performance Monitor, see the documentation for your operating system.

Viewing Events (Windows NT)

---

In addition to logging errors to the server error log (see Viewing the Error Log File), iPlanet Web Server logs severe system errors to the Event Viewer. The Event Viewer lets you monitor events on your system. Use the Event Viewer to see errors resulting from fundamental configuration problems, which can occur before the error log can be opened.

To use the Event Viewer:

1. From the Start menu, select Programs and then Administrative Tools. Choose Event Viewer in the Administrative Tools program group.

2. Choose Application from the Log menu.

   The Application log appears in the Event Viewer. Errors from iPlanet Web Server has a source label of https-serverid or WebServer4.1.

3. Choose Find from the View menu to search for one of these labels in the log. Choose Refresh from the View menu to see updated log entries.

For more information about Event Viewer, consult your system documentation.