Chapter 21   logconv.pl


The logconv.pl (log converter) tool is a Perl script that analyzes the access logs of an iPlanet Directory Server to extract usage statistics and count the occurrences of significant events. It is compatible with log formats from iPlanet Directory Server 3.x , 4.x , and 5.x .

The logconv.pl script requires Perl version 5.005_03 or later. See "Where to Find Additional Information" for links to Perl resources.

If you customize the logconv.pl script for added functionality, we encourage you to share your work with other LDAP users. Please post a message to the iplanet.server.idsrk public newsgroup with your ideas or your code.

This chapter contains the following sections:

• Command Usage

• Sample Output

Command Usage

---

The logconv.pl tool displays three types of statistics that administrators will find useful for monitoring and optimizing directory usage:

• Simple counts of events such as the total number of binds and the total number of searches.

• Lists of the most frequently occurring parameters in LDAP requests, for example, lists of the top ten bind DNs, base DNs, filter strings, and attributes returned. These lists are optional because they are computation intensive: specify only the command-line options for those you need.

• A count of occurrences of any given error code, along with a listing of the corresponding error messages in the log file.

The tool will extract the following information from access logs:

Number of restarts Total number of connections Total operations requested Total results returned Results to requests ratio Number of searches Number of modifications Number of adds Number of deletes Number of modified RDNs For iPlanet DS 5.x logs only: Persistent searches Internal operations (with verbose logs) Entry operations (with verbose logs) Extended operations Abandoned requests Smart referrals received (verbose logs) VLV (virtual list view) operations VLV unindexed searches Server-side sorting operations SSL connections Performance lowering operations: Entire database searches Unindexed searches (details optional)

FDs (file descriptors) taken FDs returned Highest FD taken Disruptions: Broken pipes Connections reset by peer Unavailable resources (and detail) Total binds, types of binds, expired password logins, and failed binds Most frequent occurrence lists (optional): Error and return codes Failed logins Connection codes Client IP addresses and connection codes Bind DNs Base DNs for searching Search filters Etimes (elapsed operation time) Longest etimes Nentries (number of entries in result) Largest Nentries Extended operations (DS 5.x only) Most requested attributes (DS 5.x only) Abandoned operation details (DS 4.15) Recommendations (optional)

Some information that is extracted by the logconv.pl script is available only in iPlanet Directory Server 5.x logs: the corresponding values will be zero when analyzing logs from other versions. In addition, some information will only be present in the logs if verbose logging is enabled in your directory server. For more information, see "nsslapd-accesslog-level" in the iPlanet Directory Server Command, Configuration and File Reference.

The following issues will affect the output and performance of this tool:

• Some data extracted from logs depend on connection and operation numbers that are reset and no longer unique after a server restarts. Therefore, to obtain the most accurate counts, the logs to be analyzed should not span the restart of the directory server.

• Due to changes in access logs formats in Directory Server 5.0 that also affect operation numbers, the tool will be more accurate on 5.x logs when processing large amounts of access logs.

• For performance reasons, it is not recommended to run more than one gigabyte of access logs through the script at any one time.

The logconv.pl script is supported when using Perl version 5.005_03. See "Where to Find Additional Information" for links to Perl resources.

If you customize the ldiffer.pl script for added functionality, we encourage you to share your work with other LDAP users. Please post a message to the iplanet.server.idsrk public newsgroup with your ideas or your code.

Syntax

The logconv.pl tool has the following syntax:

logconv.pl [options ] [-efcibaltnxgju | -E errorCode ] accessLog ...

Where:

• options and [-efcibaltnxgju] are the command-line options described in the next section.

• errorCode will display a report only on occurrences of the given error number.

• accessLog is the name of a file that contains the access log of your iPlanet Directory Server. You may use wildcards in the filename or specify multiple filenames. However, the statistics are computed over the set of all logs, so all logs should pertain to the same directory server. The tool will ignore any file with the name access.rotationinfo.

The logconv.pl -h command will display the usage help text that briefly describes all options.

Options

The logconv.pl command-line options are described in the following table.

The parameters without a preceding dash (-) at the end of the table will enable the optional lists of occurrences. Specify only those you need to limit the output and improve execution speed. You may specify any number of these parameters in any order, but they must all be given together as a single option on the command line, for example: -abcefg.

Regardless of the order of options on the command line, the lists will appear in the output in the order they are listed in this table. Use the -V option to display all optional output. Also, use the -s number option to control the length of these lists.

Table 21-1    Command-Line Options for the logconv.pl Script 

Option	Parameter	Purpose
-d	mgrDN	Specify the DN (distinguished name) of the directory manger in the logs being analyzed. This allows the tool to collect statistics for this special user. The mgrDN parameter should be given in double quotes ("") for the shell. When this parameter is omitted, logconv.pl will use the default manager DN of iPlanet Directory Server: "cn=directory manager".
-N		Enable DNS lookup on IP addresses found in the log file so that machine names appear in the output instead.
-X	IPaddress	Specify the IP address of a client to exclude from the statistics. This client will not appear in lists of IP addresses (the i flag), and the connection codes it generates will not be tallied in the total connections (default statistic) nor in the connection code details (the c flag). For example, you may wish to ignore the effect of a load balancer that connects to the directory server a regular intervals. This option may be repeated to exclude multiple IP addresses.
-v		Display the version number of the logconv.pl script.
-h		Display the usage help text that briefly describes all options.
-E	errorCode	Display only the information about the given, numeric error code. When using this option, the tool will not display the summary of all log information, it will only give the count and the list of the given error's occurrences. This option is incompatible with all of the following options.
-s	number	Specify the number of items in each of the list options below. The default is 20 when this parameter is omitted. For example, -s 10 -i will list the ten client machines that access the server most often. This parameter applies to all lists that are enabled.
-V		Enable the most verbose output. With this option, logconv.pl will compute and display all of the optional lists described below.
e		List the most frequent error and return codes.
f		List the bind DNs with the most failed logins (invalid password).
c		List the number of occurrences for each type of connection code.
i		List the IP addresses and connection codes of the clients with the most connections. This option helps to detect clients that may be trying to compromise security.
b		List the most frequently used bind DNs.
a		List the most frequent base DNs when performing operations.
l		List the most frequently used filter strings for searches.
t		List the longest and most frequent etimes (elapsed operation time).
n		List the largest and most frequent nentries (entries per result).
x		List the number and OID of all extended operations (DS 5.x only).
r		List the names of the most requested attributes (DS 5.x only).
g		List the details of all abandoned operations.
j		Give recommendations based on data collected from the log file.
u		Give operation details about unindexed searches.

Sample Output

---

The following example shows how error code listings can help track security issues. The output shows the listing of a specific error code (-E 49), corresponding to a failed login with a bad password. The listing contains machine names (-N) instead of IP addresses to make it more readable.

$ perl logconv.pl -N -E 49 logs/access

...

Error (49) Count: 5
Parsing & Sorting...

Number  Operation  Client                  Object
===================================================================
3          BIND    (test.siroe.com)        cn=Directory Manager
1          BIND    (localhost)             cn=Directory Manager
1          BIND    (test.siroe.com)        uid=bjensen

The number in the first column is the number of repeated occurrences of the same error message.

The following example shows the verbose output (-V) of the logconv.pl tool. It will read all access logs in the logs directory, ignoring access.rotationinfo files. After processing the log files, it displays all of the access statistics and event counters. Then it shows all of the lists of most frequent connection and operation values, with the top 10 in each category (-s 10). It ends with a set of general recommendations triggered by certain values or events.

$ perl logconv.pl -V -s 10 \
                  /usr/iplanet/servers/slapd-serverID /logs/access*

verbose output enabled

Log Analyzer 4.11

Initializing Variables...

Processing 3 Access Log(s)...

access (Total Lines: 5870)
        1000 Lines Processed
        2000 Lines Processed
        3000 Lines Processed
        4000 Lines Processed
        5000 Lines Processed
*       5870 Lines Processed      Total Lines Processed:  5870

access.20010713-130613 (Total Lines: 7912)
        1000 Lines Processed
        2000 Lines Processed
        3000 Lines Processed
        4000 Lines Processed
        5000 Lines Processed
        6000 Lines Processed
        7000 Lines Processed
*       7912 Lines Processed      Total Lines Processed: 13782

access.20010714-150617 (Total Lines: 6338)
        1000 Lines Processed
        2000 Lines Processed
        3000 Lines Processed
        4000 Lines Processed
        5000 Lines Processed
        6000 Lines Processed
*       6338 Lines Processed      Total Lines Processed: 20120

* Total Lines Analyzed: 20120

----------- Access Log Output ------------

Start of Log:  18/Jul/2001:13:08:18
End of Log:    18/Jul/2001:17:05:07

Restarts: 1

Total Connections: 4002
Total Operations: 14818
Total Results: 14908
Overall Performance: 100.6%

Searches: 4354
Modifications: 27
Adds: 26
Deletes: 30
Mod RDNs: 0

5.x Stats
Persistent Searches: 1
Internal Operations: 0
Entry Operations: 0
Extended Operations: 6935
Abandoned Requests: 29
Smart Referrals Received: 0

VLV Operations: 49
VLV Unindexed Searches: 49
SORT Operations: 44
SSL Connections: 0

Entire Search Base Queries: 3912
Unindexed Searches: 1

  Unindexed Search #1
  - Date/Time: 18/Jul/2001:13:33:19
  - Connection Number: 2926
  - Operation Number: 1
  - Etime: 0
  - Nentries: 4001
  - IP Address: 192.18.122.229
  - Bind DN: cn=directory manager
  - Search Filter: (objectclass=*)

FDs Taken: 3448
FDs Returned: 3446
Highest FD Taken: 89

Broken Pipes: 0
Connections Reset By Peer: 0
Resource Unavailable: 1
  - 1 (T1) Idle Timeout Exceeded

Binds: 3446
Unbinds: 3438

 LDAP v2 Binds: 1
 LDAP v3 Binds: 3445
 Expired Password Logins:     0
 SSL Client Binds: 0
 Failed SSL Client Binds: 0
 SASL Binds: 1
  1 DIGEST-MD5

 Directory Manager Binds: 16
 Anonymous Binds: 1
 Other Binds: 3429

----- Errors -----

err=0 14737 Successful Operations
err=32 75 No Such Object
err=12 62 Unavailable Critical Extension
err=10 3 Referral Received
err=49 1 Invalid Credentials (Bad Password)
err=65 1 Objectclass Violation

----- Top 10 Failed Logins ------

1 uid=rmanager,cn=config

----- Total Connection Codes -----

U1 3437 Cleanly Closed Connections
B1 8 Bad Ber Tag Encountered
T1 1 Idle Timeout Exceeded

----- Top 10 Clients -----

Number of Clients: 2

3440 123.456.789.001
                  3429 - U1 Cleanly Closed Connections
                     8 - B1 Bad Ber Tag Encountered
                     1 - T1 Idle Timeout Exceeded

8 127.0.0.1
                     8 - U1 Cleanly Closed Connections

----- Top 10 Bind DN's -----

Number of Unique Bind DN's: 8

3422 uid=rmanager,cn=config
14 cn=dm
5 uid=aa,cn=config
1 Anonymous Binds
1 uid=rmanager
1 cn=dma,cn=config
1 dc=dm
1 cn=dma

----- Top 10 Search Bases -----

Number of Unique Search Bases: 73

3519            root dse
256             ou=people,dc=siroe,dc=com
82              cn=ldbm database, cn=plugins, cn=config
57              cn=monitor
51              dc=siroe,dc=com
48              cn=config
30              cn=mapping tree,cn=config
28              cn=Babs Jensen,ou=peopled,c=siroe,dc=com
22              cn=plugins,cn=config
20              cn=features,cn=config

----- Top 10 Search Filters -----

Number of Unique Search Filters: 31

3502 (objectclass=*)
408 (|(objectclass=*)(objectclass=ldapsubentry))
119 (uid=*)
88 (objectclass=nsbackendinstance)
6 (nsslapd-backend=userroot)
6 (nsslapd-plugintype=database)
4 (uid=bjensen)
4 (objectclass=subschema)
4 (objectclass=nsindex)
3 (cn=config)

----- Top 10 Most Frequent etimes -----

14634 etime=0
229 etime=1
9 etime=2
3 etime=7
1 etime=8
1 etime=3
1 etime=4
1 etime=5

----- Top 10 Longest etimes -----

etime=8 1
etime=7 3
etime=5 1
etime=4 1
etime=3 1
etime=2 9
etime=1 229
etime=0 14634

----- Top 10 Largest nentries -----

nentries=25 5
nentries=11 2
nentries=10 2
nentries=9 2
nentries=8 1
nentries=5 1
nentries=4 16
nentries=3 62
nentries=2 37
nentries=1 3986
nentries=0 239

----- Top 10 Most returned nentries -----

3986 nentries=1
239 nentries=0
62 nentries=3
37 nentries=2
16 nentries=4
5 nentries=25
2 nentries=10
2 nentries=11
2 nentries=9
1 nentries=5

----- 5.x Extended Operations -----

3454    2.16.840.1.113730.3.5.3    Start Replication Request
                                     (incremental update)

3438    2.16.840.1.113730.3.5.5    End Replication Request
                                   (incremental update)

43      2.16.840.1.113730.3.5.6    Replication Entry Request

----- Top 10 Most Requested Attributes -----

3420 supportedControl
3420 supportedExtension
360 All Attributes
341 numSubordinates
328 objectClass
315 nsAccountLock
144 nsBackendSuffix
104 nsslapd-suffix
36 dn
32 cn

----- Abandon Request Stats -----

 - SRCH conn=2 op=10 msgid=1092 client=127.0.0.1
 - BIND conn=2 op=0 msgid=1119 client=127.0.0.1

----- Recommendations -----

 1. You have unindexed searches, this can be caused from a search on a unindexed attribute, or your returned results exceeded the allidsthreshold. Unindexed searches are not acceptable, please make any configuration changes necessary to resolve these searches!

 2. You have some connections that are are being closed by the idletimeout setting. You may want to increase the idletimeout if it is set low.

 3. You have a high number of searches that query the entire search base. Although this is not necessarily bad, it could be resource intensive if the search base contains many entries.

There are many possible recommendations depending on the statistics and occurrences of certain events. The recommendations are based on general administration guidelines and should be adapted to fit the actual usage of your directory server.