1. Monitoring and Tuning the System
  2. Configuring and Using Auditing

6 Configuring and Using Auditing

Auditing collects data at the kernel level that you can then analyze to identify unauthorized activity. Auditing collects data in greater detail than system logging does. The process of examining audit trails to find events of interest can be challenging. Therefore, you might want to consider automating this process.

Some definitions in the audit configuration file, /etc/audit/auditd.conf, include the following:

  • Data retention policy

  • Maximum size of the audit volume

  • Action to take if the capacity of the audit volume is exceeded

  • Locations of local and remote audit trail volumes

The default audit trail volume is /var/log/audit/audit.log. See the auditd.conf(5) manual page for more information.

By default, auditing captures specific events such as system logins, modifications to accounts, and sudo actions. You can configure auditing to capture detailed system call activity or modifications to certain files. The kernel audit daemon (auditd) records the events that you configure, including the event type, a timestamp, the associated user ID, and success or failure of the system call.

The entries in the audit rules file, /etc/audit/audit.rules, decides which events are audited. Each rule is a command line option that's passed to the auditctl command. Configure this file to match organization's security policy.

The following are examples of rules that you might set in the /etc/audit/audit.rules file:

  • Record all unsuccessful exits from open and truncate system calls for files and store the information in the /etc directory hierarchy. 

    -a exit,always -S open -S truncate -F /etc -F success=0

  • Record all files opened by a user with UID 10.

    -a exit,always -S open -F uid=10

  • Record all files that have been revised or whose attributes were changed by any user who originally signed in with a UID of 500 or greater. 

    -a exit,always -S open -F auid>=500 -F perm=wa

  • Record requests for write or for file attribute change access. Store the records in the /etc/sudoers file and tag such a record with the string sudoers-change. 

    -w /etc/sudoers -p wa -k sudoers-change

  • Record requests for write and for file attribute change access and store records in the /etc directory hierarchy. 

    -w /etc/ -p wa

  • Require a reboot after changing the audit configuration. 

    -e 2

    Note:

    We recommend that you define rules to reboot at the end of the /etc/audit/audit.rules file.

For more examples of audit rules, see also the auditctl(8) and audit.rules(7) manual pages.

Stringent auditing requirements generate large amounts of audit data and can impose a significant performance overhead. Some site security policies stipulate that a system must shut down if events can't be recorded because the audit volumes have exceeded their capacity. We recommend that you direct audit data to separate file systems in rotation to prevent overspill and to ease backups.

If you tag audit records, then searching an audit volume with the ausearch command becomes easier by referring to those tags. For example, to examine records that are tagged with the string sudoers-change, you would type: 

sudo ausearch -k sudoers-change

The aureport command generates summaries of audit data. For example, the following command generates a report that shows every sign-in event from 1 second after midnight on the previous day until the current time: 

sudo aureport -l -i -ts yesterday -te now

You can set up cron jobs or systemd timers that run aureport periodically to generate reports of interest. See Use the Crontab Utility to Schedule Tasks on Oracle Linux and Use systemd on Oracle Linux.

See the ausearch(8) and aureport(8) manual pages for more information.

For a hands-on tutorial on using the auditing tools on Oracle Linux, see Audit Oracle Linux with Auditd.

Working With System Log files

The log files contain messages about the system, kernel, services, and applications. The journald logging daemon, which is part of systemd, records system messages in nonpersistent journal files in memory. The journald daemon forwards messages to the system logging daemon, rsyslog.

Files in /run are volatile. Thus, the log data is lost after a reboot unless you create the directory /var/log/journal. You can use the journalctl command to query the journal logs.

For more information, see the journalctl(1) and systemd-journald.service(8) manual pages.

For a hands-on tutorial introducing system logging tools, see System Logging on Oracle Linux.

About Logging Configuration (/etc/rsyslog.conf)

The /etc/rsyslog.conf file is the configuration file for the rsyslogd daemon. The file contains global directives, module directives, and rules for the daemon or service. By default, rsyslog processes and archives only syslog messages. However, if required, you can configure rsyslog to archive any other messages that journald forwards, including kernel, boot, initrd, stdout, and stderr messages.

To obtain the latest information about rsyslog, refer to https://www.rsyslog.com/doc/. Alternatively, you can install the rsyslog-doc package locally. 

sudo dnf install -y rsyslog-doc

The documentation is installed in /usr/share/doc/rsyslog/html.

Important:

The format to configure parameters in /etc/rsyslog.conf has changed. The following formats are supported which enables backward compatibility with previous configuration:

  • Basic or sysklogd format, which has been used since the beginning of system logging.
  • Legacy format, where directives are defined on their own specific lines in the file, with each directive being preceded by the dollar ($) sign, such as $MainMsgQueueSize.
  • Advanced format, which uses the RainerScript scripting language for configuring rsyslog.

For more information about these formats, see the relevant sections in https://www.rsyslog.com/doc/.

The /etc/rsyslog.conf file is divided into the following main sections:

Modules

Modules contain configuration parameters for processing messages. The processed or transformed messages can then be transmitted to various targets as required. Modules are classified into different categories, such as output, input, parser, library, and so on. For a complete list of these module classes, see the appropriate section in https://www.rsyslog.com/doc/. For a list of the modules, see the rsyslog.conf(5) manual page.

Modules enable different rsyslog functionalities to become operative, provided that those modules are loaded. Modules are loaded through the module load directive as follows: 

module(load="module-name")

Note:

The directive uses the advanced format for loading a module and replaces the $ModLoad module-name legacy format.

Global directives

Global directives specify configuration options that apply to the rsyslogd daemon. A directive might specify the location of auxiliary files. A directive can also be a module(load" ") statement that applies global settings, such as the timestamp format to use for all messages, as shown in the following example: 

module(load="builtin:omfile" Template=RSYSLOG_TraditionalFileFormat")

Because the module applies to all messages, the directive is specified under the Global Directives section.

Rules

Rules or rule sets determine how logged messages are managed.

A rule consists of two fields: a selector field and an action field. The two fields are separated by one more spaces or tabs.

  • The selector field has two parts, separated by a period: a facility keyword and a priority keyword. Facility keywords include auth, authpriv, cron, daemon, kern, and so on. Priority keywords include debug, info, notice, warning, and so on. Thus, kern.* selects kernel messages of all priority levels, while kern.emerg selects emergency kernel messages only.

    For a list of both facility and priority selectors, see the rsyslog.conf(5) manual page.

  • The action field typically indicates to which log file the message content is written. For example, the following rule indicates that cron messages are to be logged in /var/log/cron: 

    cron.*   /var/log/cron

You can customize rsyslog configuration in two ways:

  • Directly change the /etc/rsyslog.conf file itself.
  • Create a configuration file and store it in /etc/rsyslog.d. You might choose this option to prevent custom configurations from being overwritten in case system packages are updated.

Some changes are simple to implement on the /etc/rsyslog.conf file, such as changing the log for a specific selector. For example, to change the log for cron messages to cron_new, you would enter the following: 

cron.*     /var/log/cron_new

You then restart the rsyslog service for the change to take effect.

Other changes might require more parameter definitions and steps.

Suppose that you want to create a rule that would use TCP to forward messages to another server where the messages are logged. The following steps show you how to implement this sample rule.

  1. Create a separate file, for example, /etc/rsyslog.d/forwarding, where you would set the parameters for TCP forwarding, similar to the following: 

    *.* action(type="omfwd"
      queue.filename="fwdRule1"
      queue.maxdiskspace="1g"
      queue.saveOnShutdown="on"
      queue.type="linkedlist"
      action.resumeRetryCount="-1"
      target="remote-host.com" port="30514" protocol="tcp"
     )
    queue.filename

    Prefix to be attached to the backup files. The prefixed backup files are created in the location as specified by the workDir global directive, for example, global(workDirectory="/var/log").

    queue.maxdiskspace

    Space limit for log files.

    queue.saveOnShutdown

    Saves data in memory if rsyslog shuts down.

    queue.type

    Enables a LinkedList in-memory queue.

    action.resumeRetryCount

    A setting of -1 means to retry indefinitely if the host is unavailable.

    target

    Can be a host name or an IP address.

    Based on the sample configuration, rsyslog forwards messages to the remote server remote-host.com. The rsyslog service also keeps the message in memory in case the remote server is unavailable. If rsyslog shuts down or has exhausted allotted memory, then rsyslog creates files on disk with the appropriate prefix to the file names.

  2. Open the /etc/rsyslog.conf and verify the following:

    • Ensure that the module for TCP syslog reception is loaded. Verify that the comment marks are removed from the following lines:

      module(load="imtcp")
input(type="imtcp" port="514")

    • Ensure that the global directive to include /etc/rsyslog.d files in rsyslog configuration is enabled. Verify that the following line is not commented out: 

      include(file="/etc/rsyslog.d/*.conf" mode="optional")

  3. Save the file and exit.

  4. Restart the rsyslog service. 

    systemctl restart rsyslog

To manage the rotation and archival of the correct logs, edit /etc/logrotate.d/syslog so that it references each of the log files that are defined in the RULES section of /etc/rsyslog.conf. You can configure how often the logs are rotated and how many past copies of the logs are archived by editing /etc/logrotate.conf.

For more information, see the logrotate(8), logwatch(8), rsyslogd(8) and rsyslog.conf(5) manual pages.

Configuring Logwatch

Logwatch is a monitoring system that you can configure to report on areas of interest in the system logs. After you install the logwatch package, the /etc/cron.daily/0logwatch script runs every night and sends an email report to root. You can set local configuration options in /etc/logwatch/conf/logwatch.conf that override the main configuration file /usr/share/logwatch/default.conf/logwatch.conf, including the following:

  • Log files to monitor, including log files that are stored for other hosts.

  • Names of services to monitor, or to be excluded from monitoring.

  • Level of detail to report.

  • User to be sent an emailed report.

As best practice, configure Logwatch on a log server to monitor the logs for suspicious messages, and disable Logwatch on log clients. However, if you do use Logwatch, disable high precision timestamps by adding the following entry to the GLOBAL DIRECTIVES section of /etc/rsyslog.conf on each system: 

module(load="builtin:omfile" Template=RSYSLOG_TraditionalFileFormat")

You can also run logwatch directly from the command line.

For more information, see the logwatch(8) manual page.

Using Process Accounting

The psacct package implements the process accounting service in addition to the following utilities that you can use to monitor process activities:
ac

Displays connection times in hours for a user as recorded in the wtmp file (by default, /var/log/wtmp).

accton

Turns on process accounting to the specified file. If you don't specify a file name argument, process accounting is stopped. The default system accounting file is /var/account/pacct.

lastcomm

Displays information about command history as recorded in the system accounting file.

sa

Summarizes information about command history as recorded in the system accounting file.

Note:

As for any logging activity, ensure that the file system has enough space to store the system accounting and wtmp files. Monitor the size of the files and truncate them as needed.

For more information, see the ac(1), accton(8), lastcomm(1), and sa(8) manual pages.