Note:
- This tutorial is available in an Oracle-provided free lab environment.
- It uses example values for Oracle Cloud Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your cloud environment.
Work with system logging on Oracle Linux
Introduction
In this tutorial, you configure system logging, use rsyslog
templates to format log messages, install and run logwatch
, view the journald journal
, and configure persistent journald
storage.
Background
System log files contain messages about the system, kernel, services, and applications. The journald
logging daemon, which is part of systemd
, records system messages in non-persistent journal logs in memory and in the /run/log/journal
directory. journald
forwards messages to the system logging daemon, rsyslog
. As files in /run
are volatile, the log data is lost after a reboot unless you create the directory /var/log/journal
. You can use the journalctl
command to query the journal logs.
Objectives
- Explore the
rsylog.conf
file - Explore the
logrotate.conf
file - Configure and use
rsyslog
templates - Install
logwatch
and run thelogwatch
utility - Explore and use
journald
What Do You Need?
A fully patched Oracle Linux 8 or later system.
Note: When using the free lab environment, see Oracle Linux Lab Basics for connection and other usage instructions.
Explore and Modify the rsyslog.conf
File
Oracle Linux 8 relies on the service, rsyslogd
, to facilitate logging. This daemon permits the processing and forwarding of logs on your system. /etc/rsyslog.conf
file is the main configuration file for system logging.
Watch the video below for an overview on rsyslog
.
-
Use the
less
command to explore the system logging configuration file.less /etc/rsyslog.conf
Use the up-arrow and down-arrow keys to view the various sections of the file.
- The
MODULES
section uses the$ModLoad
directive to load the modules. - The
GLOBAL DIRECTIVES
section specifies configuration options. - The
RULES
section defines a facility (facility.priority
) and an action.
Press the
q
key to exit theless
command. - The
-
Use the
ls
command to list thecron*
files in the/var/log
directory.ls /var/log/cron*
You might not have any other log files at this time, or files with a date stamp.
-
Use the
vi
editor to changecron
logging to use a new log file:/var/log/cron_new
.vi /etc/rsyslog.conf
- Search for
cron stuff
in the file. You can do that by typing/
; followed by,cron stuff
.
cron stuff
- Enter insert mode by typing
i
. - On the next line, change
/var/log/cron
to/var/log/cron_new
, as shown:
/var/log/cron_new
- Save and close the file by typing
ESC
,:wq!
andENTER
.
- Search for
-
Use the
systemctl
command to restart thersyslog
service.systemctl restart rsyslog
-
Use the
crontab -e
command to create acron
job that runs thels
command every minute.crontab -e
- Enter insert mode by typing
i
. - Enter the following:
* * * * * ls
Type
ESC
,:wq!
andENTER
to save and close the file. - Enter insert mode by typing
-
Use the
ls
command to list thecron
files in the/var/log
directory.ls /var/log/cron*
Note: The command shows the new
cron
log file. -
Use the
tail
command to view the last entries in thecron
log file.tail -2 /var/log/cron
Notice from the date_time stamps of the log entries.
-
Use the
tail
command to view the lastest entries in thecron_new
log file.tail -2 /var/log/cron_new
Notice from the date_time stamps that the new log entries are being written to
cron_new
. -
Use the
vi
editor to changecron
logging back to/var/log/cron
.vi /etc/rsyslog.conf
- Enter the following:
/var/log/cron
-
Use the
systemctl
command to restart thersyslog
service.systemctl restart rsyslog
-
Use the
tail
command to ensure thatcron
is now logging events to/var/log/cron
. Compare the output from running the command on the following files:tail -2 /var/log/cron
tail -2 /var/log/cron_new
Notice that the date_time stamp for
cron
is a later time stamp thancron_new
. -
Use the
rm
command to delete thecron_new
log file.rm /var/log/cron_new
-
Use
crontab –r
to remove the crontab.crontab -r
-
Use the
vi
command to modify the system logging configuration file by adding an entry at the bottom of the file to log all debug messages to/var/log/debug
.vi /etc/rsyslog.conf
- Enter insert mode.
- Enter the following:
*.debug /var/log/debug
-
Run the command
systemctl
to restart thersyslog
service.systemctl restart rsyslog
-
Use the
logger
command to generate an informational log message. Thelogger
command logs enteries in the system log.logger -p info "This is an info-priority message"
The
logger
command is a quick way to add messages to the/var/log/syslog
file from the command line` -
Run the
tail
command to view the log files.tail /var/log/messages
tail /var/log/debug
The message appears in both log files.
-
Use the
logger
command to generate a debug-level log message.logger -p debug "This is an debug-priority message"
-
Run the
tail
command to view the log files.tail /var/log/messages
tail /var/log/debug
The message appears in only the
/var/log/debug
log file.
Explore and Modify Log Rotation
Oracle linux installs the logrotate
utility to rotate logs. logrotate
places a logrotate.conf
configuration file the /etc
directory at install time. The logwatch
configuration file contains the directives for the default rotation frequency. You can modify the frequency of the log rotation by editing this configuration file.
-
Use the
ls
command to view represtations of rotated logs in the/var/log
directory.ls /var/log/messages*
ls /var/log/maillog*
ls /var/log/cron*
- In a production system, files with numbers at the end of the file name represent rotated logs with time stamps added to the log file names.
- The rotated log files provide a history of events, and the files without time stamps provide a record of the most current events.
- The number of rotated files depend on the how long the system has been running.
-
Use the
vi
editor to modify the/etc/logrotate.conf
configuration file to change the frequency of the default log file rotation from weekly to daily.vi /etc/logrotate.conf
- Enter the following:
daily
-
You can create separate configuration files for any specific log file in the
/etc/logrotate.d
directory. These files can define any of the configuration options. These options override the global options in/etc/logrotate.conf
. Oracle Linux provides a few separate configuration files by default.Use the
cat
command to explore the options used in these files.cat /etc/logrotate.d/chrony
cat /etc/logrotate.d/syslog
cat /etc/logrotate.d/firewalld
The options in the
/etc/logrotate.d/
configuration files are described as follows:missingok
: If the log file is missing, do not issue an error message.nocreate
: New log files are not created.postrotate
/endscript
:The lines between these directives are executed after the log file is rotated.sharedscripts
:Thepostrotatescript
runs only once, not once for each log that is rotated.
Note: For a full list of directives and configuration options, refer to the
logrotate(8)
man page.
Use rsyslog
Templates
Templates allow you to specify and format rsyslogd
output the way a user might want. A template consists of a template directive, a name, the actual template text and optional options.
A sample syntax is:
$template TEMPLATE_NAME,"text %PROPERTY% text", [OPTION]
The fields are:
$template
: Directive that notifiesrsyslog
this line is a templateTEMPLATE_NAME
: Defines the name of the template"text"
: Actual template text surrounded by quotation marks%PROPERTY%
: Specific message content surrounded by percent signsOPTION
: Specifies options that modify the template functionality
-
Use the command
vi /etc/rsyslog.conf
to define a template.- Add the template definition line shown in the code box to the bottom of the file.
- This entry creates a template named
lab
. - Do not exit the
vi
editor.
vi /etc/rsyslog.conf
- Enter the following:
$template lab,"Message: %msg%\n"
-
Continue editing
/etc/rsyslog.conf
and create a log file that uses the template.-
Add a new line after the template definition line.
-
This entry writes all messages to the
/var/log/lab.log
file and formats the entries by using thelog
template. -
Exit the
vi
editor and save the file after adding the new line. -
Enter the following:
*.* /var/log/lab.log;lab
-
-
After saving the changes to
/etc/rsyslog.conf
, use thesystemctl
command to restart thersyslog
service.systemctl restart rsyslog
-
Use the
cat
command to view the/var/log/lab.log
file.cat /var/log/lab.log
The
lab.log
file contains all entries preceded by the text “Message:
” followed by the actual message, as defined in thelab
template. -
Use the
vi
editor and change thelab
template definition in/etc/rsyslog.conf
, as shown:vi /etc/rsyslog.conf
- Enter the following:
$template lab, "Time: %timestamp%, Facility: %syslogfacility-text%, Priority: %syslogpriority-text%, Hostname: %hostname%, Message: %msg%\n"
-
After saving the changes, restart the
rsyslog
service.systemctl restart rsyslog
-
Rerun the
cat
command and view the entries now inlab.log
.The newest entries now include the
Time
,Facility
,Priority
,Hostname
, andMessage
properties, as defined in the class template.
Install and Use logwatch
logwatch
is a customizable log monitoring system. It analyzes system logs for a given time period and reports on specific areas of interest.
It might be necessary to install the logwatch
packages. After installing it, logwatch
is configured by default to run each night as defined in /etc/cron.daily/0logwatch
and email a report to the root
user.
Watch the video below for an overview on logwatch
.
-
Run the command
dnf list logwatch
to verify its availability.dnf list logwatch
The output lists the
logwatch
packages asAvailable Pachages
only, which means you need to install the utiltiy. The packages are availabe in theol8_baseos_latest
repository located in the/etc/yum.repos.d/oracle-linux-ol8.repo
file. This repository is enabled by default. -
Run the command
dnf install logwatch -y
to install thelogwatch
packages.dnf install logwatch -y
-
Use the
find
command to list alllogwatch
files.find / -name "*logwatch*"
The output lists several files:
- The main configuration file is
/usr/share/logwatch/default.conf/logwatch.conf
. - Local configuration options can be set in
/etc/logwatch/conf/logwatch.conf
. - A
HOWTO-Customize-Logwatchfile
file exists in the/usr/share/doc/logwatch/
directory. This file describes the structure of the Logwatch files in this version, how to modify the configuration files for your system, and how to create new service filters.
- The main configuration file is
-
Use the
less
command to view thelogwatch
configuration file.less /usr/share/logwatch/default.conf/logwatch.conf
Review this file and note the various configuarable items, such as:
LogDir
: Is the default log directory, and all files are placed relative to this directoryTmpDir
: Is a temporary directory you define to override the default/tmp
directoryMailTo
: Is the user to mail the reports to.root
is the default recipient.MailFrom
: Is the default person to mail reports from.Logwatch
is the default sender.Range
: Is the default time range for the report.yesterday
is the defaultrange
.Detail
: Is the default detail level for the report.Detail
levels can be set to:Low
orO
,Med
or5
, orHigh
or10
.Low
is the default detail level.Service
: Is the default services to report on as defined in/usr/share/logwatch/scripts/services/
.All
services is the default. You can also disable certain services even when specifyingAll
.
-
Use the
less
command to view thelogwatch cron
file.less /etc/cron.daily/0logwatch
-
Run
logwatch --help
command to view thelogwatch
help.logwatch --help
-
Run
logwatch
with the--range
option and a date_range parameter oftoday
. This will process log data from the current day. Sample output is displayed.logwatch --range today
- Sample output:
################### Logwatch 7.4.3 (04/27/16) #################### Processing Initiated: Thu Oct 27 14:55:54 2021 Date Range Processed: today ( 2021-Oct-27 ) Period is day. Detail Level of Output: 0 Type of Output/Format: stdout / text Logfiles for Host: ol8-server ##################################################################
Use journald
Log files can also be managed by the journald
daemon, which is part of systemd
. The full service name is systemd-journald.service
, and the full daemon name is systemd-journald
.
You use journalctl
to query the systemd
journal logs. By default, the listed entries include a time stamp, the host name, the application involved, and the given message.
The main configuration file for systemd-journald
is /etc/systemd/journald.conf
.
In this practice, you use the journalctl
command to query the systemd
journal, view journald
metadata, and enable persistent journald
storage. Your log output might vary.
Watch the video below for an overview on journald and journalctl
.
-
Use the
journalctl
command with no options or arguments to query thesystemd
journal.journalctl
Note the following:
- Enter
q
to exitjournalctl
. - All log data is displayed, including rotated logs.
- The beginning of the boot process is indicated with a special entry.
- Entries with error priority and higher are in red.
- Entries with notice and warning priority are in bold font (which might not be visible in your terminal window; opening a terminal window in the GNOME GUI will show the bold font).
- Enter
-
Run the
journalctl -h
command to display usage and query options.journalctl -h
-
Run the
journalctl -r
command to display the newest log entries first.journalctl -r
-
Run the
journalctl –n 3
command to display the three newest log entries.journalctl -n 3
The
-n
option displays the ` ` of the most recent log entries specified. -
Run the
journalctl
with the--since
option to view log entries in the last10
minutes.journalctl --since "5 minutes ago"
The output displays log enteries within the specified range.
-
You can use the
-u
option to display log entries for anysystemd
unit.journalctl -u crond
-
Type the
journalctl
command, add a space, and then press the Tab key twice to display the metadata fields.journalctl <TAB> <TAB>
Sample output:
_AUDIT_LOGINUID= JOB_RESULT= SSSD_DOMAIN= _AUDIT_SESSION= JOB_TYPE= _STREAM_ID= AVAILABLE= JOURNAL_NAME= SYSLOG_FACILITY= AVAILABLE_PRETTY= JOURNAL_PATH= SYSLOG_IDENTIFIER=
-
Run the
journalctl -o verbose
command to display log entries in verbose format.- Verbose format shows the metadata fields and values for all journal entries.
- You can use the
-o
option to display log entries in any supported format. Supported format options areshort
,short-iso
,short-precise
,short-monotonic
,verbose
,export
,json
,json-pretty
, andcat
.
journalctl -o verbose
-
Run the
journalctl -f
command to display log entries as they are being added.journalctl -f
Enter CTRL + C to terminate the command.
-
Run the
journalctl –p crit
command to display log entries with a priority ofcrit
,alert
, oremerg
.-
It is likely your lab system has not logged entries with any of these priorities.
-
You can use the
-p
option to display log entries of any priority. Valid priorities aredebug
,info
,notice
,warning
,err
,crit
,alert
, andemerg
. -
Without specifying a range with the
-p
option, log entries at the given level and those with greater significance are shown (a range has the formFROM..TO
, for example,crit..alert
).
journalctl -p crit
-
-
Run the
journalctl -p warning
command to view the entries reported.journalctl -p warning
-
Run the
journalctl –F _UID
command to display unique values for the_UID
metadata field.journalctl -F _UID
-
Run the
journalctl _UID=<value>
command to show only log entries that match the condition. The example uses the_UID=0
journalctl _UID=0
-
Use the
mount
command, pipe the output togrep
, and search for the string “run
”.mount | grep run
The output shows that the file system type mounted on /run is
tmpfs
. -
Use the
ls -l
command to view the/run/log/journal/
directory.ls -l /run/log/journal/
By default, journal data is stored in this directory. Also notice the date_time stamp on this directory.
-
Use the
mkdir -p
command to make the/var/log/journal
directory.mkdir -p /var/log/journal
-
Use the
systemctl
command to restart thesystemd-journald
service.systemctl restart systemd-journald
-
Use the
ls -l
command to view the/var/log/journal
directory.ls -l /var/log/journal/
Journal data is now stored in this directory. Also notice the date_time stamp on this directory is more recent than the date_time stamp on
/run/log/journal
.
More Learning Resources
Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.
For product documentation, visit Oracle Help Center.
Work with system logging on Oracle Linux
F49762-04
May 2022
Copyright © 2021, Oracle and/or its affiliates.