How To Use Journalctl to View and Manipulate Systemd Logs

Introduction

Some of the

most compelling advantages of systemd are those related to process and system registration. When using other tools, logs are usually scattered throughout the system, handled by different daemons and processes, and can be quite difficult to interpret when spanning multiple applications. SystemD attempts to address these issues by providing a centralized management solution to register all kernel and user-space processes. The system that collects and manages these logs is known as a journal.

The journal is implemented with the journald daemon, which handles all messages produced by the kernel, initrd, services, etc. In this guide, we will discuss how to use the journalctl utility, which can be used to access and manipulate data stored within the journal.

General idea

One of the

impetus behind the systemd diary is to centralize the management of records, regardless of where messages originate. Since much of the boot process and service management are handled by the systemd process, it makes sense to standardize how logs are collected and accessed. The journald daemon collects data from all available sources and stores it in a binary format for easy and dynamic manipulation.

This gives us a number of significant advantages. By interacting with data using a single utility, administrators can dynamically display log data according to their needs. This can be as simple as viewing boot data from three boots ago or combining registry entries sequentially from two related services to debug a communication problem.

Storing the log data in

a binary format also means that the data can be displayed in arbitrary output formats depending on what you need right now. For example, for day-to-day log management, you may be used to viewing logs in the standard syslog format, but if you decide to graph service outages later, you can generate each entry as a JSON object to make it consumable for your graphics service. Because data is not written to disk in plain text, no conversion is needed when a different on-demand format is needed.

The systemd journal can be used with an existing syslog implementation, or you can replace syslog functionality, depending on your needs. While the systemd journal will cover most of the administrator’s logging needs, it can also complement existing logging mechanisms. For example, you can have a centralized syslog server that you use to compile data from multiple servers, but you can also collate logs from multiple services into a single system with the systemd journal. You can do both by combining these technologies.

One

of the

advantages of using a binary journal for logging is the ability to view logs in UTC or local time at will. By default, systemd will display results in local time.

Because of this, before we start with the diary, we will make sure that the time zone is set correctly. The systemd suite actually comes with a tool called timedatectl that can help with this.

First, see what

time zones are available with the list-timezones option:

timedatectl list-timezones

This will display the time zones available on your system. When you find the one that matches your server location, you can configure it

using the set-timezone option: sudo timedatectl

set-timezone

zone To make sure your machine is using the correct time now, use the timedatectl command alone or with the status option. The screen will be the same

timedatectl status

Departure Local time: Fri 2021-07-09 14:44:30 EDT Universal Time: Fri 2021-07-09 18:44:30 UTC RTC Time: Fri 2021-07-09 18:44:31 Time zone: America/New_York (EDT, -0400) System clock synchronized: Yes NTP service: active PSTN in local TZ: no

The first line should display the correct time.

Basic

log viewing To view the logs that the journald daemon has collected, use the journalctl command.

When used alone, every journal entry that is in the system will be displayed within a locator (usually less) so you can navigate. Older entries will be above

journalctl

Output- Logs start on Tue 2015-02-03 21:48:52 UTC, end on Tue 2015-02-03 22:29:38 UTC. – Feb 03 21:48:52 localhost.localdomain systemd-journal[243]: The runtime journal is using 6.2M (maximum allowed 49. Feb 03 21:48:52 localhost.localdomain systemd-journal[243]: Runtime journal is using 6.2M (maximum allowed 49. Feb 03 21:48:52 localhost.localdomain systemd-journald[139]: Received SIGTERM from PID 1 (systemd). Feb 03 21:48:52 localhost.localdomain kernel: audit: type=1404 audit(1423000132.274:2): enforcing=1 old_en Feb 03 21:48:52 localhost.localdomain kernel: SELinux: 2048 avtab hash slots, 104131 rules. Feb 03 21:48:52 localhost.localdomain kernel: SELinux: 2048 avtab hash slots, 104131 rules. Feb 03 21:48:52 localhost.localdomain kernel: input: ImExPS/2 Generic Explorer Mouse as /devices/platform/ Feb 03 21:48:52 localhost.localdomain kernel: SELinux: 8 users, 102 roles, 4976 types, 294 bools, 1 sens, Feb 03 21:48:52 localhost.localdomain kernel: SELinux: 83 classes, rules 104131 . . .

You likely have pages and pages of data to scroll through, which can be tens or hundreds of thousands of lines long if systemd has been on your system for a long time. This demonstrates how much data is available in the journal’s database.

The format will be familiar to those who are used to standard syslog logging. However, this actually collects data from more sources than traditional syslog implementations are capable of. It includes logs of the early boot process, kernel, initrd, and application standard error and output. All of these are available in the journal.

You may notice that all timestamps displayed are local time. This is available for each log entry now that we have our local time set correctly in our system. All records are displayed using this new information.

you want to display timestamps in UTC, you can use the

-utc flag:

journalctl -utc

Filtering journals by time

While having access to such a large collection of data is definitely useful, such a large amount of information can be difficult or impossible to inspect and process manually. Because of this, one of the most important features of journalctl is its filtering options.

Viewing current boot logs

The most basic of these that you can use on a daily basis, is the -b flag. This will show you all journal entries that have been collected since the most recent reboot.

journalctl -b

This will help you identify and manage information that is relevant to your current environment

In cases where you are not using this feature and showing more than one day of boots, you will see that journalctl has inserted a line that looks like this every time the system goes down:

Departure . . . – Restart – . . .

This can be used to help you logically separate information in boot sessions.

While

you’ll commonly want to display the current boot information, there are certainly times when past boots would come in handy as well. The journal can save information from many previous boots, so journalctl can be made to display information easily.

Some distributions allow you to save previous boot information by default, while others disable this feature. To enable persistent boot information, you can create the directory to store the journal by typing

: sudo

mkdir -p /var/log

/journal

Or you can edit the journal configuration file

sudo nano /etc/systemd/journald.conf

In the [Journal] section, set the Storage= option to “persistent” to enable persistent logging:

. . . [Journal] Storage=persistent

When saving previous boots is enabled on the server, journalctl provides some commands to help you work with boots as a splitting unit. To see the boots journald knows, use the

-list-boots option with journalctl: journalctl -list-boots Output-2 caf0524a1d394ce0bdbcff75b94444fe Tue-2015-02-03 21:03 48:52 UTC—Tue 2015-02-03 22:17:00 UTC -1 13883d180dc0420db0abcb5fa26d6198 Mar 2015-02-03 22:17:03 UTC—Tue 2015-02-03 22:19:08 UTC 0 bed718b17a73415fade0e4e7f4bea609 Mar 2015-02-03 22:19:12 UTC—Tue 2015-02-03 23:01:01 UTC

This will display a line for each boot. The first column is the boot offset that can be used to easily reference journalctl boot. If you need an absolute reference, the boot ID is in the second column. You can know the time that the boot session refers to with the two time specifications listed towards the end.

To display information about these boots, you can use the information in the first or second column.

For example, to view the previous boot journal, use the

relative pointer

-1 with the flag -b: journalctl -b -1

You can also use the boot ID to retrieve data from a boot:

journalctl -b caf0524a1d394ce0bdbcff75b94444fe

Windows Time

While

viewing the log entries per boot is incredibly useful, you may often want to request time windows that don’t align well with the System boots. This can be especially true when it comes to long-lived servers with significant uptime.

You can filter by arbitrary time limits using the -since and -up options, which restrict displayed entries to those displayed after or before the specified time, respectively.

Time values can come in a variety of formats. For absolute time values, you must use the following

format:

YYYY-MM-DD HH:MM:SS

For example, we can see all entries from January

10, 2015 at 5:15 PM by typing:

journalctl -since “2015-01-10 17:15:00”

If components of the previous format are left out, some default values will apply. For example, if the date is omitted, the current date will be assumed. If the time component is missing, “00:00:00” (midnight) will be replaced. The seconds field can also be left off by default at

“00”: journalctl -since “2015-01-10” -until “

2015-01-11 03:00″

The journal also comprises some relative values and named shortcuts. For example, you can use the words “yesterday,” “today,” “tomorrow,” or “now.” You can make relative tenses by prefixing “-” or “+” to a numbered value or by using words like “ago” in a sentence construction.

To get

yesterday’s data, you can type: journalctl -since yesterday

If you received reports of a service outage starting at 9:00 AM and continuing until an

hour ago, you can type:

journalctl -from

09:00 -until “1 hour ago”

As you can see, it’s relatively simple to define flexible time windows to filter the entries you want to see.

Filtering by interest

From the message Earlier, we learned some ways you can filter journal data using time constraints. In this section we will discuss how to filter according to the service or component you are interested in. The systemd journal provides a variety of ways to do this.

Perhaps

the

most useful way to filter is by the unit you’re interested in. We can use the -u option to filter this way.

For example, to

view all the logs of an Nginx drive in our system, we can type:

journalctl -u nginx.service

Usually, you probably also want to filter by time to show the lines that interest you. For example, to check how the service is running today, you can type:

journalctl -u nginx.service -since today

This type of approach becomes extremely useful when you take advantage of the journal’s ability to collate records from multiple units. For example, if your Nginx process is connected to a PHP-FPM unit to process dynamic content, you can combine the entries of both in chronological order by specifying both units:

journalctl -u nginx.service -u php-fpm.service -from today

This can make it much easier to detect interactions between different programs and debugging systems rather than individual processes.

By process, User or group ID

Some services generate a variety of child processes to get the job done. If you’ve explored the exact PID of the process you’re interested in, you can filter by that as well.

To do this we can filter by specifying the _PID field. For example, if the PID we are interested in is 8088, we could write:

journalctl _PID=8088

At other times, you may want to display all registered entries for a specific user or group. This can be done with _UID or _GID filters. For example, if your web server is running under

the user www-data, you can find the user ID by typing:

id -u www-data

Output33

You can then use the ID that was returned to filter the journal results:

journalctl _UID=33-from today

The systemd journal has many fields that can be used for filtering. Some of them are passed from the process being recorded and others are applied by journal using the information you collect from the system at the time of registration.

The initial underline indicates that the _PID field is of the latter type. The journal automatically records and indexes the PID of the process you are recording for later filtering. You can get information about all available journal fields by typing:

man systemd.journal-fields

We will discuss some of these in this guide. For now, however, we will review a more useful option that has to do with filtering by these fields. The -F option can be used to display all available values for a given journal field.

For example, to see for which group ID the systemd journal has entries, you can type:

journalctl -F _GID

Output32 99 102 133 81 84 100 0 124 87

This will show you all the values that the journal has stored for the Group ID field. This can help you build your filters.

By component path

We can also filter by providing a path location

If the path leads to an executable, journalctl will display all entries involving the executable in question. For example, to find those entries that involve the bash executable, you can type:

journalctl /usr/bin/bash

Usually, if a drive is available for the executable, that method is cleaner and provides better information (associated child process inputs, etc.). Sometimes, however, this is not possible.

Viewing

kernel messages Kernel messages,

those usually found in the dmesg output, can also be retrieved from the journal

To display only these messages, we can add the -k or -dmesg flags to our command:

journalctl -k

By default, this will display kernel messages from the current boot. You can specify an alternate boot using the normal boot selection flags described above. For example, to get messages from five boots ago, you can type:

journalctl -k -b -5

By

priority

One filter that system administrators are often interested in is the priority of the message. While it is often useful to record information at a very detailed level, when the available information is actually digested, low-priority records can be distracting and confusing.

You can use journalctl to display only messages of a specified priority or higher by using the -p option. This allows you to filter out lower-priority messages.

For example, to display only entries logged at error level or higher, you can type:

journalctl -p err -b

This will show you all messages marked as error, critical, alert, or emergency. The journal implements standard syslog message levels. You can use the priority name or its corresponding numeric value. In order from highest to lowest priority, these are

0: emerg
1: alert
2: crit
3: err
4: warning
5: notice
6: info
7: debugging

The above numbers or names can be used interchangeably with the -p option. When you select a priority, messages marked at the specified level and those above it are displayed.

Modifying

the Journal Display

Above, we demonstrate input selection through filtering. However, there are other ways we can modify the output. We can adjust the journalctl display to meet various needs.

Truncate

or expand the output We can adjust the way journalctl

displays the data by telling it to reduce or expand the output

By default, journalctl will display the entire entry in the pager, allowing entries to be offset to the right of the screen. This information can be accessed by pressing the right arrow key.

If you prefer to have the output truncated, inserting ellipses where the information has been deleted, you can use the

-no-full option:

journalctl -no-full

Output . . . Feb 04 20:54:13 journalme sshd[937]: Password failed for root of 83.234.207.60…h2 Feb 04 20:54:13 journalme sshd[937]: Connection closed by 83.234.207.60 [preauth] Feb 04 20:54:13 journalme sshd[937]: PAM 2 plus authentication failures; Registration Name… You

can also go in the opposite direction with this and tell journalctl to display all your information, regardless of whether it includes non-printing characters. We can do this with the indicator

-a: journalctl -a

Output to standard output By

default,

journalctl

displays the output on a locator for ease of consumption. However, if you plan to process the data with text manipulation tools, you probably want to be able to generate standard output.

You can do this with the

-no-pager option:

journalctl -no-pager

This can be immediately piped to a processing utility or redirected to a file on disk, depending on your needs

If

you are processing journal entries, as mentioned above, you will most likely find it easier to analyze the data if it is in a more consumable format. Fortunately, the journal can be displayed in a variety of formats as needed. You can do this using the -o option with a format specifier.

For example, you can generate the journal entries in JSON by typing

journalctl -b -u nginx -o json

Output{ “__CURSOR” : “s=13a21661cf4948289c63075db6c25c00;i=116f1;b=81b58db8fd9046ab9f847ddb82a2fa2d;m=19f0daa;t=50e33c33587ae;x=e307daadb4858635”, “__REALTIME_TIMESTAMP” : “1422990364739502”, “__MONOTONIC_TIMESTAMP” : “27200938”, “_BOOT_ID” : “81b58db8fd9046ab9f847ddb82a2d”, “PRIORITY” : “6”, “_UID” : “0”, “_GID” : “0”, “_CAP_ EFFECTIVE” : “3ffffffff”, “_MACHINE_ID” : “752737531a9d1a9c1e3cb52a4ab967ee”, “_HOSTNAME” : “desktop”, “SYSLOG_FACILITY” : “3”, “CODE_FILE” : “src/core/unit. C”, “CODE_LINE” : “1402”, “CODE_FUNCTION” : “unit_status_log_starting_stopping_reloading”, “SYSLOG_IDENTIFIER” : “systemd”, “MESSAGE_ID” : “7d4958e842da4a758f6c1cdc7b36dcc5”, “_TRANSPORT” : “journal”, “_PID” : “1”, “_COMM” : “systemd”, “_EXE” : “/usr/lib/systemd/systemd”, “_ CMDLINE” : “/usr/lib/systemd/systemd”, “_SYSTEMD_CGROUP” : “/”, “UNIT” : “nginx.service”, “MESSAGE” : “Starting a high-performance web server and proxy server inverse…”, “_SOURCE_REALTIME_TIMESTAMP” : “1422990364737973” } . . .

This is useful for analyzing with utilities. You can use the json-pretty format to better control the data structure before passing it to the JSON consumer

journalctl -b -u nginx -o json-pretty

Output{ “__CURSOR” : “s=13a21661cf4948289c63075db6c25c00;;i=116f1;b=81b58db8fd9046ab9f847ddb82a2fa2d;m=19f0daa;t=50e33c33587ae;x=e307daadb4858635”, “__REALTIME_TIMESTAMP” : “1422990364739502”, “__MONOTONIC_TIMESTAMP” : “27200938”, “_BOOT_ID” : “81b58db8fd9046ab9f847ddb82a2fa2d”, “PRIORITY ” : “6 “, “_UID” : “0”, “_GID” : “0”, “_CAP_EFFECTIVE” : “3fffff”, “_MACHINE_ID” : “752737531a9d1a9c1e3cb52a4ab967ee”, “_HOSTNAME” : “desktop”, “SYSLOG_FACILITY” : “3”, “CODE_FILE” : “src/core/unit.. ” C”, “CODE_LINE” : “1402”, “CODE_FUNCTION” : “unit_status_log_starting_stopping_reloading”, “SYSLOG_IDENTIFIER” : “systemd”, “MESSAGE_ID” : “7d4958e842da4a758f6c1cdc7b36dcc5”, “_TRANSPORT” : “journal”, “_PID” : “1”, “_COMM” : “systemd”, “_EXE” : “/usr/lib/systemd/systemd”, “_CMDLINE” : “/usr/lib/systemd/systemd”, “_SYSTEMD_CGROUP” : “/”, “UNIT” : “nginx.service”, “MESSAGE” : “Starting a high-performance web server and proxy server inverse…”, “_SOURCE_REALTIME_TIMESTAMP” : “1422990364737973” } . . .

The following formats can be used for display

cat: Displays only the message field itself

export: A binary format suitable for transferring or backing up

. json: Standard JSON with one input per line. json-pretty: JSON format for

better human readability json-sse
: output formatted JSON wrapped to make adding server-sent events compatible

short: The default

syslog-style output short-iso: The default format

increased to display the timestamps of the

ISO 8601 wall clock. short-monotonic: The default format with monotonous timestamps. short-precise

The default format with detailed microsecond accuracy:
Displays all journal fields available for entry, including those that are usually hidden internally.

These options allow you to display journal entries in the format that best suits your current needs.

Active

process monitoring

The journalctl command mimics how many administrators use tail to monitor active or recent activity. This functionality is built into journalctl, allowing you to access these features without having to pipe to another tool.

Viewing

recent records To display a certain number of records, you can use the -n

option, which works exactly like tail

-n.

By default, it will display the 10 most recent entries

: journalctl -n

You can specify the number of entries you want to see with a number after

-n:

journalctl –

n 20

Next records

To actively follow records as they are written, you can use the -f flag. Again, this works as you would expect if you are experienced using tail

-f:

journalctl -f

To exit this command, type CTRL+C

You

might be wondering about the cost of storing all the data we’ve seen so far. Also, it can be interesting in cleaning up some old records and freeing up space.

Finding current

disk usage

You can find out how much disk space the journal currently occupies by using the –

disk-usage:

journalctl -disk-usage

Output flagArchived and active journals occupy 8.0 million in the file system.

Deleting old records

If you want to reduce your diary,

you can do it in two different ways (available with systemd version 218 and later). If

you

use the -vacuum-size option, you can reduce your diary by indicating a size. This will delete old entries until the total occupied journal space on disk is the requested size:

sudo journalctl -vacuum-size=1G

Another way to shrink the journal is to provide a timeout with the -vacuum-time option. Any entries beyond that time are deleted. This allows you to keep entries that have been created after a specific time.

For example, to keep last year’s entries, you can type:

sudo journalctl -vacuum-time=1years

Limit journal expansion

You can configure your server to limit the amount of space the journal can occupy. This can be done by editing the /etc/systemd/journald.conf file.

The following items can be used to limit journal growth

: SystemMaxUse=: Specifies the maximum disk space that the journal can use on persistent storage.

SystemKeepFree=: Specifies the amount of space that the journal should leave free when adding journal entries to persistent storage

SystemMaxFileSize=: Controls the size of individual journal files that can grow into persistent storage before spinning them.

RuntimeMaxUse=: Specifies the maximum disk space that can be used on volatile storage (within the /run file system). RuntimeKeepFree=: Specifies the amount of space

to reserve for other uses when writing data to volatile storage (within the /run file system). RuntimeMaxFileSize=: Specifies the amount of space

that an individual journal

file can occupy on volatile storage (within
the system of /run files) before rotating it.

By setting these values, you can control how journaling consumes and conserves space on the server. Note that SystemMaxFileSize and RuntimeMaxFileSize will target archived files to reach the set limits. It is important to remember this when interpreting file counts after an aspiration operation.

Conclusion

As you can see, the systemd journal is incredibly useful for collecting and managing your system and application data. Most of the flexibility comes from the extensive automatically recorded metadata and centralized nature of the registry. The journalctl command makes it easy to take advantage of advanced journal features and perform extensive analysis and relational debugging of different application components.