Collecting kernel messages

There are several methods of collecting kernel messages, some of them more efficient than the others, and each of them has some drawbacks and advantages.

3.1 Syslog, console and dmesg

The most popular method is to use klogd, the daemon that writes the kernel messages to a log file, usually included in the syslog package. Depending on your distribution’s configuration of klogd, the kernel messages can be written to /var/log/kern.log, /var/log/messages or /var/log/kernel, or some more complicated approach may be used (eg. on OpenSUSE systems boot messages are saved in the file /var/log/boot.msg and the messages received by klogd after the system has reached the required runlevel are written to /var/log/messages). The advantage of this method is that practically every distribution enables klogd by default and you do not really have to configure it. Unfortunately, it usually is started quite late during the boot process and stopped early during the system shutdown, which often makes it impossible to use klogd to collect messages appearing early during the system start or very late when it is going down. Moreover, if there is a run-time error that causes the kernel to crash (ie. to decide that it cannot continue running), the corresponding message cannot be written into a file and klogd may not be able to process it. For this reason, even if you use klogd, which is recommended, you will need an additional means of collecting critical messages from the kernel.

In principle, you can use the system console for this purpose. Generally speaking, the console is where the most important kernel messages appear. By default it is the current virtual terminal, but in fact it can be many places. The kernel’s command line parameter console= can be used to direct kernel messages to specific virtual terminal or to specified serial port (see the file Documentation/admin-guide/kernel-parameters.txt in the kernel sources for more details). When klogd starts, it may redirect kernel messages to some other destinations, usually specified in /etc/syslog.conf (for more information about this file refer to the documentation of syslog provided in your distribution).

The ”traditional” method of collecting console messages is to read them from the system console and write them down, either on a piece of paper, or using a text editor on a second computer, provided that you have one. Of course, this is time-consuming and nobody likes to do this, because it usually is very difficult to rewrite the message sufficiently precisely. Sometimes the most important part of the message ”escapes” from the screen and you cannot really help it, except for increasing the console resolution, either by booting the kernel with the vga=1 command line parameter (80x50 console), or by using a frame buffer console (see the file Documentation/fb/fbcon.rst in the kernel sources). Unfortunately, ”interesting” errors often make the kernel print huge messages and even the increased console resolution need not be sufficient in such cases.

Obviously, instead of rewriting messages from the console manually, you can simply photograph them and make such a ”screen dump” available from a web page (if you change the resolution of the picture to 1024x768, for example, and save it in a grey scale, it will be more ”download-friendly”). You should not, however, send the picture directly to the kernel developers, unless someone asks you to do this. In particular, the LKML has the message size limit of 100 KB and it does not make sense to send e-mail attachments larger than 100 KB to it.

Another problem with the system console is that it may not be always visible. For example, if you use the X Window System, you will not see the console, unless you run the xconsole program (or its equivalent). Moreover, even if you run it, the kernel may crash in a spectacular way and the related messages need not make it to xconsole. In that case you also may not be able to switch from X to the text console to read the error messages. For this and the above reasons, it often is necessary to send kernel messages out of the system that they are generated on and some methods of doing it are described below.

Sometimes you may need to read the kernel messages even if it does not crash and you can use the dmesg utility for this purpose. By default dmesg reads all messages from the kernel’s ring buffer and sends them to the standard output, so you can redirect the output of dmesg to a file or to your text viewer of choice. The number of messages printed by dmesg depends on the size of the kernel’s ring buffer which is configurable at compilation time

Kernel hacking --->
Kernel debugging --->
Kernel log buffer size (16 => 64KB, 17 => 128KB) --->

It also depends on the size of the buffer used by dmesg that can be adjusted with the help of the -s option (see the dmesg man page for more details). If you report a bug to the kernel developers, they may ask you to send the output of dmesg, so you should be familiar with it.

3.2 Serial console

The method of collecting kernel messages with the help of the serial console has two important drawbacks. First, it requires a serial port to be present in the computer running the tested kernel (”test bed computer”) that will generate the messages of interest, so it cannot be used, for example, with the majority of contemporary notebooks. Second, it requires you to use another computer for receiving the messages. On the other hand, the serial console can be used for collecting messages generated in the early stage of the kernel’s life time.

To use the serial console you need to connect one of the test bed computer’s serial ports (often referred to as a COM ports) to a serial port installed in the other computer (this serial port may be emulated, for example with the help of a USB-to-serial converter) with a so-called null modem cable.

You also need to configure the tested kernel to support the serial console:

Device Drivers --->
Character devices --->
Serial drivers --->
<*> 8250/16550 and compatible serial support
[*]      Console on 8250/16550 and compatible serial port

and you need to tell the kernel to actually use it, for example by appending the following parameters to the kernel’s command line:

console=ttyS0,115200n8 console=tty0

(the console=tty0 means that we want the kernel to use the virtual terminal 0 as a console apart from the serial one). Additionally, if you use GRUB as the boot loader, you can teach it to receive commands via the serial link from the other machine, for example by adding the following lines to its configuration file (usually /boot/grub/menu.lst):

serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
terminal --timeout=5 serial console

Still, this is not necessary if you have a ”normal” keyboard and monitor attached to the test bed machine (refer to the documentation of GRUB for more information about the options used above).

Usually, it also is a good idea to configure the test bed system to use the serial console as one of its terminals. In Fedora you can do it according to the instructions below, but the other distributions may require you to modify the system configuration in a different way.

First, edit the file /etc/sysconfig/init and assign the value serial to the variable BOOTUP . Next, in the file /etc/sysconfig/kudzu set SAFE=yes and in /etc/securetty}} add {{{ttyS0. Now, add the line S1:23:respawn:/sbin/mgetty -L ttyS0 115200 vt100}} to {{{etc/inittab (this requires you to have the mgetty package installed).

It is recommended to apply these changes to all systems on which you intend to use the serial terminal (you can always comment out the settings that enable it when it is no longer necessary).

Of course, the machine that will receive the messages over the serial link has to be configured too, but this usually is quite straightforward. First, run

# minicom -o -C log.txt

on it (you need to do this as root). Most probably you will get the error message Device /dev/modem/ acces failed: (...) which means that minicom could not find the special device file /dev/modem . To overcome this problem you can, for example, create a symbolic link from ttyS0 to it

# ln -s /dev/ttyS0 /dev/modem

and then minicom should run. It still is necessary to set up minicom itself, but this can be done with the help of its configuration menu. By default minicom is configured to use modem lines, so you have to change this setting. You also need to set the serial link’s transmission parameters to reflect the settings that you have done on the test bed computer (note that some serial ports do not work with the highest baud rates and in such cases you will need to decrease the baud rate and this has to be done on both ends of the serial link, because they both must use exactly the same parameters of transmission).

For more information about using the serial console with the Linux kernel refer to the file Documentation/serial-console.txt included in the kernel sources.

3.3 Network console

If there are no serial ports in the computer on which new kernels are going to be tested, you will need to use some other means of collecting kernel messages, such as the network console.

Of course, for the network console to work an additional computer is necessary, on which the messages will appear. Moreover, this computer should be connected to the same Ethernet LAN as your test bed machine (unfortunately, as of today Ethernet is the only type of network that the network console can be used with), which may be regarded as a drawback. Another drawback of the network console is that it will not work before the networking has been initialized on the test bed system, so it cannot be used for transmitting some early kernel messages. In turn, an undoubted advantage of it is that the distance between the test bed computer and the machine used for collecting images may be relatively large (at least in comparison with the serial console).

To make the tested kernel support the network console, you need configure it appropriately

Device Drivers --->
Network device support --->
<*>      Network console logging support (EXPERIMENTAL)

(for more information about the configuration of the kernel see Section 1.6). Next, to tell it that you want it to send the messages over the network, you can append the netconsole= parameter to its command line, eg.

netconsole=4444@192.168.100.1/eth0,6666@192.168.100.2/00:14:38:C3:3F:C4

where the first three settings are related to the test bed system, namely

• 4444 is the number of the UDP port,
• 192.168.100.1 is the IP address of the network interface (obviously, it has to correspond to an Ethernet card),
• eth0 is the name of the the network interface,

that will be used for sending kernel messages, and the remaining three ones are related to the other system, ie.

• 6666 is the number of the UDP port,
• 192.168.100.2 is the IP address of the network interface,
• 00:14:38:C3:3F:C4 is the MAC (ie. hardware Ethernet) address of the network adapter,

that will be used for receiving them. Remember to make sure that the last two settings reflect the actual configuration of the machine supposed to receive the messages over the network.

The computer that will be used for collecting messages from the tested kernel need not be configured in any special way, except that it should be running a program that will receive the messages and save them or print them on the screen. For this purpose you can use the netcat utility (http://netcat.sourceforge.net/), for example in the following way:

$ netcat -u -l -p 6666

where the port number (6666 in this example) has to be the one that you have provided to the tested kernel.

It is worthy of noting that the network console driver can be built as a module and loaded at run time, but we are not going to cover this case. More information about that can be found in the file Documentation/networking/netconsole.txt included in the kernel sources.