Collecting kernel messages

There are several methods of collecting kernel messages, some of them more efficient than the others, and each of them has some drawbacks and advantages.

3.1 Syslog, console and dmesg

The most popular method is to use klogd, the daemon that writes the kernel messages to a log file, usually included in the syslog package. Depending on your distribution’s configuration of klogd, the kernel messages can be written to /var/log/kern.log, /var/log/messages or /var/log/kernel, or some more compilcated approach may be used (eg. on OpenSUSE systems boot messages are saved in the file /var/log/boot.msg and the messages received by klogd after the system has reached the required runlevel are written to /var/log/messages). The advantage of this method is that practically every distribution enables klogd by default and you do not really have to configure it. Unfortunately, it usually is started quite late during the boot process and stopped early during the system shutdown, which often makes it impossible to use klogd to collect messages appearing early during the system start or very late when it is going down. Moreover, if there is a run-time error that causes the kernel to crash (ie. to decide that it cannot continue running), the corresponding message cannot be written into a file and klogd may not be able to process it. For this reason, even if you use klogd, which is recommended, you will need an additional means of collecting critical messages from the kernel.

In principle, you can use the system console for this purpose. Generally speaking, the console is where the most important kernel messages appear. By default it is the current virtual terminal, but in fact it can be many places. The kernel’s command line parameter console= can be used to direct kernel messages to specific virtual terminal or to specified serial port (see the file Documentation/kernel-parameters.txt in the kernel sources for more details). When klogd starts, it may redirect kernel messages to some other destinations, usually specified in /etc/syslog.conf (for more information about this file refer to the documentation of syslog provided in your distribution).

The ”traditional” method of collecting console messages is to read them from the system console and write them down, either on a piece of paper, or using a text editor on a second computer, provided that you have one. Of course, this is time-consuming and nobody likes to do this, because it usually is very difficult to rewrite the message sufficiently precisely. Sometimes the most important part of the message ”escapes” from the screen and you cannot really help it, except for increasing the console resolution, either by booting the kernel with the vga=1 command line parameter (80x50 console), or by using a frame buffer console (see the file Documentation/fb/fbcon.txt in the kernel sources). Unfortunately, ”interesting” errors often make the kernel print huge messages and even the increased console resolution need not be sufficient in such cases.

Obviously, instead of rewriting messages from the console manually, you can simply photograph them and make such a ”screen dump” available from a web page (if you change the resolution of the picture to 1024x768, for example, and save it in a grey scale, it will be more ”download-friendly”). You should not, however, send the picture directly to the kernel developers, unless someone asks you to do this. In particular, the LKML has the message size limit of 100 KB and it does not make sense to send e-mail attachments larger than 100 KB to it.

Another problem with the system console is that it may not be always visible. For example, if you use the X Window System, you will not see the console, unless you run the xconsole program (or its equivalent). Morever, even if you run it, the kernel may crash in a spectacular way and the related messages need not make it to xconsole. In that case you also may not be able to switch from X to the text console to read the error messages. For this and the above reasons, it often is necessary to send kernel messages out of the system that they are generated on and some methods of doing it are described below.

Sometimes you may need to read the kernel messages even if it does not crash and you can use the dmesg utility for this purpose. By default dmesg reads all messages from the kernel’s ring buffer and sends them to the standard output, so you can redirect the output of dmesg to a file or to your text viewer of choice. The number of messages printed by dmesg depends on the size of the kernel’s ring buffer which is configurable at compilation time

Kernel hacking --->
Kernel debugging --->
Kernel log buffer size (16 => 64KB, 17 => 128KB) --->

It also depends on the size of the buffer used by dmesg that can be adjusted with the help of the -s option (see the dmesg man page for more details). If you report a bug to the kernel developers, they may ask you to send the output of dmesg, so you should be familiar with it.