• Immutable Page
  • Info
  • Attachments

Diff for "KernelDebug"

Differences between revisions 5 and 6

Deletions are marked like this. Additions are marked like this.
Line 123: Line 123:
 * [http://mail.nl.linux.org/kernelnewbies/2003-08/msg00347.html How to debugging kernel OOPs and hangs]  * [http://mail.nl.linux.org/kernelnewbies/2003-08/msg00347.html How to debug kernel OOPs and hangs]

Intro

Some bugs are simple to spot, like a kernel crash. Some bugs are harder to reproduce.

How to capture dmesg

Sometimes, you can just copy lines from /var/log/kern.log or /var/log/messages or simply run

dmesg | tee log-file.txt

and then reproduce the error.

This only works if your machine doesn't totally crash when you hit the bug. If the system freezes, the dmesg may not be written to disk. In that case, your only alternative is to use either serial console or netconsole to capture the dmesg when the crash happens.

Kernel options to turn on

You may need to turn on several different kernel configuration options in order to capture all of the dmesg necessary to debug your kernel issue. Here's some examples

  • CONFIG_PRINTK_TIME - add time stamps to dmesg

  • CONFIG_DEBUG_KERNEL - turn on kernel debugging

  • CONFIG_DETECT_HUNG_TASK - good for figuring out what's causing a kernel freeze

  • CONFIG_DEBUG_INFO - ensures you can decode kernel oops symbols

  • CONFIG_EARLY_PRINTK

  • CONFIG_LOG_BUF_SHIFT=21 - sets the kernel buffer log size to the biggest buffer

  • CONFIG_NETCONSOLE=m - compiles netconsole as a module, see tutorial below.

Other kernel subsystems often have debug config options you can turn on for more verbose debug. For example, to enable verbose USB 3.0 debug, you should turn on CONFIG_USB_DEBUG and CONFIG_USB_XHCI_HCD_DEBUGGING.

Netconsole

Netconsole is a powerful Linux kernel debugging tool. The dmesg output from a machine under test is transferred over an ethernet link (via UDP packets) to another machine. That means that you can see the debugging messages from the test machine on the screen of another machine. Netconsole isn't good for debugging early kernel panics, but it is very useful if your new kernel driver hangs your system.

Netconsole is a kernel module, so you will need to compile a custom kernel with CONFIG_NETCONSOLE=m. If you need help compiling a custom kernel, follow the directions on KernelBuild. Which kernel you choose to compile depends on which kernel you want to reproduce the bug on. You may want to download the source of your distribution kernel, or attempt to reproduce the bug on the latest stable kernel.

Prework

First, you need to have some tools installed. You'll need netcat, ping, and (optionally) wireshark. You'll also need to have netconsole compiled as a module on the source box. Netconsole has to be a module so you can load it after you get the system set up.

Make sure the ethernet driver for both machines supports netpoll. Also make sure that the machines are both plugged into the same router or subnet.

Configuring Netconsole

In this section, I'll refer to the computer under test that is generating the dmesg output as the source machine. The computer that receives the debugging messages is called the target machine.

First, on the source machine, make sure you have the daemon that routes kernel messages (sysctl) set up so that messages of all priority types will end up in /var/log/messages. You can do this by running

sudo dmesg -n 8

or

sudo sysctl -w kernel.printk="8 8 8 8"

Next, on the source machine, figure out which ethernet port is connected. It may be referred to as ethN (e.g. eth0 or eth1). Run ifconfig to find out:

sarah@xanatos:~$ sudo ifconfig 
eth1      Link encap:Ethernet  HWaddr 12:34:56:78:90:12  
          inet addr:10.7.201.12  Bcast:10.7.201.255  Mask:255.255.255.0
          inet6 addr: fe80::3e97:eff:fe39:d710/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:157762 errors:0 dropped:0 overruns:0 frame:0
          TX packets:39377 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:64231841 (64.2 MB)  TX bytes:6863064 (6.8 MB)
          Interrupt:20 Memory:f2500000-f2520000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:4672 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4672 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:394237 (394.2 KB)  TX bytes:394237 (394.2 KB)

Next, on the target machine, use the same command to find its IPv4 address and mac address. (Netconsole does not currently support IPv6 addresses.) Those are the inet and HWaddr numbers. Let's assume the IP address is 10.7.201.20 and the mac address is 12:34:56:78:90:20.

Now, make sure both boxes can ping each other. This ensures both boxes are on the same subnet and can send UDP packets. In this example, on the target machine, we would run

ping 10.7.201.12

and on the source machine, we would run

ping 10.7.201.20

If you aren't getting packet transmissions from the pings, go debug your network. Wireshark is a useful tool in this case.

Starting netconsole

On the target machine, start netconsole to capture the dmesg output:

nc -l -u 6666 | tee ~/dmesg-`date +%Y-%m-%d-%H-%M`.txt

This will create a new file in your home directory with the dmesg output, while allowing you to see what's going into the file.

Next, on the source machine, load the netconsole module. You'll need to load the module with an extra 'netconsole' module parameter. The netconsole documentation describes how you use the module parameters to tell netconsole how to send dmesg packets to your target machine. The parameter format is currently netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]. In our example, we're leaving off the source port and source IP, since the default is fine. The source device (dev) is eth1, from our ifconfig example above. The target IP address is 10.7.201.20, and the target MAC address is 12:34:56:78:90:20, which we found from running ifconfig on the target machine. Thus, the modprobe command would be:

sudo modprobe netconsole netconsole=@/eth1,@10.7.201.20/12:34:56:78:90:20

If you've followed the directions correctly, on your target machine, you should see the terminal with netconsole spit out some dmesg:

[ 2009.373932] netpoll: netconsole: local port 6665
[ 2009.373941] netpoll: netconsole: local IP 0.0.0.0
[ 2009.373945] netpoll: netconsole: interface 'eth1'
[ 2009.373949] netpoll: netconsole: remote port 6666
[ 2009.373952] netpoll: netconsole: remote IP 10.7.201.20
[ 2009.373956] netpoll: netconsole: remote ethernet address 12:34:56:78:90:20
[ 2009.373962] netpoll: netconsole: local IP 10.7.201.12
[ 2009.375261] console [netcon0] enabled
[ 2009.375307] netconsole: network logging started

Now you can go trigger the kernel crash and capture the dmesg from the crash on your target machine.

Decoding kernel oops

Tell others about this page:

last edited 2013-03-15 20:28:37 by SarahSharp