The kernel, patches, trees and compilation
1.1 The kernel
The current version of the Linux kernel can always be downloaded from The Linux Kernel Archives web page (http://www.kernel.org/) in the form of a large tar ﬁle compressed with either gzip or bzip2, which is called the kernel tarball (it is worthy of noting that the bz2 ﬁles are smaller than the gz ones, but gzip is generally faster than bzip2).
After downloading the tarball you can extract the kernel’s source code from it in many diﬀerent ways (the virtue of UNIX-like systems). Our favorite method is to run either
$ tar xjvf linux-2.6.x.y.tar.bz2
for the tarballs compressed with bzip2, or
$ tar xzvf linux-2.6.x.y.tar.gz
for the ones compressed with gzip, in the directory that we want to contain the kernel sources. In principle, it can be an arbitrary directory, but it should be located on a partition with as much as 2 GB of free space.
Of course, you can also follow the kernel’s README literally and extract the source code by executing one of the following instructions:
$ bzip2 -dc linux-2.6.x.y.tar.bz2 | tar xvf - $ gzip -cd linux-2.6.x.y.tar.gz | tar xvf -
depending on what kind of a tarball you have downloaded. In any case it is not recommended to unpack the kernel tarball as root.
As a result of unpacking the kernel tarball we get a directory the name of which corresponds to the version of the kernel that will be obtained by compiling the downloaded source code (eg. linux-2.6.18). The contents of this directory are often referred to as the kernel tree. For this reason the word tree is used for naming diﬀerent development branches of the kernel. For instance, instead of saying the development branch of the kernel maintained by Andrew Morton it is more convenient to say the -mm tree.
It often is necessary to update the kernel source code, but you should not download the entire tarball every time, since that would be bandwidth-consuming and tedious. Usually, it is suﬃcient to download a patch that can be applied to an existing kernel tree from within the directory in which the tree is located. For example, if the patch in question has been compressed with bzip2, this can be done in one of the following ways:
$ bzip2 -cd /path/to/patch-2.6.x.bz2 | patch -p1 $ bzcat /path/to/patch-2.6.x.bz2 | patch -p1
If the patch has been compressed with gzip, you can apply it analogously:
$ gzip -cd /path/to/patch-2.6.x.gz | patch -p1 $ zcat /path/to/patch-2.6.x.gz | patch -p1
and for uncompressed patches you can use cat instead of either zcat or bzcat. Of course, these are only the simplest methods of applying patches and it is quite easy to invent some more interesting ones.
It is very useful to be able to revert patches that are no longer needed (or buggy). It can be accomplished by adding the -R ﬂag to the patch command line:
$ bzcat /path/to/patch-2.6.x.bz2 | patch -p1 -R
It is also possible to check if the patch will apply cleanly (ie. it does not contain any pieces of code that will be rejected) with the help of the --dry-run command-line option of patch:
$ bzcat /path/to/patch-2.6.x.bz2 | patch -p1 --dry-run
Then, if there are no messages similar to
1 out of 1 hunk FAILED -- saving rejects to file xxx
the patch can be safely applied.
Most of diﬀerent versions of the kernel source code are available as sets of patches (patchsets) or even as individual patches that should be applied on top of speciﬁc kernel trees. For this reason the patches are labeled in reference with the kernel trees to which they should be applied. Namely, since stable kernel versions are labeled as 2.6.x or 2.6.x.y, development trees are called 2.6.x-git*, 2.6.z-rc* and 2.6.z-rc*-git*, where the -rc patch with z equal to x plus one should be applied on top of the stable kernel 2.6.x. This means, for example, that the patch called patch-2.6.21-rc3.bz2 should be applied on top of the linux-2.6.20 kernel tree and the patch labeled as patch-2.6.21-rc3-git3.bz2 should be applied on top of patch-2.6.21-rc3.bz2.
The following example will hopefully make everyting clear. Assume that we have the stable kernel source code labeled as linux-220.127.116.11.tar.bz2 and we want to obtain the development tree 2.6.20-rc1-git1. First, we unpack the tarball:
$ tar xjvf linux-18.104.22.168.tar.bz2
Next, we fetch the patch called patch-22.214.171.124.bz2 from ftp://ftp.kernel.org and revert it:
$ bzip2 -cd /path/to/patch-126.96.36.199.bz2 | patch -p1 -R
Now, we have the sources of the 2.6.16 kernel in our work directory. This is what we wanted, since the stable and development patches only apply cleanly to the 2.6.x kernel trees, as they can contain some code that is also included in the 2.6.x.y versions. Accordingly, we can apply patch-2.6.17.bz2, patch-2.6.18.bz2 and patch-2.6.19.bz2 to our tree:
$ bzip2 -cd /path/to/patch-2.6.17.bz2 | patch -p1 $ bzip2 -cd /path/to/patch-2.6.18.bz2 | patch -p1 $ bzip2 -cd /path/to/patch-2.6.19.bz2 | patch -p1
to obtain the 2.6.19 kernel source code. Finally, we need to download two development patches, patch-2.6.20-rc1.bz2 and patch-2.6.20-rc1-git1.bz2, and apply them in the order speciﬁed.
Isn’t that easy? Well, it is, but it also is tedious and that is why some people who do not like to lose time use the tool called ketchup (http://www.selenic.com/ketchup/), described in the next section.
It should be noted that patches can only be applied if they are properly formatted. If you receive them via email or download them directly from a mailing list archives, the formatting is often changed in a way that makes them unusable. In such a case patch will refuse to apply the patch and you will see a message like patch: **** malformed patch at line. For instance,
$ cat ../sched-2.patch | patch -p1 -R --dry-run patching file kernel/sched.c patch: **** malformed patch at line 33: if (next == rq->idle)
means that the patch called sched-2.patch is damaged and cannot be reverted. Fortunately, if the patch has been posted to the Linux Kernel Mailing List (LKML), it can be downloaded from the archives at http://marc.theaimsgroup.com/?l=linux-kernel in the original form. For this purpose you only need to ﬁnd the message that contains the patch and click on the link Download message RAW. Still, even this will not work if the original raw message containing the patch is malformed.
For more information on downloading and applying patches please see the ﬁles README and Documentation/applying-patches.txt in the kernel sources.
The ketchup tool allows us to download any version of the kernel source code quickly and easily as well as to transform given kernel tree into another one. It does this quite intelligently, checking the current version and downloading only the required patches.
The users of Debian testing/unstable can simply install the ketchup package with the help of their favorite tool, eg.
# apt-get install ketchup
Still, for the users of Debian stable or another distribution in which the ketchup tool is not available the easiest way of installing it is to do:
$ mkdir ~/bin
unless the subdirectory bin is already present in the user’s home directory, and
$ cd ~/bin $ wget http://www.selenic.com/ketchup/ketchup-0.9.8.tar.bz2 $ tar xjvf ketchup-0.9.8.tar.bz2
It is also necessary to install the python package.
You should remember that ketchup is able to check the signatures of the downloaded patches. For this reason it is recommended to visit the web page at http://www.kernel.org/signature.html and save the current key in a text ﬁle (we have never managed to make $ gpg --keyserver wwwkeys.pgp.net --recv-keys 0x517D0F0E work on our systems). Next, you can do
$ gpg --import the_file_containing_the_kernel.org_key
and you are ready to check how the whole thing works. Namely, you can do
$ mkdir ~/tree $ cd ~/tree $ ketchup -m
and you should get an error message similar to the following one:
Traceback (most recent call last): File "/usr/bin/ketchup", line 695, in ? lprint(get_ver("Makefile")) File "/usr/bin/ketchup", line 160, in get_ver m = open(makefile) IOError: [Errno 2] No such file or directory: ’Makefile’
This only means that there is no Linux kernel tree in the current directory. You can order ketchup to download the latest stable version of the kernel by using the command ”ketchup 2.6-tip”:
$ ketchup 2.6-tip None -> 188.8.131.52 Unpacking linux-2.6.15.tar.bz2 Applying patch-2.6.16.bz2 Downloading patch-184.108.40.206.bz2 --21:11:47-- http://www.kernel.org/pub/linux/kernel/v2.6/patch-220.127.116.11.bz2 => ‘/home/michal/.ketchup/patch-18.104.22.168.bz2.partial’ Resolving www.kernel.org... 22.214.171.124, 126.96.36.199 Connecting to www.kernel.org|188.8.131.52|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 5,284 (5.2K) [application/x-bzip2] 100%[====================================>] 5,284 6.19K/s 21:11:49 (6.18 KB/s) - ‘/home/michal/.ketchup/patch-184.108.40.206.bz2.partial’ saved [5284/5284] Downloading patch-220.127.116.11.bz2.sign --21:11:49-- http://www.kernel.org/pub/linux/kernel/v2.6/ patch-18.104.22.168.bz2.sign => ‘/home/michal/.ketchup/patch-22.214.171.124.bz2.sign.partial’ Resolving www.kernel.org... 126.96.36.199, 188.8.131.52 Connecting to www.kernel.org|184.108.40.206|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 250 [application/pgp-signature] 100%[====================================>] 250 --.--K/s 21:11:49 (18.34 MB/s) - ‘/home/michal/.ketchup/ patch-220.127.116.11.bz2.sign.partial’ saved [250/250] Verifying signature... gpg: Signature made wto 28 mar 2006 09:37:31 CEST using DSA key ID 517D0F0E gpg: Good signature from "Linux Kernel Archives Verification Key <firstname.lastname@example.org>" gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: C75D C40A 11D7 AF88 9981 ED5B C86B A06A 517D 0F0E Applying patch-18.104.22.168.bz2
Now you can check if you really have the latest stable tree:
$ ketchup -m 22.214.171.124
As you can see, ketchup automatically downloads the necessary patches. It can also take patches from the ~/.ketchup subdirectory of the user’s home directory, so it is a good idea to put the patches that you already have in there.
Making a transition to another kernel tree is absolutely not a problem for ketchup. For this purpose you can simply use the command ”ketchup kernel_version”, eg.
$ ketchup 2.6.16-rc4-mm1 126.96.36.199 -> 2.6.16-rc4-mm1 Applying patch-188.8.131.52.bz2 -R Applying patch-2.6.16.bz2 -R Applying patch-2.6.16-rc4.bz2 Applying 2.6.16-rc4-mm1.bz2
Similarly, if you want to get the latest kernel version from the -rt branch, you can do
$ ketchup 2.6-rt
and the tool will automatically apply or revert the patches for you:
2.6.16-rc4-mm1 -> 2.6.16-rt12 Applying 2.6.16-rc4-mm1.bz2 -R Applying patch-2.6.16-rc4.bz2 -R Applying patch-2.6.16.bz2 Applying patch-2.6.16-rt12
It is also possible to teach ketchup to use patches included in the latest -mm tree (described below), which can be done in the following way:
$ wget -c ‘ketchup -u 2.6-mm | sed "s/.bz2/-broken-out.tar.bz2/"‘
As follows from the above examples, ketchup can recognize several development branches of the Linux kernel source code. In particular, it can handle the following trees:
2.6-tip – the latest "stable" kernel
2.6-rc – the latest "release candidate" kernel
2.6-git – the latest development -git kernel
2.6-rt – the Ingo Molnar’s "real-time" tree
2.6-mm – the Andrew Morton’s tree
2.6-ck – the Con Kolivas’ tree (desktop)
2.6-cks – the Con Kolivas’ tree (server)
Now, you may be wondering why we have shown you all of the commands used for applying and reverting patches. The reason is really simple: if you ﬁnd and report a bug, you will probably get a ﬁx in the form of a patch and you should be able to correctly apply it.
We have already used the term ”kernel tree” for quite a few times and we have told you that it means ”the contents of the directory containing the kernel source code”. However, it also has another meaning which is ”a development branch of the kernel source code”. Usually, such a branch is equivalent to a patchset that can be applied on top of a speciﬁc version of the kernel sources and diﬀerent branches are named diﬀerently, so that they can be easily distinguished from each other. The more popular branches, also referred to as ”trees”, are the following:
- The -mm tree (maintained by Andrew Morton), which is built from other trees (ie. development branches of the kernel) and from many individual, often experimental, patches.
- The -rt (real-time) tree (maintained by Ingo Molnar), which contains patches that turn Linux into a real time system.
- The -ck tree (maintained by Con Kolivas), focused on the improvements of the system’s performance.
Many developers and the majority of kernel subsystems maintain their own trees used in the development process. After some time speciﬁc parts of these trees are included into the main kernel tree maintained by Linus Torvalds, called the mainline. The -git patches discussed in the previous sections represent the snapshots of the ever-evolving Linus’ tree taken at various instants of time.
1.5 The -mm tree
We have already stated that the -mm tree is a combination of some other trees and individual experimental patches. More importantly, it also is the main battleﬁeld of the ﬁght against bugs in the Linux kernel.
The importance of this tree stems from the fact that it combines the mainline with multiple trees used by the maintainers of kernel subsystems (subsystem trees) including patches considered as experimental and not ready for merging with the mainline. In other words, each release of the -mm tree is sort of a future snapshot of the mainline with some additional changes and with some bugs that (hopefully) are going to be ﬁxed before the patches containing them will get merged. Thus by testing the current -mm kernel we check how the future mainline (ie. stable) kernels may behave and if any bugs are found at this stage, they can be ﬁxed before hitting the mainline.
Still, some patches reach the mainline without appearing in the -mm tree, which is a consequence of the ﬂow of changes in the kernel source code. Namely, the majority of patches merged with the mainline comes from diﬀerent subsystem trees and they are included in the Linus’ tree when he decides to pull the patches considered as ready for the inclusion from given subsystem tree. It is possible, and it sometimes really happens, that Linus pulls the ”ready to go” patches from certain subsystem tree before they have a chance to appear in -mm (in that case the patches will appear in the -mm tree anyway, but they will come into it from the mainline rather than from the subsystem tree). For this reason it is not suﬃcient to test the -mm kernels alone and the development -rc or -git kernels should be tested as well.
On the other hand, there are kernel subsystems that do not have their own development trees and many patches are sent directly to Andrew Morton. These patches are ﬁrst included in the -mm tree and then they go either to one of the subsystem trees or to the mainline. Thus the -mm tree is a place in which the patches coming from many diﬀerent sources ﬁrst meet together and are combined into an approximation of a future stable kernel that can be tested.
Since the -mm tree is a combination of many other trees, including the mainline, that always evolve, it really should be regarded as a set of patches that can be put together from time to time. When this happens, Andrew releases the so-called -mm patches to be applied on top of speciﬁc -rc or stable kernel trees. Each of them is also available as a series of individual patches combined in order to obtain it, including some ﬁxes introduced by Andrew himself. This makes the identiﬁcation of patches that contain bugs easier, as they can be found using the quilt tool with the help of the bisection technique (for more information about it see Chapter 4).
In the directory containing the -mm patch and the series of individual patches it consists of (at ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.X-rcY/2.6.X-rcY-mmZ/ or at ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.X/2.6.X-mmZ/, where X, Y, Z are numbers) there always is a directory called hot-fixes containing the patches that ﬁx known bugs in this version of the -mm tree and should be applied on top of the corresponding -mm patch before testing it (the bugs that have already been identiﬁed need not be found once again).
The individual patches included in the -mm tree are sent to the mm-commits mailing list (http://vger.kernel.org/vger-lists.html#mm-commits). By following this list you can get an idea about the current contents of the -mm tree. It may also help you identify patches that introduce bugs.
From time to time Andrew sends to mm-commits a message with the subject like mm snapshot broken-out-date-time.tar.gz uploaded announcing that new -mm tree snapshot has been made available from the server. These snapshots are considered as less usable than the regular -mm patches and are to be used by people who want to help Andrew put the next -mm patch together.
1.6 Compilation and installation
1.6.1 Kernel compilation
First, your system should be prepared for the conﬁguration and compilation of the kernel. You will need gcc, make, binutils, util-linux and the package containing the ncurses header ﬁles (libncurses5-dev in Debian or ncurses-devel in Fedora Core and OpenSUSE). If you want to use a graphical tool for conﬁguring the kernel, you will also need the Qt library or GTK+ and the corresponding development packages. The ﬁle Documentation/Changes in the kernel sources directory contains the list of all programs that you may need.
Now, we assume that you have already downloaded the kernel source code and patched it to obtain the required version. To compile it, change the current directory to the one that contains the kernel sources and run the following sequence of commands:
make menuconfig (allows you to adjust the kernel conﬁguration to suit you; there are many documents on the web describing the conﬁguration of the kernel, one of them you can ﬁnd at http://www.tlug.org.za/old/guides/lkcg/lkcg_config.html)
make (starts the compilation of the kernel; if you have more than one processor core in your machine, you may want to use the -j option of make, for example ”make -j5”)
make modules_install (installs the kernel modules in the /lib/modules/ directory, creating the subdirectory named after the version of the newly compiled kernel)
cp arch/i386/boot/bzImage /boot/vmlinuz-<kernel_version> (copy the kernel to the /boot directory; <kernel_version> should be the label reﬂecting the version of the kernel being installed)
cp System.map /boot/System.map-<kernel_version> (copy the map of kernel symbols to the /boot directory)
Instead of ”make menuconfig” you can use ”make config” that will cause the kernel build system to ask you a lot of questions regarding its conﬁguration, which is ineﬃcient and tedious, but can be done without ncurses.
If you have already conﬁgured the kernel, you can use the conﬁguration ﬁle .config from the previous compilation. For this purpose copy it to the directory containing the current kernel sources and run ”make oldconfig” (it works like ”make config”, but does not ask so many questions). Alternatively, you can use the conﬁguration of the distribution kernel (usually it contains many things you will never need and that is one of the reasons why you may want to learn how to conﬁgure the kernel):
$ zcat /proc/config.gz > .config $ make oldconfig
1.6.2 Useful make options
There are some useful options you can pass to make during the kernel compilation:
make O=/directory/ (saves the result of the compilation in given directory; this is particularly useful if you do not want to litter the kernel source directory with *.o files, but in that case you should also use ”make O=/directory/ menuconfig” instead of ”make menuconfig” and ”make O=/katalog/ modules_install” etc.)
make CC=<compiler> (allows you to specify the name of the compiler to use; for example, in Debian there are several versions of gcc, and if you want to use gcc-3.4 to compile the kernel it is suﬃcient to use ”make CC=gcc-3.4”)
make C=1 (before compilation the kernel sources are checked by the sparse tool)
make -j<number> (sets the number of parallel processes that will be run during the kernel compilation)
make xconfig (graphical conﬁguration using Qt)
make gconfig (graphical conﬁguration using GTK+)
1.6.3 Kernel mo dules
If you want to use kernel modules, in which case some parts of the kernel will be loaded into memory only if they are needed, select the options:
Loadable module support ---> [*] Enable loadable module support [*] Automatic kernel module loading
The modularization allows the kernel to be smaller and work faster.
To compile a piece of the kernel code as a module, you need to mark it with M during the conﬁguration, eg. <M> Check for non-fatal errors on AMD Athlon/Duron / Intel Pentium 4 If the ”Automatic kernel module loading” option has been selected, the modules should be automatically loaded by the kernel when they are needed. That is, for example, if you want to mount a CD-ROM, the kernel should automatically load the module isofs.ko and this modules can be removed from memory after the CD-ROM has been unmounted. You can also load and unload kernel modules manually, using the commands modprobe, insmod or rmmod, eg.
modprobe isofs (loads the module isofs.ko)
modprobe -r isofs (removes isofs.ko from memory)
All of the modules available to the currently running kernel should be located in various subdirectories of /lib/modules/<kernel_version>/kernel .
1.6.4 Kernel hacking options
If you build the kernel to test it, you should seriously consider setting the options located in the ”Kernel hacking” menu. At least, you should set:
[*] Kernel debugging [*] Compile the kernel with debug info
The other options should also be set, if possible, because they help debug speciﬁc functionalities of the kernel, but some of them hurt performance, so you may want to leave them unset:
[*] Debug shared IRQ handlers [*] Detect Soft Lockups [*] Debug slab memory allocations [*] Slab memory leak debugging [*] RT Mutex debugging, deadlock detection [*] Built-in scriptable tester for rt-mutexes [*] Lock debugging: prove locking correctness [*] Lock dependency engine debugging [*] Locking API boot-time self-tests [*] Highmem debugging [*] Debug VM
1.6.5 Magic SysRq
The option [*] Magic SysRq key is particularly useful, because it often allows you to reduce the impact of a kernel failure and to obtain some additional information about the state of the system. It turns on some keyboard shortcuts allowing you to pass speciﬁc instructions directly to the kernel:
Alt + SysRq + b – reboot the system immediately, without unmounting ﬁlesystems and writing the contents of disk buﬀers to the storage (DANGEROUS)
Alt + SysRq + c – crashdump
Alt + SysRq + e – send the TERM signal to all processes except for INIT
Alt + SysRq + i – send the KILL signal to all processes except for INIT
Alt + SysRq + k – SAK (Secure Access Key); kill all processes bound to the currently active virtual console (eg. emergency shutdown of the X server when Ctrl + Alt + Backspace does not work)
Alt + SysRq + l – send the KILL signal to all processes including INIT
Alt + SysRq + m – print information about the state of memory
Alt + SysRq + o – system shutdown
Alt + SysRq + p – print the contents of CPU registers and ﬂags
Alt + SysRq + r – swith the keyboard to the RAW mode (eg. when Ctrl + Alt + Backspace does not work)
Alt + SysRq + s – synchronize ﬁlesystems and write the contents of disk buﬀers to the storage
Alt + SysRq + t – print the list of all tasks
Alt + SysRq + u – remount all ﬁlesystems read-only
Alt + SysRq + h – print help
Which key is the SysRq? That depends on the architecture of your machine:
- x86 – Print Screen
- SPARC – STOP
- serial console – Break
- PowerPC – Print Screen (or F13)
On all architectures you can trigger the action assigned to given key (say t) by echoing the corresponding character to the ﬁle /proc/sysrq-trigger, eg. # echo t > /proc/sysrq-trigger
What is the safest way to restart the computer?
First, you should try to press the combination of Ctrl + Alt + Backspace (X Window system) or Ctrl + Alt + Del (text console). If it does not work, you can:
press Alt + SysRq + k to kill all processes using the current console,
More information related to the ”Magic SysRq key” mechanism can be found in the ﬁle Documentation/sysrq.txt included in the kernel sources.
Some distributions allow you to install a newly compiled kernel by running
# make install
as root from within the directory that contains the kernel sources. Usually, however, you will need to change the boot loader configuration file manually and if you use lilo, you will need to run /sbin/lilo to make the new kernel available to it.
Moreover, if you use a modern distribution, you will need to generate the initrd image corresponding to the new kernel. The initrd images contain the kernel modules that should be loaded before the root ﬁlesystem is mounted as well as some scripts and utilities that should be executed before the kernel lets init run. To create the initrd image for your kernel and copy it to the system’s /boot directory, you can do:
# mkinitrd -k vmlinuz-<kernel_version> -i initrd-<kernel_version>
where <kernel_version> is the kernel’s release string. The kernel itself has to be copied to the ﬁle /boot/vmlinuz-<kernel_version> and the map of its symbols has to be saved in the ﬁle /boot/System.map-<kernel_version> before mkinitrd is run. The release string of the running kernel can be obtained by executing
$ uname -r
It usually corresponds to the version of the source code from which the kernel has been built.
After generating the initrd image for the new kernel, you have to adjust the conﬁguration of the boot loader so that it can use this kernel. If you use GRUB, the conﬁguration ﬁle should be /boot/grub/menu.lst. Open it in your favorite text editor and add the following lines:
title Linux <kernel_version> root (hdX,Y) kernel /boot/vmlinuz-<kernel_version> ro root=/dev/<root_partition> # If initrd is used initrd /boot/initrd-<kernel_version>
where X and Y are numbers used by GRUB to identify the partition that contains your system’s /boot directory (please refer to the documentation of GRUB for more details), <kernel_version> is the kernel’s release string and <root_partition> is the identiﬁcation of you root partition that will be used by the kernel to mount the root ﬁlesystem and run init.
The users of LILO should add the following lines to its conﬁguration ﬁle /etc/lilo.conf:
image=/boot/vmlinuz-<kernel_version> label=linux # If initrd is used initrd=/boot/initrd-<kernel_version> read-only root=/dev/<root_partition>
It also is necessary to make lilo actually use the new kernel by running /sbin/lilo (please refer to the documentation of LILO for more information).