The kernel, patches, trees and compilation
1.1 The kernel
The current version of the Linux kernel can always be downloaded from The Linux Kernel Archives web page (http://www.kernel.org/) in the form of a large tar ﬁle compressed with either gzip or bzip2, which is called the kernel tarball (it is worthy of noting that the bz2 ﬁles are smaller than the gz ones, but gzip is generally faster than bzip2).
for the ones compressed with gzip, in the directory that we want to contain the kernel sources. In principle, it can be an arbitrary directory, but it should be located on a partition with as much as 2 GB of free space.
As a result of unpacking the kernel tarball we get a directory the name of which corresponds to the version of the kernel that will be obtained by compiling the downloaded source code (eg. linux-2.6.18). The contents of this directory are often referred to as the kernel tree. For this reason the word tree is used for naming different development branches of the kernel. For instance, instead of saying the development branch of the kernel maintained by Andrew Morton it is more convenient to say the -mm tree.
It often is necessary to update the kernel source code, but you should not download the entire tarball every time, since that would be bandwidth-consuming and tedious. Usually, it is sufficient to download a patch that can be applied to an existing kernel tree from within the directory in which the tree is located. For example, if the patch in question has been compressed with bzip2, this can be done in one of the following ways:
and for uncompressed patches you can use cat instead of either zcat or bzcat. Of course, these are only the simplest methods of applying patches and it is quite easy to invent some more interesting ones.
Most of different versions of the kernel source code are available as sets of patches (patchsets) or even as individual patches that should be applied on top of specific kernel trees. For this reason the patches are labeled in reference with the kernel trees to which they should be applied. Namely, since stable kernel versions are labeled as 2.6.x or 2.6.x.y, development trees are called 2.6.x-git*, 2.6.z-rc* and 2.6.z-rc*-git*, where the -rc patch with z equal to x plus one should be applied on top of the stable kernel 2.6.x. This means, for example, that the patch called patch-2.6.21-rc3.bz2 should be applied on top of the linux-2.6.20 kernel tree and the patch labeled as patch-2.6.21-rc3-git3.bz2 should be applied on top of patch-2.6.21-rc3.bz2.
The following example will hopefully make everything clear. Assume that we have the stable kernel source code labeled as linux-22.214.171.124.tar.bz2 and we want to obtain the development tree 2.6.20-rc1-git1. First, we unpack the tarball:
Now, we have the sources of the 2.6.16 kernel in our work directory. This is what we wanted, since the stable and development patches only apply cleanly to the 2.6.x kernel trees, as they can contain some code that is also included in the 2.6.x.y versions. Accordingly, we can apply patch-2.6.17.bz2, patch-2.6.18.bz2 and patch-2.6.19.bz2 to our tree:
$ bzip2 -cd /path/to/patch-2.6.17.bz2 | patch -p1 $ bzip2 -cd /path/to/patch-2.6.18.bz2 | patch -p1 $ bzip2 -cd /path/to/patch-2.6.19.bz2 | patch -p1
Isn’t that easy? Well, it is, but it also is tedious and that is why some people who do not like to lose time use the tool called ketchup (http://www.selenic.com/ketchup/), described in the next section.
It should be noted that patches can only be applied if they are properly formatted. If you receive them via email or download them directly from a mailing list archives, the formatting is often changed in a way that makes them unusable. In such a case patch will refuse to apply the patch and you will see a message like patch: **** malformed patch at line. For instance,
$ cat ../sched-2.patch | patch -p1 -R --dry-run patching file kernel/sched.c patch: **** malformed patch at line 33: if (next == rq->idle)
means that the patch called sched-2.patch is damaged and cannot be reverted. Fortunately, if the patch has been posted to the Linux Kernel Mailing List (LKML), it can be downloaded from the archives at http://marc.theaimsgroup.com/?l=linux-kernel in the original form. For this purpose you only need to ﬁnd the message that contains the patch and click on the link Download message RAW. Still, even this will not work if the original raw message containing the patch is malformed.
The ketchup tool allows us to download any version of the kernel source code quickly and easily as well as to transform given kernel tree into another one. It does this quite intelligently, checking the current version and downloading only the required patches.
$ cd ~/bin $ wget http://www.selenic.com/ketchup/ketchup-0.9.8.tar.bz2 $ tar xjvf ketchup-0.9.8.tar.bz2
You should remember that ketchup is able to check the signatures of the downloaded patches. For this reason it is recommended to visit the web page at http://www.kernel.org/signature.html and save the current key in a text ﬁle (we have never managed to make $ gpg --keyserver wwwkeys.pgp.net --recv-keys 0x517D0F0E work on our systems). Next, you can do
Traceback (most recent call last): File "/usr/bin/ketchup", line 695, in ? lprint(get_ver("Makefile")) File "/usr/bin/ketchup", line 160, in get_ver m = open(makefile) IOError: [Errno 2] No such file or directory: ’Makefile’
$ ketchup 2.6-tip None -> 126.96.36.199 Unpacking linux-2.6.15.tar.bz2 Applying patch-2.6.16.bz2 Downloading patch-188.8.131.52.bz2 --21:11:47-- http://www.kernel.org/pub/linux/kernel/v2.6/patch-184.108.40.206.bz2 => ‘/home/michal/.ketchup/patch-220.127.116.11.bz2.partial’ Resolving www.kernel.org... 18.104.22.168, 22.214.171.124 Connecting to www.kernel.org|126.96.36.199|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 5,284 (5.2K) [application/x-bzip2] 100%[====================================>] 5,284 6.19K/s 21:11:49 (6.18 KB/s) - ‘/home/michal/.ketchup/patch-188.8.131.52.bz2.partial’ saved [5284/5284] Downloading patch-184.108.40.206.bz2.sign --21:11:49-- http://www.kernel.org/pub/linux/kernel/v2.6/ patch-220.127.116.11.bz2.sign => ‘/home/michal/.ketchup/patch-18.104.22.168.bz2.sign.partial’ Resolving www.kernel.org... 22.214.171.124, 126.96.36.199 Connecting to www.kernel.org|188.8.131.52|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 250 [application/pgp-signature] 100%[====================================>] 250 --.--K/s 21:11:49 (18.34 MB/s) - ‘/home/michal/.ketchup/ patch-184.108.40.206.bz2.sign.partial’ saved [250/250] Verifying signature... gpg: Signature made wto 28 mar 2006 09:37:31 CEST using DSA key ID 517D0F0E gpg: Good signature from "Linux Kernel Archives Verification Key <firstname.lastname@example.org>" gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: C75D C40A 11D7 AF88 9981 ED5B C86B A06A 517D 0F0E Applying patch-220.127.116.11.bz2
As you can see, ketchup automatically downloads the necessary patches. It can also take patches from the ~/.ketchup subdirectory of the user’s home directory, so it is a good idea to put the patches that you already have in there.
$ ketchup 2.6.16-rc4-mm1 18.104.22.168 -> 2.6.16-rc4-mm1 Applying patch-22.214.171.124.bz2 -R Applying patch-2.6.16.bz2 -R Applying patch-2.6.16-rc4.bz2 Applying 2.6.16-rc4-mm1.bz2
2.6.16-rc4-mm1 -> 2.6.16-rt12 Applying 2.6.16-rc4-mm1.bz2 -R Applying patch-2.6.16-rc4.bz2 -R Applying patch-2.6.16.bz2 Applying patch-2.6.16-rt12
Now, you may be wondering why we have shown you all of the commands used for applying and reverting patches. The reason is really simple: if you ﬁnd and report a bug, you will probably get a ﬁx in the form of a patch and you should be able to correctly apply it.
We have already used the term ”kernel tree” for quite a few times and we have told you that it means ”the contents of the directory containing the kernel source code”. However, it also has another meaning which is ”a development branch of the kernel source code”. Usually, such a branch is equivalent to a patchset that can be applied on top of a speciﬁc version of the kernel sources and different branches are named differently, so that they can be easily distinguished from each other. The more popular branches, also referred to as ”trees”, are the following:
Many developers and the majority of kernel subsystems maintain their own trees used in the development process. After some time speciﬁc parts of these trees are included into the main kernel tree maintained by Linus Torvalds, called the mainline. The -git patches discussed in the previous sections represent the snapshots of the ever-evolving Linus’ tree taken at various instants of time.
1.5 The -mm tree
We have already stated that the -mm tree is a combination of some other trees and individual experimental patches. More importantly, it also is the main battleﬁeld of the ﬁght against bugs in the Linux kernel.
The importance of this tree stems from the fact that it combines the mainline with multiple trees used by the maintainers of kernel subsystems (subsystem trees) including patches considered as experimental and not ready for merging with the mainline. In other words, each release of the -mm tree is sort of a future snapshot of the mainline with some additional changes and with some bugs that (hopefully) are going to be ﬁxed before the patches containing them will get merged. Thus by testing the current -mm kernel we check how the future mainline (ie. stable) kernels may behave and if any bugs are found at this stage, they can be ﬁxed before hitting the mainline.
Still, some patches reach the mainline without appearing in the -mm tree, which is a consequence of the ﬂow of changes in the kernel source code. Namely, the majority of patches merged with the mainline comes from different subsystem trees and they are included in the Linus’ tree when he decides to pull the patches considered as ready for the inclusion from given subsystem tree. It is possible, and it sometimes really happens, that Linus pulls the ”ready to go” patches from certain subsystem tree before they have a chance to appear in -mm (in that case the patches will appear in the -mm tree anyway, but they will come into it from the mainline rather than from the subsystem tree). For this reason it is not sufficient to test the -mm kernels alone and the development -rc or -git kernels should be tested as well.
On the other hand, there are kernel subsystems that do not have their own development trees and many patches are sent directly to Andrew Morton. These patches are ﬁrst included in the -mm tree and then they go either to one of the subsystem trees or to the mainline. Thus the -mm tree is a place in which the patches coming from many different sources ﬁrst meet together and are combined into an approximation of a future stable kernel that can be tested.
Since the -mm tree is a combination of many other trees, including the mainline, that always evolve, it really should be regarded as a set of patches that can be put together from time to time. When this happens, Andrew releases the so-called -mm patches to be applied on top of specific -rc or stable kernel trees. Each of them is also available as a series of individual patches combined in order to obtain it, including some ﬁxes introduced by Andrew himself. This makes the identification of patches that contain bugs easier, as they can be found using the quilt tool with the help of the bisection technique (for more information about it see Chapter 4).
In the directory containing the -mm patch and the series of individual patches it consists of (at ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.X-rcY/2.6.X-rcY-mmZ/ or at ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.X/2.6.X-mmZ/, where X, Y, Z are numbers) there always is a directory called hot-fixes containing the patches that ﬁx known bugs in this version of the -mm tree and should be applied on top of the corresponding -mm patch before testing it (the bugs that have already been identified need not be found once again).
The individual patches included in the -mm tree are sent to the mm-commits mailing list (http://vger.kernel.org/vger-lists.html#mm-commits). By following this list you can get an idea about the current contents of the -mm tree. It may also help you identify patches that introduce bugs.
From time to time Andrew sends to mm-commits a message with the subject like mm snapshot broken-out-date-time.tar.gz uploaded announcing that new -mm tree snapshot has been made available from the server. These snapshots are considered as less usable than the regular -mm patches and are to be used by people who want to help Andrew put the next -mm patch together.
1.6 Compilation and installation
1.6.1 Kernel compilation
First, your system should be prepared for the conﬁguration and compilation of the kernel. You will need gcc, make, binutils, util-linux and the package containing the ncurses header ﬁles (libncurses5-dev in Debian or ncurses-devel in Fedora Core and OpenSUSE). If you want to use a graphical tool for conﬁguring the kernel, you will also need the Qt library or GTK+ and the corresponding development packages. The ﬁle Documentation/Changes in the kernel sources directory contains the list of all programs that you may need.
Now, we assume that you have already downloaded the kernel source code and patched it to obtain the required version. To compile it, change the current directory to the one that contains the kernel sources and run the following sequence of commands:
make menuconfig (allows you to adjust the kernel conﬁguration to suit you; there are many documents on the web describing the conﬁguration of the kernel, one of them you can ﬁnd at http://www.tlug.org.za/old/guides/lkcg/lkcg_config.html)
Instead of ”make menuconfig” you can use ”make config” that will cause the kernel build system to ask you a lot of questions regarding its conﬁguration, which is inefficient and tedious, but can be done without ncurses.
If you have already conﬁgured the kernel, you can use the conﬁguration ﬁle .config from the previous compilation. For this purpose copy it to the directory containing the current kernel sources and run ”make oldconfig” (it works like ”make config”, but does not ask so many questions). Alternatively, you can use the conﬁguration of the distribution kernel (usually it contains many things you will never need and that is one of the reasons why you may want to learn how to conﬁgure the kernel):
1.6.2 Useful make options
make O=/directory/ (saves the result of the compilation in given directory; this is particularly useful if you do not want to litter the kernel source directory with *.o files, but in that case you should also use ”make O=/directory/ menuconfig” instead of ”make menuconfig” and ”make O=/katalog/ modules_install” etc.)
make CC=<compiler> (allows you to specify the name of the compiler to use; for example, in Debian there are several versions of gcc, and if you want to use gcc-3.4 to compile the kernel it is sufficient to use ”make CC=gcc-3.4”)
1.6.3 Kernel modules
To compile a piece of the kernel code as a module, you need to mark it with M during the conﬁguration, eg. <M> Check for non-fatal errors on AMD Athlon/Duron / Intel Pentium 4 If the ”Automatic kernel module loading” option has been selected, the modules should be automatically loaded by the kernel when they are needed. That is, for example, if you want to mount a CD-ROM, the kernel should automatically load the module isofs.ko and this modules can be removed from memory after the CD-ROM has been unmounted. You can also load and unload kernel modules manually, using the commands modprobe, insmod or rmmod, eg.
1.6.4 Kernel hacking options
[*] Debug shared IRQ handlers [*] Detect Soft Lockups [*] Debug slab memory allocations [*] Slab memory leak debugging [*] RT Mutex debugging, deadlock detection [*] Built-in scriptable tester for rt-mutexes [*] Lock debugging: prove locking correctness [*] Lock dependency engine debugging [*] Locking API boot-time self-tests [*] Highmem debugging [*] Debug VM
1.6.5 Magic SysRq
The option [*] Magic SysRq key is particularly useful, because it often allows you to reduce the impact of a kernel failure and to obtain some additional information about the state of the system. It turns on some keyboard shortcuts allowing you to pass speciﬁc instructions directly to the kernel:
Which key is the SysRq? That depends on the architecture of your machine:
press Alt + SysRq + k to kill all processes using the current console,
as root from within the directory that contains the kernel sources. Usually, however, you will need to change the boot loader configuration file manually and if you use lilo, you will need to run /sbin/lilo to make the new kernel available to it.
Moreover, if you use a modern distribution, you will need to generate the initrd image corresponding to the new kernel. The initrd images contain the kernel modules that should be loaded before the root ﬁlesystem is mounted as well as some scripts and utilities that should be executed before the kernel lets init run. To create the initrd image for your kernel and copy it to the system’s /boot directory, you can do:
where <kernel_version> is the kernel’s release string. The kernel itself has to be copied to the ﬁle /boot/vmlinuz-<kernel_version> and the map of its symbols has to be saved in the ﬁle /boot/System.map-<kernel_version> before mkinitrd is run. The release string of the running kernel can be obtained by executing
After generating the initrd image for the new kernel, you have to adjust the conﬁguration of the boot loader so that it can use this kernel. If you use GRUB, the conﬁguration ﬁle should be /boot/grub/menu.lst. Open it in your favorite text editor and add the following lines:
title Linux <kernel_version> root (hdX,Y) kernel /boot/vmlinuz-<kernel_version> ro root=/dev/<root_partition> # If initrd is used initrd /boot/initrd-<kernel_version>
where X and Y are numbers used by GRUB to identify the partition that contains your system’s /boot directory (please refer to the documentation of GRUB for more details), <kernel_version> is the kernel’s release string and <root_partition> is the identification of you root partition that will be used by the kernel to mount the root ﬁlesystem and run init.
image=/boot/vmlinuz-<kernel_version> label=linux # If initrd is used initrd=/boot/initrd-<kernel_version> read-only root=/dev/<root_partition>
1.6.7 Automated conﬁguration and installation
Some of the above steps or even all of them may be completed by the kernel installation script available in your distribution. Still, very often they have to be done manually. In such a case you can use the following script, which allows you to download, compile and install the latest stable kernel, after adjusting it to suit your system conﬁguration:
#! /bin/sh # Patch to the kernel source directory SRC_PATH="/usr/src/kernel/linux-stable" OBJ_PATH="$SRC_PATH-obj/" cd $SRC_PATH # Download the latest -stable ketchup 2.6 # Save the version VER=‘ketchup -m‘ # Generate the configuration file (based on the old one) make O=$OBJ_PATH oldconfig # Build the kernel make O=$OBJ_PATH # Install modules sudo make O=$OBJ_PATH modules_install # Copy the compressed kernel image into /boot sudo cp $OBJ_PATH/arch/i386/boot/bzImage /boot/vmlinuz-$VER # Copy System.map into /boot sudo cp $OBJ_PATH/System.map /boot/System.map-$VER # If you use Fedory, you have to generate the Fedora initrd sudo /sbin/new-kernel-pkg --make-default --mkinitrd --depmod --install $VER
Generally, each distribution has its own preferred method of building and installing new kernels. The following table contains the links to the kernel installation instructions for several selected distributions:
To learn more about the conﬁguration and compilation of the Linux kernel you can refer to the book Linux Kernel in a Nutshell by Greg Kroah-Hartman (http://www.kroah.com/lkn/).