KernelNewbies:

The kernel, patches, trees and compilation

1.1 The kernel

The current version of the Linux kernel can always be downloaded from The Linux Kernel Archives web page (http://www.kernel.org/) in the form of a large tar file compressed with either gzip or xz, which is called the kernel tarball (it is worthy of noting that the xz files are smaller than the gz ones).

After downloading the tarball you can extract the kernel’s source code from it in many different ways (the virtue of UNIX-like systems). Our favorite method is to run either

$ tar xJvf linux-2.6.x.y.tar.xz

for the tarballs compressed with xz, or

$ tar xzvf linux-2.6.x.y.tar.gz

for the ones compressed with gzip, in the directory that we want to contain the kernel sources. In principle, it can be an arbitrary directory, but it should be located on a partition with as much as 2 GB of free space.

Of course, you can also follow the kernel’s README literally and extract the source code by executing one of the following instructions: 

$ xz -dc linux-2.6.x.y.tar.bz2 | tar xvf -
$ gzip -cd linux-2.6.x.y.tar.gz | tar xvf -

depending on what kind of a tarball you have downloaded. In any case it is not recommended to unpack the kernel tarball as root.

As a result of unpacking the kernel tarball we get a directory the name of which corresponds to the version of the kernel that will be obtained by compiling the downloaded source code (e.g. linux-2.6.18). The contents of this directory are often referred to as the kernel tree. For this reason the word tree is used for naming different development branches of the kernel. For instance, instead of saying the development branch of the kernel maintained by Andrew Morton it is more convenient to say the -mm tree.

1.2 Patches

It often is necessary to update the kernel source code, but you should not download the entire tarball every time, since that would be bandwidth-consuming and tedious. Usually, it is sufficient to download a patch that can be applied to an existing kernel tree from within the directory in which the tree is located. For example, if the patch in question has been compressed with xz, this can be done in one of the following ways: 

$ xz -cd /path/to/patch-2.6.x.bz2 | patch -p1
$ xzcat /path/to/patch-2.6.x.bz2 | patch -p1

If the patch has been compressed with gzip, you can apply it analogously: 

$ gzip -cd /path/to/patch-2.6.x.gz | patch -p1
$ zcat /path/to/patch-2.6.x.gz | patch -p1

and for uncompressed patches you can use cat instead of either zcat or bzcat. Of course, these are only the simplest methods of applying patches and it is quite easy to invent some more interesting ones.

It is very useful to be able to revert patches that are no longer needed (or buggy). It can be accomplished by adding the -R flag to the patch command line:

$ zcat /path/to/patch-2.6.x.gz | patch -p1 -R

It is also possible to check if the patch will apply cleanly (i.e. it does not contain any pieces of code that will be rejected) with the help of the --dry-run command-line option of patch:

$ zcat /path/to/patch-2.6.x.gz | patch -p1 --dry-run

Then, if there are no messages similar to

1 out of 1 hunk FAILED -- saving rejects to file xxx

the patch can be safely applied.

Most of different versions of the kernel source code are available as sets of patches (patchsets) or even as individual patches that should be applied on top of specific kernel trees. For this reason the patches are labeled in reference with the kernel trees to which they should be applied. Namely, since stable kernel versions are labeled as 2.6.x or 2.6.x.y, development trees are called 2.6.x-git*, 2.6.z-rc* and 2.6.z-rc*-git*, where the -rc patch with z equal to x plus one should be applied on top of the stable kernel 2.6.x. This means, for example, that the patch called patch-2.6.21-rc3.gz should be applied on top of the linux-2.6.20 kernel tree and the patch labeled as patch-2.6.21-rc3-git3.gz should be applied on top of patch-2.6.21-rc3.gz.

The following example will hopefully make everything clear. Assume that we have the stable kernel source code labeled as linux-2.6.16.25.tar.xz and we want to obtain the development tree 2.6.20-rc1-git1. First, we unpack the tarball:

$ tar xJvf linux-2.6.16.25.tar.xz

Next, we fetch the patch called patch-2.6.16.25.xz from http://www.kernel.org and revert it:

$ xz -cd /path/to/patch-2.6.16.25.xz | patch -p1 -R

Now, we have the sources of the 2.6.16 kernel in our work directory. This is what we wanted, since the stable and development patches only apply cleanly to the 2.6.x kernel trees, as they can contain some code that is also included in the 2.6.x.y versions. Accordingly, we can apply patch-2.6.17.xz, patch-2.6.18.xz and patch-2.6.19.xz to our tree: 

$ xz -cd /path/to/patch-2.6.17.bz2 | patch -p1
$ xz -cd /path/to/patch-2.6.18.bz2 | patch -p1
$ xz -cd /path/to/patch-2.6.19.bz2 | patch -p1

to obtain the 2.6.19 kernel source code. Finally, we need to download two development patches, patch-2.6.20-rc1.xz and patch-2.6.20-rc1-git1.xz, and apply them in the order specified.

Isn’t that easy? Well, it is, but it also is tedious and that is why some people who do not like to lose time use the tool called ketchup (http://www.selenic.com/ketchup/), described in the next section.

It should be noted that patches can only be applied if they are properly formatted. If you receive them via email or download them directly from a mailing list archives, the formatting is often changed in a way that makes them unusable. In such a case patch will refuse to apply the patch and you will see a message like patch: **** malformed patch at line. For instance, 

$ cat ../sched-2.patch | patch -p1 -R --dry-run
patching file kernel/sched.c
patch: **** malformed patch at line 33:                     if (next == rq->idle)

means that the patch called sched-2.patch is damaged and cannot be reverted. Fortunately, if the patch has been posted to the Linux Kernel Mailing List (LKML), it can be downloaded from the archives at http://marc.info/?l=linux-kernel or https://lore.kernel.org/lkml/ in the original form. For this purpose you only need to find the message that contains the patch and click on the link Download message RAW (or just raw). Still, even this will not work if the original raw message containing the patch is malformed.

For more information on downloading and applying patches please see the files README and Documentation/process/applying-patches.rst in the kernel sources.

1.3 Ketchup

The ketchup tool allows us to download any version of the kernel source code quickly and easily as well as to transform given kernel tree into another one. It does this quite intelligently, checking the current version and downloading only the required patches.

The users of Debian testing/unstable can simply install the ketchup package with the help of their favorite tool, eg.

# apt-get install ketchup

Still, for the users of Debian stable or another distribution in which the ketchup tool is not available the easiest way of installing it is to do:

$ mkdir ~/bin

unless the subdirectory bin is already present in the user’s home directory, and 

$ cd ~/bin
$ wget http://www.selenic.com/ketchup/ketchup-0.9.8.tar.bz2
$ tar xjvf ketchup-0.9.8.tar.bz2

It is also necessary to install the python package.

You should remember that ketchup is able to check the signatures of the downloaded patches. For this reason it is recommended to visit the web page at http://www.kernel.org/signature.html and save the current key in a text file (we have never managed to make $ gpg --keyserver wwwkeys.pgp.net --recv-keys 0x517D0F0E work on our systems). Next, you can do

$ gpg --import the_file_containing_the_kernel.org_key

and you are ready to check how the whole thing works. Namely, you can do 

$ mkdir ~/tree
$ cd ~/tree
$ ketchup -m

and you should get an error message similar to the following one: 

Traceback (most recent call last):
  File "/usr/bin/ketchup", line 695, in ?
     lprint(get_ver("Makefile"))
   File "/usr/bin/ketchup", line 160, in get_ver
      m = open(makefile)
 IOError: [Errno 2] No such file or directory: ’Makefile’

This only means that there is no Linux kernel tree in the current directory. You can order ketchup to download the latest stable version of the kernel by using the command ”ketchup 2.6-tip”: 

$ ketchup 2.6-tip
None -> 2.6.16.1
Unpacking linux-2.6.15.tar.bz2
Applying patch-2.6.16.bz2
Downloading patch-2.6.16.1.bz2
--21:11:47-- http://www.kernel.org/pub/linux/kernel/v2.6/patch-2.6.16.1.bz2
           => ‘/home/michal/.ketchup/patch-2.6.16.1.bz2.partial’
Resolving www.kernel.org... 204.152.191.5, 204.152.191.37
Connecting to www.kernel.org|204.152.191.5|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5,284 (5.2K) [application/x-bzip2]
100%[====================================>] 5,284          6.19K/s
21:11:49 (6.18 KB/s) - ‘/home/michal/.ketchup/patch-2.6.16.1.bz2.partial’
saved [5284/5284]
Downloading patch-2.6.16.1.bz2.sign
--21:11:49-- http://www.kernel.org/pub/linux/kernel/v2.6/
patch-2.6.16.1.bz2.sign
           => ‘/home/michal/.ketchup/patch-2.6.16.1.bz2.sign.partial’
Resolving www.kernel.org... 204.152.191.37, 204.152.191.5
Connecting to www.kernel.org|204.152.191.37|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 250 [application/pgp-signature]
100%[====================================>] 250           --.--K/s
21:11:49 (18.34 MB/s) - ‘/home/michal/.ketchup/
patch-2.6.16.1.bz2.sign.partial’ saved [250/250]
Verifying signature...
gpg: Signature made wto 28 mar 2006 09:37:31 CEST using DSA key ID 517D0F0E
gpg: Good signature from "Linux Kernel Archives Verification Key
 <ftpadmin@kernel.org>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: C75D C40A 11D7 AF88 9981 ED5B C86B A06A 517D 0F0E
Applying patch-2.6.16.1.bz2

Now you can check if you really have the latest stable tree: 

$ ketchup -m
2.6.16.1

As you can see, ketchup automatically downloads the necessary patches. It can also take patches from the ~/.ketchup subdirectory of the user’s home directory, so it is a good idea to put the patches that you already have in there.

Making a transition to another kernel tree is absolutely not a problem for ketchup. For this purpose you can simply use the command ”ketchup kernel_version”, eg. 

$ ketchup 2.6.16-rc4-mm1
2.6.16.1 -> 2.6.16-rc4-mm1
Applying patch-2.6.16.1.bz2 -R
Applying patch-2.6.16.bz2 -R
Applying patch-2.6.16-rc4.bz2
Applying 2.6.16-rc4-mm1.bz2

Similarly, if you want to get the latest kernel version from the -rt branch, you can do

$ ketchup 2.6-rt

and the tool will automatically apply or revert the patches for you: 

2.6.16-rc4-mm1 -> 2.6.16-rt12
Applying 2.6.16-rc4-mm1.bz2 -R
Applying patch-2.6.16-rc4.bz2 -R
Applying patch-2.6.16.bz2
Applying patch-2.6.16-rt12

It is also possible to teach ketchup to use patches included in the latest -mm tree (described below), which can be done in the following way:

$ wget -c ‘ketchup -u 2.6-mm | sed "s/.bz2/-broken-out.tar.bz2/"‘

As follows from the above examples, ketchup can recognize several development branches of the Linux kernel source code. In particular, it can handle the following trees:

Now, you may be wondering why we have shown you all of the commands used for applying and reverting patches. The reason is really simple: if you find and report a bug, you will probably get a fix in the form of a patch and you should be able to correctly apply it.

1.4 Trees

We have already used the term ”kernel tree” for quite a few times and we have told you that it means ”the contents of the directory containing the kernel source code”. However, it also has another meaning which is ”a development branch of the kernel source code”. Usually, such a branch is equivalent to a patchset that can be applied on top of a specific version of the kernel sources and different branches are named differently, so that they can be easily distinguished from each other. The more popular branches, also referred to as ”trees”, are the following:

Many developers and the majority of kernel subsystems maintain their own trees used in the development process. After some time specific parts of these trees are included into the main kernel tree maintained by Linus Torvalds, called the mainline. The -git patches discussed in the previous sections represent the snapshots of the ever-evolving Linus’ tree taken at various instants of time.

1.5 The -mm tree

We have already stated that the -mm tree is a combination of some other trees and individual experimental patches. More importantly, it also is the main battlefield of the fight against bugs in the Linux kernel.

The importance of this tree stems from the fact that it combines the mainline with multiple trees used by the maintainers of kernel subsystems (subsystem trees) including patches considered as experimental and not ready for merging with the mainline. In other words, each release of the -mm tree is sort of a future snapshot of the mainline with some additional changes and with some bugs that (hopefully) are going to be fixed before the patches containing them will get merged. Thus by testing the current -mm kernel we check how the future mainline (ie. stable) kernels may behave and if any bugs are found at this stage, they can be fixed before hitting the mainline.

Still, some patches reach the mainline without appearing in the -mm tree, which is a consequence of the flow of changes in the kernel source code. Namely, the majority of patches merged with the mainline comes from different subsystem trees and they are included in the Linus’ tree when he decides to pull the patches considered as ready for the inclusion from given subsystem tree. It is possible, and it sometimes really happens, that Linus pulls the ”ready to go” patches from certain subsystem tree before they have a chance to appear in -mm (in that case the patches will appear in the -mm tree anyway, but they will come into it from the mainline rather than from the subsystem tree). For this reason it is not sufficient to test the -mm kernels alone and the development -rc or -git kernels should be tested as well.

On the other hand, there are kernel subsystems that do not have their own development trees and many patches are sent directly to Andrew Morton. These patches are first included in the -mm tree and then they go either to one of the subsystem trees or to the mainline. Thus the -mm tree is a place in which the patches coming from many different sources first meet together and are combined into an approximation of a future stable kernel that can be tested.

Since the -mm tree is a combination of many other trees, including the mainline, that always evolve, it really should be regarded as a set of patches that can be put together from time to time. When this happens, Andrew releases the so-called -mm patches to be applied on top of specific -rc or stable kernel trees. Each of them is also available as a series of individual patches combined in order to obtain it, including some fixes introduced by Andrew himself. This makes the identification of patches that contain bugs easier, as they can be found using the quilt tool with the help of the bisection technique (for more information about it see Chapter 4).

The individual patches included in the -mm tree are sent to the mm-commits mailing list (http://vger.kernel.org/vger-lists.html#mm-commits). By following this list you can get an idea about the current contents of the -mm tree. It may also help you identify patches that introduce bugs.

From time to time Andrew sends to mm-commits a message with the subject like mm snapshot broken-out-date-time.tar.gz uploaded announcing that new -mm tree snapshot has been made available from the server. These snapshots are considered as less usable than the regular -mm patches and are to be used by people who want to help Andrew put the next -mm patch together. The -mm (or mmotm: mm of the moment) snapshots are found at https://www.ozlabs.org/~akpm/mmotm/.

1.6 Compilation and installation

1.6.1 Kernel compilation

First, your system should be prepared for the configuration and compilation of the kernel. You will need gcc, make, binutils, util-linux and the package containing the ncurses header files (libncurses5-dev in Debian or ncurses-devel in Fedora Core and OpenSUSE). If you want to use a graphical tool for configuring the kernel, you will also need the Qt library or GTK+ and the corresponding development packages. The file Documentation/Changes in the kernel sources directory contains the list of all programs that you may need.

Now, we assume that you have already downloaded the kernel source code and patched it to obtain the required version. To compile it, change the current directory to the one that contains the kernel sources and run the following sequence of commands:

Instead of ”make menuconfig” you can use ”make config” that will cause the kernel build system to ask you a lot of questions regarding its configuration, which is inefficient and tedious, but can be done without ncurses.

If you have already configured the kernel, you can use the configuration file .config from the previous compilation. For this purpose copy it to the directory containing the current kernel sources and run ”make oldconfig” (it works like ”make config”, but does not ask so many questions). Alternatively, you can use the configuration of the distribution kernel (usually it contains many things you will never need and that is one of the reasons why you may want to learn how to configure the kernel): 

$ zcat /proc/config.gz > .config
$ make oldconfig

1.6.2 Useful make options

There are some useful options you can pass to make during the kernel compilation:

1.6.3 Kernel modules

If you want to use kernel modules, in which case some parts of the kernel will be loaded into memory only if they are needed, select the options: 

Loadable module support --->
[*] Enable loadable module support
[*]     Automatic kernel module loading

The modularization allows the kernel to be smaller and work faster.

To compile a piece of the kernel code as a module, you need to mark it with M during the configuration, eg. <M>     Check for non-fatal errors on AMD Athlon/Duron / Intel Pentium 4 If the ”Automatic kernel module loading” option has been selected, the modules should be automatically loaded by the kernel when they are needed. That is, for example, if you want to mount a CD-ROM, the kernel should automatically load the module isofs.ko and this modules can be removed from memory after the CD-ROM has been unmounted. You can also load and unload kernel modules manually, using the commands modprobe, insmod or rmmod, eg.

All of the modules available to the currently running kernel should be located in various subdirectories of /lib/modules/<kernel_version>/kernel .

1.6.4 Kernel hacking options

If you build the kernel to test it, you should seriously consider setting the options located in the ”Kernel hacking” menu. At least, you should set: 

[*] Kernel debugging
[*]     Compile the kernel with debug info

The other options should also be set, if possible, because they help debug specific functionalities of the kernel, but some of them hurt performance, so you may want to leave them unset: 

[*]     Debug shared IRQ handlers
[*]     Detect Soft Lockups
[*]     Debug slab memory allocations
[*]        Slab memory leak debugging
[*]     RT Mutex debugging, deadlock detection
[*]     Built-in scriptable tester for rt-mutexes
[*]     Lock debugging: prove locking correctness
[*]     Lock dependency engine debugging
[*]     Locking API boot-time self-tests
[*]     Highmem debugging
[*]     Debug VM

1.6.5 Magic SysRq

The option [*] Magic SysRq key is particularly useful, because it often allows you to reduce the impact of a kernel failure and to obtain some additional information about the state of the system. It turns on some keyboard shortcuts allowing you to pass specific instructions directly to the kernel:

Which key is the SysRq? That depends on the architecture of your machine:

On all architectures you can trigger the action assigned to given key (say t) by echoing the corresponding character to the file /proc/sysrq-trigger, eg. # echo t > /proc/sysrq-trigger

What is the safest way to restart the computer?

First, you should try to press the combination of Ctrl + Alt + Backspace (X Window system) or Ctrl + Alt + Del (text console). If it does not work, you can:

More information related to the ”Magic SysRq key” mechanism can be found in the file Documentation/admin-guide/sysrq.rst included in the kernel sources.

1.6.6 Installation

Some distributions allow you to install a newly compiled kernel by running

# make install

as root from within the directory that contains the kernel sources. Usually, however, you will need to change the boot loader configuration file manually and if you use lilo, you will need to run /sbin/lilo to make the new kernel available to it.

Moreover, if you use a modern distribution, you will need to generate the initrd image corresponding to the new kernel. The initrd images contain the kernel modules that should be loaded before the root filesystem is mounted as well as some scripts and utilities that should be executed before the kernel lets init run. To create the initrd image for your kernel and copy it to the system’s /boot directory, you can do:

# mkinitrd -k vmlinuz-<kernel_version> -i initrd-<kernel_version>

where <kernel_version> is the kernel’s release string. The kernel itself has to be copied to the file /boot/vmlinuz-<kernel_version> and the map of its symbols has to be saved in the file /boot/System.map-<kernel_version> before mkinitrd is run. The release string of the running kernel can be obtained by executing

$ uname -r

It usually corresponds to the version of the source code from which the kernel has been built.

After generating the initrd image for the new kernel, you have to adjust the configuration of the boot loader so that it can use this kernel. If you use GRUB, the configuration file should be /boot/grub/menu.lst. Open it in your favorite text editor and add the following lines: 

title Linux <kernel_version>
         root (hdX,Y)
         kernel /boot/vmlinuz-<kernel_version> ro root=/dev/<root_partition>
#        If initrd is used
         initrd /boot/initrd-<kernel_version>

where X and Y are numbers used by GRUB to identify the partition that contains your system’s /boot directory (please refer to the documentation of GRUB for more details), <kernel_version> is the kernel’s release string and <root_partition> is the identification of you root partition that will be used by the kernel to mount the root filesystem and run init.

The users of LILO should add the following lines to its configuration file /etc/lilo.conf: 

image=/boot/vmlinuz-<kernel_version>
label=linux
# If initrd is used
initrd=/boot/initrd-<kernel_version>
read-only
root=/dev/<root_partition>

It also is necessary to make lilo actually use the new kernel by running /sbin/lilo (please refer to the documentation of LILO for more information).

1.6.7 Automated configuration and installation

Some of the above steps or even all of them may be completed by the kernel installation script available in your distribution. Still, very often they have to be done manually. In such a case you can use the following script, which allows you to download, compile and install the latest stable kernel, after adjusting it to suit your system configuration: 

# Patch to the kernel source directory
SRC_PATH="/usr/src/kernel/linux-stable"
OBJ_PATH="$SRC_PATH-obj/"
cd $SRC_PATH
# Download the latest -stable
ketchup 2.6
# Save the version
VER=‘ketchup -m‘
# Generate the configuration file (based on the old one)
make O=$OBJ_PATH oldconfig
# Build the kernel
make O=$OBJ_PATH
# Install modules
sudo make O=$OBJ_PATH modules_install
# Copy the compressed kernel image into /boot
sudo cp $OBJ_PATH/arch/i386/boot/bzImage /boot/vmlinuz-$VER
# Copy System.map into /boot
sudo cp $OBJ_PATH/System.map /boot/System.map-$VER
# If you use Fedory, you have to generate the Fedora initrd
sudo /sbin/new-kernel-pkg --make-default --mkinitrd --depmod --install $VER

Generally, each distribution has its own preferred method of building and installing new kernels. The following table contains the links to the kernel installation instructions for several selected distributions:

To learn more about the configuration and compilation of the Linux kernel you can refer to the book Linux Kernel in a Nutshell by Greg Kroah-Hartman (http://www.kroah.com/lkn/).

KernelNewbies: Linux_Kernel_Tester's_Guide_Chapter1 (last edited 2021-01-14 03:05:47 by RandyDunlap)