Testing

Generally, there are many ways in which you can test the Linux kernel, but we will concentrate on the following four approaches:

1. Using a test version of the kernel for normal work.
2. Running special test suites, like LTP, on the new kernel.
3. Doing unusual things with the new kernel installed.
4. Measuring the system performance with the new kernel installed.

Of course, all of them can be used within one combined test procedure, so they can be regarded as different phases of the testing process.

2.1 Phase One

The first phase of kernel testing is simple: we try to boot the kernel and use it for normal work.

• Before starting the system in a fully functional configuration it is recommended to boot the kernel with the init=/bin/bash command line argument, which makes it start only one bash process. From there you can check if the filesystems are mounted and unmounted properly and you can test some more complex kernel functions, like the suspend to disk or to RAM, in the minimal configuration. In that case the only kernel modules loaded are the ones present in the initrd image mentioned in Subsection 1.6.6. Generally, you should refer to the documentation of your boot loader for more information about manual passing command line arguments to the kernel (in our opinion it is easier if GRUB is used).

• Next, it is advisable to start the system in the runlevel 2 (usually, by passing the number 2 to the kernel as the last command line argument), in which case network servers and the X server are not started (your system may be configured to use another runlevel for this purpose, although this is very unusual, so you should look into /etc/inittab for confidence). In this configuration you can check if the network interfaces work and you can try to run the X server manually to make sure that it does not crash.

• Finally, you can boot the system into the runlevel 5 (ie. fully functional) or 3 (ie. fully functional withot X), depending on your needs.

Now, you are ready to use the system in a normal way for some time. Still, if you want to test the kernel quickly, you can carry out some typical operations, like downloading some files, reading email, browsing some web pages, ripping some audio tracks (from a legally bought audio CD, we presume), burning a CD or DVD etc., in a row to check if any of them fail in a way that would indicate a kernel problem.

2.2 Phase Two (AutoTest)

In the next phase of testing we use special programs designed for checking if specific kernel subsystems work correctly. We also carry out regression and performance tests of the kernel. The latter are particularly important for kernel developers (and for us), since they allow us to identify changes that hurt performance. For example, if the performance of one of our filesystems is 10% worse after we have upgraded the 2.6.x-rc1 kernel to the 2.6.x-rc2 one, it is definitely a good idea to find the patch that causes this to happen.

For automated kernel testing we recommend you to use the AutoTest suite (http://test.kernel.org/autotest/) consisting of many test applications and profiling tools combined with a fairly simple user interface.

To install AutoTest you can go into the /usr/local directory (as root) and run

# svn checkout svn://test.kernel.org/autotest/trunk autotest

Although it normally is not recommended to run such commands as root, this particular one should be safe, unless you cannot trust your DNS server, because it only downloads some files and saves them in /usr/local . Besides, you will need to run AutoTest as root, since some of its tests require superuser privileges to complete. For this reason you should not use AutoTest on a production system: in extreme cases the data stored in the system the privileged tests are run on can be damaged or even destroyed, and we believe that you would not like this to happen to your production data.

By design, AutoTest is noninteractive, so once started, it will not require your attention (of course, if something goes really wrong, you will have to recover the system, but this is a different kettle of fish). To start it you can go to /usr/local/autotest/client (we assume that AutoTest has been installed in /usr/local) and execute (as root)

# bin/autotest tests/test_name/control

where test_name is the name of the directory in /usr/local/autotest/client/tests that contains the test you want to run. The control file tests/test_name/control contains instructions for AutoTest. In the simplest cases only one such instruction is needed, namely

job.run_test(’test_name’)

where test_name is the name of the directory that contains the control file. The contents of more sophisticated control files can look like this:

job.run_test(’pktgen’, ’eth0’, 50000, 0, tag=’clone_skb_off’)
job.run_test(’pktgen’, ’eth0’, 50000, 1, tag=’clone_skb_on’)

where the strings after the test name represent arguments that should be passed to the test application. You can modify these arguments, but first you should read the documentation of the test application as well as the script tests/test_name/test_name.py (eg. tests/pktgen/pktgen.py) used by AutoTest to actually run the test (as you have probably noticed, the AutoTest scripts are written in Python). The results of the execution of the script tests/test_name/test_name.py are saved in the directory results/default/test_name/, where the status file contains the information indicating whether or not the test has been completed successfully. To cancel the test, press Ctrl+C while it is being executed.

If you want to run several tests in a row, it is best to prepare a single file containing multiple instructions for AutoTest. The instructions in this file should be similar to the ones contained in the above-mentioned control files. For example, the file samples/all_tests contains instructions for running all of the available tests and its first five lines are the following:

job.run_test(’aiostress’)
job.run_test(’bonnie’)
job.run_test(’dbench’)
job.run_test(’fio’)
job.run_test(’fsx’)

To run all of the tests requested by the instructions in this file, you can use the command bin/autotest samples/all_tests but you should remember that it will take a lot of time to complete. Analogously, to run a custom selection of tests, put the instructions for AutoTest into one file and provide its name as a command line argument to autotest.

To run several tests in parallel, you will need to prepare a special control file containing instructions like these:

def kernbench():
          job.run_test(’kernbench’, 2, 5)
def dbench():
          job.run_test(’dbench’)
job.parallel([kernbench], [dbench])

While the tests are being executed, you can stop them by pressing Ctrl+C at any time.

For people who do not like the command line and configuration files, ATCC (AutoTest Control Center ) has been created. If you run it, for example by using the command ui/menu, you will be provided with a simple menu-driven user interface allowing you to select tests and profiling tools, view the results of their execution and, to a limited extent, configure them.

If you are bored with the selection of tools available in the AutoTest package, you can visit the web page http://ltp.sourceforge.net/tooltable.php containing a comprehensive list of tools that can be used for testing the Linux kernel.

2.3 Phase Three

Has your new kernel passed the first two phases of testing? Now, you can start to experiment. That is, to do stupid things that nobody sane will do during the normal work, so no one knows that they can crash the kernel. What exactly should be done? Well, if there had been a ”standard” procedure, it would have certainly been included in some test suite.

The third phase can be started, for example, from unplugging and replugging USB devices. While in theory the replugging of a USB device should not change anything, at least from the user’s point of view, doing it many times in a row may cause the kernel to crash if there is a bug in the USB subsystem (this may only cause the problem to appear provided that no one has ever tried this on a similarly configured system). Note, however, that this is also stressful to your hardware, so such experiments should better be carried out on add-on cards rather than on the USB ports attached directly to your computer’s mainboard.

Next, you can write a script that will read the contents of files from the /proc directory in a loop or some such. In short, in the third phase you should do things that are never done by normal users (or that are done very rarely: why would anyone mount and unmount cartain filesystem in an infinite loop? :)).

2.4 Measuring performance

As we have already mentioned, it is good to check the effects of the changes made to the kernel on the performance of the entire system (by the way, this may be an excellent task for beginner testers, who do not want to deal with development kernels yet, although they eagerly want to help develop the kernel). Still, to do this efficiently, you need to know how to do it and where to begin.

To start with, it is recommended to choose one subsystem that you will test regularly, since in that case your reports will be more valuable to the kernel developers. Namely, from time to time messages like ”Hello, I’ve noticed that the performance of my network adapter decreased substantially after I had upgraded from 2.6.8 to 2.6.20. Can anyone help me?” appear on the LKML. Of course, in such cases usually no one has a slightest idea of what could happen, because the kernel 2.6.20 was released two and a half years (and gazillion random patches) after 2.6.8. Now, in turn, if you report that the performance of your network adapter has dropped 50% between the kernels 2.6.x-rc3 and 2.6.x-rc4, it will be relatively easy to find out why. For this reason it is important to carry out the measurements of performance regularly.

Another thing that you should pay attention to is how your tests actually work. Ideally, you should learn as much as possible about the benchmark that you want to use, so that you know how to obtain reliable results from it. For example, in some Internet or press publications you can find the opinion that running

$ time make

in the kernel source directory is a good test of performance, as it allows you to measure how much time it takes to build the kernel on given system. While it is true that you can use this kind of tests to get some general idea of how ”fast” (or how ”slow”) the system is, they generally should not be regarded as measurements of performance, since they are not sufficiently precise. In particular, the kernel compilation time depends not only on the ”speed” of the CPU and memory, but also on the time needed to load the necessary data into memory from the hard disk, which in turn may depend on where exactly these data are physically located. In fact, the kernel compilation is quite I/O-intensive and the time needed to complete it may depend on some more or less random factors. Moreover, if you run it twice in a row, the first run usually takes more time to complete than the second one, since the kernel caches the necessary data in memory during the first run and afterwards they can simply be read from there. Thus in order to obtain reproducible results, it is necessary to suppress the impact of the I/O, for example by forcing the kernel to load the data into memory before running the test. Generally, if you want to carry out the ”time make” kind of benchmarks, it is best to use the kernbench script (http://ck.kolivas.org/kernbench/), a newer version of which is included in the AutoTest suite (see Section 2.2). Several good benchmarks are also available from http://ltp.sourceforge.net/tooltable.php (some of them are included in AutoTest too). Still, if you are interested in testing the kernel rather than in testing hardware, you should carefully read the documentation of the chosen benchmark, because it usually contains some information that you may need.

The next important thing that you should always remember about is the stability (ie. invariableness) of the environment in which the measurements are carried out. In particular, if you test the kernel, you should not change anything but the kernel in your system, since otherwise you would test two (or more) things at a time and it would not be easy to identify the influence of each of them on the results. For instance, if the measurement is based on building the kernel, it should always be run against the same kernel tree with exactly the same configuration file, using the same compiler and the other necessary tools (of course, once you have upgraded at least one of these tools, the results that you will obtain from this moment on should not be compared with the results obtained before the upgrade).

Generally, you should always do your best to compare apples to apples. For example, if you want to test the performance of three different file systems, you should not install them on three different partitions of the same disk, since the time needed to read (or write) data from (or to) the disk generally depends on where exactly the operation takes place. Instead, you should create one partition on which you will install each of the tested filesystems. Moreover, in such a case it is better to restart the system between consecutive measurements in order to suppress the effect of the caching of data.

Concluding, we can say that, as far as the measurements of performance are concerned, it is important to

• carry out the tests regularly
• know the ”details” allowing one to obtain reliable results
• ensure the stability of the test environment
• compare things that are directly comparable

If all of these conditions are met, the resulting data will be very valuable source of information on the performance of given kernel subsystem.