Testing
Generally, there are many ways in which you can test the Linux kernel, but we will concentrate on the following four approaches:
- Using a test version of the kernel for normal work.
- Running special test suites, like LTP, on the new kernel.
- Doing unusual things with the new kernel installed.
- Measuring the system performance with the new kernel installed.
Of course, all of them can be used within one combined test procedure, so they can be regarded as different phases of the testing process.
2.1 Phase One
The first phase of kernel testing is simple: we try to boot the kernel and use it for normal work.
- Before starting the system in a fully functional configuration it is recommended to boot the kernel with the init=/bin/bash command line argument, which makes it start only one bash process. From there you can check if the filesystems are mounted and unmounted properly and you can test some more complex kernel functions, like the suspend to disk or to RAM, in the minimal configuration. In that case the only kernel modules loaded are the ones present in the initrd image mentioned in Subsection 1.6.6. Generally, you should refer to the documentation of your boot loader for more information about manual passing command line arguments to the kernel (in our opinion it is easier if GRUB is used).
Next, it is advisable to start the system in the runlevel 2 (usually, by passing the number 2 to the kernel as the last command line argument), in which case network servers and the X server are not started (your system may be configured to use another runlevel for this purpose, although this is very unusual, so you should look into /etc/inittab for confidence). In this configuration you can check if the network interfaces work and you can try to run the X server manually to make sure that it does not crash.
- Finally, you can boot the system into the runlevel 5 (ie. fully functional) or 3 (ie. fully functional withot X), depending on your needs.
Now, you are ready to use the system in a normal way for some time. Still, if you want to test the kernel quickly, you can carry out some typical operations, like downloading some files, reading email, browsing some web pages, ripping some audio tracks (from a legally bought audio CD, we presume), burning a CD or DVD etc., in a row to check if any of them fail in a way that would indicate a kernel problem.
2.2 Phase Two (AutoTest)
In the next phase of testing we use special programs designed for checking if specific kernel subsystems work correctly. We also carry out regression and performance tests of the kernel. The latter are particularly important for kernel developers (and for us), since they allow us to identify changes that hurt performance. For example, if the performance of one of our filesystems is 10% worse after we have upgraded the 2.6.x-rc1 kernel to the 2.6.x-rc2 one, it is definitely a good idea to find the patch that causes this to happen.
For automated kernel testing we recommend you to use the AutoTest suite (http://test.kernel.org/autotest/) consisting of many test applications and profiling tools combined with a fairly simple user interface.
To install AutoTest you can go into the /usr/local directory (as root) and run
# svn checkout svn://test.kernel.org/autotest/trunk autotest
Although it normally is not recommended to run such commands as root, this particular one should be safe, unless you cannot trust your DNS server, because it only downloads some files and saves them in /usr/local . Besides, you will need to run AutoTest as root, since some of its tests require superuser privileges to complete. For this reason you should not use AutoTest on a production system: in extreme cases the data stored in the system the privileged tests are run on can be damaged or even destroyed, and we believe that you would not like this to happen to your production data.
By design, AutoTest is noninteractive, so once started, it will not require your attention (of course, if something goes really wrong, you will have to recover the system, but this is a different kettle of fish). To start it you can go to /usr/local/autotest/client (we assume that AutoTest has been installed in /usr/local) and execute (as root)
# bin/autotest tests/test_name/control
where test_name is the name of the directory in /usr/local/autotest/client/tests that contains the test you want to run. The control file tests/test_name/control contains instructions for AutoTest. In the simplest cases only one such instruction is needed, namely
job.run_test(’test_name’)
where test_name is the name of the directory that contains the control file. The contents of more sophisticated control files can look like this:
job.run_test(’pktgen’, ’eth0’, 50000, 0, tag=’clone_skb_off’) job.run_test(’pktgen’, ’eth0’, 50000, 1, tag=’clone_skb_on’)
where the strings after the test name represent arguments that should be passed to the test application. You can modify these arguments, but first you should read the documentation of the test application as well as the script tests/test_name/test_name.py (eg. tests/pktgen/pktgen.py) used by AutoTest to actually run the test (as you have probably noticed, the AutoTest scripts are written in Python). The results of the execution of the script tests/test_name/test_name.py are saved in the directory results/default/test_name/, where the status file contains the information indicating whether or not the test has been completed successfully. To cancel the test, press Ctrl+C while it is being executed.
If you want to run several tests in a row, it is best to prepare a single file containing multiple instructions for AutoTest. The instructions in this file should be similar to the ones contained in the above-mentioned control files. For example, the file samples/all_tests contains instructions for running all of the available tests and its first five lines are the following:
job.run_test(’aiostress’) job.run_test(’bonnie’) job.run_test(’dbench’) job.run_test(’fio’) job.run_test(’fsx’)
To run all of the tests requested by the instructions in this file, you can use the command bin/autotest samples/all_tests but you should remember that it will take a lot of time to complete. Analogously, to run a custom selection of tests, put the instructions for AutoTest into one file and provide its name as a command line argument to autotest.
To run several tests in parallel, you will need to prepare a special control file containing instructions like these:
def kernbench():
job.run_test(’kernbench’, 2, 5)
def dbench():
job.run_test(’dbench’)
job.parallel([kernbench], [dbench])While the tests are being executed, you can stop them by pressing Ctrl+C at any time.
For people who do not like the command line and configuration files, ATCC (AutoTest Control Center ) has been created. If you run it, for example by using the command ui/menu, you will be provided with a simple menu-driven user interface allowing you to select tests and profiling tools, view the results of their execution and, to a limited extent, configure them.
If you are bored with the selection of tools available in the AutoTest package, you can visit the web page http://ltp.sourceforge.net/tooltable.php containing a comprehensive list of tools that can be used for testing the Linux kernel.
2.3 Phase Three
Has your new kernel passed the first two phases of testing? Now, you can start to experiment. That is, to do stupid things that nobody sane will do during the normal work, so no one knows that they can crash the kernel. What exactly should be done? Well, if there had been a ”standard” procedure, it would have certainly been included in some test suite.
The third phase can be started, for example, from unplugging and replugging USB devices. While in theory the replugging of a USB device should not change anything, at least from the user’s point of view, doing it many times in a row may cause the kernel to crash if there is a bug in the USB subsystem (this may only cause the problem to appear provided that no one has ever tried this on a similarly configured system). Note, however, that this is also stressful to your hardware, so such experiments should better be carried out on add-on cards rather than on the USB ports attached directly to your computer’s mainboard.
Next, you can write a script that will read the contents of files from the /proc directory in a loop or some such. In short, in the third phase you should do things that are never done by normal users (or that are done very rarely: why would anyone mount and unmount cartain filesystem in an infinite loop? :)).