Introduction
One big problem we face in kernel development are regressions - a kernel that worked for the user and a more recent kernel no longer works for the user.
Another problem is for newbies to get into kernel development. There aren't many useful coding tasks for newbies available.
Testing the kernel and learning how to debug problems both brings immediate value to the kernel and can be a good way to get started with kernel development.
Regressions are bad since they make it harder for users to upgrade the kernel. They cause problems for people when upgrading the kernel and result in many people running old kernels with known security holes.
There are easy to find regressions like compile errors, but the real problems are things like for example: - system doesn't boot - system crashes with a specific workload - mp3 playback now stutters when copying files in the background - when doing this or that the system feels worse
Automated tests can find some of these problems, but many of the problems that affect only some specific hardware or the interactive feeling of the computer can not be found automatically.
And a helpful property of regressions is that for a reproducible regression it's relatively easy to figure out what broke it through bisection.
There are usually two months between some kernel developer adding a new regression to the kernel and the regression reaching a stable kernel, and it's important to catch as many of them as early as possible.
The next step is to also test -next and -mm kernels to identify regressions before they even reach Linus' tree.
Mailing list
http://vger.kernel.org/vger-lists.html#kernel-testers
The list is primarily for people starting at testing kernels, but if people with experiences in kernel testing and/or kernel debugging want to join that's highly appreciated.
Webpage
http://kernelnewbies.org/KernelTesters
Guide
A PDF version of the Kernel Tester Guide is linked on http://lwn.net/Articles/238838/ . The guide covers topics ranging from compiling the kernel, applying patches, additional options to enable in kernel configuration, using git/quilt/ketchup, reporting bugs and sending patches.
FAQ
Q: What will be discussed on the mailing list?
A: - which kernels to test - how to test them - how to turn observed problems into proper bugreports
Q: What are the prerequisites for participating?
A: You should already know how to build and install your own kernel (http://www.kroah.com/lkn/ is a good introduction).
And be willing to spend quite some time with debugging problems you discover.
Q: How to start?
A: Run 2.6.25 for a week. If you already did this, check what the latest snapshot listed at http://www.kernel.org/ is and try this kernel.
Q: The snapshot worked just fine for some days.
A: The kernel was less broken than expected. 
(Well, this actually should be the normal case...)
Try a more recent snapshot and follow the discussions on the mailing list what to test next.
Q: Is it dangerous to test such kernels?
A: Most of the times there are no problems.
But your system might crash, and in the worst case it might turn the contents of your disks into garbage.
But you anyway have a backup, don't you?
If you need your system to work for reaching some deadline in real life you obviously shouldn't test any kernels.
Q: I found a regression!
A: Great!
Please tell the details on the mailing list.
Q: What about bugs that are not regressions?
A: Such bugs can also be discussed, but they are not the primary target.
Tools for testing/benchmarking
What are these tools and how do we use these tools?
a. filesystem testing: fsstress, fsfuzz
b. memory testing: memtest
c. usb testing:
d. scheduler testing/benchmarking: hackbench
e. kdump
f. kprobes
g. systemtap
h. oprofile