KernelNewbies:

Intro

This tutorial will cover how to get your first patch submitted. You have three choices for how to complete this tutorial:

  1. Run Linux in VMPlayer from Windows.
  2. Run Linux natively on your own machine.
  3. Run Linux within VMPlayer on Linux.
  4. Run Linux within Virt-Manager/Libvirt on Linux.
  5. Run Linux within Qemu on Linux.

We recommend running Linux natively. Most Linux kernel developers run Linux natively, so you may as well get used to it. :) If you want to run Linux in VMPlayer, follow these directions. Note, you will not be able to compile the Linux kernel on a Mac, because the filesystem defaults to case-insensitive. For running Linux in Virt-Manager/Libvirt follow OutreachyfirstpatchAlt_libvirtvm_draft and for Qemu, follow OutreachyfirstpatchAlt_qemu_draft.

This tutorial assumes you are running Ubuntu or Debian. If you are running Fedora, Suse, Arch, or Gentoo, the package installation commands or package names may be slightly different. Ask for help on #kernelnewbies on irc.oftc.net if you get stuck. If (and only if) you are an applicant for Outreachy, then you should ask for help on #kernel-outreachy on irc.oftc.net.

Additionally, we highly recommend that applicants have a stable internet connection, with no download caps. Communication over IRC can be difficult if your internet connection keeps dropping or has a big lag time, so you need a stable internet connection. Downloading the initial kernel will use over 5 GB of data, which will easily blow through a standard 3G capped plan. We recommend making sure you have cable internet, or an unlimited 3G plan.

Overview

This tutorial will show you how to:

Setup your tools

You'll need to install, configure, and download some software to get started. You should follow the setup directions here.

Setup vim

(Note, if you're running a native Linux install and you're used to another editor like emacs or nano, you can still use that editor, and you can skip this step. You may not be able to use gvim.)

First, we need to make sure to enable the C indentation module in our default text editor (vim). Turning on this module will ensure that lines automatically get indented to the right level as you're editing. It saves you from hitting <tab> a lot. You can turn on automatic indentation based on the file type. First run: 

vim ~/.vimrc

Then add this line: 

filetype plugin indent on

You'll also want to add a couple more lines, to turn syntax highlighting on, and show the file name in the terminal title bar: 

syntax on
set title

Most distributions compile vim so that 8 space tabs are the default. If you find they're not the default, you will need to add the following line to your .vimrc: 

set tabstop=8
set softtabstop=8
set shiftwidth=8
set noexpandtab

That's equivalent to running this command in vim: 

:set tabstop=8 softtabstop=8 shiftwidth=8 noexpandtab

Setup vim as your default editor

Next, we'll need to set up mutt to use vim as the default editor, instead of nano. Run: 

sudo update-alternatives --config editor

and chose /usr/bin/vim.basic as the default editor.

Set up email

To be able to send Linux kernel patches, you'll need to be able to send email from the Linux VM image (or your computer that is natively running Linux). The VM image comes installed with esmtp, and if you were following the native Linux install directions you should have that installed on your computer as well. Esmtp is a mail transport agent. It routes email to your mail server, such as gmail. To know what information to give esmtp, you will need to look up your mail server settings. Alternatively the newer msmtp can be used.

You will also need to install an approved email client, and use it to respond to message on the outreachy-kernel mailing list. These instructions assume you're using mutt, but you may find a GUI mail client like Evolution to be easier to use.

Gmail set up

In gmail, go click the gear icon, go to "Settings", go to the tab "Forwarding POP/IMAP", and click "Enable IMAP", then click on "Save Changes".

Then click the "Configuration instructions" link at the very bottom of the page. Note the outgoing mail server information in "Step 2", and copy it into the .esmtprc file, as shown in the next section.

If your gmail account uses two-step verification, then you will need to generate and use App Password.

Yahoo set up

In yahoo, go click on your account icon (top right, above "Settings" and to the left of "Home"). Click on "Account info" and then go to "Account security". (You may have to sign in again for this step.) Scroll down to the setting "Allow apps that use less secure sign-in" and turn it on. If you have two-step verification or account key enabled, you will also need to use App Password.

Configure esmtp

First, create a .esmtprc file with the right permissions: 

touch ~/.esmtprc
chmod g-rwx ~/.esmtprc
chmod o-rwx ~/.esmtprc

Edit the .esmtprc in your home directory, and add lines like this: 

identity "my.email@gmail.com"
hostname smtp.gmail.com:587
username "my.email@gmail.com"
password "ThisIsNotARealPassWord"
starttls required

(For Yahoo mail, replace hostname line with: 

hostname smtp.mail.yahoo.com:587

Configure msmtp

First, create a .msmtprc file with the right permissions: 

touch ~/.msmtprc
chmod g-rwx ~/.msmtprc
chmod o-rwx ~/.msmtprc

Edit the .msmtprc in your home directory, and add lines like this: 

defaults
tls on
auth on
logfile ~/.msmtp.log

account username-gmail
tls_starttls on
host smtp.gmail.com
port 587
from my.email@gmail.com
user my.email@gmail.com
pass "ThisIsNotARealPassWord"

account default: username-gmail

(For Yahoo mail, replace host line with: 

host smtp.mail.yahoo.com

Next, set up the mail client, mutt, with some defaults, by creating a .muttrc file in your homedirectory: 

set sendmail="/usr/bin/esmtp"
set envelope_from=yes
set from="Your Name <my.email@gmail.com>"
set use_from=yes
set edit_headers=yes

(For msmtp, replace esmtp line with: 

set sendmail="/usr/bin/msmtp"

Test your email setup

Next, let's send a test email message with mutt. Run this command: 

mutt

Say "no" to creating an inbox for now. Type 'm' to create a new message. Specify your own email address (or a secondary email) to send the test message to. Set the Subject however you want to. Type a message in the body, and then save and quit. Hit 'y' to send the message, hit 'e' to edit the message again, or hit 'q' to abort sending the message.

Look in your email to double check you received a message. If you send the email to yourself, for some mail services like gmail, the message will not show up in your inbox, and you will have to look in your Sent Mail folder.

If mutt is not working, try 

mutt -d 2

This will cause the creation of files with names beginning .muttdebug, followed by 0, 1, etc., that can help you find the problem.

Setup git

First, you need to tell git what your name and email address is, so that it can be used in the authorship information in the git commit. Create a file called .gitconfig and add lines like these to it: 

[user]
   name = Your Name
   email = your.email@example.com

Make sure that the email you specify here is the same email you used to set up sending mail. The Linux kernel developers will not accept a patch where the "From" email differs from the "Signed-off-by" line, which is what will happen if these two emails do not match.

Make sure you store your full, legal name in the 'name' line. By adding your Signed-off-by line to a patch, you are certifying that you have read and understood the Developer's Certificate of Origin. Please read through that document before you send patches to the kernel.

Explore the kernel tree

First, open a terminal, by clicking the black screen icon with the ">_" text in it.

Change directories to your git checkout you set up earlier: 

cd git/kernels/staging/

This is the Linux kernel tree. You can explore it by using the `ls` and `cd` commands. If you run `ls`, you'll see several different folders: 

intern@ubuntu:~/git/kernels/staging$ ls
COPYING  Documentation  Kconfig      Makefile  arch   certs   drivers   fs       init  kernel  mm   samples  security  tools  virt
CREDITS  Kbuild         MAINTAINERS  README    block  crypto  firmware  include  ipc   lib     net  scripts  sound     usr

There's more to this directory than meets the eye! If you run ls -A, you'll see there's a hidden directory called .git. This contains all the meta information that git uses to track branches, remote repositories, and changes to files in the local directory.

You can view the commit history by running 

git log

If you want a more compact form, you can run a command to see just the "short description" for each commit, with an abbreviated git commit ID: 

git log --pretty=oneline --abbrev-commit

Play with some git basics

Git is a distributed revision control system, which means you can hack on your version of the code without having to coordinate with other developers. Think of your git checkout as a separate copy of the kernel respository.

Git includes support for branches. Each branch can contain a completely different set of patches. Kernel developers typically use one branch per patchset. For example, you might have one branch that includes bug fixes, and another branch that contains commits for a new feature you're working on.

You can run `git branch` to see which branch you're on, and what other branches are available: 

intern@ubuntu:~/git/kernels/staging$ git branch
 * staging-testing

In this case, there is only one branch, called staging-testing. The star indicates that the "staging-testing" branch is the one you are currently on. In git speak, we say that you currently have the master branch "checked out".

Create a new branch called 'first-patch', and checkout that branch by running: 

git checkout -b first-patch

Now if you run git branch, you'll see that there are two branches, and you are currently on the "first-patch" branch: 

intern@ubuntu:~/git/kernels/staging$ git branch
  staging-testing
* first-patch

You can also use the git branch command to show branches on the staging remote repository. Run the command: 

intern@ubuntu:~/git/kernels/staging$ git branch -a
* first-patch
  staging-testing
  remotes/origin/HEAD -> origin/master
  remotes/origin/master
  remotes/origin/staging-linus
  remotes/origin/staging-next
  remotes/origin/staging-testing
  remotes/origin/test

The first remote repository that is used to create the git checkout is called "origin". For now, just remember that "origin" means Greg Kroah-Hartman's staging git repository. Here, you can see the staging remote has five branches: master, staging-linus, staging-next, staging-testing, and test. The staging-linus branch contains bug fix patches for the current kernel release candidate, and the staging-next branch contains patches for the next kernel release. Your patches will all go into staging-testing (since they will be code clean up, not bug fixes), so you want to base all your branches on the staging-testing branch. Greg first applies patches to staging-testing. They are moved to staging-next shortly thereafter.

Update your kernel

When you create your first application clean-up patches, you want to create them on top of the latest commit from the staging-testing tree. If your patch is out-of-date and doesn't apply to the latest tree, it may be rejected. You'll need to use git to fetch the latest changes: 

git fetch origin

The third word in that command is the name of the remote repository you are fetching from. In this case, it's origin, which is the remote repository we initially cloned from (the staging repository).

That command will fetch the changes from the remote, but it won't actually change in files in the working copy (i.e. the files in this directory). If you run: 

git log

You will see that your current working directory still points to the original commit. So where are the staging tree current changes?

The answer is that git stores the changes in a special hidden directory called .git. You can view the history of the staging repository by giving git log the "staging-testing" branch of the "origin" remote repository: 

git log origin/staging-testing

Next, we need to update our branch to include the changes in the staging tree. The safest way to do this is to "rebase" your branch. This means that if you have any commits on your branch, they will be placed on top of the staging tree commits. Sometimes you may have to edit your commits if there are conflicts, but you should ask your mentor for help with this. For now, run: 

git rebase origin/staging-testing

If you run `git log` to show your staging branch history and then `git log origin/staging-testing` to show the staging-testing branch history, you should see that they have exactly the same commits.

Configure the kernel

The next step is to create a configuration file, compile the new kernel, and install it.

The first thing to know is that the Linux kernel is completely configurable. Each driver can be separately configured to be installed or not. There are three choices for driver installation:

If you build the driver into the main kernel file, it will be loaded at boot time. The downside is that the kernel will have to load more code at boot for drivers that may not even correspond to hardware on the system. To avoid this, kernel developers often compile drivers as "modules". A module is a stand-alone .ko driver file that is loaded when the kernel detects hardware that matches the driver. For example, you could configure your wifi driver as a module, and the kernel will load it when it detects the wifi card.

The Linux kernel make system uses a special file called .config that stores what drivers are compiled in, or compiled as modules. Most Linux distributions store the .config file they used to compile your distro kernel in the /boot/ directory: 

intern@ubuntu:~/git/kernels/staging$ ls /boot/
System.map-4.8.0-2-amd64  System.map-4.9.0-rc8-amd64  config-4.9.0-1-amd64    grub                      initrd.img-4.9.0-1-amd64    lost+found             vmlinuz-4.9.0-1-amd64
System.map-4.9.0-1-amd64  config-4.8.0-2-amd64        config-4.9.0-rc8-amd64  initrd.img-4.8.0-2-amd64  initrd.img-4.9.0-rc8-amd64  vmlinuz-4.8.0-2-amd64  vmlinuz-4.9.0-rc8-amd64

You can duplicate the distro's configuration by copying one of the config-* files to a .config file in your git tree. This has already been done for you in the VM image.

You can read more about configuring a kernel here.

Compile the kernel

Next, you'll need to run make to compile your new kernel. Optionally, make can take a flag that indicates how many threads to spawn to start separate compilations. Usually you want to pick a number that is equal to the number of CPUs you have in your machine. For example, if you had a dual core system, you would run: 

make -j2

That may take a while. I would suggest reading some of the Linux Device Drivers book while you're waiting.

Make a driver change

These next couple of steps will allow you to make a change to a driver, and test that you've correctly compiled and installed the modified driver.

Modifying a driver under the VM

If you are running Linux in a VM, follow these directions. Otherwise, if you are running Linux natively on your machine, go to the next section.

One driver that's included in all VM images is the e1000 driver, the Intel ethernet driver. If you're running Linux natively, you will need to find a different driver. See the next driver section for how to find an appropriate driver.

The e1000 driver is found in the networking portion of the kernel: 

intern@ubuntu:~/git/kernels/staging$ ls drivers/net/ethernet/intel/e1000/
e1000_ethtool.c  e1000.h  e1000_hw.c  e1000_hw.h  e1000_main.c  e1000_osdep.h  e1000_param.c  Makefile

Let's make a small change to the probe function of the e1000 driver. A probe function is called when the driver is loaded. Let's edit e1000_main.c: 

vim drivers/net/ethernet/intel/e1000/e1000_main.c

Next, find the probe function. You can search for text by typing '/' in standard mode. Once you've found the probe function, add a printk line to it: 

static int e1000_probe(struct pci_dev *pdev, const struct pci_device_id *ent) {
         struct net_device *netdev;
         struct e1000_adapter *adapter;
         struct e1000_hw *hw;

         printk(KERN_DEBUG "I can modify the Linux kernel!\n");
         static int cards_found = 0;

Then type :wq<enter> to save the file and quit.

A printk function causes a message to be written to the kernel log buffer, which can then be viewed using the dmesg command.

Modifying a driver on native Linux

Your native Linux system will have many different drivers than the ones loaded on a Linux system running in VMPlayer. You will not necessarily have hardware that the e1000 driver can run on. Instead, you must find out which drivers are loaded on your system, and modify one of them.

First, use lsmod to see what drivers are loaded, and pick a name from that list to modify. If you have a variant of the e1000 driver, like e1000e, you may want to use that. Or you can find your wireless driver and modify that.

Once you have the name of a driver, it's time to find out where the .c and .h files for that driver are in the Linux kernel repository. You can do that by searching through the Makefiles to find out what C files are compiled into the driver binary. The best way to do that is with `git grep`. Unlike normal grep, git grep only searches through checked-in files in the repository. Normal grep will also search the binary files, and even the .git directory, which isn't useful and wastes your time.

For example, if you wanted to search for the files that are compiled into the xhci-hcd driver, you would run this command: 

git grep xhci-hcd -- '*Makefile'

Once you've identified the files for your driver, you will need to make a change to the probe function as described in the previous section.

Compile your changes

Recompile your kernel, by running make (with an optional -jN flag): 

make -j2

You may need to fix some compilation errors. Also fix any new warnings that are due to your changes. In the Linux kernel, we strive to make sure that drivers do not produce warnings on anyone's system (this includes 32-bit and 64-bit systems, as well as different architectures, such as PPC, ARM, or x86). New features or bug fix patches that add additional warnings may not get merged.

Install your changes

After you've compiled the driver, you need to install your changes by running: 

sudo make modules_install install

Test your changes

Since you've compiled a completely new kernel, you need to reboot into that new kernel in order to test your module changes. Reboot your VM (or computer), and then run: 

dmesg | less

Search for your printk in the log file by typing "/I can modify". You should be able to find this message within the driver output during boot. If you don't see this message, ask for help on the #kernel-outreachy IRC channel on irc.oftc.net, or on the outreachy-kernel mailing list

Revert your changes

Since that was just a simple test, and you probably don't want to commit that change, you can revert your changes. Exit out of your editor by typing :q<enter> and running this command: 

git reset --hard HEAD

That will revert ALL FILES in your current working directory to the last known commit (the HEAD commit), wiping out all your uncommited changes. Read the git reset manual for more information on ways to reset the state for specific files.

Start creating your first patch

Next, you'll get to do some useful modifications to the kernel, create your first git commit, and send out your first patch. Before you make your first commit using git, you'll need to do some setup.

Git post-commit hooks

Git includes some "hooks" for scripts that can be run before or after specific git commands are executed. The "post-commit" hook is run after you make a git commit with the `git commit` command.

Linux kernel developers have very stringent guidelines for coding style. They're so picky, they created a script called checkpatch.pl that you can run over your patches to make sure the patch complies with the kernel coding style.

To ensure that you create good commits that comply with the coding style, you can run checkpatch.pl over your commit with the "post-commit" hook. Note that running checkpatch after the commit, checks the patch file you've created - not just the source code diff. Therefore it will catch more issues - spelling errors in log messages, spacing in log messages, warnings about adding/removing files, etc.

If you already have a .git/hooks/post-commit file, move it to .git/hooks/post-commit.sample. git will not execute files with the .sample extension.

Then edit the .git/hooks/post-commit file to contain only the following two lines: 

#!/bin/sh
exec git show --format=email HEAD | ./scripts/checkpatch.pl --strict --codespell

Make sure the file is executable: 

chmod a+x .git/hooks/post-commit

If you don't already have /usr/share/codespell/dictionary.txt, get it: 

apt-get install codespell

The following Python's libraries are required by checkpatch. 

apt-get install python-ply python-git

After you commit, this hook will output any checkpatch errors or warnings that your patch creates. If you see warnings or errors that you know you added, you can amend the commit by changing the file, using git add to add the changes, and then using git commit --amend to commit the changes.

Checking that the post-commit hook works

You can check that the post-commit hook is working by adding a deliberate change that will trigger checkpatch (such as adding a really long line or adding trailing whitespace to a line), and then attempting to run git commit. After you commit, you should see the additional checkpatch.pl warning or error.

Note that you will need to place this hook in any/every tree in which you build patches.

If your post-commit hook is not working, please ask for help on IRC.

Understand patch best practices

Before you create your patch, you need to understand how to create good patches. Otherwise your patches will be less likely to be accepted by maintainers, and you will have to go through more revisions than necessary.

First, read about PatchPhilosophy. That document will help you create patches that are easy to read, and have a better chance of being applied by maintainers. Please also read CodingStyle (which is also available in the kernel code repository under Documentation). This will help you understand how to write code that the Linux kernel community will accept, and the rules here are the basis for the script checkpatch.pl.

Note that checkpatch.pl is failable! Use your best judgement when deciding whether it makes sense to make the change checkpatch.pl suggests. The end goal is for the code to be more readable. If checkpatch.pl suggests a change and you think the end result is not more readable, don't make the change. For example, if a line is 81 characters long, but breaking it makes the resulting code look ugly, don't break that line. Please read the CheckpatchTips page for how to avoid common mistakes when cleaning up drivers.

Find a driver to clean up

The staging tree, in drivers/staging/ is full of drivers that are not quite up to kernel coding style, or that use deprecated API. Drivers get placed here in order to get cleaned up. Some drivers have a TODO file in their parent directory, that lists things that need to be done to it: 

find drivers/staging -name TODO

You can either tackle one of those TODO items, or you can do a simple coding style cleanup.

Note: Please avoid sending patches for the octeon and octeon-usb driver as building it is quite complicated and not really suitable for the newbies.

drivers/staging contains both drivers that are on their way into the kernel and those that are on their way out of the kernel. It would be better to avoid working on the latter. Some drivers that are on their way out of the kernel as of February 2015 are 

i2o
line6
media/parport
media/tlg2300
media/vino

Running checkpatch.pl

If you pick a driver in staging, you can run the script that checks whether a file conforms to kernel coding style: 

perl scripts/checkpatch.pl -f drivers/staging/android/* | less

Pick a warning, and try to fix it. For your first patch, only pick one warning. In the future you can group multiple changes into one patch, but only if you follow the PatchPhilosophy of breaking each patch into log

KernelNewbies: FirstKernelPatch_draft (last edited 2020-04-07 13:21:55 by veera)