503 Service Unavailable

2010-01-20

Managing Linux kernel sources using Git

Filed under: Software — rg3 @ 21:53

This will be a short and easy tutorial on how to use Git to manage your kernel sources.

Before Git, the easiest way to manage your kernel sources was to download the kernel using the provided tarballs from kernel.org and update them downloading the provided patches between releases, which was very important to keep the download size small, instead of downloading complete tarballs each time. Also, by applying patches, you only needed to rebuild stuff that changed between releases instead of the full kernel once more. This is a good method that can be applied today and will probably never disappear. Simple HTTP and FTP downloads are very convenient in many situations.

However, with the arrival of kernel 2.6, its stable branches (e.g. the 2.6.32.y branch) and Git, there have been some changes. First of all, the process is now a bit more complicated. Stable patches are applied against the base release. If you have the kernel sources for version 2.6.32.1 and want to jump to version 2.6.32.2, you first have to revert the changes of release 2.6.32.1 (patch --reverse) and then apply the 2.6.32.2 patch. Slightly less convenient and, furthermore, you’ll modify every file that changed with every patch until that moment. This will affect the compilation process that would follow afterwards. In other words, if patch 2.6.32.1 meant (hypothetically speaking) a long build because it changed stuff that affected a lot of systems, so will be the build process for any other subsequent release in the 2.6.32.y branch. It was this small glitch that prompted me to manage my kernel sources the way I’m going to describe. Also, using Git is fun. :)

We will try to achieve the following:

-------------------------------------------------> Linus Torvalds' master branch
           \                   \
            \                   \
             A stable release    Another stable release

We will have a master branch that will follow Torvalds’ master branch and will be updated from time to time, or when he releases a new stable version of the Linux kernel (e.g. 2.6.32).

We will have other local branches that follow the stable releases by Greg K-H (e.g. 2.6.30.y, 2.6.31.y, 2.6.32.y, etc).

Git is very flexible and simple, and allows more than one way to do things. I will try to explain why I do things this way and why they make sense to me, and will try to avoid shortcuts, i.e. I will use one command for each action even if two actions could be compressed into a single command.

First, we will create a directory to hold the kernel sources. Let’s name it /path/to/kernel. In it we’ll have a directory named “src” that will hold the unmodified kernel sources and a second directory named “build” that we’ll use to build the kernel and keep the sources intact, for clarity. We start by cloning Torvalds’ branch:

cd /path/to/kernel
mkdir build
git clone 'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' src

This will create a directory named “src” with the sources. Take into account you’ll be downloading the full repository with a lot of revision history. It’s a relatively long download that requires a lot of patience or a good broadband connection. Whatever you have at hand. At the moment I’m writing this, it’s several hundred MBs but less than 1 GB, if I recall correctly.

If you issue a “git branch” command you’ll see you only have a local branch named “master”. This local branch follows Torvalds’ master branch. You can update your kernel sources when you are in this branch issuing a simple “git pull” command.

Now, we will add a second local branch to follow the stable 2.6.32.y kernel. In other words, our master branch follows Torvalds’ master branch and our “branch_2.6.32.y” (let’s call it that way) will have to follow the master branch in the stable 2.6.32.y repository.

First, we create a shortcut to the 2.6.32.y repository for convenience:

git remote add remote_2.6.32.y \
    'git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.32.y.git'

The name “remote_2.6.32.y” is arbitrary. At this moment, that name is only like an alias for that long URL and barely anything more. The next step is very important so that the name becomes something more and the following git commands understand what you mean when you use it. It will download data to your repository under that name.

git fetch remote_2.6.32.y

After you run that, which will take considerably less time that the full repository clone we did previously, remote_2.6.32.y will have a meaning in your hard drive. You can then use the following command:

git branch --track branch_2.6.32.y remote_2.6.32.y/master

This will create a new branch in your local repository that will be tracking the master branch at the 2.6.32.y repository. If you issue a “git branch” command you’ll now see you have two branches. Being a “tracking branch” means several things. You can change between the master branch and the new branch using “git checkout <branch name>” and, in each branch, you can perform a simple “git pull” to retrieve changes to that branch from the remote repository. From this point you’re on your own using Git to manage the sources and perform more operations if you need them, but the many tutorials available on the web will get you going in the basics of Git and that’s the only thing needed to manage the kernel sources with the only purpose of easing the downloading and building process.

Note that, between release 2.6.32.1 and 2.6.32.2, for example, you will only download the changes between those releases and a painful build for 2.6.32.1 does not have to mean a painful build for 2.6.32.2 if you update your sources this way.

Finally, we had created a “build” directory previously, in parallel to the “src” directory, in order to keep the sources directory clean. We can use this directory easily. When we are at the “src” directory, any “make command” we use can and would have to be replaced by “make O=../build”. To avoid mistakes, I have created a global alias in my system called “kmake”, aliased precisely to “make O=../build”. It affects the regular user account that I use to compile the kernel sources and the root account that I use in the installation step, to perform the “modules_install”, “firmware_install” and “install” operations.

As a regular user account:

  • kmake menuconfig
  • kmake
  • kmake oldconfig
  • etc

As the root account:

  • kmake modules_install
  • kmake firmware_install
  • kmake install

These aliases could be tuned further to install the kernel image, modules, firmware, etc to a sandbox directory if you intend to create packages with them, for example. The README file in the kernel sources directory has more information about this topic.

Advertisements

3 Comments »

  1. Hi,

    Nice article explaning how git is used for linux kernel development. Wanted to how can you checkout a particular branch without creating a clone.

    Comment by chait83 — 2010-03-10 @ 21:23 | Reply

    • I had to Google this and test a few things in my own machine. I think the best method to do it, or at least the most understandable for me, is:

      mkdir partial-clone
      cd partial-clone
      git init
      git remote add origin <url or path>
      git pull origin <branch of interest>
      git branch --set-upstream master origin/<branch of interest>

      In other words, you create an empty repository, then add a remote reference named “origin”, which is the name usually given automatically to the repository you clone, when you clone. After that you tell git to pull from a specific branch from the origin, and it will merge it with your current branch, which is none, so the master branch will be created and the contents of the remote branch merged with it. After that, you only need to specify that your master branch is supposed to follow the remote branch of interest, which will help you when you want to type “git pull” or “git push” without specifying where to pull from or push to.

      Comment by rg03 — 2010-03-11 @ 01:04 | Reply

  2. Thanks for the clear explanation, worked flawlessly for me.

    Comment by njc — 2010-06-04 @ 18:03 | Reply


RSS feed for comments on this post.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: