When one of my addresses is compromised, I can change it right away, yet I prefer not to change them so often to avoid annoying people wanting to contact me from time to time. For this reason, when I receive a spam message to one of my accounts, the first thing I do is reporting it. If I received 100 spam messages a day, I couldn’t do this, but as I receive one a month, I don’t mind spending 5 minutes reporting the message. Only if the spam doesn’t stop and apparently increases, I change that address.

Reporting a message is quite straightforward, but I don’t have it automated. I could, but I haven’t bothered yet. Basically, I view the source of the message and look for “Received from” headers. I find the first one, in chronological order, that appears to be a valid public SMTP server that people should trust. Then, I run “whois” on that IP address and find the ISP or organization owning that network block, and report the message to the abuse address they provide as part of the “whois” reply. If they don’t provide an abuse address, usually I send it to the technical contact that appears in the “whois” reply, and also to the “abuse@” address of the company’s main domain, just in case it actually exists and is being read.

In my e-mail client I have a template to report spam. I fire a new message from the template, fill the “To” field with the addresses just mentioned and copy-paste the full spam message source at the end of my message, which consists of a very brief message to the person that could be reading it, saying I received a spam message apparently coming from their network block. As I said, this takes 5 minutes and could be automated.

Sometimes, the spam message comes from a Yahoo! account, using their servers, and I follow the same procedure, emailing abuse@yahoo.com. This is the case of the latest spam message I received, two days ago. I proceeded to report the spam as I always do and received a reply from Yahoo! with the following contents.

Thank you for your email, but this address now only accepts messages in
Abuse Reporting Format (http://tools.ietf.org/html/rfc5965)

To report abuse manually (or to get help with security or abuse related
issues), please go to Yahoo! Abuse:
http://abuse.yahoo.com

For questions about using Yahoo! services, please visit Yahoo Help:
http://help.yahoo.com

Thank you,
– Yahoo! Customer Care

Note: Please do not reply to this email as replies will not be answered.

A quick Google search revealed a few people upset by this. Apparently, Yahoo! is applying this policy since the beginning of December. The RFC they mention in that first paragraph is from August. People are upset for several reasons. The RFC is so recent there are almost no tools to handle or create reports in that format yet. For that reason, they are cutting people out of the loop. The second option is going to their website and reporting the spam message there. This means two things: that you have to treat Yahoo! in a special way when reporting spam and that you have to be annoyed by their web form to report spam. It’s annoying because the landing page has no direct form to report spam. As of today, you have to click on “I want to report spam” (this opens a new window or tab), then copy, on separate locations, the full email headers on one box, and the message contents on another one. Fantastic. So you can’t simply upload the message for paste the full contents to a form. No, no. You have to carefully select the message headers first, then copy them, then paste them on the form, then copy the message body, then paste it on the form, then pass a captcha.

I was also a bit upset by this, so I read RFC 5965 a little bit. It looked simple if you only wanted to fill the required parts, and had a simple report example at the end, so I searched for a tool that would convert an e-mail message to a report based on these parameters. I didn’t find any tool immediately. I realized Python has a very comprehensive and easy to use package to handle e-mail messages, so I investigated a little bit and decided to spend the rest of the evening trying to create such a tool. The result has been uploaded to github as the spamreport repository, but don’t try to use it immediately. I have some bad news. Python’s e-mail library is amazingly simple and, in the end, including all the code to check program options and such, the program is exactly 100 lines long, so it’s very short and straightforward, and should work perfectly. However, it doesn’t work.

I have tried submitting an abuse report to Yahoo! in that format several times, making minor changes to the code, tweaking my program here and there, and every time the report has been rejected. Yahoo! does not explain why the report is being rejected in their reply, which, by the way, is a bit against the RFC itself. Section 4:

When an agent that accepts and handles ARF messages receives a
message that purports (by MIME type) to be an ARF message but
syntactically deviates from this specification, that agent SHOULD
ignore or reject the message. Where rejection is performed, the
rejection notice (either via an SMTP reply or generation of a
DSN) SHOULD identify the specific cause for the rejection.

As they are replying via SMTP with a rejection, they SHOULD explain the reason but they’re not doing it, and that’s why this is so frustrating. At first, I thought GMail was mangling the reports so I sent one to my own accounts at another e-mail provider, and it came out unmangled on the other end. GMail is not manipulating the reports. Just so you get an idea, here’s a screenshot from a test case. I took the simple report example they give in the RFC and attempted to create a similar report with my tool, using the same spam message and the same notification text, just to see what the differences were. Click on the image to view it in full size.

As you can see, apparently the only differences are:

1. The header order for “To”, “From”, “Date” and “Subject” differs (this should be irrelevant).
2. The words “feedback-report” are quoted in my output because Python writes them that way. This should also be irrelevant.
3. The MIME boundary markers differ (irrelevant and are generated randomly for each message).
4. The words “us-ascii” are in lowercase in my output. Python writes them in lowercase even if I put them in uppercase, and this should be irrelevant too.
5. The User-Agent string changes (obviously).

Yet the reports are being rejected by Yahoo! I’m puzzled at this moment and won’t tag the release as 1.0.0 until the reports are accepted or proved to be correct, but I don’t know what more to check. I suspect there’s a minor flaw I haven’t detected. If you spot it, please let me know. The code is on the net.

https://github.com/rg3/spamreport/

As you may know, freetype is normally configured by creating files in /etc/fonts/conf.avail and creating symlinks to those files in /etc/fonts/conf.d. Normally, separating each configuration parameter or parameter group to individual files lets you easily enable and disable specific font-rendering features by creating and destroying symlinks. One of these configurable features usually enabled in any distribution is to parse the file ~/.fonts.conf to allow every user to set their own font rendering parameters. For example, when KDE configures the font rendering features from the “System Settings” panel, it overwrites your ~/.fonts.conf. If you want to disable antialiasing for a specific font in freetype, you can either create a new config file in /etc/fonts/conf.avail and link to it in /etc/fonts/conf.d, setting it for any user, or adding the setting in your own ~/.fonts.conf. If you do the later, be sure to back file up somewhere, because fiddling with the font settings in your Destkop Environment may overwrite the file.

Going to specific details, I recently installed the Tahoma font from my Windows installation and wanted to use it with the bytecode interpreter and without antialiasing in the GUI, so it would look like this:

However, the rest of the fonts look ugly with those settings, so I wanted to disable antialiasing for the Tahoma font only, and only in sizes of 10 points or less. For bigger sizes, antialiasing would be enabled. Long story short, here are the settings that need to be integrated into your personal ~/.fonts.conf or put in an individual file in /etc/fonts/conf.{avail,d}. I’ll explain the contents next.

<?xml version='1.0'?>
<!DOCTYPE fontconfig SYSTEM 'fonts.dtd'>
<fontconfig>
  <match target="font">
    <test qual="any" name="family">
      <string>Tahoma</string>
    </test>
    <!-- pixelsize or size -->
    <test compare="more_eq" name="size" qual="any">
      <double>1</double>
    </test>
    <test compare="less_eq" name="size" qual="any">
      <double>10</double>
    </test>
    <edit mode="assign" name="antialias">
      <bool>false</bool>
    </edit>
    <edit name="autohint" mode="assign"><bool>false</bool></edit>
  </match>
  <match target="font">
    <test qual="any" name="family">
      <string>Tahoma</string>
    </test>
    <!-- pixelsize or size -->
    <test compare="more_eq" name="pixelsize" qual="any">
      <double>1</double>
    </test>
    <test compare="less_eq" name="pixelsize" qual="any">
      <double>14</double>
    </test>
    <edit mode="assign" name="antialias">
      <bool>false</bool>
    </edit>
    <edit name="autohint" mode="assign"><bool>false</bool></edit>
  </match>
</fontconfig>

I don’t want to go into specific details about the rules above. There is an XML header that needs to be present in any configuration file, and it contains a “fontconfig” section. Inside that section, you can put any number of “match” sections among other things, and we need two. One specifies the rules in terms of point size and another one in terms of pixel size. Both are needed for some reason.

The matches look for fonts named Tahoma and disable antialiasing and autohinting for them in some specific sizes. The exact point and pixel sizes depend on your X server and/or Xft settings. Most people set the DPI value to 75, 96 or 100. In KDE, you can override the current setting from the style configuration window. DPI stands for “Dots Per Inch”. In this case, pixels per inch. Normally it should really match your monitor. That is, if you have a 22″ screen with a specific resolution in pixels, you’d specify a DPI setting that would match the real DPI. However, like I said, most people use 75, 96 or 100 (I set it to 96 myself) and it DOES NOT match the real DPI. Depending on the DPI setting, your fonts will look bigger or smaller at the same size in points. In my case, I was interested in sizes lower than 10. Hence the match you can read above.

To write the pixel size match you need to calculate the equivalent of those point-values in pixels. This is easily calculated knowing two constants: the DPI value you’re currently using and knowing that an inch has exactly 72 points. So the equivalent in pixels of a 10-point distance in a 96 DPI screen is the following:

10 points, in inches: 10 / 72 = 0.1388

With 96 pixels per inch, those are: 0.1388 * 96 = 13.33, or 14 pixels rounding the number up, which is what you see in the config file I pasted above.

I should start by apologizing to anyone who thinks this is a bad move either because they may have to rebase all their work in a new repository, bringing all their changes back, or simply registered at bitbucket.org to follow the project. It currently has 17 forks and 100 followers, and I’m pretty sure some of them registered there just to follow youtube-dl, and the move to github.com is, if anything, a problem because they would have to create an account somewhere else to continue following the project. Again, apologies to anyone for whom the move has no practical aspects.

That said, I’d like to explain why I made the move. You may recall I wrote an article some time ago about Mercurial vs. Git. Apart from explaining what I considered were the main differences between the two, I also wanted to express my indecision about which one was better. While I think Mercurial is and was great, the balance has been leaning towards Git for some time now, and I tend to use Git for all my personal projects. Many of the reasons, if not all of them, have been expressed by other people in the past. It’s a good moment to quote a very well known blog post from Theodore Tso, written in 2007 when he was still planning to migrate e2fsprogs to Git from Mercurial:

The main reason why I’ve come out in favor of git is that I see its potential as being greater than hg, and so while it definitely has some ease-of-use and documentation shortcomings, in the long run I think it has “more legs” than hg, [...]

I think that paragraph describes with great accuracy what I think too. In the medium and long run, Git’s problems almost vanish. Its documentation was a bit poor back then, but people have been writing more and more about Git and there are a few very good resources to learn its internals and basic features. Furthermore, once you have a simple idea about its internals and use it daily, you no longer need that much documentation. If you’re not sure how to do something, chances are a simple web search will tell you how to do what you wanted to achieve.

Also, as many people know, Mercurial was and is mostly about not modifying the project’s history, while Git has a lot of commands that directly modify the project’s history. With time, I’ve come to realize that modifying the project’s history is simply more practical in many cases and in a range of situations it leads to less confusion. In my day job, we are slowly moving from CVS to Subversion to manage the sources of a very old and important project, which exists since about 1984. At the same time, we are modifying our work flow here and there to take advantage of Subversion, and we’re heavily using branching and merging despite the fact that’s not one of Subversion’s greatest strengths, as you may know. That’s giving us some problems and it’s amazing how many times I caught myself thinking “this would be much easier if we were using git, because we would simply do this and that and job done”. Many of those actions would modify the project’s history and clean it up. I repeat, in real situations with a lot of people working on something and not doing everything exactly as it should be done, it’s only a matter of time that you miss a Git feature.

The only thing I don’t like about Git is its staging area. From a technical perspective, the staging area makes a lot of sense, and you can build many neat features based on it. However, one thing is having a staging area and a second thing is exposing it to end users. I think you can have a staging area and all the features it provides while hiding it from users in their most common work flows. Still, it’s something you get used to and everybody knows that, when your project is a bit mature, you spend way more time browsing the source code, debugging, running it and testing it than actually committing changes to the source tree. The staging area is not a big issue and “git commit -a” covers the most common cases.

Apart from Git itself, the move was partly motivated by the site, github.com. When I started using bitbucket.org I liked it a bit more than github.com, but things have changed slowly. github.com fixed a rendering bug that hid part of project top bar, got rid of its Flash-based file uploader and got an internal issue tracker with a web interface that works really really well. The site is very nice and the “pages” feature, that allows you to set up a simple web page for the project, is still not provided by bitbucket.org as far as I know. In addition, with the arrival of Firesheep, it quickly moved to using SSL for everything. It’s fantastic.

bitbucket.org was recently bought by Atlassian and their plans are indeed better. For me, however, the number of private repositories and private collaborators is not an issue, because all the projects I host on github.com are public. Still, it’s fair to mention their plans because it could be a deciding factor for some people.

I wouldn’t like to close this article without mentioning the big improvement that both sites bring to the typical free and open source software developer. I still host a few projects on sourceforge.net, and I can tell you I’m not going back to it despite the great service they have provided for years for which I thank them sincerely.

It’s been months since I last used it so I apologize if things have changed without me noticing, but back then it was very hard to get your code on sourceforge.net. You didn’t perceive it was hard because there was no github.com. Once you try github.com or bitbucket.org, you realize how much the process can be simplified. Two key aspects to note. First, the project name doesn’t have to be unique. It only needs to have a unique name among your own projects, which is much easier and simplifies choosing the project name a lot. Second, once the project is created and has a basic description, without filling any form and without having to wait for anything, you are only a few commands away from uploading your code to the Internet. It can literally take less than 5 minutes to create a project and have your code publicly available, and that’s fantastic and motivating. You don’t need to find time to upload your code or thinking if the process is worth it for the size of the project. You simply do it. That’s good news for everyone.

Let me finish by apologizing again to anyone for the inconveniences created by the move. I sincerely hope this will remain the project location for many years to come.

# iptables -v -L
Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 663K  905M ACCEPT     all  --  any    any     anywhere             anywhere            state RELATED,ESTABLISHED
  105  6300 ACCEPT     all  --  lo     any     anywhere             anywhere
    0     0 ACCEPT     icmp --  any    any     anywhere             anywhere            icmp destination-unreachable
    0     0 ACCEPT     icmp --  any    any     anywhere             anywhere            icmp time-exceeded
    0     0 ACCEPT     icmp --  any    any     anywhere             anywhere            icmp source-quench
    0     0 ACCEPT     icmp --  any    any     anywhere             anywhere            icmp parameter-problem
    0     0 DROP       tcp  --  any    any     anywhere             anywhere            tcp flags:!FIN,SYN,RST,ACK/SYN state NEW

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

We’ll start by the most obvious rules. The FORWARD chain has a policy of “DROP” and no specific rules. A desktop computer isn’t usually employed as a router or to share an Internet connection, so there’s no reason in allowing forwarding.

The OUTPUT chain has a policy of “ACCEPT” and no rules. Basically, we are allowing everything going out of our computer. While this isn’t the most secure policy at all, it’s usually enough for a desktop computer. Many paranoid people would not let everything out. For example, to prevent their computers from being used to send spam due to a mistake somewhere else, sometimes people forbid from sending traffic from the source port 25, or in general from source ports below 1024, where most common services are. We could do that, but I think it’s not really needed for a desktop computer. We’ll put more effort blocking incoming traffic, and we can keep a relaxed policy on outgoing traffic.

Finally, the guts of the rules. The INPUT chain has a policy of DROP. That is, everything not explicitly allowed will be forbidden. If anything passes through all the rules, the traffic will be discarded silently without making noise.

The rules in the INPUT chain are sorted according to the typical frequency of hits. “Popular” and frequent traffic will be quickly accepted instead of having to check many rules before. That’s why the first rule is to allow RELATED and ESTABLISHED traffic, for any protocol. The any part is important. This is the rule that, basically, allows us to receive replies and normal traffic for connections we start ourselves. For example, when we open a web page with our web browser, we’ll send traffic one way and when we receive the reply, the connection will be ESTABLISHED and we’ll see the reply. This first rule is the most important one because, just due to it, we can use the computer “normally”.

The stateful packet firewall in Linux is quite clever and understands established connections even when the underlying protocol has no notion of connections. For example, that first rule allows us to receive DNS replies from queries we made ourselves, using the UDP protocol, or allows receiving ICMP echo replies from our own requests. In other words, we can ping other computers thanks to that rule.

On to the second rule, it looks like it would accept any traffic from anywhere, but the keyword here is lo:

  105  6300 ACCEPT     all  --  lo     any     anywhere             anywhere

This rule accepts all incoming traffic from interface “lo”, which is the loopback interface. This rule allows us to connect to services on our own machine by pointing to 127.0.0.1, or ::1 in IPv6. This rule would allow connecting to the CUPS printing service, for example, if we had a printer connected to our computer. A variant of this rule that can be frequently found on the Internet is to include a further check to verify the destination IP is 127.0.0.1, just to be more paranoid and forbid strange traffic. While this can increase security, I don’t think you need that further check generally. Just to clarify, browsing unsafe web pages with Javascript and/or Flash is more dangerous than not checking if traffic coming through “lo” is really directed to 127.0.0.1, so it’s not a priority.

Then, you can see I allow some specific types of ICMP packets that usually signal network problems. None of those require a reply to be sent, so we accept them and try to interpret what they would mean if they ever come in. I don’t think it’s possible to get anything more than a DoS attack with those rules, but comments are welcome. And, of course, you can be DoS’ed just by someone saturating you with incoming traffic. Again, this is a matter of getting your priorities sorted. If you feel paranoid, well, drop those rules.

Finally, at the end of the chain we have the famous specific rule to block incoming traffic with state “NEW” and the SYN flag not set in TCP. This rule is quite specific and an explanation for it can be found in many iptables manuals, FAQs and tutorials. I put the rule in the end because the first rule is not affected by it, because the second rule isn’t either (we are allowing ALL traffic coming from “lo”, after all), and the ICMP rules are not affected either.

However, we still keep it there even if the traffic was going to be dropped anyway due to the chain policy, because when we want to create a hole in these rules, we do it by adding more rules at the end of the INPUT chain. For example, sometimes I want to allow incoming traffic to a specific port where I have configured a server that is supposed to be reached from other machines, to serve a specific content in a specific point in time. For that, I have created a couple of scripts called “service-open” and “service-close”, that can be used followed by a list of service names or port numbers. For example, when I start a web server to allow someone in my home network to get a file from my computer, I usually run the command “service-open 8080″ (the server would be listening on that port). Once the file is served, I run “service-close 8080″ and shut the server down. Those commands add and remove rules at the end of the INPUT chain, so that’s why I put the last rule there, so it’s present before any holes I punch through my firewall in those special cases. If you frequently run a P2P application on your computer, you may want to open a hole permanently to some port and save it as part of your usual rules. I don’t, so I keep everything closed.

The content of my scripts are:

# cat /usr/local/sbin/service-open
#!/bin/sh
if test $# -eq 0; then
        echo usage: $( basename $0 ) service ... 1>&2
        exit 1
fi
while test $# -ne 0; do
        /usr/sbin/iptables -A INPUT -p tcp --dport "$1" -j ACCEPT
        /usr/sbin/iptables -A INPUT -p udp --dport "$1" -j ACCEPT
        shift
done

# cat /usr/local/sbin/service-close
#!/bin/sh
if test $# -eq 0; then
        echo usage: $( basename $0 ) service ... 1>&2
        exit 1
fi
while test $# -ne 0; do
        /usr/sbin/iptables -D INPUT -p tcp --dport "$1" -j ACCEPT
        /usr/sbin/iptables -D INPUT -p udp --dport "$1" -j ACCEPT
        shift
done

Those scripts play nicely with my set of rules because they are designed with my rules in mind. Also, you can see they are dead simple.

With the set of rules I have described, you can use your computer normally, you can easily let more traffic through in specific cases and, more importantly, you’ll be “invisible” on the network. Nobody will know if your computer is really there or not unless you send them traffic or if they found out by other means. And, also, it’s a very small set of rules and it’s very easy to remember and understand, and to create scripts that modify it easily.

Edit: The commands needed to create those rules:

iptables -P FORWARD DROP
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -p icmp -m icmp --icmp-type 3 -j ACCEPT
iptables -A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT
iptables -A INPUT -p icmp -m icmp --icmp-type 4 -j ACCEPT
iptables -A INPUT -p icmp -m icmp --icmp-type 12 -j ACCEPT
iptables -A INPUT -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW -j DROP
iptables -P INPUT DROP

The first impression I got from the movie was so-so. It’s fun and, for a computer engineer, the references to mainframes, IO ports, programs and games are enjoyable. After watching the movie I went directly to disc 2 and watched the documentary on how it was done. It was then when I started enjoying the film much more. The documentary helps you appreciate TRON as the piece of art it really is and all the attention paid to the different details in the movie.

By coincidence, I watched the film last Christmas with a friend of mine and we both share a fun interpretation of its script. TRON is a movie you can enjoy because, as in many other good fantasy and science-fiction films, the bad guys win in the end. Now, before you jump at me and wonder what I’ve been smoking to say that, just think about it. Do you really think Dillinger and the Master Control Program are the bad guys in the movie? The bad guy is Flynn! And that little program, TRON! It’s a tragic and realistic story.

A company, ENCOM, and two programmers: Ed Dillinger and Kevin Flynn. Ed is the good guy in the company. Working hard to improve technology in his cubicle until late hours, motivated by the need to create something bigger, better and never seen before. He’s a shy guy with brilliant ideas and creates a program, called the Master Control Program, based originally on a chess program, with several features that will be a breakthrough in computing history. First, the Master Control Program allows for real multitasking. Programmers don’t interfere with each other and they no longer have direct access to the computer hardware. The modern operating system is born, also with a built-in firewall to monitor and control connections to and from external systems. Second, this program is powered by an incredibly advanced AI system capable of developing primitive feelings, and also features natural language parsing via audio input and replies in the same language, with a voice synthesizer.

The Master Control Program is amazing and could push ENCOM from being a medium-sized company into a big corporation in every field of technology. However, management are too short-sighted to pay attention to it and the shy guy who created it, and are amused by the extrovert programmer Kevin Flynn. Much younger than Ed Dillinger, as we can see in the film, he enjoys creating video games and breaking into different systems and, with such a personality, the company board is waiting for their golden boy to do something spectacular that will never really arrive, because Flynn uses the company resources to create games that he will keep for himself. He won’t let the company see the real good games and will be jumping ship as soon as he finds a good deal with a big game publisher.

Dillinger, untalented for creating popular games, sees envy grow at the core of his heart and one day decides to steal the good games from Flynn and presents them to the company board as his work. He shouldn’t have done that, but poor Dillinger thought that was the only way to get attention from the board. From then on, they finally pay attention to him and he can push the Master Control Program forward as a way to manage the company’s computing resources and is promoted to the position he really deserves. They even start investigating on teleportation. Of course, TRON (the program) is rejected by Dillinger and the MCP. After all, TRON is redundant and its tasks are already being performed by the MCP. No good engineer would tolerate such an evident duplication in functionality. Alan Bradley suffers from the NIH syndrome.

All that technology never reaches the market because, in the film, we see the bad guys preying on this good guy for his only mistake until he is defeated, his programs are deleted forever from the hard drive and he, probably, he’s fired from the company.

I’ll be watching Tron Legacy to follow the adventures of Flynn and the result of his evil and ego-driven plot to control the world with his videogames, unable to realize he lacks the talent Dillinger had. If you watch the trailers released so far you’ll see Flynn is really evil, as he has always been.

Before Git, the easiest way to manage your kernel sources was to download the kernel using the provided tarballs from kernel.org and update them downloading the provided patches between releases, which was very important to keep the download size small, instead of downloading complete tarballs each time. Also, by applying patches, you only needed to rebuild stuff that changed between releases instead of the full kernel once more. This is a good method that can be applied today and will probably never disappear. Simple HTTP and FTP downloads are very convenient in many situations.

However, with the arrival of kernel 2.6, its stable branches (e.g. the 2.6.32.y branch) and Git, there have been some changes. First of all, the process is now a bit more complicated. Stable patches are applied against the base release. If you have the kernel sources for version 2.6.32.1 and want to jump to version 2.6.32.2, you first have to revert the changes of release 2.6.32.1 (patch --reverse) and then apply the 2.6.32.2 patch. Slightly less convenient and, furthermore, you’ll modify every file that changed with every patch until that moment. This will affect the compilation process that would follow afterwards. In other words, if patch 2.6.32.1 meant (hypothetically speaking) a long build because it changed stuff that affected a lot of systems, so will be the build process for any other subsequent release in the 2.6.32.y branch. It was this small glitch that prompted me to manage my kernel sources the way I’m going to describe. Also, using Git is fun. :)

We will try to achieve the following:

-------------------------------------------------> Linus Torvalds' master branch
           \                   \
            \                   \
             A stable release    Another stable release

We will have a master branch that will follow Torvalds’ master branch and will be updated from time to time, or when he releases a new stable version of the Linux kernel (e.g. 2.6.32).

We will have other local branches that follow the stable releases by Greg K-H (e.g. 2.6.30.y, 2.6.31.y, 2.6.32.y, etc).

Git is very flexible and simple, and allows more than one way to do things. I will try to explain why I do things this way and why they make sense to me, and will try to avoid shortcuts, i.e. I will use one command for each action even if two actions could be compressed into a single command.

First, we will create a directory to hold the kernel sources. Let’s name it /path/to/kernel. In it we’ll have a directory named “src” that will hold the unmodified kernel sources and a second directory named “build” that we’ll use to build the kernel and keep the sources intact, for clarity. We start by cloning Torvalds’ branch:

cd /path/to/kernel
mkdir build
git clone 'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' src

This will create a directory named “src” with the sources. Take into account you’ll be downloading the full repository with a lot of revision history. It’s a relatively long download that requires a lot of patience or a good broadband connection. Whatever you have at hand. At the moment I’m writing this, it’s several hundred MBs but less than 1 GB, if I recall correctly.

If you issue a “git branch” command you’ll see you only have a local branch named “master”. This local branch follows Torvalds’ master branch. You can update your kernel sources when you are in this branch issuing a simple “git pull” command.

Now, we will add a second local branch to follow the stable 2.6.32.y kernel. In other words, our master branch follows Torvalds’ master branch and our “branch_2.6.32.y” (let’s call it that way) will have to follow the master branch in the stable 2.6.32.y repository.

First, we create a shortcut to the 2.6.32.y repository for convenience:

git remote add remote_2.6.32.y \
    'git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.32.y.git'

The name “remote_2.6.32.y” is arbitrary. At this moment, that name is only like an alias for that long URL and barely anything more. The next step is very important so that the name becomes something more and the following git commands understand what you mean when you use it. It will download data to your repository under that name.

git fetch remote_2.6.32.y

After you run that, which will take considerably less time that the full repository clone we did previously, remote_2.6.32.y will have a meaning in your hard drive. You can then use the following command:

git branch --track branch_2.6.32.y remote_2.6.32.y/master

This will create a new branch in your local repository that will be tracking the master branch at the 2.6.32.y repository. If you issue a “git branch” command you’ll now see you have two branches. Being a “tracking branch” means several things. You can change between the master branch and the new branch using “git checkout <branch name>” and, in each branch, you can perform a simple “git pull” to retrieve changes to that branch from the remote repository. From this point you’re on your own using Git to manage the sources and perform more operations if you need them, but the many tutorials available on the web will get you going in the basics of Git and that’s the only thing needed to manage the kernel sources with the only purpose of easing the downloading and building process.

Note that, between release 2.6.32.1 and 2.6.32.2, for example, you will only download the changes between those releases and a painful build for 2.6.32.1 does not have to mean a painful build for 2.6.32.2 if you update your sources this way.

Finally, we had created a “build” directory previously, in parallel to the “src” directory, in order to keep the sources directory clean. We can use this directory easily. When we are at the “src” directory, any “make command” we use can and would have to be replaced by “make O=../build”. To avoid mistakes, I have created a global alias in my system called “kmake”, aliased precisely to “make O=../build”. It affects the regular user account that I use to compile the kernel sources and the root account that I use in the installation step, to perform the “modules_install”, “firmware_install” and “install” operations.

As a regular user account:

• kmake menuconfig
• kmake
• kmake oldconfig
• etc

As the root account:

• kmake modules_install
• kmake firmware_install
• kmake install

These aliases could be tuned further to install the kernel image, modules, firmware, etc to a sandbox directory if you intend to create packages with them, for example. The README file in the kernel sources directory has more information about this topic.

I had to change the strategy and get the list of devices I could buy, and then search for them on the Internet, trying to know if any of them were supported by an out-of-tree driver or something similar. After a couple of returns, thanks to some manufacturers changing devices internally while keeping the product name unchanged, I finally arrived home with a working, hybrid, TV USB stick, the Pinnacle PCVT Hybrid Pro Stick, sold in some countries as model 330e. It costed just over 100 euros.

My main target being digital TV, I quickly got it working with an out-of-tree driver by Markus Rechberger. This out-of-tree driver was part of a project that tried to create the possibility of having user-space tuners for TV cards. While I am nobody to judge if that’s a good or bad idea, it was different enough to not make it into the main kernel tree. The author, then, appeared to change the approach and created a different out-of-tree driver called “em28xx-new”, based on the in-kernel “em28xx” driver that he had already contributed. This driver used a more traditional approach, and worked like a charm too. Unfortunately, it never made it into the vanilla kernel either, for whatever reasons.

I contacted Markus Rechberger a couple of times, if I recall correctly. I thanked him for his efforts and time put into creating the driver and asked a couple of questions once, and also sent him a patch for the build scripts some time later. I don’t recall if the patch was applied or not. He was always very nice and polite.

However, one day I had just compiled a new kernel and was about to build the driver for it. Before doing that, I always downloaded the latest copy of the driver source code from its Mercurial repository. This time, when I ran Mercurial it exited with a confusing error message, saying the remote tree was not the same repository I had in my hard drive. I supposed the author would have created a new repository for the driver, so I cloned it to a new directory. It turned out there was only a README file in the repository. I opened it and… uh oh. A note saying the old driver had been pulled from the Internet and giving a URL that led to the web site of a TV card manufacturer offering products that were supposedly supported by Linux. The equivalent USB stick costed just about 100 euros, like the one I had. But, of course, it was too late for me to return the one I had bought. I had been using the device for months.

I searched on the Internet again trying to find the reason that led to the driver being pulled from the web site, and everything I got was the site of an Arch Linux user that uploaded the latest version he got from the repository and even offers some patches to make the code work with more recent kernels. However, as of the time I’m writing this, the latest patch is for kernel 2.6.30 and the driver does not compile for the recently released kernel 2.6.32. So the status of this device is that it works, but only if you have a specific kernel version. At the top of that page, you can see a huge banner that reads like this:

DISCLAIMER: Don’t bother me or the original author, Markus Rechberger, with any questions about problems with this driver, because Markus Rechberger deleted it because of these questions and because I just host these files.

I thought the driver may have been pulled from the Internet for some kind of legal reasons, but the disclaimer suggests a different reason. I don’t know if I buy the reason. I’m not sure it’s entirely credible but there’s no point in not believing those words are true. Markus Rechberger, for all we know, got burned out maintaining the driver and decided not to maintain it any longer.

A story published months ago at lwn.net explains this case with more details and further information. The situation for people owning this device and wanting to use it under a recent kernel is that you are supposed to be using the in-kernel em28xx driver. However, as the linuxtv.org page for the device says, the difficulty in supporting digital TV for it has its source in the Micronas DRX3975D DVB-T chipset it features. This chipset already has an in-kernel driver, which can be located at Device Drivers > Multimedia support > DVB/ATSC adapters > Customize the frontend modules to build > Customize DVB Frontends > Micronas DRX3975D/DRX3977D based. The location may change in the future (2.6.32 as I’m writing this).

Unfortunately, the driver cannot be used by now. As its help text mentions, this driver needs external firmware which currently cannot be obtained. Marked as “TODO” in the help text, you are told to run “<kerneldir>/Documentation/dvb/get_dvb_firmware drx397xD”. But, if you try, you’ll get an error saying that drx397xD is not a know component.

It’s an appropriate moment to thank and encourage the developers that are working on this, being the last missing piece. Devin Heitmueller has done a good job trying to keep people up-to-date with information on the progress and the difficulties encountered. The last comment on that blog post is from December 6 and says:

Unfortunately, at this point the answer is “not right now”. I’m waiting for the DVB generator to arrive, at which point I should be able to complete the work.

Again, thanks for working on this, keep up the good work and we’re eager to make our 330e USB cards work again with recent kernels, Devin!

While reflecting on the driver situation and putting together the different pieces of this soap opera, it all reminded me of the situation we professional programmers face from time to time while maintaining open source software. Many of us really love programming and we have tried to make it our job, successfully. There’s a difference, however, when you change from student to professional programmer.

When you are a student, you have a lot of time in your hands. It’s a wonderful experience going to college and learning new things everyday, buy books, read about different languages and technology, and the amount of spare time to learn and have fun programming is incredible. Later, however, you become professional and you start working for a company in a full time job. You leave home before dawn everyday and, at least in winter and in my case, you arrive home after sunset. It’s incredibly depressing if you think about it. You spend the day coding, fixing issues in programs, debugging, testing, etc. This kind of life doesn’t make it impossible to enjoy programming again, but if you arrive home and find that you have a popular open source program in your hands with users reporting bugs and requesting new features you may feel as if you were still at the office.

My advice here is obvious. Don’t stop coding in your spare time, but do it for fun. If you don’t feel like adding a new feature someone requested, don’t add it. It’s very important to say “no” often so your program will still be your program, the product you wanted. If a user or a group of users are still in the fortunate situation in which they are students and have a lot of spare time, they can always fork your code. That is the beauty of free and open source software.

I couldn’t care less if some people would like to use all this text above to attack FOSS and say bad things about it: non-working drivers, unresponsive maintainers, lack of documentation, user unfriendliness. Mental health of the people writing the code is more important. Don’t burn out. Produce something and let others make better things out of it if you don’t have the time. Start new projects all the time. Handle maintenance of old projects to new people. Have fun. Enjoy. Code. Help others. Submit patches with bug reports if possible. Appreciate the effort of others and thank them for the work they provide you. Try to be kind and explain your users the reasons behind your “noes”.

A Hacker News user, anyway, pointed out something interesting. You can easily get information about the needed shared library dependencies for a program or library using “objdump”, so I spent a few hours writing and tweaking a small script called lddsafe that prints almost the same information as “ldd” using “objdump” and avoiding the security problems, as it doesn’t have to run the program. Two major caveats at this point in time:

• It requires bash and, more specifically, bash version 4 or later. I needed to use associative arrays to make the program reasonably fast and they are only available in bash 4.
• It’s only been tested under Slackware Linux. However, bug reports and patches are welcome if it doesn’t run properly in other distributions.

Future improvements may include rewriting it in Perl so as not to require bash 4, knowing that Perl is present in most Unix systems.

A picture is worth a thousand words:

$ lddsafe /usr/bin/xcalc
        libXaw.so.7 => /usr/lib/libXaw.so.7
        libXmu.so.6 => /usr/lib/libXmu.so.6
        libXt.so.6 => /usr/lib/libXt.so.6
        libSM.so.6 => /usr/lib/libSM.so.6
        libICE.so.6 => /usr/lib/libICE.so.6
        libc.so.6 => /lib/libc.so.6
        ld-linux.so.2 => /lib/ld-linux.so.2
        libuuid.so.1 => /lib/libuuid.so.1
        libX11.so.6 => /usr/lib/libX11.so.6
        libxcb.so.1 => /usr/lib/libxcb.so.1
        libXau.so.6 => /usr/lib/libXau.so.6
        libXdmcp.so.6 => /usr/lib/libXdmcp.so.6
        libdl.so.2 => /lib/libdl.so.2
        libXext.so.6 => /usr/lib/libXext.so.6
        libXpm.so.4 => /usr/lib/libXpm.so.4
        libm.so.6 => /lib/libm.so.6

Our program has soft real-time requirements and is written mostly in Ada, with a few calls to C library functions via pragmas. Due to the nature of the program, it avoids memory allocation as frequently as possible and uses static-sized arrays for its data structures, and O(1) algorithms whenever possible to behave properly in the context it is being used.

I will completely skip the part of the story dealing with analysis of the source code in vain to find where the program was leaking memory, but I can tell you it took a lot of time and did not give any positive results, making us quite angry and desperate. We would have been unable to find the problem this way. As I said before, the leak was in a standard library function from an expensive software development kit for Ada, and had nothing to do with memory allocation functions. To be more precise, the memory leak was in Text_IO.Reset, a procedure that resets the state of a text file, very similar to rewind in C. I will head to the final steps, what I considered interesting.

The program runs on Solaris, so we monitored the process using pmap. This gave us precise information and told us clearly that the memory region that was growing was the heap, where memory allocation happens. I thought that, if our program was barely doing any memory allocation operations, and normally it should be doing none, according to the code, we had a good chance of catching it leaking memory. When a program in Unix needs more memory, it calls either mmap, brk or sbrk. I could not come up with more system calls that allocated memory. Normally, when you program in C or C++ you use malloc, free, new and delete. These language operators or library functions in turn manage memory blocks but request more memory to the operating system with the previous system calls I mentioned. It is explained in many books and tutorials over the Internet but I would say, and maybe I am wrong, that it is not exactly common knowledge.

My first approach, which did not work, involved creating a shared library that I would load using LD_PRELOAD, which intercepted calls to sbrk, brk and mmap. When intercepted, it would call pstack on the current process (a program that prints the call stack of any process given its pid), save the call stack to a text file and proceed with the normal system call. Hackish and clever, I thought while laughing like a maniac when I was coding that. Well, that did not work, I repeat. While the program was indeed calling sbrk as confirmed by truss (for Linux users, truss is very similar to strace), it was not calling, apparently, any function called sbrk that I could intercept. I created a test program to see if my library worked and it did, but it did not create any stack trace for the program in question.

Still, I had already started using truss to verify the program was allocating memory with sbrk, so I dived into the truss manual to see if I could use it for something else. This way I discovered that truss was able to stop the program execution when it made any system call I specified. My new approach was, then, tracing the program with truss, stopping in sbrk, then calling pstack on the PID and then telling the program to continue running. This almost worked. The printed stack did not have any symbols, probably due to the Ada compiler not populating the executable file with the debugging information as the C compiler did. So close yet so far. Our programs were indeed compiled with debugging information, and a minor change to the strategy was enough. Instead of printing the stack with pstack, I would attach the Ada debugger to the program and print the call stack. This way, I finally witnessed the program leaking memory in what seemed to be a call to Text_IO.Reset.

I thought this could be wrong, so I created a test program that read a file over and over again, calling Text_IO.Reset when reaching EOF. The test program, indeed, leaked memory at an alarming rate. Case closed, smile and surprise in my face. Well, to be honest, we replaced the calls to Text_IO.Reset with something else and tested again, to confirm the program had stopped leaking memory. But I already knew the problem had been found after running the test program.

When I came home I wondered if I could have done something similar in my Linux system. I read the man page for strace to see if it could stop programs when a specific system call was made, but I found no way of doing so. Apparently, the solution and the strategy I had employed was Solaris (or maybe UNIX) specific. Several Google searches did not give me any clue about doing the same in Linux.

Yesterday, however, GDB 7.0 was released. I took a look at the new features and I found this little sentence at the end of it:

* New command to stop execution when a system call is made

According to the documentation, you only need to set a cathpoint for the program like catch syscall sbrk to achieve the same. Two months ago, I would have read the features and forgotten them five minutes later. But, yesterday, I again smiled, then laughed like a maniac and shouted “BEGONE, MEMORY LEAKS!!!”.

Thanks to someone on the ##slackware FreeNode IRC channel that mentioned the problem some weeks ago, I discovered that GNU grep is very slow when working on UTF-8 files, and possibly other Unicode encodings. This, apparently, is a long-standing bug that hasn’t been officially fixed yet. The problem manifests itself when you run grep using locale settings that involve using UTF-8. Let’s see the following example:

$ echo $LANG
en_US.UTF-8
$ time grep '^....' /usr/share/dict/words >/dev/null 

real    2m16.795s
user    2m10.536s
sys     0m0.087s
$ export LANG=C
$ time grep '^....' /usr/share/dict/words >/dev/null 

real    0m0.031s
user    0m0.028s
sys     0m0.003s

In the previous text, /usr/share/dict/words is a file part of the bsd-games package in my Slackware system. It contains a list of English words and it’s not too long. It has below 40000 lines, each line having a word, and weights about 345 KB. Still, as you can see in the previous example, it takes more than 2 minutes in my computer to search for words having at least 4 characters. When I change my locale settings to “C” (ASCII), it only takes 31 milliseconds. The difference is amazing. Does grep behave differently in both cases? The answer is yes.

When grep runs in UTF-8 mode, the dot character, for example, represents any multi-byte character, while in ASCII mode the dot represents a single byte. See for example the following, using an accented Spanish character to form a 5-letter word.

$ echo ámbar | LANG=C grep '^.....$'
$ echo ámbar | LANG=en_US.UTF-8 grep '^.....$'
ámbar

The á character is represented using two bytes in UTF-8. Using the UTF-8 locale, grep correctly identifies it as a single character. Hence, my search for a 5-character word inside the file correctly returns 1 result. With LANG=C, no results are found. This feature is not, however, worth making grep so slow.

If you try to reproduce the problem above, probably you will not succeed, at least in your Linux system. This is because most Linux distributions are well aware of the problem and ship a patched GNU grep, and have been doing so for years. Debian does it (and with it, Ubuntu), Archlinux does it, Fedora does it, etc. Other distributions like Slackware traditionally ship software as vanilla as possible, and the problem shows, as seen above. Slackware’s GNU grep is completely vanilla. Most distributions use slightly different versions of the same patch, which replaces the MBS (Multi-Byte Sequence) treatment almost completely.

In my most recent scripts, I avoid GNU grep altogether, and use the fantastic and very efficient PCRE library (Perl Compatible Regular Expressions), used by many open source software projects (e.g. the Apache web server). The pcre package is present in most Linux distributions and BSD ports systems. It will probably ship the pcregrep tool inside. This is an alternative grep which features compatibility option-wise with the most common POSIX and GNU options, like -n, -l, -r, -w, etc. It expects, however, a Perl regular expression. They are, in the most common cases, like every other regular expression syntax out there, but closer to egrep than grep. By default, pcregrep behaves like grep with the LANG=C locale, even if your locale specifies that you are using UTF-8. It’s this fast:

$ time pcregrep '^....' /usr/share/dict/words >/dev/null 

real    0m0.061s
user    0m0.042s
sys     0m0.003s

A bit slower than grep with C locale, yes, but not a problem. In addition, you can activate UTF-8 mode to enable compatibility with multi-byte characters by using the -u option, explicitly. In this mode, pcregrep is not much slower:

$ time pcregrep -u '^....' /usr/share/dict/words >/dev/null

real    0m0.068s
user    0m0.049s
sys     0m0.002s

Of course, it’s able to behave correctly in the previous UTF-8 test with the -u flag:

$ echo ámbar | pcregrep -u '^.....$'
ámbar

Moving away from GNU grep to pcregrep is not a bad option. You get consistently fast behavior, regular expression syntax compatible with Perl, and get to choose if you want UTF-8 compatibility or not by providing an explicit option. So long, GNU grep! Welcome, pcregrep!

Final note: GNU awk suffers from this problem too, but its behavior with a UTF-8 locale is more or less equivalent to a patched grep. Still a bit slow, though.

$ time awk '/^..../' /usr/share/dict/words >/dev/null

real    0m0.373s
user    0m0.342s
sys     0m0.003s
$ export LANG=C
$ time awk '/^..../' /usr/share/dict/words >/dev/null

real    0m0.075s
user    0m0.055s
sys     0m0.002s