503 Service Unavailable

2007-01-28

Free data

Filed under: Programming,Software — rg3 @ 23:49

With a couple of posts talking about Slackware, you may know by now that I’m a Slackware Linux user. It’s the system I run on my main computer, which happens to be an Acer Aspire 3003WLMi laptop. Hence my published review of that computer.

Knowing that, I felt I had to clarify my view on free software, open source software and proprietary software, so the few people who read me or will read me actually know the type of person that text is comming from. I always avoid discussing about “gray area” topics and tend to accept other people’s point of view easily. For that reason, I’ll try to avoid non-technical topics as much as possible, this post being an exception to that rule, but I’ll try not to break it again.

The following statements are my opinion. They’re not the truth and I don’t think everybody else should have that same opinion. All warnings in place, let’s begin.

The Free Software Foundation believes all software should be free. By free they mean software which gives its users four basic freedoms, which are the freedom to run the program, the freedom to study its internal workings and adapt it to your needs, the freedom to redistribute the program and the freedom to improve the program and release those improvements to the public. On the other hand, the open source apologists try to see it from a practical point of view and consider open source software better because it has development advantages that should increase the quality of the final program, but by no means they consider proprietary software inmoral, and concede sometimes the best tool for a given job may be a proprietary program.

My vision is different to both. I like freedom, and I think freedom is one of the most important qualities a human being should try to achieve. It must be noted that being completely free is probably impossible and it’s a topic that can act as an starting point of very long philosophical discussions. While working with computers, the most important and inmediate user freedom is the ability to decide which software (s)he wants to run. This is probably the only important freedom, in my humble opinion.

In order to achieve that, two conditions must be met. First, the user needs general purpose, open hardware available. Computers with open specifications what can run any software designed to run on them and that won’t check which software they’re running. Those computers would let you write and run your own programs if you decide to do so, and provide the appropriate hardware platform to be free in the sense I mentioned.

Finally, the user and the industry in general needs open protocols, formats and specifications. Because, at the end of the day, what really matters is the data, not the programs. The programs are tools to solve problems, to generate data that can be manipulated, stored and retrieved in the future and by yourself or another person. If we want everyone to have the freedom to choose the software they run, we need to communicate using open protocols. When I send someone a “document” I try to do it in an open format. I want to make sure they will be able to work with that piece of data (view it and/or manipulate it) no matter what platform they choose, be it Linux, Windows, Mac, FreeBSD, NetBSD, OpenBSD, Solaris, HP-UX or whatever. I know software is complex and that a normal computer runs a whole stack of it starting from the kernel to the different libraries and programs. Sometimes I may send someone a piece of data they can’t work with under the platform they have chosen, because it lacks the appropriate tool. However, by using an open format I can at least guarantee that someone may develop a tool of his/her taste in the future, and I’m showing them my desire that they keep that freedom of choice they have.

Free software advocates often mention goverments, schools and any other public body should use free software. I don’t think that way. They need to run the best tools, the tools that let them do the job as conveniently and cheap as possible. However, one of the requirements for those tools should be to store the important data in standarized, open or otherwise perfectly documented and tested formats.

It’s been years since I stopped advocating Linux and other free software. It doesn’t really matter. I don’t mind which platforms the people choose. Repeating again, it’s the data, not the programs. Of course, this has small exceptions. After all, aren’t the programs data themselves? Yes. That’s why, when developping a program, I try to do it as open as possible using standarized or open languages and libraries. At the same time, I know sometimes this is not feasible.

Considering programs data themselves is important in some other situations, like when running the program is the objective by itself (think of games) or when the program produces data but it’s not generated by itself, so it has no responsibility in creating it under an open specification (think of youtube-dl, which simply downloads the video data). These programs ideally should be platform independent, thus the need for open multiplatform libraries, languages and tools providing the foundation to build on them and creating any type of program (open source software, free software or proprietary software) and make it available for as many platforms as needed and let people choose freely.

It’s a complex topic with rough edges. However, I think I have outlined my general line of thought in the paragraphs above.

2007-01-20

Making Sendmail listen on the loopback interface only under Slackware

Filed under: Software — rg3 @ 16:26

Slackware is one of the few “important” distributions which ships Sendmail by default instead of another MTA like Postfix, qmail or Exim. In any case, the important detail is that it’s usually a good idea to run an MTA in your computer. Many programs like cron use the sendmail command to mail the output of the programs they run. This output is generally empty but when it’s not, cron mails it because it will probably contain error messages. Another good idea is to have the root email account aliased to your normal user, so you don’t forget to check these error messages and detect problems as soon as possible. Further setup may involve setting up your email client (be it KMail, Thunderbird or whatever) to check this local mail account.

In any case, let’s suppose you don’t mind which MTA Slackware installs as long as you have your ass covered, so you have Sendmail installed but, by running netstat --listen --tcp --udp or a similar command you discover Sendmail is listening to the world in not one but probably two different ports. You may want to disable the second port, which may be important if your machine was a real mail server, but it’s not. You may also want to make Sendmail listen on the loopback interface only. It’s quite simple.

You need to install the m4 and sendmail-cf packages if they’re not already installed, apart from the obvious sendmail package which will probably be already present in your system. Head to the /usr/share/sendmail/cf/cf/ directory. You will find a file called sendmail-slackware.mc there, which stores the default configuration values. I’d recommend making a copy of the file instead of modifying it directly, with a meaningful name like localhost.mc or loopback.mc.

Open the copy and add the following line in the “FEATURE” block:

FEATURE(`no_default_msa')

Also, before the “MAILER” lines, add the following line:

DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1,Name=MTA')

Processing that file with the m4 command from that directory will output a configuration file suitable to replace the default sendmail.cf present in /etc/mail/. Hence, run the following command replacing loopback.mc with the name you chose:

m4 loopback.mc >/etc/mail/sendmail.cf

And finally restart Sendmail by running :

/etc/rc.d/rc.sendmail restart

If you now run netstat again, you should see Sendmail listening only on port number 25, on the loopback interface and unreachable from the outside world. I’m not really sure if you need to add the lines in those specific positions. Probably, a Sendmail expert will know if you can or cannot, but I haven’t tried. It works without errors in the positions I mentioned above.

2007-01-17

Less loops, less bugs

Filed under: Programming — rg3 @ 18:07

In programming, it’s a known fact that the more lines a program has, the more bugs it has. Several studies reached this conclusion in the past and some of them even went as far as trying to predict the average number of programming mistakes per line. The number is not important, but the conclusion is, and it has led to the creation of the motto “less code, less bugs“.

An intuition I have, without any proof, is that we could also conclude “less loops, less bugs“, and I’ll explain why. Loops are the most complex constructions programming languages have. It’s obvious that a list of ordered steps are easy to understand and follow. Those are the typical sentence sequences in the form “do this, then do that, later that other thing“. Increasing in complexity, we have the if/else constructions. Those are more complex but, unless the bodies are very long and nested, you can still mentally trace what you are doing when you read a program from the top to the bottom. Loops are a different thing, however. A loop requires planning and does “more” than what is writen. Except for simple loops, many times you can’t read the loop and know what it does. You need to trace the loop behaviour, in your mind or on paper. There’s a condition on when to stop, and variables that change its contents based on previous steps, until the loop stops and concludes its assigned job. Most programmers at least put comments above loops to indicate what they do, and in which variables they leave the desired result. Some others go beyond that and explicitly write the loop invariant, so as to have the proof that the loop does what it says it does, like if the loop involved some kind of “magic”.

In any case, it’s obvious that writing a loop requires planning and reading it requires an extra effort. This difficulty makes loops, in my opinion, more error-prone than any other programming construction. So, basically, we have “less code, less bugs” and “less loops, less bugs“.

My three favourite programming languages are, in no particular order, C, C++ and Python. I like each one of them for a different reason. However, from those three, it’s Python the one that fits better with both goals. What allows you to have less code? A large standard library. Programming languages with large standard libraries or a large set of unofficial libraries or modules let you program closer to the problem definition and cuts code/time costs in two ways. First, less code needs to be writen to accomplish a given task because there may be a function or object that abstracts the behaviour you’re looking for. Second, when large abstract libraries are available, you don’t have to design them. Smaller design, less code, less bugs, less time, smaller cost.

And, finally, if the language allows you to have functional-programming-like constructions, or is itself a functional programming language, you can avoid writing too many loops. In C++, the “functional” header includes templates for common loops, so you can avoid writing them. Many C++ programmers will tell you not to write a loop if that is already present in the standard library. Going a little beyond, but not too much, I’d tell you to avoid writing a loop if you can do the same as a series of sequence transformations. Transform the elements from one sequence into another sequence, copy or remove elements if they match certain conditions, etc. Python’s list comprehensions and lambda functions allow you to do this. A series of sequence transformations many times let you perform the equivalent job a loop would do, but in the shape of the simplest constructions: a sequence of sentences.

There will always be exceptions to both mottos, but I think they should be taken into account when writing any program and choosing the appropriate programming language to do the job.

2007-01-14

Backing up DVD movies under Slackware

Filed under: Software — rg3 @ 13:58

Warning: you won’t find an easy tutorial using graphical tools below, but a description on how to do this using several command line tools. Also, it seems doing the following is illegal in some countries, for some stupid reason.

Let’s suppose you have a possibly encrypted DVD movie that you want to back up to one of those 4.4 GB DVD-R or DVD+R discs you can buy in a store. To do this, you must solve two problems. The first one is about the DVD movie being encrypted, that will probably make a simple copy with dd fail, reporting an Input/Output error while trying to create the disc image. The second problem you may have is that the DVD movie disc is too big to be copied directly. For example, my American Beauty original disc has around 7 GB of data and I can’t copy it directly.

So let’s start describing the available tools in general, before diving directly into Slackware. While trying to solve those two problems I reached a Gentoo HOWTO on backing up DVDs. It mentions several ways of handling encrypted DVDs. The tricks to make dd work didn’t give me good results. dd didn’t spit any error message creating the DVD image after I had played the movie with MPlayer, but the resulting image wasn’t properly created and failed to play in MPlayer itself and in the home DVD player. The second best solution in that list looks to be dvdbackup. You’d need to install libdvdcss, libdvdread and dvdbackup itself. Another good solution is to find out which patches, if any, Gentoo uses to make K3B handle encrypted DVDs and finally install the two already mentioned libraries and recompile K3B so it can copy encrypted DVDs too. None of those solutions solve the second problem, anyway.

There are some very good graphical tools out there that let you author DVD movie discs. One of the best is, apparently, ManDVD. But let’s have a look at the list of requirements and see which ones are not present in the Slackware official packages, if you were to use ManDVD:

  • DVD Slideshow >= 0.7.5 (not provided by Slackware)
  • mplayer/mencoder (not provided by Slackware)
  • mkisofs >= 2.01
  • xine > 0.99.4
  • lame >= 3.97 (not provided by Slackware)
  • dvdauthor >= 0.6.11 (not provided by Slackware)
  • mjpegtools >= 1.8.0 (not provided by Slackware)
  • netpbm >= 10.29 (not provided by Slackware)
  • ImageMagick >= 6.2.4
  • transcode >= 1.0.2 (not provided by Slackware)
  • dvd+rw-tools >= 5.21.4

For me, that’s too complicated. My solution was to use two unofficial packages only, MPlayer/Mencoder and dvdauthor, as outlined in a Linux.com article on creating video DVDs, which itself is partly based on some excellent sections from the MPlayer documentation. Specifically, the section titled “Using MEncoder to create VCD/SVCD/DVD-compliant files“. Two packages, which are not dependant on each other, are easier to manage in my humble opinion.

The outline of what you need to do is to create an MPEG file containing an AC3 audio stream and an MPEG-2 video stream, and use dvdauthor to create a DVD video disc with it. The Linux.com article describes how to convert any video to this format. This is even easier if you are creating the MPEG file from the DVD disc itself because it will probably already have the audio stream in that format, and the video stream in the proper format for sure.

Start by playing the DVD movie with MPlayer in order to locate the proper video title and select the proper audio stream with the -alang option, and also to get information on the audio stream format. For example, I’m interested in backing up my original American Beauty disc, using the Spanish audio stream. I play the movie with mplayer -alang es dvd://1.

MPlayer reports I’m using the Spanish audio stream and gives me the following useful information about the video and audio streams:

[...]
Selected DVD audio channel: 129 language: es
MPEG-PS file format detected.
VIDEO:  MPEG2  720x576  [...]
[...]
Opening audio decoder: [liba52] AC3 decoding with liba52
Using SSE optimized IMDCT transform
Using MMX optimized resampler
AUDIO: 48000 Hz, 2 ch, s16le, 448.0 kbit/29.17% [...]
[...]

Great, everything under the expected format, so I can try to copy the video and audio streams directly. The options needed to have a DVD-compliant MPEG file are -of mpeg -mpegopts format=dvd or, also, -of mpeg -mpegopts format=dvd:tsaf. The tsaf format option inserts timestamps on every frame if possible. This may be useful, so let’s use it. Do I need to reencode the video or audio streams? Not for now. Let’s run mencoder copying the audio and video streams and produce an MPEG file:

mencoder \
    -of mpeg -mpegopts format=dvd:tsaf \
    -alang es -oac copy -ovc copy \
    -o american-beauty.mpg dvd://1

The resulting file should be quite big. It takes some time to generate it but not too much. This is because we’re not reencoding the audio or video streams, but copying the streams as fast as we can. In my computer the process runs at about 175-200 frames per second. The movie plays at 25 frames per second in the PAL system. This means a two-hour long movie should be ready in 15-18 minutes. During the process, mencoder estimates how big the resulting file will be. In my case, it predicted a final size around 4500 MB, that is, 4.42 GB. Would that fit in the disc? We’ll see.

The next step is to create an XML file for dvdauthor, named american-beauty.xml in my case, with the following content:

<dvdauthor>
  <vmgm />
  <titleset>
    <titles>
      <pgc>
        <vob file="american-beauty.mpg" />
      </pgc>
    </titles>
  </titleset>
</dvdauthor>

And, finally, let’s use dvdauthor to produce a video DVD image:

mkdir image
dvdauthor -o image -x american-beauty.xml

This will take some time too, it will use another 4+ GB of disc space. When it finishes, the image directory we just created will contain a VIDEO_TS and an AUDIO_TS subdirectories, with everything ready. Run K3B and click on File > New project > New video DVD project. It will create a new project with the AUDIO_TS and VIDEO_TS directories. Clic on AUDIO_TS and drag the files from the AUDIO_TS directory in your harddrive to it, and repeat the operation with the VIDEO_TS directory. If the files fit in the disc, put a disk label and burn it.

Two side notes: there may be no files inside the AUDIO_TS directory, so don’t worry if it’s empty. You can also check the result before burning it, using mplayer -dvd-device ./image dvd://1.

What if the files didn’t fit?

If they don’t fit, you have several solutions. First off, you can buy a double-layer disc and use that instead. It will probably fit and I think the price is worth it. If you can’t burn double-layer discs, you can’t losslessly copy the video and audio streams as we did. You’d need to reencode the audio stream, or the video stream, or both. In my opinion, the first candidate should always be the audio stream. In the example above, you could see the audio stream was in AC3 format, 2 channels and 448 kbit. If it had 6 channels (5.1 surround sound), you could start by changing it to stereo sound, using the -channels 2 option and reencode the audio to a lower bitrate, replacing -oac copy by -oac lavc -lavcopts acodec=ac3:abitrate=192. I doubt you’ll hear the difference and it will save you some space. For example, I was off by a few MB and reencoding the audio stream was enough. On top of that, reencoding the audio stream is a cheap operation that won’t take much time, as opposed to reencoding the video stream.

If that’s not enough, you’ll have to reencode the video stream. You can replace -ovc copy using the options present in the MPlayer documentation webpage, using -ovc lavc and adding the video options to the -lavcopts command-line option.

Result

Once you’ve finally made it, the resulting disc will only contain the movie in unencrypted form with the selected audio stream. Insert the disc in your home DVD player and the movie should automatically start playing. No ads, no menus, no wait. Just the movie. I hope you like it that way.

2007-01-11

Imitating Yahoo’s AddressGuard using GMail

Filed under: Communication — rg3 @ 20:21

Since some days ago, I’ve been moving all my mail from Yahoo! to GMail. Both have their advantages and disadvantages, considering that I use an e-mail client to store all my mail in my hard drive. With that, I mean that I don’t use the web interface. I download all my mail using POP3 access. My main reason to stay with Yahoo! is that I had been [ab]using AddressGuard intensely. AddressGuard was still free in the Spanish version of Yahoo! Mail, as opposed to the USA and other countries in which it was only available via Yahoo! Mail Plus, which costs like 20 or 25 USD a year.

There’s a flash demo in Yahoo’s website that explains what AddressGuard is and how to use it. Essentially, it’s a mechanism that allows you to have disposable e-mail addresses in a very convenient way. You have to create some sort of secondary Yahoo! ID which is used as a prefix for your disposable addresses, which have the form prefix-suffix@yahoo.com, replacing the .com domain by the relevant one. That secondary ID, which does not relate to your original Yahoo! ID is used as the prefix, and you could create suffixes quickly on demand. It’s like a web interface to qmail’s alias system.

You may know that one of the best ways of avoiding spam (probably the best) is not letting the spammers get your real e-mail address, and this is what AddressGuard facilitates you. Instead of giving people your real, true address, you gave them secondaryid-suffix@yahoo.com, and they couldn’t, from that, reach or know your primaryid@yahoo.com address. Mail sent to secondaryid@yahoo.com is not accepted. In fact, only mail sent to valid suffixes is accepted and ends in the inbox of your Yahoo! mail account. So you have your Internet newbie friend John who asks you for your e-mail address and you give him secondaryid-john@yahoo.com. In the moment John makes a mistake by sending you a chain-letter email, or is infected by a virus or malware which leaks that address to the public and you start receiving spam, you delete that suffix. You may inform John of what has happened, using the level of violence you consider appropriate, but you won’t receive more spam. This can also be used for other friends, websites, online stores, forums, mailing lists, etc.

I had abused the system so much that I had around 50 different suffixes in around 50 different places. This fact together with knowing that I wouldn’t have a similar system in GMail is the reason I considered to stay with Yahoo.

On the other hand, GMail had its own advantages. For example, GMail’s POP3 and SMTP access is secure. This meant that my username and password were going to be transmitted over an encrypted connection, detail that I like. It’s not about the e-mail message itself (I would use GPG for that), but about the username and password. Also, GMail doesn’t put ads at the end of the message like Yahoo does (unless you’re paying for Yahoo! Mail Plus), which is better when you send mail because having those ads at the end of the message looks very unprofessional. Also, GMail’s attachment size limits are higher. I don’t remember the exact numbers, but they are.

Anyway, I thought of a way to imitate Yahoo’s AddressGuard with GMail and then I moved all my mail. It was a long task, converting a handful of addresses each day until I finished. The system is not perfect for several reasons. I’ll explain it all.

I created 2 accounts. One was going to be the equivalent of Yahoo’s primary ID. The second one would imitate the secondary ID. GMail also has a suffix system, but it’s different to the one from Yahoo. In GMail, mail sent to yourid+suffix@gmail.com is also delivered to your address. This smells like a web interface to Postfix’s alias system. However, GMail doesn’t let you decide which suffixes are accepted, and it won’t hide your real ID. I chose to redirect every known secondaryid+suffix@gmail.com address to primaryid@gmail.com using GMail filters. GMail doesn’t impose a limit in the number of filters. Caveats in this system, which may be important but so long they haven’t been for me?

  • GMail “To:” filters let you put the address there, but they will only look at the To and Cc e-mail headers. If someone includes you in their Bcc header, it won’t be forwarded. Note that the Bcc header won’t be present when you receive the message either. The same happens when someone doesn’t explicitly put your address in the message. This is typical in mailing lists. Mail is sent to the list address, but doesn’t mention who is receiving the message. This second case has a solution, which is to include an explicit filter so that mail sent to the mailing list (identified by the To header, words present in the subject or whatever) is also forwarded. The first case is unresolved. I mailed the GMail team so the system would use the Delivered-To header instead, which mentions your address, but so far I haven’t received a reply about it.
  • If an e-mail message is classified as Spam, it won’t hit the message filters. This means you have to check your mail using the web interface from time to time, just in case an important message is not filtered. Currently, there’s no way to disable GMail’s spam filtering features.
  • You can’t use the SMTP server and use any From header. If you use your primary ID account to authenticate to the SMTP server, the From header will be replaced. If you not only want to receive messages to secondaryid+suffix@gmail.com, but also to send them, that is, use that address in the From header, you have to explicitly tell GMail to allow you, and you have to pass a simple verification process to do so.
Next Page »

The Rubric Theme. Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.