503 Service Unavailable

2007-10-29

Life of a software package

Filed under: Software — rg3 @ 01:12

Note: This text has been copied almost verbatim from the slackwiki entry “Life of a Software Package”, which was also written by me. I’m replicating it here out of interest and to have a backup. ;)

Introduction

This article explains how a program someone writes in one side of the world ends up being managed in your system. It’s meant to be easy to understand for a novice user coming from Windows, and only requires some basic knowledge of Unix systems. Specifically, the reader should know:

  • How basic Unix permissions work.
  • How to interpret the basic output of the ls command.
  • How a command line interface works.

It only contains general ideas that could help a novice user understand the existing differences when installing software under Windows and under Unix, but no specific information about how to do it. The operating system manual will give you the specific details you need, and may be a good read after you have read this article.

From source code to machine language

Note: You do not need to run any of the commands in this section. It’s enough to understand the text and part of the output.

When a programmer creates a program, it’s very common to write the program in a so-called high level language. In other words, he doesn’t create the program executable directly specifying the intructions for the computer to run. He writes the program in a language that allows him to represent the program structure and logic, and then that logic is translated to a language the bare machine can understand. Let’s suppose someone wants to write a Hello, World program in the C programming language. This is a program that prints a message to the screen and finishes, in order to check your system can translate the language you have used to machine language and to test the basic stuff works, as well as to give the novice programmer an idea on how the high level language works. This would be a very simple Hello, World program written in C:

#include <stdio.h>

int main()
{
	printf("Hello, World!\n");
	return 0;
}

You do not need to understand that program. The above text is what is commonly known as the source code of the program. The source code of a program in C is usually spread over one or more source files, so this program could very well be stored in a plain text file called hello.c. You could view this file using a plain text editor like the Windows notepad.

This file cannot be executed directly because it’s not a real program. First off, it doesn’t have execution permissions, and if we tried to give it execution permissions and run it, we would get an error message:

$ ls -l hello.c
-rw------- 1 rg3 users 74 2007-10-15 19:27 hello.c
$ chmod +x hello.c
$ ./hello.c
./hello.c: line 3: syntax error near unexpected token `('
./hello.c: line 3: `int main()'

We need to use a program called compiler to get a binary and executable file. A file in a specific format that the operating system (Linux in our case) can understand. When you run it, the operating system reads the file, copies the different program components to the computer’s memory and starts the program execution. If our machine has everything ready to compile C programs and we want to use the GNU C Compiler (gcc), we could do something as simple as:

$ gcc -o hello hello.c

And we would get a file named hello which is our program ready to be run. As you see, our program is simply called hello and not hello.exe, which would be a common name if we were working under Windows. In Unix systems, the convention is that programs do not have any file extension in their name (like EXE). We could then run the program and see that it does what we wanted.

$ ./hello
Hello, World!
$

Complex programs use libraries

Most programs do a more sophisticated task than printing a message on the screen and finishing. The source code of the program above has 7 lines. It is not uncommon for a simple program to have thousands of lines, and there are a good amount of complex programs out there that have millions of lines of source code. It is also a very common practice to use items called libraries (usually shared libraries) to build your program. Libraries are files that contain the machine code to perform several different tasks. For example, let’s suppose you were going to create a program that needs to download some data from the Internet, via HTTP (the web) or via FTP or another network service. And also let’s suppose you don’t have much knowledge on how to create a program that talks to others using the network, or that the focus of your program is on solving some other problem and you don’t want to lose time or create a lot of code just because you want it to be able to download a file. Fortunately for you, there is a library called libcurl that makes retrieving files over the network very easy. The library contains all the code you need, so the source code you are going to create will not contain anything specific to be able to use the network. You simply indicate that you want to use libcurl and call the library functions everytime you want to download a file. Do you need to learn to sail or fly a plane or a different language if you want to send a letter to a friend in a different continent? No, you put the letter in the mailbox and the postal service does it for you. Libraries work like this.

In the moment you decide to do this, your program starts depending on libcurl. Some pieces of the library need to be present in your system if you want to compile that program to create an executable, and some other pieces need to be present in the moment you want to run the program. Else, the program will not compile or will not be able to run. Libraries are convenient because, if managed well, you can install them in your system once and they will be used by every program that needs them. This is why libraries are regarded as a good thing or a good idea in the programming world, in most cases.

In Windows, shared libraries are called DLLs, as the library files usually have the DLL extension. In Unix, it’s common for them to have the .so suffix, or some other containing it. For example, I have libcurl in my system, and the shared library is located in the file /usr/lib/libcurl.so.4.0.0.

Libraries and programs all over the place need a package manager

So your system is going to be populated by a lot of programs, many of them using many different libraries for different tasks, some of them having some libraries in common, others having nothing in common. As you can guess, this situation can evolve into a pretty chaotic system. Let’s describe how Windows did this in the past, and how Unix systems have been trying to handle the situation for a good amount of years now.

In Windows, most people distribute programs already compiled. You get a group of files or a single file that holds your program already prepared to be run. You extract those contents and place them somewhere in your hard drive, usually all of them under a specific directory (folder). You could then create some shortcuts in the start menu and the program is ready to be run. A installer program usually does all of this for you, asking some questions. What happens when the program uses a library? If the library is not very common and cannot be assumed to exist in a standard Windows installation, the common practice is to include a copy of the library with the program. If it’s a relatively uncommon library, the installer usually puts it in the same place as the program, and when it is run and requests the library, the system first looks in the folder holding the program and finds the library there, and starts to use it. If it’s not an uncommon library but you need a specific version of it, the installer may try to install it in a common place so all the programs can use it. It was very typical, when you had a system in which you had installed a lot of software as time passed, that the installer would ask you “I am trying to install the following library, but it appears to be present in your system in a newer version. Do you want me to replace the copy of it with my copy or do I leave it as it is?”. And, in the same line, when you removed the program it would say “I was going to remove this library from the common place, but other programs may want to use it. Can I remove it or should I keep it there?”. This chaos was called “DLL Hell”.

Unix tries to avoid this problem in several ways, and its solutions bring the need of a package manager as we will explain. First off, in Unix the files on your hard drive are not grouped by program, but by their function. All binaries are stored in two or three folders, and the same for libraries or help documents. If all the documentation and help for the different programs is installed in a common location, it’s easier to create a help system from which you can browse the documentation of every program installed on the computer, for example. This is generally considered a good idea and it’s the tradition, but of course the idea has its detractors. Anyway, a second difference is that in Unix programs are distributed alone. If a program needs a library to be run, it needs to specify that somewhere, but only under special circumstances it’s recommended to include the library as part of the program. In most cases, the library is distributed apart. Those two differences avoid the DLL hell. By installing libraries, programs, documentation and other data to common system locations, you avoid duplicating data. If there’s a security problem using a library and every program using it could become compromised and make the system vulnerable, you update the library once and all the programs that use it are automatically protected, as each program doesn’t include its own copy in its directory. By separating the programs from the libraries they use and distributing them apart, you make sure programs do not overwrite the libraries used by others or remove common libraries when they are removed.

However, the solution itself brings some new problems. For example, if a program installs files all over common system directories and I later want to remove it, how do I know which files need to be removed? And if a program requires a library to run, can I or should I specify that fact somewhere? Package managers are the answer. Under Unix, software is many times distributed as packages. Packages are groups of files that contain programs, libraries, documentation or simply data. Under Windows, to install a program many times you download an installer file, run it and the program is installed. This installer file that holds inside all the files the program needs and extracts them to the proper location could be considered a form of package, so you get an idea. Packages in Unix are usually managed by a package manager. A package manager is a program that allows you to install packages, check the list of installed packages, remove packages and many more complex tasks depending on how powerful and featureful the package manager itself is.

When you install a program using a package, the package holds the program binary, the program data and the program documentation, typically, along with information on where those files should be installed in the system, all over the place. Fortunately, prior, during or after the installation, after copying the files to your system, the package manager records somewhere the name and version of the package and the files it installed. This is the trick that allows you to later remove the package using the package manager without having to remember which files had been installed where. In addition, the package may hold information about other packages it needs installed for it to run, and this information may be used by the package manager to automatically download and install them too. Hopefully, you now start to understand the practical vision behind packages and package managers.

Too many package managers

The problem with package managers is that many Unix systems and even many different Linux distributions use many different package managers. Each one uses different package formats that cannot understand each other. Slackware uses pkgtools, Debian and Ubuntu use apt-get, Red Hat uses yum, Mandriva uses urpmi, Arch uses pacman, Gentoo uses portage, etc.

You are a programmer and created the Hello, World program we saw at the beginning. How do you distribute your program? You have several solutions. If you don’t want people to get the source code of your program, you need to distribute the program already compiled and probably packaged. To do that, you could provide your own program to install and uninstall the package cleanly from any system, and distribute it somehow and break some rules to achieve maximum compatibility, so the program will run on many different systems and distributions. Many commercial games are shipped this way. Unreal Tournament 2004 for Linux is distributed this way, for example. You could also provide it as a package for each of a subset of supported systems. Many companies do this. They give you Debian, Red Hat, Suse and Mandriva packages for you to choose, for example, each in the proper format to be used with the package manager from that distribution. If you want to use the software under other system you are out of luck. You can try some tricks but it’s not guaranteed to work.

If, on the other hand, your program is open source and you don’t mind people reading the source code of your program, the common case is to avoid creating a package for anything. You simply distribute a tarball (similar to a ZIP or RAR file) containing the source code and instructions to compile it. If someone wants a Debian package to install it, someone will have to compile your program under Debian, and make a package with the result. This is a very very very common case. In fact, distributions like Ubuntu or Debian heavily rely on package repositories, network locations from where you can download thousands of packages for your system, created by a myriad of official and unofficial packagers (people that create packages for the system). For example, if you want to install a program under Ubuntu, it’s very infrequent for the program not to exist already packaged in a repository, and you can download and install it, together with its dependencies, in a couple of mouse clicks.

Summing up

  • Many times the programmer creates programs using source code that must be compiled.
  • They use libraries to make writing programs easy.
  • Distributing the resulting program is easier using a package manager.
  • Many programmers only give you the source code due to the diversity of package managers and systems.
  • Someone else is responsible for creating a package for a specific system.

Slackware specifics

Slackware is, as you may know, a very simple system. Being simple doesn’t mean it’s simple to use. On the contrary, a system with a simple design and simple tools usually requires the user to do more things to achieve a goal. The advantage of a simple design is that it’s easier to understand if you want to know how your system works, and sometimes it’s also more stable and has less bugs. As part of its simple design, the package manager in Slackware is also very simple, and its packages are also very simple. Slackware packages are tarballs (again, something like ZIP files) that, if extracted in the right place, will populate the system with the package files, and it also holds some special files with information about the package itself. As they are simple tarballs, Slackware doesn’t try to hide this fact, and Slackware packages have the tgz extension (short for tar.gz), contrary to other systems in which packages have a special extension to make it clear that they are packages, like rpm or deb.

This is not a problem, but sometimes this confuses novice users. They go to the program webpage and download the program source code in a tarball (usually a file with tar.gz extension) and think “Hey, if Slackware packages are tarballs and this is a tarball, I’m going to install this file with the package manager”. Wrong! Even when the package manager complains that the package name does not end in tgz but on tar.gz, they many times rename the file and try again. Those are two mistakes in a row. The package manager will try to extact the tarball contents to that special location we mentioned earlier and nothing will happen, as the files inside the tarball are not structured as they need to be, but this is the small problem. The big one is that what you are trying to install is the source code of the program, and not the program itself! Remember, you need to compile it first in the majority of cases.

Under Slackware, you should first check if there is an official package for the program. If there is not, you could try to to download a ready to use package from a place or someone you trust. Else, you could compile the program yourself and create a package for it, and then install the package. The compilation and package creation can be automated sometimes for ease of use, for example using SlackBuild scripts.

Advertisements

2007-10-05

Formulae to raster images

Filed under: Software — rg3 @ 10:58

Related to the previous post, AsciiDoc is missing a feature some people will consider important, which is the ability to include formulas as part of any document. It features ASCIIMathML, a JavaScript trick that allows it to transform mathematical formulae in plain text to HTML code representing it. This option, however, is only available using the “xhtml11” backend and there are no other methods, at least for now. David Jones also mentioned MathTran, which can be used for similar purposes with (X)HTML backends.

If you want to include your mathematical formulae in any type of document, one of the best options is to convert the formulae to a raster PNG image and include them in the document as an image.

After Googling a little bit trying to find a program that would simply convert a formula to PNG or GIF or another raster format, I found several scripts to do the job. The general procedure is described as follows:

  1. Write the formula using LaTeX notation.
  2. Embed it in a temporary file containing a full LaTeX document.
  3. Process that file with LaTeX to produce DVI output.
  4. Convert the DVI to EPS using the “dvips” command.
  5. Convert the resulting EPS to PNG using ImageMagick’s “convert” tool.

I copied one of them to my computer and tried to use it, but it didn’t work as I expected. This is because I was interested in DocBook output instead of (X)HTML, so I tried to create very big image files so when embedded in the final PDF the formula would have enough quality to be printed as part of the rest of the text without anyone noticing the pixels. Using the scripts I found on the net, I was able to create very big images, yet when embedded in the final PDF they looked giantic. Even a normal image would look too big and pixelated.

I tracked the problem down to being an issue with the density of the image. The density or resolution of the image, as opposed to its size, is a metadata field that can be specified in some image formats (PNG being one of them) that registers the number of pixels in the image that must be used per physical measure unit when putting the image on a physical medium. Typically, it’s given in pixels per inch. In the procedure described previously, you can see that we are working with a vectorial image until the last step, in which it’s converted to a raster format. It’s in this last step when you specify the image density. In theory, when you specify a large density like 1200×1200 for printed output, that density is recorded as part of the PNG image. Most scripts out there, however, forget to pass an additional parameter to the “convert” tool, called “-units”, to specify the units of the image resolution. This option is mandatory to get the expected result automatically when embedding the image in a DocBook document and generating the final PDF using “fop”. You can view the horizontal and vertical resolution of a PNG image using the “identify” tool from ImageMagick.

identify -format 'H: %x\nV: %y' sample_equation.png

If the image has been correctly generated, you should see something like this:

H: 472.44 PixelsPerCentimeter
V: 472.44 PixelsPerCentimeter

On the other hand, when you have forgotten to specify the units in a 1200×1200 ppi conversion, you will see the units are undefined:

H: 1200 Undefined
V: 1200 Undefined

Such an image will not be displayed properly when embedding it in a PDF document.

As a result of this, I created my own script to convert formulae to raster images. As input, it needs a list of files with formulae you want to convert, one per file, in LaTeX format.

Source code (public domain)

#!/bin/sh

usage_and_exit() {
	echo "Usage: $0 [-d density] LATEX_FORMULA_FILE ..." 1>&2
	exit 1
}

formula_file_to_png() {
	# File names taking part of this
	texfile="${1}.tex"
	dvifile="${1}.dvi"
	epsfile="${1}.eps"
	pngfile="${1}.png"
	auxfile="${1}.aux"
	logfile="${1}.log"

	test -f "$1" -a -s "$1"
	if test $? -ne 0; then
		echo "Skipping ${1} ..."
		return 1
	fi

	echo -n "Creating ${pngfile} ..."

	# Create complete TeX file with formula
	cat 2>/dev/null >"$texfile" <<-EOF
	\\documentclass{article}
	\\pagestyle{empty}
	\\usepackage{euler}
	\\begin{document}

	\\[ $( cat "$1" ) \\]

	\\end{document}
	EOF

	# Generate DVI -> EPS -> PNG
	latex "$texfile" </dev/null >/dev/null 2>/dev/null \
	&& dvips -D 72 -E "$dvifile" -o "$epsfile" >/dev/null 2>/dev/null \
	&& convert -units PixelsPerInch -density "$density" "$epsfile" "$pngfile" 2>/dev/null \
	&& rm -f "$texfile" "$dvifile" "$epsfile" "$auxfile" "$logfile"

	if test $? -ne 0; then
		echo " failed"
		return 1
	fi
	echo " done"
	return 0
}

if test "$1" == "-d"; then
	shift
	density="$1"
	shift || usage_and_exit
else
	density=1200x1200
fi

if test $# -lt 1; then
	usage_and_exit
fi

echo "Density: $density"
while test $# -ne 0; do
	formula_file_to_png "$1"
	shift
done

Example

You can find many guides online if you are unfamiliarized with the LaTeX formulae notation. See for example the contents of sample_equation:

x = \frac{-b \pm \sqrt{b^{2} - 4ac}}{2a}

If you save the previous script to a file named formula2png in your PATH, you would run something like:

$ formula2png sample_equation
Density: 1200x1200
Creating sample_equation.png ... done

The default density is 1200×1200, more than enough for the formula to be printed flawlessly in a home inkjet or even laser printer. You can modify the image density (and get a smaller image suitable for the web if you prefer) by using the -d script option, like:

$ formula2png -d 150x150 sample_equation

This is the output using a density of 150×150:

Remarks

Some similar scripts out there use additional “convert” options like “-trim” (which I found unneeded) and “-transparent #FFFFFF” to make the white color transparent and get a transparent background image. This, at least in my case and probably everyone else’s, doesn’t work as expected. Only the completely white pixels are made transparent. As “convert” blurs the font borders a little bit to make them look good, the text is surrounded by a narrow almost-white border that looks wrong if you put the image on a non-white background. That’s why I didn’t bother and generate white background images.

I also tried to use a tool present in my system, called “dvipng” to translate the LaTeX DVI output directly to PNG. It works fine but the resulting image has big white margins with the formula located in the bottom right corner of the image. When you translate the DVI document to EPS you avoid this, as the generated EPS file has strictly the needed margins, so I kept the original method. If you want to use “dvipng” you would need to run ImageMagick’s “convert” or “mogrify” tool on the resulting image, using the “-trim” option, which would be useful in this case.

2007-10-01

Creating structured documents from plain text

Filed under: Software — rg3 @ 21:08

Since personal computers appeared, they have been trying to solve the problem of text composition. The goal is to be able to write and format a long piece of text like a book or an article with minimum work. Ideally, you would only have to write the text itself and nothing more. This is impossible, since you need to provide information about how the text should be formatted, either by providing the formatting information yourself or providing clues about the document structure so the computer, which is smarter than a typewriter, can format the text itself. Or maybe you could provide a mixture of the two, the structure and some directives about how you want it to look like.

Typewriters

Think for a moment about typewriters. They are simple devices, with many limitations, but they are easy to use and let you concentrate on the contents of what you’re writing. The most basic models can’t make text bold or italic, and they can’t change the size of the text. These restrictions force you to play by some rules and forget a bit about the appearance of text. Naturally, they have disadvantages derived from being a mechanical device, in the sense that you type the text directly to the final physical medium, mistakes are hard to correct and you have to think beforehand how you are going to number the document sections and other aspects. But, apart from that minimal preparation, writing the text is straightforward.

Word processors

Word processors are the inmediate and typical computer solution to the problem. They are sophisticated typewriters, as powerful as presses. Word processors, with time, have become big programs with many options and features, trying to become the swiss army knife of text composition. It’s unfortunate that many people use them taking every decision on the document format without paying attention to the structure of the document itself. It’s very common to use two Enter key presses to separate paragraphs, using spaces or tabs to indent the first line of them or to somehow align them at some position. Those documents lose all the advantages of being managed by a computer, because in reality they are “typewriter documents” that, on top. have styles everywhere that have been applied by hand and can’t be changed easily.

It is true, however, that most word processors let you define paragraph styles (document title, section title, subsection title, normal paragraph, etc) and write your document quickly. Everytime you write something you indicate what it is, and define easily somewhere else how that type of paragraph is to be represented. If later you want to make all section titles underlined, it’s a matter of changing the style of section titles instead of going over every section title. The same goes for section numbering and automatic generation of Tables of Contents (TOCs), among others. Some word processors like LyX force you to work this way. For many people, that’s a positive thing. I’ve used LyX to write some very long documents and I must admit that it’s a pleasure to work with it. LyX uses LaTeX in the background, but that’s another story.

This approach to the problem is relatively good. You barely lose time thinking about the text format. Strictly speaking, you only write the text itself. The rest is configured usually through a handful of mouse clicks. However, word processors have traditionally suffered from other problems. For example, if you want to send the document to someone else, you usually need to be sure the other person has a word processor capable of opening that type of document (maybe the same word processor you used). Until not long ago, the document files on disk were stored in particular and undocumented formats. This was specially true for Microsoft Word, the most widely used word processor. You couldn’t and in many cases can’t automatically operate on them, or export them to a format really suitable for the web. Still, like I mentioned before, when you used paragraph styles the problem was solved nicely even though the resulting document wasn’t very flexible.

Nowadays word processors try to output their documents in an XML dialect so they can be translated easily to other formats and gain the flexibility they lacked in the past. Remember that word processors want to become the swiss army knife of text composition, and always feel the need to adapt and meet the needs of everybody.

Structured plain text

On the other hand, we have structured plain text formats like TeX (or LaTeX), HTML, XHTML or DocBook. These formats tried to do the right thing from the start, more or less. You would create a plain text file with your text editor of choice. It would hold the document contents and information about the document structure. Based on this structure, you give the computer some instructions (or none) about how the document should be formatted when appearing on the final presentation medium, be it paper or the computer screen.

In other words, they provide the same solution as a word processor using styles, with some differences. First, by their plain text nature, they have or had a bit more flexibility than their word processor counterparts. If you are going to write the plain text file yourself, it’s obvious that the internal file format is open and well known and, even more important, understandable for a human. In a worst case scenario, you could send them to someone not having programs to format or display it and they could still read it. But, as the format is open, it’s much easier to find tools to represent those document no matter whatever platform you’re using, with varied degrees of accuracy. Everyone who has written a webpage knows the problems Internet Explorer has with Cascading Style Sheets (CSS), if you decide to create a sophisticated one.

Furthermore, many of these documents can be put on the web easily. HTML and XHTML are designed directly for the web, and DocBook can very easily be translated to them. There are also tools for LaTeX that allow you to transform it to HTML (latex2html being one). In the modern world where everything can be done online and people start to avoid using sheets of paper, everytime a group of people discuss about text composition somebody suggests to avoid word processors completely and write your documents in (X)HTML instead. They would be structured, they can be easily printed if needed, transformed to other formats and, with CSS, you can easily change their appearance. These people definitely have a point in what they say. The problem is… how to say it… that writing (X)HTML and DocBook tags is completely annoying, among other minor problems. You need more than a plain text editor to create them easily if you don’t want to lose your nerves writing tags. This is obviously a problem…

Formatted plain text

… because formatted plain text formats and applications wouldn’t exist otherwise. You probably use these applications more than once a day. They are present in forum software and wikis. Surrounding every paragraph with <p> or <para> makes you lose too much time, so some people has rightly thought “What if we rescue the good things typewriters had?”. Like using the Enter key to separate paragraphs. Can’t we use those formatting elements to infer the document structure from them? Furthermore, the people who use word processors using the Enter key to separate paragraphs would feel almost at home. Even with minor glitches, in my humble opinion, formatted plain text is probably one of the best ideas in text composition.

For example, much web forum software lets you break paragraphs by pressing the enter key twice. Some of them translate your text to HTML using <br /> elements. Others surround each paragraph with the proper tags. The level of sophistication varies. That change alone lets you structure your document writing much fewer tags. I’m using that at this moment, as the blog software surrounds my paragraphs with the proper tags automatically. In MediaWiki, the wiki used by Wikipedia, the formatting goes further. Titles are surrounded by equal characters (=) that are way easier to type than header tags. You could have:

= Section title =

Section paragraph.

== Subsection title ==

Subsection paragraph.

Lists can be written using asterisks or dashes prepending each list item. But wikis and forum software have a problem: it’s not very easy to get the documents out of them. So the last big question is… is there any software that lets me use plain text format as indicator of document structure easily, without the need for tags, and that is independent by itself, so it can then be translated to (X)HTML, DocBook or something else? The answer is yes: AsciiDoc. I “discovered” AsciiDoc some years ago, but it was not until today that I decided to lose some minutes reading the user guide (that explains the conventions used to format the plain text and how they are going to be interpreted by AsciiDoc). Shame on me, because it’s very easy to use. AsciiDoc turns out to be incredibly flexible and powerful, yet the default settings are more than enough for most users. I inmediately converted a couple of webpages to AsciiDoc to see the results and they are excellent. I think I’m going to keep using AsciiDoc whenever I can, because the XHTML output is flexible and customizable, and the DocBook output can be easily translated into chunked HTML or serious-looking PDF documents using FOP. Give AsciiDoc a try if you ever need to write an article, book or Unix manpage.

Blog at WordPress.com.