Bulk Update With Mercurial

Some of you may well know that I was previously an subversion user, more out of comfort than necessity. SVN was the first version control system that I became well acquainted with, so it just seemed like a natural choice for me when I thought I needed version control.

Several months ago I read a blog article by a buddy, in which he briefly discussed Mercurial. I had been meaning to give some distributed version control systems a shot after some disasters related to the centralized nature of SVN. This blog article prompted me to take a stab at Mercurial and some others.

Within a few days I was sold on Mercurial. I won't go into details simply because I'm not one for religious wars that way. Let's just say that Mercurial seemed to be perfect for my wants and needs.

There were, however, a few things about using Mercurial that I miss from the SVN world. One such thing is that you can update several "working copies" of something in SVN with a single command. For example, I keep a lot of my 3rd party Django applications in one directory. Many of these applications use SVN. Sometimes I'll just run a command like this:

svn up /path/to/third/party/apps/*

Each project that uses SVN will automatically be updated without much fuss with such a command. However, with Mercurial, it appears that you need to be in an actual Mercurial repository in order to update it. There are extensions to get around this problem, but I was looking for something a little different.

Since I use Linux almost exclusively, I didn't feel bad about just using the power within to do the work. The following command does everything I need it to:

find -name ".hg" -type d | xargs -t -i bash -c "(cd {}; hg pull; hg up)"

This command finds any directories called .hg anywhere under your current location on the filesystem. Any matches will be used in the command at the end: cd {}; hg pull; hg up

So far I haven't had any problems with this command, but your mileage may vary. To make things even easier, I made an alias for this rather long command:

alias hgupall='find -name ".hg" -type d | xargs -t -i bash -c "(cd {}; hg pull; hg up)"'

I put that line in my ~/.bashrc script, which is executed each time I log into my computer. With that in place, all I need to do is something like this:

cd /path/to/third/party/apps
hgupall

And the aliased command handles the rest. Pretty slick stuff. Hooray for Mercurial and Linux!

Groovy One-Liner

It's been a while since I wrote a blog article, so I'm using this one-liner as an excuse. In case you're new here, I do a lot of Python development. In the world of Python, you need to have a special file in a directory before you can use Python code within that directory. Yeah, yeah... that's not exactly the clearest way to explain things, but it'll have to do.

This special file is called __init__.py. Having this file in a directory that contains Python code turns that directory into what's called a "python package." We like Python packages. They make our lives so much fun!

Anyhoo, I was working on a project last night, and I wanted to create a bunch of placeholder directories that I plan to use later on. I plan on keeping Python code in these directories, so putting the special __init__.py file in them is what I was looking to do. I didn't want to have to create the __init__.py file in each directory manually, or copy/paste the file all over the place, so I investigated a way to do it quickly from the command line.

One of my buddies brought an interesting command to my attention recently: xargs. I had seen it before in various tutorials online, but I never bothered to learn about it. This seemed like as good a time as any, so I started playing. The result of my efforts follows:

find . -type d | xargs -I {} touch {}/__init__.py

What it does is:

  • recursively finds (find) all directories (-type d) within the current directory (.)
  • pipes (|) each directory to xargs, which makes sure that the __init__.py file exists in each one (touch {}/__init__.py)
  • the -I {} tells xargs what to use as a placeholder when considering each directory found by the find command

Turns out that xargs can be used for all sorts of good stuff. My friend brought it up as a way to get rid of those nasty .svn directories on his path to "Mercurial bliss."

find . -name ".svn" -type d | xargs -I {} rm -Rf {}

How beautiful!

GIT-SVN on Slackware 12.2

With all of the hype that git has been receiving lately, I started playing with it a while back to see if it suited me and my wants/needs. I found it to be an interesting utility. I won't go into any details simply because I'm not really all that knowledgeable about all the ins and outs of version control systems, but I will say that I have decided I like it. I'm still not sure whether I prefer GIT over SVN or SVN over GIT.

My problem is that basically all of my projects are based on SVN repositories. I don't want to have to start up a new GIT repository for each of my past projects. Fortunately, there is an interface for GIT to use SVN repositories called git-svn. I use this utility primarily on my EeePC because it saves a good amount of space on my small disk (the git-svn versions of the working copies are typically about half the size of their svn counterparts). Sometimes it's a little wacky, but it works well enough for my needs.

I started using this git-svn utility on a Debian-based distribution. That meant it was insanely simple to get up and running: sudo apt-get install git-svn. I recently installed Slackware 12.2, and I was surprised to find out that the git-svn utility wasn't immediately available to me.

I did some googling to see if others had encountered the same problem. There were several accurate hits, but I couldn't quite find the solution I needed. In the end, I finally got things working. The following information describes what I did to achieve this monumental success.

Trying git svn

The first roadblock that I encountered, obviously, was finding out that git-svn didn't work on my shiny new Slackware installation. After doing a bit of research, I learned that I could substitute the familiar git-svn command with git svn and continue using it as I previously had.

Installing Dependencies

Once I learned about git svn and tried it out, I got a nasty error about Alien/SVN. I've lost track of the original error, and for that I apologize. Doing a little bit of research led me to execute this command as root:

cpan Alien::SVN

I'm not sure exactly whether that step is required, but you might as well do it :).

Next, I downloaded a couple SlackBuilds to create my own Slackware packages suited for my computer.

For each SlackBuild, you must download the original source code along with the actual SlackBuild itself. For example, when retrieving the necessary files for swig, I must download both swig-1.3.35.tar.gz and swig.tar.gz from the link specified. Here are some example commands, which should be run as root:

mkdir -p ~/downloads/slackbuilds; cd ~/downloads/slackbuilds
wget http://slackbuilds.org/slackbuilds/12.2/development/swig.tar.gz
tar zxf swig.tar.gz
cd swig/
wget http://downloads.sourceforge.net/swig/swig-1.3.35.tar.gz
./swig.SlackBuild
installpkg /tmp/swig-1.3.35-i486-1_SBo.tgz

The commands above should create a new directory in /root/ called downloads/slackbuilds. Next, the SlackBuild for swig will be downloaded and extracted, after which the swig source code will be downloaded. The SlackBuild is executed, rendering an installable Slackware package. Finally, the package is installed onto the system.

The process is basically the same for the subversion-bindings SlackBuild. On my system, however, I had to modify the stock SlackBuild slightly. I didn't install Apache on my EeePC because I don't use it and it would just be taking up space. When I tried to execute the SlackBuild for subversion-bindings straight from the archive, it complained about a missing apxs file, which has something to do with Apache.

To avoid the error, I modified the subversion-bindings.SlackBuild script to ignore the apxs thingy. The original ./configure section looked like this:

CFLAGS="$SLKCFLAGS" \
./configure \
  --prefix=/usr \
  --mandir=/usr/man \
  --enable-shared \
  --disable-static \
  --with-apr=/usr \
  --with-apr-util=/usr \
  --with-apxs=/usr/sbin/apxs \
  --with-neon=/usr \
  --with-zlib=/usr \
  --with-pic \
  --with-ssl \
  --build=$ARCH-slackware-linux

I just removed the line that says --with-apxs=/usr/sbin/apxs \ and ran the SlackBuild script again. Worked like a charm.

At this point everything appeared to be able to work properly. Running git svn from the command line no longer spit out that nasty error I mentioned earlier. Instead it gave me the options I would expect to see.

That's when I tried to update an existing working copy of an SVN repository. It gave me this error:

$ git svn rebase
Authentication realm: <http://special.domain.com:80> Subversion - code
Password for 'myuser': Can't locate Term/ReadKey.pm in @INC (@INC contains:
/usr/lib/perl5/site_perl/5.10.0/i486-linux-thread-multi /usr/lib/perl5
/site_perl/5.10.0 /usr/lib/perl5/5.10.0/i486-linux-thread-multi /usr/lib
/perl5/5.10.0 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.10.0
/i486-linux-thread-multi /usr/lib/perl5/vendor_perl/5.10.0 /usr/lib/perl5
/vendor_perl .) at /usr/libexec/git-core/git-svn line 3071.

That's not very nice, now is it? The solution was fairly simple: install Perl's Term::ReadKey module. As root, execute the following command:

cpan Term::ReadKey

After doing that I was able to happily update my working copy and move on.

I don't envision that this article will be the all-knowing, all-powerful resource for how to use git-svn on Slackware, but I sure hope it will help some other folks who run into the same problems as me.

Installing Slackware 12.2 On Your EeePC (701 4G, in my case)

Welcome to my second article about installing Slackware on an Asus EeePC. This is a follow-up article to the one I posted in May 2008 soon after Slackware 12.1 was released. In this article, I will assume that you're doing a fresh installation of Slackware 12.2 and that you have access to an external USB CD/DVD ROM drive.

In all honesty, the installation process is extremely similar to what I did with 12.1. However, looking back at my previous article, I realize that my steps may not have been the most useful in the world. This time around I will try to be more helpful.

Getting Slackware

The first, and most obvious step, is to get a copy of Slackware. Simply head on over to http://www.slackware.com/getslack/ and retrieve the appropriate ISO(s) using whichever method you prefer. I downloaded the DVD version of Slackware. If you download the CD ISOs, you really only need the first 3 ISOs. The remaining 3 are source packages for the binary packages you install from the first three discs. Rarely do you need the source code for these packages.

After retrieving the Slackware ISO(s), you must burn them to a disc of some sort: ISOs that are ~650MB should be burned to CDs and anything larger should (obviously) be burned to a DVD. Be sure you burn each ISO using the "burn disc image" functionality in your disc writing software--simply burning the ISO file onto the disc in a regular data session will not do what we need.

Booting The Install Disc

After you have a good copy of the installation disc (the DVD or the first of the CDs), put the disc into your CD/DVD ROM drive and reboot your computer. To ensure that your computer boots from the disc rather than the hard drive, hit F2 when you see the initial boot screen. Then go to the "Boot" tab and verify that your external CD/DVD drive takes precedence over the internal SSD. While we're in the BIOS, let's hop over to the "Advanced" tab and set "OS Installation" to "Start". This will increase the chances that your external drive will be recognized or something.... mine didn't work until I made that change. When you're all done with that, exit your BIOS, saving your changes.

The computer will reboot, and it should access your installation disc immediately after the initial boot screen disappears. Once you boot from the installation disc, you should be presented with a screen which allows you to pass some settings to the installation kernel.

The installation boot screen

To make the installation go faster, use the following boot string:

hugesmp.s hdc=noprobe

This makes it so the installation will see the internal SSD as /dev/sda instead of /dev/hdc, which also boosts the read/write times by about 13 times.

During the boot process you will be asked to specify your keyboard map. Unless you want something special here, just hit the enter key to proceed.

Partition Your SSD

Next you will need to login as root and partition your SSD. You can do this using one of the following two commands:

fdisk /dev/sda
cfdisk /dev/sda

Here are some steps in case you're not familiar with these utilities:

  1. Remove all partitions (unless you know what you're doing)
    1. fdisk: d to delete (you may have to select multiple partitions to delete if you have more than one for some reason)
    2. cfdisk: Select all partitions individually with up/down arrow keys and use the left/right arrow keys to select delete from the menu at the bottom. Hit enter to run the delete command when it's highlighted.
  2. Create one partition that takes the whole SSD (again, unless you know what you're doing)
    1. fdisk: n (for new); enter; p (for primary); enter; 1 (for the first primary partition); enter; enter (to start at the beginning of the drive); enter (to select the end of the drive)
    2. cfdisk: Select the new command with the left/right arrow keys and hit enter when it's selected. Make it a primary partition, and have it take the whole SSD (3997.49MB in my case).
  3. Set the type of the new partition to be Linux
    1. fdisk: t (for type); enter; 83 (for Linux); enter
    2. cfdisk: Use the left/right arrow keys to select the type command at the bottom and hit enter when it's selected. Choose 83.
  4. Set the new partition (or the first, if you decided to make more than one) to be bootable
    1. fdisk: a (for bootable); enter; 1 (for primary partition 1); enter
    2. cfdisk: Select the bootable command from the bottom using the left/right arrow keys. Hit enter when it's selected.
  5. Write the changes to the partition table and quit
    1. fdisk: w
    2. cfdisk: Use the left/right arrow keys to select the write command from the bottom. Hit enter when it's selected. Type 'yes' to verify your intent, acknowledging that your previous data will be "gone". Then select the quit command.

Installing Slackware

As soon as your partitioning has finished, go ahead and run setup to begin the actual installation program.

The first screen of the installation program

Since we don't have a swap partition, can jump straight to the TARGET option. Use the arrow keys to highlight this option and hit enter. Select /dev/sda1 from the list, and format it with ext2. On the EeePC, most people prefer this format since it is a non-journaling filesystem. That means fewer writes to the SSD, which supposedly translates to a longer lifetime.

After the SSD is formatted, you will be asked to select the installation source. Again, I'm assuming that you want to use your fresh Slackware 12.2 disc, but you are free to choose what you want if you know what you're doing.

Selecting the installation source

I went with the default "Install from a Slackware CD or DVD" and told it to auto scan for my disc drive. It was found at /dev/sr0.

Choosing Your Packages

Next, you are given the opportunity to tweak the package series which will be installed on your EeePC. I chose the following series: A, AP, K, L, N, TCL, X, and XAP. I planned on using XFCE instead of KDE on my EeePC simply because it is much more light-weight and still capable of what I need. If you want KDE, be sure to check the appropriate series.

Selecting the packages to install

Once you mark each of the package series you wish to install, hit the "OK" button. You'll then have to choose which prompting mode to use. I chose menu, simply to be a little more picky about which packages I wanted installed. Installation took approximately 28 minutes with my package selection and setup.

Configuring Your System

When all of the packages are done being installed, you will be presented with some other screens to finish up the installation process.

  1. Choose whether or not you want to make a bootable USB... I skipped it.
  2. Choose how you wish to install LILO. I chose simple.
  3. Choose your frame buffer mode for the console. I chose 640x480x256.
  4. Specify any optional kernel parameters. Ensure that the hdc=noprobe from earlier is here to speed up your system considerably.
  5. Specify whether you wish to use UTF-8 on the console. I chose no.
  6. Specify where to install LILO. I chose MBR.
  7. Specify your mouse type. I chose imps2.
  8. Specify whether or not you wish to have gpm run at boot, which allows you to use your mouse in the console. I chose yes.
  9. Configure your network.
  10. Give your EeePC a hostname. This can be whatever you'd like.
  11. Specify the domain for your network. This can be whatever you'd like as well.
  12. Configure your IP address information. I just chose DHCP.
  13. Set the DHCP hostname. I left this blank.
  14. Review and confirm your network settings.
  15. Choose which services you wish to have running immediately after booting.
  16. See if you want to try custom screen fonts. I usually don't bother.
  17. Specify whether your hardware clock is set to local time or UTC.
  18. Choose your timezone.
  19. Select your preferred window manager. I chose XFCE.
  20. Set the root password.

At this point Slackware has been installed on your EeePC and you can exit the setup menu and hit Ctrl-Alt-Delete to reboot your computer.

First Boot

You should now go back into your BIOS and set "OS Installation" back to "Finished", exit and save changes, and reboot again.

Slackware's default LILO boot screen

You should then see the Slackware boot screen. By default, it has a 2-minute timeout, which seems absolutely absurd to me, so we'll change that later. Just hit enter for now and watch your new Slackware boot. The first boot will usually take a bit longer than subsequent reboots because all sorts of things need to generate their first configuration file.

When your system is ready, you'll be presented with a login prompt. Just login as root, using the password you specified in the last step of the installation process.

Tweaking Your Slackware

Here are some of the first things I do when I install a new copy of Slackware:

Add An Unprivileged User

This step is very important, because one thing that sets Linux apart from other operating systems is security ;). If you run your Linux system as root all the time, you're begging for problems.

To create a new unprivileged user, I use the adduser command. It walks you through the process of creating a user. This is the user you should use to do your day-to-day computing. Only use the root user when performing system administration tasks. Trust me :)

Tell X Windows to Start Automatically

I have no problem with the command line interface in Linux. I actually enjoy it quite a bit. However, on a device such as the EeePC, not having a GUI just doesn't seem all that practical. It's also not very impressive to your potential converts when they look over your shoulder and see that your tiny gadget just displays a black and white screen when you turn it on...

So, to help ourselves be a little more productive and to impress our followers, let's tell X Windows to start up automatically when we turn on the computer. To do that, we want to edit /etc/inittab and change the following line:

id:3:initdefault:

to be:

id:4:initdefault:

You can use whatever program you feel comfortable with, such as vi or nano. The next time you reboot your computer, you should see a GUI as soon as all of the services are fully loaded.

Along with this step, I suppose we can mention the configuration of X Windows. I usually run xorgsetup as root to get things up and running. Usually there is also a bit of tweaking to get things like the scroll wheel on the mouse to function. This part in particular took quite some time for me to figure out.

Enable The Scroll Wheel on the Trackpad

Some of you might be able to live without being able to scroll a page or whatever without using the scroll feature on most mouse devices these days, but I'm not one of them. Here is my entire /etc/X11/xorg.conf file:

Section "ServerLayout"
    Identifier     "X.org Configured"
    Screen      0  "Screen0" 0 0
    InputDevice    "Mouse0" "CorePointer"
    InputDevice    "SynapticMouse" "AlwaysCore"
    InputDevice    "Keyboard0" "CoreKeyboard"
EndSection

Section "Files"
    RgbPath      "/usr/share/X11/rgb"
    ModulePath   "/usr/lib/xorg/modules"
    FontPath     "/usr/share/fonts/TTF"
    FontPath     "/usr/share/fonts/OTF"
    FontPath     "/usr/share/fonts/Type1"
    FontPath     "/usr/share/fonts/misc"
    FontPath     "/usr/share/fonts/75dpi/:unscaled"
EndSection

Section "Module"
    Load  "xtrap"
    Load  "GLcore"
    Load  "record"
    Load  "dri"
    Load  "dbe"
    Load  "extmod"
    Load  "glx"
    Load  "freetype"
    Load  "type1"
    Load  "synaptics"
EndSection

Section "InputDevice"
    Identifier  "Keyboard0"
    Driver      "kbd"
    Option       "XkbModel"  "pc104"
    Option       "XkbLayout"  "us"
EndSection

Section "InputDevice"
    Identifier  "Mouse0"
    Driver "mouse"
    Option "Device" "/dev/input/mice"
    Option "Protocol" "IMPS/2"
    Option "Buttons" "5"
    Option "zAxisMapping" "4 5"
    Option "SHMConfig" "on"
EndSection

Section "InputDevice"
    Identifier "SynapticMouse"
    Driver "synaptics"
    Option "Device" "/dev/input/mice"
    Option "Protocol" "auto-dev"
    Option "SHMConfig" "on"
EndSection

Section "Monitor"
    Identifier   "Monitor0"
    VendorName   "Monitor Vendor"
    ModelName    "Monitor Model"
EndSection

Section "Device"
        ### Available Driver options are:-
        ### Values: <i>: integer, <f>: float, <bool>: "True"/"False",
        ### <string>: "String", <freq>: "<f> Hz/kHz/MHz"
        ### [arg]: arg optional
        #Option     "NoAccel"               # [<bool>]
        #Option     "SWcursor"              # [<bool>]
        #Option     "ColorKey"              # <i>
        #Option     "CacheLines"            # <i>
        #Option     "Dac6Bit"               # [<bool>]
        #Option     "DRI"                   # [<bool>]
        #Option     "NoDDC"                 # [<bool>]
        #Option     "ShowCache"             # [<bool>]
        #Option     "XvMCSurfaces"          # <i>
        #Option     "PageFlip"              # [<bool>]
    Identifier  "Card0"
    Driver      "intel"
    VendorName  "Intel Corporation"
    BoardName   "Mobile 915GM/GMS/910GML Express Graphics Controller"
    BusID       "PCI:0:2:0"
EndSection

Section "Screen"
    Identifier "Screen0"
    Device     "Card0"
    Monitor    "Monitor0"
    DefaultDepth 24
    SubSection "Display"
        Viewport   0 0
        Depth     1
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     4
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     8
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     15
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     16
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     24
    EndSubSection
EndSection

A lot of that stuff might not be necessary, but it's what works for me. Normally the process for enabling the scroll wheel is pretty easy, but something seems to have changed in this respect with the release of Slackware 12.2. I had to edit the /etc/modprobe.d/psmouse script to make this line:

options psmouse proto=imps

look like:

options psmouse proto=any

After making that change, things seemed to work a lot better.

Make LILO to Boot Faster

There are a couple tricks we can use to make LILO boot our EeePC slightly faster. The first is to add the compact option somewhere, and the second is to decrease the menu timeout.

Open up /etc/lilo.conf with a text editor of your choosing as root. Add a single line with the word compact somewhere. I put it under the line that says boot = /dev/sda so the top of lilo.conf looks like this:

# LILO configuration file
# generated by 'liloconfig'
#
# Start LILO global section
# Append any additional kernel parameters:
append="hdc=noprobe vt.default_utf8=8"
boot = /dev/sda
compact

I also changed the line that said timeout = 1200 to be timeout = 50 to make LILO only hang around for 5 seconds instead of 2 minutes.

After making these changes, we must reinstall LILO to the MBR with the new settings:

lilo -v

Here's my /etc/lilo.conf with most of the commented lines removed:

# LILO configuration file
# generated by 'liloconfig'
#
# Start LILO global section
# Append any additional kernel parameters:
append="hdc=noprobe vt.default_utf8=0"
boot = /dev/sda
compact

# Boot BMP Image.
# Bitmap in BMP format: 640x480x8
bitmap = /boot/slack.bmp
bmp-colors = 255,0,255,0,255,0
bmp-table = 60,6,1,16
bmp-timer = 65,27,0,255

prompt
timeout = 50
change-rules
reset
vga = normal
# End LILO global section
# Linux bootable partition config begins
image = /boot/vmlinuz
root = /dev/sda1
label = Linux
read-only
# Linux bootable partition config ends

Network Tweaking

While the wireless adapter seemed to work great for me out of the box this time, the ethernet adapter is still not functional. I compiled and installed the atl2 driver to solve the problem. You can get it from http://people.redhat.com/csnook/atl2/atl2-2.0.4.tar.bz2. Here are the steps I took to install it:

wget http://people.redhat.com/csnook/atl2/atl2-2.0.4.tar.bz2
tar jxf atl2-2.0.4.tar.bz2
cd atl2-2.0.4
make
cp atl2.ko /lib/modules/`uname -r`/kernel/drivers/net/
depmod -a
modprobe atl2
ifconfig

The next tweak I added for networking was to boost boot times... The DHCP address request hangs the entire boot process out of the box if you don't have an ethernet cable plugged in while booting. To remedy this problem, add the following line to the first section of your /etc/rc.d/rc.inet1.conf:

DHCP_TIMEOUT[0]="5"

This will tell your computer to continue booting if an IP address hasn't been assigned after 5 seconds of waiting.

Enable Frequency Scaling

We all like out battery to last a long time, right? Well, the EeePC 701 doesn't have the greatest battery in the world, but we can help increase the battery life by enabling the CPU frequency modules. I put this stuff in my /etc/rc.d/rc.local script:

#!/bin/sh
#
# /etc/rc.d/rc.local:  Local system initialization script.
#
# Put any local startup commands in here.  Also, if you have
# anything that needs to be run at shutdown time you can
# make an /etc/rc.d/rc.local_shutdown script and put those
# commands in there.

modprobe p4-clockmod
modprobe cpufreq_ondemand
modprobe cpufreq_conservative
modprobe cpufreq_powersave
modprobe cpufreq_performance

cpufreq-set -g ondemand -d 450Mhz -u 900Mhz

Add Your SD Card to /etc/fstab

I have an SD card that I leave in my EeePC all the time, and it's formatted with ext2 just like the internal SSD. Without this tweak, I have to mount the SD card each time I turn on the computer, which gets bothersome. My fix is to add the SD card to /etc/fstab, which takes care of mounting the device at boot.

First, you should make a directory that will be used to mount the device. I made one as such:

mkdir /mnt/sd

Now you need to determine your SD card's UUID. I started out by unmounting my SD card and taking it out of the slot. Then I executed this command:

ls /dev/disk/by-uuid

Next, I popped the SD card back in and executed that command again. The UUID that appears the second time but not the first time is your SD card's UUID.

It's time to add the magic line to your /etc/fstab. Add a line such as:

UUID=[your SD card's UUID] /mnt/sd ext2 defaults,noatime 1 1

somewhere in the file. While we're digging around in /etc/fstab, we might as well add the noatime option to the internal SSD to help reduce disk writes. Save the file and exit the editor. Then mount everything (using mount -a) or just the SD card (using mount /mnt/sd).

For posterity's sake, here's my entire /etc/fstab file:

/dev/sda1        /                ext2        defaults,noatime         1   1
UUID=30293ff4-5bee-457a-8528-ec296f099e9a /mnt/sd ext2 defaults,noatime 1 1
#/dev/cdrom      /mnt/cdrom       auto        noauto,owner,ro  0   0
/dev/fd0         /mnt/floppy      auto        noauto,owner     0   0
devpts           /dev/pts         devpts      gid=5,mode=620   0   0
proc             /proc            proc        defaults         0   0
tmpfs            /dev/shm         tmpfs       defaults         0   0

Preventing Shutdown Hangs

Sometimes the sound card seems to make Slackware hang when you're shutting down. Everything seems to turn off fine, but the little green power LED still shines bright. The solution to this problem appears to be adding the following line:

modprobe -r snd_hda_intel

to /etc/rc.d/rc.6 right before the "Unmounting local file systems." line (around line 195).

Enable Volume Hotkeys and Sleeping

Slackware 12.2 is already listening for ACPI events by default, so we just need to insert our custom stuff into /etc/acpi/acpi_handler.sh:

#!/bin/sh

IFS=${IFS}/
set $@

#logger "ACPI Event $1, $2, $3, $4, $5"

case "$1" in
    button)
        case "$2" in
            power) /sbin/init 0;;
            sleep) /etc/acpi/actions/lid.sh;;
            lid)
                if grep -q closed /proc/acpi/button/lid/LID/state
                then
                    /etc/acpi/actions/lid.sh
                fi
                ;;
            *) logger "ACPI action $2 is not defined";;
        esac
        ;;
    hotkey)
        case "$3" in
            # Fn+F2 Wireless/Bluetooth button
            # Fn+F7 Mute button
            00000013) amixer set Master toggle;;
            # Fn+F8 Volume down
            00000014) amixer set Master 10%-;;
            # Fn+F9 Volume up
            00000015) amixer set Master 10%+;;
        esac
        ;;
    *) logger "ACPI group $1 / action $2 is not defined";;
esac

And to handle the closing of the lid or pressing the sleep button, we need to create a new script in /etc/acpi/actions/ called lid.sh:

#!/bin/sh
# script by Fluxx from linuxquestions slackware forum
# discover video card's ID
ID=`/sbin/lspci | grep VGA | awk '{ print $1 }' | sed -e 's@:@/@'`

# securely create a temporary file
TMP_FILE=`mktemp /tmp/video_state.XXXXXX`
trap 'rm -f $TMP_FILE' 0 1 15

# switch to virtual terminal 1 to avoid graphics
# corruption in X
chvt 1

/sbin/hwclock --systohc

# remove the webcam module
rmmod uvcvideo

# write all unwritten data (just in case)
sync

# dump current data from the video card to the
# temporary file
cat /proc/bus/pci/$ID > $TMP_FILE

# suspend-to-ram
# (samwise) not using this it stuffs up the screen brightness
echo -n mem > /sys/power/state

# suspend-to-disk
#echo -n disk > /sys/power/state

# standby
#echo -n standby > /sys/power/state

# force on for now...
xset dpms force on

/sbin/hwclock --hctosys

# restore the webcam module
modprobe uvcvideo

# restore video card data from the temporary file
# on resume
cat $TMP_FILE > /proc/bus/pci/$ID

# switch back to virtual terminal 2 (running X)
chvt 6; sleep 2
chvt 2

# remove temporary file
rm -f $TMP_FILE

And we need to make sure the script is executable:

chmod +x /etc/acpi/actions/lid.sh

These scripts should enable us to use the mute key, the increase/decrease volume keys, and the sleep key. They should also allow us to close the lid of the EeePC to put it to sleep. Occasionally, when you wake up the computer, you will just see a blank black screen. To get around this, switch back to VT2 by using the keystroke Ctrl+Alt+F2.

Install Special Packages

Slackware comes with a lot of awesome stuff right out of the box, but it is missing some very important utilities at the same time. Included in this list, for me, is a program called wicd, or a network connectivity manager. This is similar to the "Network Manager" utility found in other mainstream distributions like Ubuntu, Fedora, and openSuSE. Slackware has yet to include such a utility by default.

Anyway, wicd can be found in the extra directory on the Slackware DVD or the 3rd (?) CD. To install it, find the package on the disc (or download it from the Internet) and execute the following command:

installpkg wicd-1.5.6-noarch-2.tgz

Be sure to check out the extra directory on the Slackware install disc. There are some neat tools in there. Some excellent resources for Slackware packages include:

There are some utilities out there to help you in your quest to resolve package dependencies. Two of the major ones that I've used in the past are swaret and slapt-get.

Using Slackware 12.2

My Slackware 12.2-powered EeePC 701 4G

I have to give the Linux kernel hackers props--the 2.6.27.7 kernel is amazingly fast! I'm sure the fact that I'm running a fairly stock Slackware installation (as opposed to something like Ubuntu) helps the speed quite a bit too. This past semester I had Linux Mint 5 (XFCE edition) installed on my EeePC, and that seemed fairly responsive. Slackware blew me away though, and I can still do everything I want to do!

The webcam and sound card work out of the box, just like the wireless. I rarely use the webcam, but it's fun to play with, and my mom appreciates seeing me on Skype occasionally. The wireless connection quality exceeds what it was with the madwifi driver I was using with Slackware 12.1 and other distros like Linux Mint. Programs are ultra speedy and responsive, even with the processor clocked at 450Mhz. I love it!!!

Boot times could be better, but I'm not too concerned with it. My setup takes approximately 50 seconds from boot to a useable desktop interface. Not horrible by any means, but perhaps not the best for a netbook when all you want to do is check your e-mail.

I would like to see the Network Manager that so many other distributions offer in Slackware some day. The wicd application is nice, but it's not nearly as intuitive as Network Manager, and it seems to be relatively limited in its capabilities in comparison. I know I'm not alone in my desire to see Network Manager included, or at least available, for Slackware. It would be tremendously beneficial in a world where wireless networking and laptops are more and more pervasive. Using the command line to adjust your wireless connection settings each time you have to hop to a new access point is just annoying.

In the end, I'm excited to have Slackware on my EeePC once again. I think it will be around for quite a while this time.

Please comment with any advice or problems that you have in regards to installing Slackware 12.2 on an EeePC.

Syntax Highlighting, ReST, Pygments, and Django

Some of you regulars out there may have noticed an interesting change in the presentation of some of my articles: source code highlighting. I've been interested in doing this for quite some time, I just never really got around to implementing it until last night.

I found this implementation process to be a bit more complicatd than I had anticipated. For my own benefit as well as for anyone else who wants to do the same thing, I thought I'd document my findings in a thorough article for how to add syntax highlighting to an existing Django- and reStructuredText-powered Web site.

The power behind the syntax highlighting is:

Python is a huge player in this feature because reStructuredText (ReST) was built for Python, Pygments is the source highlighter (written in Python), and Django is written in Python (and my site is powered by Django). Some of you may recall that I converted all of my articles to ReST not too long ago because it suited my needs better than Textile, my previous markup processor. At the time, I was not aware that the conversion to ReST would make it all the easier for me to implement the syntax highlighting, but last night I figured out that that conversion probably saved me a lot of frustration. Cascading Stylesheets (CSS) are responsible for making the source code actually look good, while Pygments takes care of assigning classes to various parts of the designated source code and generating the CSS.

So, the first set of requirements, which I will not document in this article, are that you already have a Django site up and running and that you're familiar with ReST syntax. If you have the django.contrib.flatpages application installed already, you can type up some ReST documents there and apply the concepts discussed in this article.

Next, you should ensure that you have Pygments installed. There are a variety of ways to install this. Perhaps the easiest and most platform-independent method is to use easy_install:

$ easy_install pygments

This command should work essentially the same on Windows, Linux, and Macintosh computers. If you don't have it installed, you can get it from its website. If you're using a Debian-based distribution of Linux, such as Ubuntu, you could do something like this:

$ sudo apt-get install python-pygments

...and it should take care of downloading and installing Pygments. Alternatively, you can download it straight from the PyPI page and install it manually.

Now we need to install the Pygments ReST directive. A ReST directive is basically like a special command to the ReST processor. I think this part was the most difficult aspect of the implementation, simply because I didn't know where to find the Pygments directive or how to write my own. Eventually, I ended up downloading the Pygments-1.0.tar.gz file from PyPI, opening the Pygments-1.0/external/rst-directive.py file from the archive, and copying the stuff in there into a new file within my site.

For my own purposes, I made some small adjustments to the directive over what come with the Pygments distribution. I think it would save us all a lot of hassle if I just copied and pasted the directive, as I currently have it, so you can see it first-hand.

"""
    The Pygments reStructuredText directive
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    This fragment is a Docutils_ 0.4 directive that renders source code
    (to HTML only, currently) via Pygments.

    To use it, adjust the options below and copy the code into a module
    that you import on initialization.  The code then automatically
    registers a ``code-block`` directive that you can use instead of
    normal code blocks like this::

    .. code:: python

            My code goes here.

    If you want to have different code styles, e.g. one with line numbers
    and one without, add formatters with their names in the VARIANTS dict
    below.  You can invoke them instead of the DEFAULT one by using a
    directive option::

    .. code:: python
       :number-lines:

            My code goes here.

    Look at the `directive documentation`_ to get all the gory details.

    .. _Docutils: http://docutils.sf.net/
    .. _directive documentation:
       http://docutils.sourceforge.net/docs/howto/rst-directives.html

    :copyright: 2007 by Georg Brandl.
    :license: BSD, see LICENSE for more details.
"""

# Options
# ~~~~~~~

# Set to True if you want inline CSS styles instead of classes
INLINESTYLES = False

from pygments.formatters import HtmlFormatter

# The default formatter
DEFAULT = HtmlFormatter(noclasses=INLINESTYLES)

# Add name -> formatter pairs for every variant you want to use
VARIANTS = {
    'linenos': HtmlFormatter(noclasses=INLINESTYLES, linenos=True),
}


from docutils import nodes
from docutils.parsers.rst import directives

from pygments import highlight
from pygments.lexers import get_lexer_by_name, TextLexer

def pygments_directive(name, arguments, options, content, lineno,
                       content_offset, block_text, state, state_machine):
    try:
        lexer = get_lexer_by_name(arguments[0])
    except ValueError:
        # no lexer found - use the text one instead of an exception
        lexer = TextLexer()
    # take an arbitrary option if more than one is given
    formatter = options and VARIANTS[options.keys()[0]] or DEFAULT
    parsed = highlight(u'\n'.join(content), lexer, formatter)
    parsed = '<div class="codeblock">%s</div>' % parsed
    return [nodes.raw('', parsed, format='html')]

pygments_directive.arguments = (1, 0, 1)
pygments_directive.content = 1
pygments_directive.options = dict([(key, directives.flag) for key in VARIANTS])

directives.register_directive('code-block', pygments_directive)

I won't explain what that code means, because, quite frankly, I'm still a little hazy on the inner workings of ReST directives myself. Suffice it to say that this snippet allows you to easily highlight blocks of code on ReST-powered pages.

The question now is: where do I put this snippet? As far as I'm aware, this code can be located anywhere so long as it is loaded at one point or another before you start your ReST processing. For the sake of simplicity, I just stuffed it in the __init__.py file of my Django site. This is the __init__.py file that lives in the same directory as manage.py and settings.py. Putting it in that file just makes sure it's loaded each time you start your Django site.

To make Pygments highlight a block of code, all you need to do is something like this:

.. code:: python

    print 'Hello world!'

...which would look like...

print 'Hello world!'

If you have a longer block of code and would like line numbers, use the :number-lines: option:

.. code:: python
    :number-lines:

    for i in range(100):
        print i

...which should look like this...

for i in range(100):
    print i

That's all fine and dandy, but it probably doesn't look like the code is highlighted at all just yet (on your site, not mine). It's just been marked up by Pygments to have some pretty CSS styles applied to it. But how do you know which styles mean what?

Luckily enough, Pygments takes care of generating the CSS files for you as well. There are several attractive styles that come with Pygments. I would recommend going to the Pygments demo to see which one suits you best. You can also roll your own styles, but I haven't braved that yet so I'll leave that for another day.

Once you choose a style (I chose native for Code Koala), you can run the following commands:

$ pygmentize -S native -f html > native.css
$ cp native.css /path/to/site/media/css

(obviously, you'd want to replace native with the name of the style you like the most) Finally, add a line to your HTML templates to load the newly created CSS file. In my case, it's something like this:

<link rel="stylesheet" type="text/css" href="/static/styles/native.css" />

Now you should be able to see nicely-formatted source code on your Web pages (assuming you've already got ReST processing your content).

If you haven't been using ReST to generate nicely-formatted pages, you should make sure a couple of things are in place. First, you must have the django.contrib.markup application installed. Second, your templates should be setup to process ReST markup into HTML. Here's a sample templates/flatpages/default.html:

{% extends 'base.html' %}
{% load markup %}

{% block title %}{{ flatpage.title }}{% endblock %}

{% block content %}
<h2>{{ flatpage.title }}</h2>

{{ flatpage.content|restructuredtext }}
{% endblock %}

So that short template should allow you to use ReST markup for your flatpages, and it should also take care of the magic behind the .. code:: python directive.

I should also note that Pygments can handle a TON of languages. Check out the Pygments demo for a list of languages it knows how to highlight.

I think that about does it. Hopefully this article will help some other poor chap who is currently in the same situation as I was last night, and hopefully it will save you a lot more time than it took me to figure out all this junk. If it looks like I've missed something, or maybe that something needs further clarification, please comment and I'll see what I can do.

Installing Python 3.0 Alongside an Existing Python

With the recent release of Python 3.0 final, I've had a crazy itch that needed to be scratched. That itch, my friends, was to do a write-up of how to install Python 3.0 alongside my existing Python 2.5.2 installation without borking things up. The reason I thought it would be useful is that I'm running a Debian-based distribution of Linux called sidux right now, and neither Python 2.6 nor Python 3.0 are in the package repositories. I assume that Ubuntu and other Debian-based distributions might be the same way, and that there are others like me who would like to tinker with these new releases.

Just before attempting to install Python 3.0 on my computer, I started getting ready to write this article. Before I knew it, Python 3.0 was installed on my system, and I had no notes to share with you! That is how stinkin easy it is to install Python 3.0 without interfering with an existing install. I mentioned to a good friend that I wasn't sure the process was worth writing an article because it was so simple, but he encouraged me to carry on. Thanks bro.

So, without any further ado, here is what I did to install Python 3.0 from source alongside my Python 2.5 installation:

Note: If you're using a Mac or Windows, you should be able to simply install the packages for your platform to accomplish this same feat. Once the packages are in the repositories it will be just as easy for Linux.

$ wget http://python.org/ftp/python/3.0/Python-3.0.tar.bz2
  • Unpack the archive:
$ tar jxf Python-3.0.tar.bz2
  • Descend into the newly extracted Python-3.0 directory:
$ cd Python-3.0
  • Install libreadline-dev. This step is necessary for the arrow keys to work, as pointed out by jazevec below in the comments. If you are on a Debian-based system, you can execute a command such as this:
  $ sudo apt-get install libreadline-dev

Other distributions may have different package names, such as ``readline-dev``.  If neither one of those package names work for your distribution, try searching your package manager for ``readline`` and install the development files.  Alternatively, you should be able to manually install what you need by installing `readline itself <http://directory.fsf.org/project/readline/>`_.
  • Configure Python 3.0 for your computer:
$ ./configure
  • Compile Python 3.0:
  $ make


**UPDATE** (9 Dec): Depending on your setup, you may or may not see a message such as this after executing the ``make`` command (thanks again to ``jazevec`` for pointing out that this can happen)::

  Failed to find the necessary bits to build these modules:
  _dbm               _gdbm              _hashlib
  _sqlite3           _ssl               _tkinter
  bz2                zlib
  To find the necessary bits, look in setup.py in detect_modules() for the module's name.

If you need any of those capabilities, you should install the appropriate development files for the missing module(s).  For example, above, we installed the ``libreadline-dev`` package.  To resolve the missing module problem for each one listed above (except ``_dbm`` because it's apparently borked on Debian right now... possibly other distros too), install these packages:

  * ``tk-dev`` to satisfy ``_tkinter``
  * ``libsqlite3-dev`` to satisfy ``_sqlite3``
  * ``libbz2-dev`` to satisfy ``bz2``
  * ``zlib1g-dev`` to satisfy ``zlib``
  * ``libssl-dev`` to satisfy ``_ssl`` and ``_hashlib``
  * ``libgdbm-dev`` to satisfy ``_gdbm``
  • Install Python 3.0:
  $ sudo make install

or:
# make install
  • Test Python 3.0:
$ python3.0
Python 3.0 (r30:67503, Dec  5 2008, 11:05:45)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print 'Hi'
File "<stdin>", line 1
    print 'Hi'
            ^
SyntaxError: invalid syntax
>>> print('Hi')
Hi
>>>
  • Make sure that your old version of Python is still around:
$ python
Python 2.5.2 (r252:60911, Sep 29 2008, 21:15:13)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print 'Hi'
Hi
>>>
  • Rejoice

That's about it folks! Extremely simple and fast, compared to what it could have been.

If you find yourself in a situation where you don't have access to sudo or straight root-level access, you can install Python 3.0 locally by doing something like this:

  • Configure Python 3.0 for your computer:
$ ./configure --prefix=$HOME/local/
  • Compile Python 3.0:
$ make
  • Install Python 3.0:
  $ make install

Not that I omitted the ``sudo`` part of the command here.
  • Symlink to Python 3.0, assuming you have a bin/ directory in your home directory (i.e. /home/[yourusername]/bin), and that said bin directory is on your PATH:
$ ln -s ~/local/bin/python3.0 ~/bin
  • Test your locally installed Python 3.0:
  $ python3.0

or, if your local ``bin`` directory isn't on your ``PATH``:
$ ~/bin/python3.0
  • Do the dance.

Please comment with any problems you find with this process, or any additional advice you can offer to newbies!

Installing Django on Shared Hosting (Site5)

This article is a related to my previously posted article about installing Django, an advanced Web framework for perfectionists, on your own computer. Now we will learn how to install Django on a shared hosting account, using Site5 and fastcgi as an example. Depending on your host, you may or may not have to request additional privileges from the support team in order to execute some of these commands.

Note: Django requires at least Python 2.3. Newer versions of Python are preferred.

Note: This HOWTO assumes familiarity with the UNIX/Linux command line.

Note: If the wget command doesn't work for you (as in you don't have permission to run it), you might try curl [url] -O instead. That's a -O as in upper-case o.

Install Python

Site5 (and many other shared hosting providers that offer SSH access) already has Python installed, but you will want to have your own copy so you can install various tools without affecting other users. So go ahead and download virtual python:

mkdir ~/downloads
cd ~/downloads
wget http://peak.telecommunity.com/dist/virtual-python.py

Virtual Python will make a local copy of the installed Python in your home directory. Now you want to make sure you execute this next command with the newest version of Python available on your host. For example, Site5 offers both Python 2.3.4 and Python 2.4.3. We want to use Python 2.4.3. To verify the version of your Python, execute the following command:

python -V

If that displays Python 2.3.x or anything earlier, try using python2.4 -V or python2.5 -V instead. Whichever command renders the most recent version of Python is the one you should use in place of python in the next command. Since python -V currently displays Python 2.4.3 on my Site5 sandbox, I will execute the following command:

python ~/downloads/virtual-python.py

Again, this is just making a local copy of the Python installation that you used to run the virtual-python.py script. Your local installation is likely in ~/lib/python2.4/ (version could vary).

Make Your Local Python Be Default

To reduce confusion and hassle, let's give our new local installation of Python precedence over the system-wide Python. To do that, open up your ~/.bashrc and make sure it contains a line similar to this:

export PATH=$HOME/bin:$PATH

If you're unfamiliar with UNIX-based text editors such as vi, here is what you would type to use vi to make the appropriate changes:

  • vi ~/.bashrc to edit the file
  • go to the end of the file by using the down arrow key or the j key
  • hit o (the letter) to tell vi you want to start typing stuff on the next line
  • type export PATH=$HOME/bin:$PATH
  • hit the escape key
  • type :x to save the changes and quit. Don't forget the : at the beginning. Alternatively, you can type :wq, which works exactly the same as :x.

Once you've made the appropriate changes to ~/.bashrc, you need to make those changes take effect in your current SSH session:

source ~/.bashrc

Now we should verify that our changes actually took place. Type the following command:

which python

If they output of that command is not something like ~/bin/python or /home/[your username]/bin/python, something probably didn't work. If that's the case, you can try again, or simply remember to use ~/bin/python instead of python throughout the rest of this HOWTO.

Install Python's setuptools

Now we should install Python's setuptools to make our lives easier down the road.

cd ~/downloads
wget http://peak.telecommunity.com/dist/ez_setup.py
python ez_setup.py

This gives us access to a script called easy_install, which makes it easy to install many useful Python tools. We will use this a bit later.

Download Django

Let's now download the most recent development version of Django. SSH into your account and execute the following commands (all commands shall be executed on your host).

svn co http://code.djangoproject.com/svn/django/trunk ~/downloads/django-trunk

Now we should make a symlink (or shortcut) to Django and put it somewhere on the Python Path. A sure-fire place is your ~/lib/python2.4/site-packages/ directory (again, that location could vary from host to host):

ln -s ~/downloads/django-trunk/django ~/lib/python2.4/site-packages
ln -s ~/downloads/django-trunk/django/bin/django-admin.py ~/bin

Now verify that Django is installed and working by executing the following command:

python -c "import django; print django.get_version()"

That command should return something like 1.0-final-SVN-8964. If you got something like that, you're good to move onto the next section. If, however, you get something more along the lines of...

Traceback (most recent call last):
    File "<string>", line 1, in ?
ImportError: No module named django

...then your Django installation didn't work. If this is the case, make sure that you have a ~/downloads/django-trunk/django directory, and also verify that ~/lib/python2.4/site-packages actually exists.

Installing Dependencies

In order for your Django projects to become useful, we need to install some other packages: PIL (Python Imaging Library, required if you want to use Django's ImageField), MySQL-python (a MySQL database driver for Python), and flup (a utility for fastcgi-powered sites).

easy_install -f http://www.pythonware.com/products/pil/ Imaging
easy_install mysql-python
easy_install flup

Sometimes, using easy_install to install PIL doesn't go over too well because of your (lack of) permissions. To circumvent this situation, you can always download the actual PIL source code and install it manually.

cd ~/downloads
wget http://effbot.org/downloads/Imaging-1.1.6.tar.gz
tar zxf Imaging-1.1.6.tar.gz
cd Imaging-1.1.6
ln -s ~/downloads/Imaging-1.1.6/PIL ~/lib/python2.4/site-packages

And to verify, you can try this command:

python -c "import PIL"

If that doesn't return anything, you're good to go. If it says something about "ImportError: No module named PIL", it didn't work. In that case, you have to come up with some other way of installing PIL.

Setting Up A Django Project

Let's attempt to setup a sample Django project.

mkdir -p ~/projects/django
cd ~/projects/django
django-admin.py startproject mysite
cd mysite
mkdir media templates

If that works, then you should be good to do the rest of your Django development on your server. If not, make sure that ~/downloads/django-trunk/django/bin/django-admin.py exists and that it has a functioning symlink (shortcut) in ~/bin. If not, you'll have to make adjustments according to your setup. Your directory structure should look something like:

  • projects
    • django
      • mysite
        • media
        • templates
        • __init__.py
        • manage.py
        • settings.py
        • urls.py

Making A Django Project Live

Now we need to make your Django project accessible from the Web. On Site5, I generally use either a subdomain or a brand new domain when setting up a Django project. If you plan on having other projects accessible on the same hosting account, I recommend you do the same. Let's assume you setup a subdomain such as mysite.mydomain.com. On Site5, you would go to ~/public_html/mysite for the next few commands. This could differ from host to host, so I won't go into much more detail than that.

Once you're in the proper place, you need to setup a few things: two symlinks, a django.fcgi, and a custom .htaccess file. Let's begin with the symlinks.

ln -s ~/projects/django/mysite/media ~/public_html/mysite/static
ln -s ~/lib/python2.4/site-packages/django/contrib/admin/media ~/public_html/mysite/media

This just makes it so you can have your media files (CSS, images, javascripts, etc) in a different location than in your public_html.

Now for the django.fcgi. This file is what tells the webserver to execute your Django project.

#!/home/[your username]/bin/python
import sys, os

# Add a custom Python path.
sys.path.insert(0, "/home/[your username]/projects/django")

# Switch to the directory of your project. (Optional.)
os.chdir("/home/[your username]/projects/django/mysite")

# Set the DJANGO_SETTINGS_MODULE environment variable.
os.environ['DJANGO_SETTINGS_MODULE'] = "mysite.settings"

from django.core.servers.fastcgi import runfastcgi
runfastcgi(method="threaded", daemonize="false")

And finally, the .htaccess file:

1
2
3
4
5
6
RewriteEngine On
RewriteBase /
RewriteRule ^(media/.*)$ - [L]
RewriteRule ^(static/.*)$ - [L]
RewriteCond %{REQUEST_URI} !(django.fcgi)
RewriteRule ^(.*)$ django.fcgi/$1 [L]

The .htaccess file makes it so that requests to http://mysite.mydomain.com/ are properly directed to your Django project. So, now you should have a directory structure that something that looks like this:

  • public_html
    • mysite
      • media
      • static
      • .htaccess
      • django.fcgi

If that looks good, go ahead and make the django.fcgi executable and non-writable by others:

chmod 755 ~/public_html/mysite/django.fcgi

After that, head over to http://mysite.mydomain.com/ (obviously, replace the mydomain accordingly). If you see a page that says you've successfully setup your Django site, you're good to go!

Afterthoughts

I've noticed that I need to "restart" my Django sites on Site5 any time I change the .py files. There are a couple methods of doing this. One includes killing off all of your python processes (killall ~/bin/python) and the other simply updates the timestamp on your django.fcgi (touch ~/public_html/mysite/django.fcgi). I find the former to be more destructive and unreliable than the latter. So, my advice is to use the touch method unless it doesn't work, in which case you can try the killall method.

Good luck!

Using Django to Design Your Database Schema

Last night I had a buddy of mine ask me how I would approach a particular database design problem. I get similar questions quite often from my peers--suggests there is something important lacking from the database classes out there. Instead of answering him directly, I decided to come up with this tutorial for using Django to design your database schema.

For those of you new to Django, this article might seem a bit advanced. In time I will have more introductory-level Django tutorials, but I hope this one is easy enough.

Create a Django Project

The first step is to create a Django project. If you already have a project that you can play with, you can skip this step. To create a project, go to a place where you want to keep your code (like C:\projects or /home/me/projects) in a command prompt/terminal and run the following command:

django-admin.py startproject myproject

This will create a new directory in your current location called myproject (you can replace myproject with whatever you'd like so long as you're consistent). This new directory will contain a few files:

  • __init__.py
  • manage.py
  • settings.py
  • urls.py

If you get an error message when running the above command, you might not have Django installed properly. See Step-by-Step: Installing Django for details on installing Django.

Create An Application

Once you have a Django project setup, you should create a new application.

Note: If you're using Windows, you will probably need to omit the ./ on the ./manage.py commands. I will include them here for everyone else who's using Linux or a Mac.

cd myproject
./manage.py startapp specialapp

This will create a new directory in your myproject directory. This new directory will contain three files: __init__.py, models.py, and views.py. We are only concerned with the models.py file in this article.

Create Your Models

Models are usually a direct representation of what your database will be. Django makes creating these models extremely easy, and Python's syntax makes them quite readable. The Django framework asks for models to be defined in the models.py file that was created in the last step. Here's an example (for my buddy who prompted the creation of this article):

from django.db import models

class Component(models.Model):
    item_number = models.CharField(max_length=20)
    name = models.CharField(max_length=50)
    size = models.CharField(max_length=10)
    quantity = models.IntegerField(default=1)
    price = models.DecimalField(max_digits=8, decimal_places=2)

class Project(models.Model):
    name = models.CharField(max_length=50)
    components = models.ManyToManyField(Component)
    instructions = models.TextField()

(for more information about models, see the Django Model API Reference)

I don't know about you, but that code seems pretty straightforward to me. I'll spare you all the details about what's going on (that can be a future article).

Install Your New Application

Once you have your models setup, we need to add our specialapp to our list of INSTALLED_APPS in order for Django to register these models. To do that, open up settings.py in your myproject directory, go to the bottom of the file, until you see something like

1
2
3
4
5
6
INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
)

When you find that, add your specialapp to the list

1
2
3
4
5
6
7
INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'specialapp'
)

Setup Your Database

Now you need to let Django know what kind of database you're using. Django currently supports MySQL, SQLite3, PostgreSQL, and Oracle natively, but you can get third-party tools that allow you to use other database (like SQL Server).

Still in your settings.py, go to the top until you see DATABASE_ENGINE and DATABASE_NAME. Set that to whatever type of database you are using:

DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = 'myproject.db'

Save your settings.py and go back to your command prompt/terminal.

Get Django's Opinion For Your Schema

Make sure you're in your myproject directory and run the following command:

./manage.py sqlall specialapp

This command will examine the models that we created previously and will generate the appropriate SQL to create the tables for your particular database. For SQLite, we get something like this for output:

BEGIN;
CREATE TABLE "specialapp_component" (
      "id" integer NOT NULL PRIMARY KEY,
      "item_number" varchar(20) NOT NULL,
      "name" varchar(50) NOT NULL,
      "size" varchar(10) NOT NULL,
      "quantity" integer NOT NULL,
      "price" decimal NOT NULL
)
;
CREATE TABLE "specialapp_project" (
      "id" integer NOT NULL PRIMARY KEY,
      "name" varchar(50) NOT NULL,
      "instructions" text NOT NULL
)
;
CREATE TABLE "specialapp_project_components" (
      "id" integer NOT NULL PRIMARY KEY,
      "project_id" integer NOT NULL REFERENCES "specialapp_project" ("id"),
      "component_id" integer NOT NULL REFERENCES "specialapp_component" ("id"),
      UNIQUE ("project_id", "component_id")
)
;
COMMIT;

Notice how Django does all sorts of nifty things, like wrapping the table creation queries in a transaction, setting up indexes, unique keys, and defining relationships between tables. The output also offers a solution to the original problem my buddy had: an intermediate table that just keeps track of relationships between projects and components (the specialapp_project_components table).

Notice that the SQL above may not work with database servers other than SQLite.

Enhancing The Intermediate Table

After my buddy reviewed this article, he asked a very interesting and valid question: What if a project needs 3 of one component? In response, I offer the following models (this requires a modern version of Django--it doesn't work on Django 0.96.1 or earlier):

from django.db import models

class Component(models.Model):
    item_number = models.CharField(max_length=20)
    name = models.CharField(max_length=50)
    size = models.CharField(max_length=10)
    quantity = models.IntegerField(default=1)
    price = models.DecimalField(max_digits=8, decimal_places=2)

class Project(models.Model):
    name = models.CharField(max_length=50)
    components = models.ManyToManyField(Component, through='ProjectComponent')
    instructions = models.TextField()

class ProjectComponent(models.Model):
    project = models.ForeignKey(Project)
    component = models.ForeignKey(Component)
    quantity = models.PositiveIntegerField()

    class Meta:
        unique_together = ['project', 'component']

Running ./manage.py sqlall specialapp now generates the following SQL:

BEGIN;
CREATE TABLE "specialapp_component" (
    "id" integer NOT NULL PRIMARY KEY,
    "item_number" varchar(20) NOT NULL,
    "name" varchar(50) NOT NULL,
    "size" varchar(10) NOT NULL,
    "quantity" integer NOT NULL,
    "price" decimal NOT NULL
)
;
CREATE TABLE "specialapp_project" (
    "id" integer NOT NULL PRIMARY KEY,
    "name" varchar(50) NOT NULL,
    "instructions" text NOT NULL
)
;
CREATE TABLE "specialapp_projectcomponent" (
    "id" integer NOT NULL PRIMARY KEY,
    "project_id" integer NOT NULL REFERENCES "specialapp_project" ("id"),
    "component_id" integer NOT NULL REFERENCES "specialapp_component" ("id"),
    "quantity" integer unsigned NOT NULL,
    UNIQUE ("project_id", "component_id")
)
;
CREATE INDEX "specialapp_projectcomponent_project_id" ON "specialapp_projectcomponent" ("project_id");
CREATE INDEX "specialapp_projectcomponent_component_id" ON "specialapp_projectcomponent" ("component_id");
COMMIT;

As you can see, most of the SQL is the same. The main difference is that the specialapp_project_components table has become specialapp_projectcomponent and it now has a quantity column. This can be used to keep track of the quantity of each component that a project requires. You can add however many fields you want to this new intermediate table's model.

Using This SQL

There are several ways you can use the SQL generated by Django. If you want to make your life really easy, you can have Django create the tables for you directly. Assuming that you have specified all of the appropriate database information in your settings.py file, you can simply run the following command:

./manage.py syncdb

This will execute the queries generated earlier directly on your database, creating the tables (if they don't already exist). Please note that this command currently will not update your schema if the table exists but is missing a column or two. You must either do that manually or drop the table in question and then execute the syncdb command.

Another option, if you want to keep your DDL(Data Definition Language) in a separate script (maybe if you want to keep it in some sort of version control) is something like:

./manage.py sqlall specialapp > specialapp-ddl-080813.sql

This just puts the output of the sqlall command into a file called specialapp-ddl-080813.sql for later use.

Benefits of Using Django To Create Your Schema

  • Simple: I personally find the syntax of Django models to be very simple and direct. There is a comprehensive API that explains and demonstrates what Django models are capable of.
  • Fast: Being that the syntax is so simple, I find that it makes designing and defining your schema much faster than trying to do it with raw SQL or using a database administration GUI.
  • Understandable: Looking at the model code in Django is not nearly as intimidating as similar solutions in other frameworks (think about Java Persistence API models).
  • Intelligent: Using the same model code, Django can generate proper Data Definition Language SQL for several popular database servers. It handles indexes, keys, relationships, transactions, etc. and can tell the difference between server types.

Downfalls of Using Django To Create Your Schema

  • The Table Prefix: Notice how all of the tables in the SQL above were prefixed with specialapp_. That's Django's safe way of making sure models from different applications in the same Django project do not interfere with each other. However, if you don't plan on using Django for your end project, the prefix could be a major annoyance. There are a couple solutions:
    • A simple "search and replace" before executing the SQL in your database
    • Define the db_table option in your models
  • Another Technology: Django (or even Python) may or may not be in your organization's current development stack. If it's not, using the methods described in this article would just become one more thing to support.

Other Thoughts

I first thought about doing the things mentioned in this application when I was working on a personal Java application. I like to use JPA when developing database-backed applications in Java because it abstracts away a lot of the database operations. However, I don't like coming up with the model classes directly, so I usually reverse engineer them from existing database tables.

Before thinking about the things discussed in this article, I created the tables by hand, making several modifications to the schema before I was satisfied with my JPA models. This proved to be quite bothersome and time-consuming.

After using Django to develop my tables, the JPA models turned out to be a lot more reliable, and they were usually designed properly from the get-go. I haven't created tables manually ever since.

If you find yourself designing database schemas often, and you find that you have to make several changes to your tables before you/the project requirements are satisfied, you might consider using Django to do the grunt work. It's worked for me, and I'm sure it will work for you too.

Good luck!

How To Compile and Install a 2.6.x Series Linux Kernel

The Linux kernel is the core component in any Linux distribution. Without a kernel, your computer would be essentially useless. It is the piece of software which allows interaction between you, your computer's applications, and your computer's hardware. With such a powerful role in your computing experience, it is important to keep your kernel up-to-date. Each new release provides more hardware support and many performance enhancements. It is also important to keep your kernel up-to-date for security purposes.

Let's upgrade our Linux kernels together. I will walk you through each of the steps I take, from beginning to end, to upgrade my kernel. Just as a warning, I prefer to do the whole process on the command line, so you might want to pull up a terminal, konsole, xterm or whatever you prefer to use for your command line operations.

First you need to download the kernel source code. Many Linux distributions provide specialized editions of the Linux kernel. Typically, you don't want to manually compile and install a custom kernel for these distributions. This does not mean that you can't, it simply means that you might be better off using the "official" kernels for your distribution, which can usually be obtained through your distribution's package manager. You can get the official, 100% free, and complete Linux kernel source code from http://www.kernel.org/. Look for "The latest stable version of the Linux kernel is:" and click the link on the F on the same line. Currently, the latest stable version is 2.6.20, and that's what I'll be using for this tutorial. Please note that commands which begin with a dollar sign ($) are executed as a regular user and commands beginning with a pound sign (#) are executed as a superuser.

$ cd /home/user/download
$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.20.tar.bz2

Now login as the superuser, and navigate to the /usr/src directory. Then extract the kernel source into that directory.

$ su -
# cd /usr/src
# tar jxf /home/user/download/linux-2.6.20.tar.bz2

You probably already have a symlink or shortcut called linux which points to your most recent kernel. If you do, delete the link and create another link to the new source tree. Then go into your kernel source tree.

# rm /usr/src/linux
# ln -s /usr/src/linux-2.6.20 /usr/src/linux
# cd /usr/src/linux

I like to identify each compile of my kernel uniquely, to make sure that I'm using the right one. To do that, you have to modify your Makefile

# vi Makefile

You will see the following lines, or something similar, at the very top of the file:

VERSION = 2
PATCHLEVEL = 6
SUBLEVEL = 20
EXTRAVERSION =
NAME = Homicidal Dwarf Hamster

Change the EXTRAVERSION property to something you want to use to identify this kernel. I will use -jcv1

EXTRAVERSION = -jcv1

The rest of the Makefile should be fine. In fact, I discourage editing Makefiles unless you know what you're doing. This next step is totally optional, but I like to do it to save some time. You can copy your existing kernel's configuration file in order to have a very similar kernel configuration. My previous kernel version was 2.6.19.1, so this is the command I use:

# cp /usr/src/linux-2.6.19.1/.config /usr/src/linux/

Then I run make oldconfig or make silentoldconfig to update my older kernel configuration file to be able to handle newer features. If you use oldconfig you are required to specify whether or not you want the new features included in your kernel, whereas silentoldconfig will use the defaults determined by kernel developers (they usually know best), asking for minimal input. Let's update our configuration file and then customize it by running make menuconfig (there are several options here, such as make xconfig and make gconfig, but I prefer the text-based menuconfig; there is another you can run by using make config, which runs through each and every option available--it's scary).

# make silentoldconfig
# make menuconfig

menuconfig is a graphical command line application which lets you navigate the features offered by the kernel. Each computer is considerably different from the next, so it really does no good to provide a list of things that I tweak. However, it is important to note what some of the symbols are in the menuconfig utility:

  • M = Module. Modules are loaded when they are required and can contribute to the speed of your system
  • * = built into the kernel. These are typically things which are necessary for your machine to function properly, such as support for your root file system.
  • X = exclusively selected. You'll see this when you select what type of processor you have, for example.

One thing to note before we go further is MAKE SURE YOU KERNEL HAS BUILT-IN SUPPORT FOR YOUR ROOT FILE SYSTEM!!!! My root file system is reiserfs. In my configuration, I made sure that reiserfs was marked with a star. If you don't do this, your kernel won't boot and you will be very frustrated. Trust me.

Your computer is probably quite different than mine, so you might want to just poke around and see if you recognize things that deal with your computer's hardware. Once you are done tweaking your kernel configuration, exit the configuration utility and make sure the configuration is stored in /usr/src/linux/.config

Next we get to build and install the kernel. After that, we have to add an entry to our boot manager so that we can try out our new kernel. The compilation part usually takes just about a half hour on my 2.2Ghz Turion64 processor with 1.25GB of RAM. It takes about 6 hours on my 300Mhz Pentium 2 with 32MB of RAM. Let's find out how long it takes for you to compile your kernel!

# time make
...
real    27m29.663s
user    23m34.476s
sys     2m56.575s

Now let's install the modules and install the appropriate files in the boot area:

# make modules_install
# make install

This is the part that always used to mess me up. I use Slackware Linux, which is more UNIX-ish than most distributions. It's actually the oldest surviving Linux distribution to date, but that's another story. For some reason, the make install command doesn't always work with Slackware. There is a process I use to setup my boot directory when I compile a new kernel. I wrote a simple shell script called fixkernelinstall to take care of it for me:

#!/bin/bash
# Configure my computer for a new kernel
# Author: Josh VanderLinden
# Assisted By: Dan Purcell

# if the user didn't supply a kernel number, ask for it
if [ $# -eq 0 ]; then
    echo -n "Kernel: "
    read kernel
else
    kernel=$1
fi

# determine root partition
echo "Determining root partition..."
rootpart=`mount -l | grep ' / ' | cut -f 1 -d\ `
echo "Root partition is $rootpart"

# copy kernel configuration file
cp /usr/src/linux/.config ./config-$kernel

# now rename everything
echo "Renaming files..."
mv System.map System.map-$kernel
mv vmlinuz vmlinuz-$kernel

# if the config file exists and it's a symlink, remove it
if [ -f 'config' -a `stat config | grep -c 'symbolic link'` = '1' ]; then
    echo "Removing link to configuration file"
    rm config
else
    # otherwise it might be important
    echo "Renaming configuration file"
    mv config config.bak
fi

# Link files
echo "Creating symlinks..."
ln -s System.map-$kernel System.map
ln -s config-$kernel config
ln -s vmlinuz-$kernel vmlinuz

# Update lilo
echo "Adding entry to /etc/lilo.conf for $kernel"
echo "image = /boot/vmlinuz-$kernel" >> /etc/lilo.conf
echo "  root = $rootpart" >> /etc/lilo.conf
echo "  label = $kernel" >> /etc/lilo.conf
echo "  read-only" >> /etc/lilo.conf
echo "Linux kernel $kernel has been configured."
echo "Please check your lilo configuration and run lilo before rebooting"

I'm not an expert on shell scripts, so please feel free to offer suggestions for doing things better if you know how. This script uses the kernel version (given by the user) to setup by /boot directory properly. In my case, I run the script as such

# cd /boot
# fixkernelinstall 2.6.20-jcv1

And the output is something like:

Determining root partition...
Root partition is /dev/hda5
Renaming files...
Renaming configuration file
Creating symlinks...
Adding entry to /etc/lilo.conf for 2.6.20-jcv1
Linux kernel 2.6.20-jcv1 has been configured.
Please check your lilo configuration and run lilo before rebooting

As you can see from the script, I use LILO instead of the arguably more popular GRUB. Either one works for me, but LILO is sufficient for my needs. If you want to use the same kind of script for a GRUB installation, just change the LILO part at the end to something like:

echo 'Adding entry to /boot/grub/menu.lst for $kernel'
echo '  title Linux on ($rootpart)' >> /boot/grub/menu.lst
echo '  root (hd0,4)' >> /boot/grub/menu.lst
echo '  kernel /boot/vmlinuz-$kernel root=$rootpart ro vga=normal' >> /boot/grub/menu.lst

Make sure you change the line with root (hd0,4) to fit your setup. With GRUB, you don't have to worry about applying changes to see the menu entry at boot. It's automatically there. With LILO, however, you have to actually apply changes each time you make them. You do this by running the lilo command as the superuser:

# lilo
Added Windows
Added Linux
Added 2.6.20-jcv1 *

The star (*) signifies the default kernel to boot. Make sure that your root partition is correctly specified in your boot loader configuration. My root partition is on /dev/hda5, but yours may be (and probably is) on a different partition. If you fail to specify the correct root partition, your system will not boot that kernel until the configuration is fixed. GRUB makes this a lot easier than LILO.

And this is the point when you start to cross your figures and hope that your computer doesn't blow up... We get to reboot our computer and hope that our configuration file plays well with our computer. So, let's do that! See you in a few minutes (hopefully).

# shutdown -r now

So here I am, back on Linux on my freshly-rolled kernel. I hope you are as successful as I have been this time around. Keep in mind that you have to reinstall custom kernel modules if you installed others while you were on your other kernel. For example, I use ndiswrapper to access wireless Internet. I have to recompile and reinstall the ndiswrapper module and device drivers before I can use wireless. Likewise, I have VMWare Server on my laptop, which installed special modules. I have to run vmware-config.pl to reconfigure VMWare Server for my new kernel before I can run any virtual machines.

To summarize, here are the commands that I used in this tutorial. Remember that lines beginning with a dollar sign ($) are executed as a non-privileged user, while lines beginning with the pound sign (#) are executed as the superuser (root).

$ cd /home/user/download
$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.20.tar.bz2
$ su -
# cd /usr/src
# tar jxf /home/user/download/linux-2.6.20.tar.bz2
# rm /usr/src/linux
# ln -s /usr/src/linux-2.6.20 /usr/src/linux
# cd /usr/src/linux
# make clean
# vi Makefile (to change EXTRAVERSION to -jcv1)
# cp ../linux-2.6.19.1/.config .
# make silentoldconfig
# make menuconfig (just to ensure settings were good)
# time make
# make modules_install
# make install
# cd /boot
# fixkernelinstall 2.6.20-jcv1
# vi /etc/lilo.conf (to make sure things were good)
# lilo
# shutdown -r now

I hope that you are able to use this tutorial to successfully install or upgrade your kernel. Good luck! Any comments or suggestions are welcome!

Use Your Linux System With A Broken Bootloader

Here are some simple steps that should allow you to make some very simple changes that could end up saving you a lot of time and possibly a reinstall of your Linux system.

  1. Acquire a Linux liveCD of your choice. I prefer SLAX, as it is very small and quick. There are countless others such as KNOPPIX, ZenWalk, Ubuntu, Fedora Live, and DSL. Each of these would easily be up to the task.
  2. Boot the live CD and get to a command line somehow. If you are in a GUI environment, this means starting a Terminal or Konsole session. If you are booted into a classic Linux login prompt, you're good to go.
  3. Make sure you're the root user (you can do this by typing whoami). If you're not root, type su - and enter the root password when prompted.
  4. Mount your linux root partition. These steps may vary depending on preference, but they should be fairly similar:
    1. cd /mnt
    2. mkdir myroot
    3. mount /dev/hda5 /mnt/myroot
    4. cd /mnt/myroot
  5. Depending on the complexity of the tasks you wish to perform, you may need to mount your device list according to what your live CD detected:
    1. mount /dev /mnt/myroot/dev
    2. mount -t proc /proc /mnt/myroot/proc
  6. Finally, switch into your installed system, specifying where your root partition is mounted and they shell you wish to use:
    1. chroot /mnt/myroot /bin/bash

At this point you should be able to do several useful things, such as reinstall your bootloader into the MBR. You can also edit hosed configuration files this way.

If you need to add Windows into your boot configuration, and you use LILO, you could try adding something like this to your /etc/lilo.conf:

other = /dev/hda1
    label = Windows
    table = /dev/hda

(be sure to run lilo again to save the change)

When you are all done with your chroot environment, simply type exit or logout or what have you. Then be sure to unmount anything you may have mounted in order to use the system.