Quick And Easy Execution Speed Testing

There have been many times when I've been programming, encounter a problem that probably involves a loop of some sort, and I think of two or more possible ways to achieve the same end result. At this point, I usually think about which one will probably be the fastest solution (execution-wise) while still being readable/maintainable. A lot of the time, the essentials of the problem can be tested in a few short lines of code.

A while back, I was perusing some Stack Overflow questions for work, and I stumbled upon what I consider one of the many hidden jewels in Python: the timeit module. Given a bit of code, this little guy will handle executing it in several loops and giving you the best time out of three trials (you can ask it to do more than 3 runs if you want). Once it completes its test, it will offer some very clean and useful output.

For example, today I encountered a piece of code that was making a comma-separated list of an arbitrary number of "%s". The code I saw essentially looked like this:

",".join(["%s"] * 50000)

Even though this code required no optimization, I thought, "Hey, that's neat... I wonder if a list comprehension could possibly be any faster." Here's an example of the contender:

",".join(["%s" for i in xrange(50000)])

I had no idea which would be faster, so timeit to the rescue!! Open up a terminal, type a couple one-line Python commands, and enjoy the results!

$ python -mtimeit 'l = ",".join(["%s"] * 50000)'
1000 loops, best of 3: 1.15 msec per loop
$ python -mtimeit 'l = ",".join(["%s" for i in xrange(50000)])'
100 loops, best of 3: 3.23 msec per loop

Hah, the list comprehension is certainly slower.

Now, for other more in-depth tests of performance, you might consider using the cProfile module. As far as I can tell, simple one-liners can't be tested directly from the command line using cProfile--they apparently need to be in a script. You can use something like:

python -mcProfile script.py

...in such situations. Or you can wrap function calls using cProfile.run():

import cProfile

def function_a():
    # something you want to profile

def function_b():
    # an alternative version of function_a to profile

if __name__ == '__main__':
    cProfile.run('function_a()')
    cProfile.run('function_b()')

I've used this technique for tests that I'd like to have "hard evidence" for in the future. The output of such a cProfile test looks something like this:

3 function calls in 6.860 CPU seconds

Ordered by: standard name

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1    0.000    0.000    6.860    6.860 <string>:1(<module>)
     1    6.860    6.860    6.860    6.860 test_enumerate.py:5(test_enumerate)
     1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

This is useful when your code is calling other functions or methods and you want to find where your bottlenecks are. Hooray for Python!

What profiling techniques do you use?

Whoa! Another Reason To Love Vim

I've been struggling with some misconfigured appliances at work for the past couple of days, and I was getting tired of manually diff-ing things. On a whim, I decided to ask Google if there is a better way. Turns out there is, and it uses what I already know and love: VIM. Here's a command that lets you diff two remote file using vimdiff:

vimdiff scp://user@host//path/to/file scp://user@otherhost//path/to/file

This is going to save me so much time! I hope it is as useful to you all as it is to me.

Review: Hacking Vim

Introduction

Some of my faithful visitors may have noticed that I have a thing for Vim, one of the oldest and most powerful text editors in the world. In the past 15 or so years that I've been developing, I have spent quite a bit of time in several different text editors. It seemed like I was continually on the quest to find the fastest, most feature-packed editor out there, while still being cross-platform compatible and having it stay out of my way. Speed has always been very important to me.

I have been using Vi and Vim regularly since about 2000, when I began dabbling with Linux. I could certainly hold my ground in either of the two programs, but I was by no means proficient. The more appealing text editors for me offered syntax highlighting and code completion. At the time, I was under the impression that Vi/Vim didn't offer either of these two features. It wasn't until around the middle of last year, however, that I really started putting effort into learning and using Vim. After asking some of my Vim-savvy friends a lot of questions to get me kickstarted, I began to see the power that lies in Vim.

Before long, Vim had replaced all other text editors as my preferred editing environment. I learned that Vim could satisfy just able every single one of my personal qualifications for the perfect editor. I dumped all other editors in favor of Vim, and I even opted to use Vim over a several hundred dollar IDE at work.

Anyway. I received a review copy of Kim Schulz' "Hacking Vim: A cookbook to get the most out of the latest Vim editor" a couple of months ago and have been rummaging through it since then. I have learned a ton of fantastic tips from this little book! Being a cookbook, you're not expected to read the entire book start to finish. Rather, you can dig right into whatever section interests you and feel right at home.

Brief Overview

Packt Publishing printed this book back in 2007, but all of the tips are still very much up-to-date. The book starts off with the obligatory history lesson (which is actually quite interesting if you're a nerd like me), and the target audience is described as such:

New users join the Vim user community every day and want to use this editor in their daily work, and even though Vim sometimes can be complex to use, they still favor it above other editors. This is a book for these Vim users.

After the history lesson, chapter 2 of the book digs right into personalizing Vim to fit your own preferences. Topics covered include:

  • changing fonts
  • changing color schemes
  • personalizing highlighting
  • customizing the status line
  • toggling menus and toolbars in gvim
  • adding your own menu items and toolbar buttons
  • customizing your work area

Chapter 3 discusses better navigation techniques. Topics covered include:

  • faster navigation in a file
  • faster navigation in the Vim help system
  • faster navigation in multiple buffers
  • in-file searching
  • searching in multiple files or buffers
  • using marks and signs

Chapter 4, titled "Production Boosters" discusses the following:

  • templates using simple template file
  • templates using abbreviations
  • auto-completion using known words and tag lists
  • auto-completion using omni-completion
  • macros
  • sessions
  • registers and undo branches
  • folding
  • vimdiff
  • opening remote files using Netrw

Chapter 5 introduces some advanced formatting tips. You can learn how to put text into nicely-formatted paragraphs, aligning text, marking headlines, and creating lists. For code, this chapter discusses several different indentation options.

Vim scripting is the topic of chapter 6, and Schulz covers a wide variety of useful tips to get anyone started on scripting Vim to do their bidding. Tips include:

  • creating syntax-coloring scripts
  • how to install and use scripts
  • different types of scripts
  • basic syntax of Vim scripts
  • how to structure Vim scripts
  • debugging a Vim script
  • using other scripting languages (Perl, Python, Ruby)

Appendix A describes how Vim can be used for much more than just text editing. Several different games, including Tetris and a Rubik's Cube are briefly introduced, along with how to use Vim as a mail client or programmer's IDE. Appendix B suggests miscellaneous configuration script maintenance tips, such as how you can maintain the same configuration script across several different machines.

My Thoughts

I was very impressed with this book. I was afraid that, being published in 2007, it might be a little too out-of-date for my personal tastes. Since the book is about Vim, though, I wasn't overly concerned (the editor has been around for decades, and it doesn't change drastically from release to release anymore).

Just like the last book I reviewed, I found several typos in this book. A lot of the typos were in the first few pages of the actual content, and some were definitely more minor than others. This sort of thing doesn't really detract much from the material covered, but it sure does stand out as a distraction for people who pay attention to details.

Here are some of the things that I truly enjoyed reading and learning about (many of which actually made my jaw drop in awe of Vim)

  • Specifying multiple fonts for GVim, just in case your first choice isn't always available:

    :set guifont=Courier\ New\ 12, Arial\ 10
    
  • Specifying different font faces based on the extension of the file you're editing:

    :autocmd BufEnter *.txt set guifont=Arial\ 12
    
  • Highlighting the line your cursor is currently on, and the column the cursor is in:

    :set cursorline
    :set cursorcolumn
    
  • Limiting the number of suggestions that the spell checker offers:

    :set spellsuggest=5
    
  • Navigating to different words based on whitespace instead of "regular" word separators:

    • W to move to the beginning of the next word
    • B to move to the beginning of the previous word
    • E to move to the beginning of the previous word

    I knew about the lowercase variations of these commands, but not the uppercase.

  • Navigating up and down in the same long, wrapped line:

    gk
    gj
    
  • Opening a file that is referenced in the current buffer:

    gf
    

    I learned that this even works on Python imports! Just like the description says, it will work on the import module, not classes or other objects from inside the module. Not quite that intelligent!

  • Incremental searching:

    :set incsearch
    
  • Searching up/down in a buffer for any occurrence of the word under the cursor:

    g#
    g*
    

    I knew about the usual # and *, but those two will only match the same exact word. When they're prefixed with g, they will match any occurrence of the word, be it whole or part of another word. For example, hitting g* while the cursor is over the word foo would would match both food and foobar, while * would match neither.

  • Using markers to jump between specific points in different open buffers (mA through mZ)

  • Prepopulating empty files based on their extension:

    :autocmd BufNewFile * silent! 0r $VIMHOME/templates/%:e.tpl
    
  • Formatting a paragraph of text:

    gqap
    
  • Formatting all paragraphs of text in a file:

    1gqG
    
  • Smart indentation:

    :set smartindent
    
  • Enabling paste mode, so smartindent doesn't try to format code that you paste into your buffer:

    :set paste
    
  • Prettifying XML and HTML using Tidy:

    :autocmd FileType xml exe ":silent 1,$!tidy --input-xml true --indent yes -q"
    :autocmd FileType html,htm exe ":silent 1,$!tidy --indent yes -q"
    

Conclusion

All in all, this is a fantastic book. I will be keeping it near my workstation as a quick reference book when I want to do something crazy with Vim. I've already recommended the book to several of my friends and acquaintances, and I will make the same recommendation here. If you are mildly familiar with Vim and at all interested in getting more out of this fabulous editor, I highly recommend picking up a copy of this book.

How I Have A Mobile & Desktop Site With Django

Part of the latest version of my site included the deployment of a mobile-friendly site. Up until recently, I hadn't even attempted to create a mobile site because I thought it would take more time than it was worth. I wanted something beyond just using CSS to hide certain elements on the page. I wanted to be able to break down the content of my site into its most basic pieces and only include what was necessary. Also, I wanted to figure it out on my own (instead of reusing wheels other people had invented before me--horrible, I know).

With these requirements, I was afraid it would require more resources than I could spare on my shared Web host. My initial impression was that I would have to leverage the django.contrib.sites framework in a fashion that would essentially require two distinct instances of my site running in RAM. Despite these feelings, I decided to embark on a mission to create a mobile-friendly site while still offering a full desktop-friendly site. It was surprisingly simple. This may not be the best way to do it, but it sure works for me, and I'm very satisfied. So satisfied, in fact, that I am going to share my solution with all of my Django-loving friends.

The first step is to add a couple of new settings to your settings.py file:

import os
DIRNAME = os.path.abspath(os.path.dirname(__file__))

TEMPLATE_DIRS = (
    os.path.join(DIRNAME, 'templates'),
)

MOBILE_TEMPLATE_DIRS = (
    os.path.join(DIRNAME, 'templates', 'mobile'),
)
DESKTOP_TEMPLATE_DIRS = (
    os.path.join(DIRNAME, 'templates', 'desktop'),
)

For those of you not used to seeing that os.path.join stuff, it's just a (very efficient) way to make your Django project more portable between different computers and even operating systems. The new variables are MOBILE_TEMPLATE_DIRS and DESKTOP_TEMPLATE_DIRS, and their respective meanings should be fairly obvious. Basically, this tells Django that it can look for templates in your_django_project/templates, your_django_project/templates/mobile, and your_django_project/templates/desktop.

Next, we need to install a middleware that takes care of determining which directory Django should pay attention to when rendering pages, between mobile and desktop. You can put this into your_django_project/middleware.py:

from django.conf import settings

class MobileTemplatesMiddleware(object):
    """Determines which set of templates to use for a mobile site"""

    ORIG_TEMPLATE_DIRS = settings.TEMPLATE_DIRS

    def process_request(self, request):
        # sets are used here, you can use other logic if you have an older version of Python
        MOBILE_SUBDOMAINS = set(['m', 'mobile'])
        domain = set(request.META.get('HTTP_HOST', '').split('.'))

        if len(MOBILE_SUBDOMAINS & domain):
            settings.TEMPLATE_DIRS = settings.MOBILE_TEMPLATE_DIRS + self.ORIG_TEMPLATE_DIRS
        else:
            settings.TEMPLATE_DIRS = settings.DESKTOP_TEMPLATE_DIRS + self.ORIG_TEMPLATE_DIRS

Now you need to install the new middleware. Back in your settings.py, find the MIDDLEWARE_CLASSES variable, and insert a line like the following:

'your_django_project.middleware.MobileTemplatesMiddleware',

Finally, if you already have a base.html template in your your_django_project/templates directory, rename it to something else, such as site_base.html. Now create two new directories: your_django_project/templates/mobile and your_django_project/templates/desktop. In both of those directories, create a new base.html template that extends site_base.html.

Example site_base.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>{% block base_title %}Code Koala{% endblock %} - {% block title %}Welcome{% endblock %}</title>
<link href="{{ MEDIA_URL }}css/common.css" rel="stylesheet" type="text/css" media="screen" />
{% block extra-head %}{% endblock %}

</head>
<body>
<div id="page-wrapper">
    {% block header %}
    <div id="logo">
        <h1><a href="/">Code Koala</a></h1>
    </div>
    <div id="header">
        <div id="menu">
            <ul>
                <li><a href="/" class="first">Home</a></li>
                <li><a href="/blog/">Blog</a></li>
                <li><a href="/about/">About</a></li>
                <li><a href="/contact/">Contact</a></li>
            </ul>
        </div>
    </div>
    {% endblock %}
    <div id="page">
        <div id="content">
            {% block content %}{% endblock %}
        </div>
        <div id="sidebar">
            {% block sidebar %}
            Stuff
            {% endblock %}
        </div>
    </div>
    <div id="footer">
        {% block footer %}
        Footer stuff
        {% endblock %}
    </div>
</div>
</body>
</html>

Example desktop/base.html

{% extends 'site_base.html' %}

{% block extra-head %}
<!-- stylesheets -->
<link href="{{ MEDIA_URL }}css/desktop.css" rel="stylesheet" type="text/css" media="screen" />

<!-- JavaScripts -->
<script type="text/javascript" src="{{ MEDIA_URL }}js/jquery.js"></script>
{% endblock %}

Example mobile/base.html

{% extends 'site_base.html' %}

{% block extra-head %}
<!-- stylesheets -->
<link href="{{ MEDIA_URL }}css/mobile.css" rel="stylesheet" type="text/css" media="screen" />
{% endblock %}

{% block sidebar %}{% endblock %}

Please forgive me if the HTML or whatever is incorrect--I butchered the actual templates I use on Code Koala for the examples. There are some neat things you can do in your pages to make them more mobile friendly, such as including something like <meta name="viewport" content="width=device-width; initial-scale=1.0; maximum-scale=1.0; user-scalable=0;" /> in your <head> tag. This is supposed to tell your visitor's browser to not scale your pages to make it all fit on the screen. You can find a lot of other such tips elsewhere on the interwebs, and I'm sure they'll be better explained elsewhere too. You can also find scripts to handle redirecting your visitors to a mobile site and whatnot. Google is your friend.

As for the Django side of things, that should be just about it. If you have other templates you want to customize based on the version of your site that visitors are viewing, simply add those templates to the your_django_project/templates/mobile or your_django_project/templates/desktop directories as necessary. For example, if you have an application called blog, and you want to override the entry_detail.html template for the mobile site, so it doesn't pull in a bunch of unnecessary information to save bandwidth, you could save your modified copy in your_django_project/templates/mobile/blog/entry_detail.html.

With this setup, all you have to do is point your main domain and a subdomain such as m.yourdomain.com to the same Django application, and the middleware will take care of the "heavy lifting". No need for an additional instance of your Django project just for the mobile site. No hackish hiding of elements using CSS. If you find this article useful and decide to use these techniques on your site, please let me know how it works in your environment and if you ran into any snags so I can update the information!

Even More New VIM Fun

It appears that my last article generated a bit of interest! I had one good friend share some enlightening tips with me after he read about my findings. He opened my eyes to the world of abbreviations in VIM. Oh boy, oh boy!!

I'll pretty much just continue today's earlier post, by showing another (super awesome) way to insert blocks into your Django templates. If you were to enter this line into your ~/.vimrc file...

ab bl <Esc>bi{% block <Esc>ea %}{% endblock %}<Esc>h%i

...you would be able to type this...

content bl

...while you're in insert mode. As soon as you hit your spacebar after the bl, VIM would turn it into...

{% block content %} {% endblock %}

...and put you in insert mode between the two tags. Perfect!

For those of you who are interested in understanding the abbreviation, first it will take you out of insert mode (<Esc>). Then it jumps to the beginning of the word before the bl abbreviation and drops back into insert mode (bi). Next it inserts {% block and hops out of insert mode long enough to move the cursor to the end of the word you typed before the bl abbreviation. It finishes inserting {% endblock %}, gets out of insert mode, goes back a character (to account for the space you typed after bl) (h), moves the cursor to the matching opening brace for the } at the end of {% endblock %} (%), and finally puts you back into insert mode (i). Whew! It's all the same old VIM stuff, just packed into one uber-powerful abbreviation!

Thank to Jonathan Geddes for the guidance on this! I find this method to be superior to the one I posted about earlier because you don't even have to leave insert mode to get the code block stuff inserted into your templates!

Django's New Comment System

There are a lot of exciting changes happening with Django right now. A lot. Some of these changes cause a lot of things to break across my sites. One such change was the integration of Thejaswi Puthraya's Summer of Code project: an improved comment system.

The first, and most obvious problem, was the change in the URLconf. This took me a while to track down for one reason or another. Here's the situation: originally, the django.contrib.comments application used a URLconf such as:

(r'^comments/', include('django.contrib.comments.urls.comments')),

This makes any comments-powered pages blow up. To solve this particular problem, just make it:

(r'^comments/', include('django.contrib.comments.urls')),

The next thing that caught me dealt with the templates for comments. Now there are actually some default ones, which is nice, but they might interfere with your own templates. I found that all I need in my templates/comments/ directory now is a single simple template called base.html:

{% extends 'base.html' %}

All of the other templates aren't needed unless you do some customized stuff (which I don't bother with).

Finally, and probably the most frustrating of all, getting an error such as:

NoReverseMatch: Reverse for '<function post_comment at 0xb504a1b4>' not found.

I'm not really sure why this problem has arisen, but my solution for it is to remove the entire django/contrib/comments/ directory and bring it back down from SVN. My guess is that some .pyc file lingering from the original comments application is interfering with the new comments application.

Feel free to post here if you have any other advice or problems!