Django-Tracking 0.3.5

I've finally gotten around to looking at a bunch of tickets that had been opened for django-tracking in the past year and a half or so. I feel horrible that it's really taken that long for me to get to them! Every time I got a ticket notification, I told myself, "Okay, I'll work on that this weekend." Many have weekends have passed without any work on any of my projects. I'm going to get better about that!

Anyway, several fixes have gone into the latest version of django-tracking. Some have to do with unicode problems (thanks ramusus!). Others have to do with overall performance, while yet others have to do with overall stability.

The first interesting change in this release is that django-tracking no longer relies on the GeoIP Python API. Instead it's now using django.contrib.gis.utils.GeoIP. I had hoped that this would remove the dependency on the GeoIP C API, but it appears that I was mistaken. Oh well.

Perhaps the biggest improvement in this new release is the use of caching. With caching in place, the middleware classes don't slam the database nearly as badly as they used to. There's still more that could be done with caching to improve performance, but I think what I've got now will be a big help.

Another noteworthy change, in my opinion, is the use of logging. I've sprinkled mildly useful logging messages throughout the code so you can learn when something bad happens that is silently handled. I hope that this will help me improve the quality of the code as it will allow anyone who uses the project (and pays attention to the log messages, of course) to tell me when bad things are happening.

Finally, the packaging code has been updated to be much more simple. Version 0.3.5 has been uploaded to PyPI and is available via pip or easy_install. If you prefer to have the latest copy of the code, the official code repositories are (in order of my personal preference):

I can't wait for your feedback!

How I Have A Mobile & Desktop Site With Django

Part of the latest version of my site included the deployment of a mobile-friendly site. Up until recently, I hadn't even attempted to create a mobile site because I thought it would take more time than it was worth. I wanted something beyond just using CSS to hide certain elements on the page. I wanted to be able to break down the content of my site into its most basic pieces and only include what was necessary. Also, I wanted to figure it out on my own (instead of reusing wheels other people had invented before me--horrible, I know).

With these requirements, I was afraid it would require more resources than I could spare on my shared Web host. My initial impression was that I would have to leverage the django.contrib.sites framework in a fashion that would essentially require two distinct instances of my site running in RAM. Despite these feelings, I decided to embark on a mission to create a mobile-friendly site while still offering a full desktop-friendly site. It was surprisingly simple. This may not be the best way to do it, but it sure works for me, and I'm very satisfied. So satisfied, in fact, that I am going to share my solution with all of my Django-loving friends.

The first step is to add a couple of new settings to your settings.py file:

import os
DIRNAME = os.path.abspath(os.path.dirname(__file__))

TEMPLATE_DIRS = (
    os.path.join(DIRNAME, 'templates'),
)

MOBILE_TEMPLATE_DIRS = (
    os.path.join(DIRNAME, 'templates', 'mobile'),
)
DESKTOP_TEMPLATE_DIRS = (
    os.path.join(DIRNAME, 'templates', 'desktop'),
)

For those of you not used to seeing that os.path.join stuff, it's just a (very efficient) way to make your Django project more portable between different computers and even operating systems. The new variables are MOBILE_TEMPLATE_DIRS and DESKTOP_TEMPLATE_DIRS, and their respective meanings should be fairly obvious. Basically, this tells Django that it can look for templates in your_django_project/templates, your_django_project/templates/mobile, and your_django_project/templates/desktop.

Next, we need to install a middleware that takes care of determining which directory Django should pay attention to when rendering pages, between mobile and desktop. You can put this into your_django_project/middleware.py:

from django.conf import settings

class MobileTemplatesMiddleware(object):
    """Determines which set of templates to use for a mobile site"""

    ORIG_TEMPLATE_DIRS = settings.TEMPLATE_DIRS

    def process_request(self, request):
        # sets are used here, you can use other logic if you have an older version of Python
        MOBILE_SUBDOMAINS = set(['m', 'mobile'])
        domain = set(request.META.get('HTTP_HOST', '').split('.'))

        if len(MOBILE_SUBDOMAINS & domain):
            settings.TEMPLATE_DIRS = settings.MOBILE_TEMPLATE_DIRS + self.ORIG_TEMPLATE_DIRS
        else:
            settings.TEMPLATE_DIRS = settings.DESKTOP_TEMPLATE_DIRS + self.ORIG_TEMPLATE_DIRS

Now you need to install the new middleware. Back in your settings.py, find the MIDDLEWARE_CLASSES variable, and insert a line like the following:

'your_django_project.middleware.MobileTemplatesMiddleware',

Finally, if you already have a base.html template in your your_django_project/templates directory, rename it to something else, such as site_base.html. Now create two new directories: your_django_project/templates/mobile and your_django_project/templates/desktop. In both of those directories, create a new base.html template that extends site_base.html.

Example site_base.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>{% block base_title %}Code Koala{% endblock %} - {% block title %}Welcome{% endblock %}</title>
<link href="{{ MEDIA_URL }}css/common.css" rel="stylesheet" type="text/css" media="screen" />
{% block extra-head %}{% endblock %}

</head>
<body>
<div id="page-wrapper">
    {% block header %}
    <div id="logo">
        <h1><a href="/">Code Koala</a></h1>
    </div>
    <div id="header">
        <div id="menu">
            <ul>
                <li><a href="/" class="first">Home</a></li>
                <li><a href="/blog/">Blog</a></li>
                <li><a href="/about/">About</a></li>
                <li><a href="/contact/">Contact</a></li>
            </ul>
        </div>
    </div>
    {% endblock %}
    <div id="page">
        <div id="content">
            {% block content %}{% endblock %}
        </div>
        <div id="sidebar">
            {% block sidebar %}
            Stuff
            {% endblock %}
        </div>
    </div>
    <div id="footer">
        {% block footer %}
        Footer stuff
        {% endblock %}
    </div>
</div>
</body>
</html>

Example desktop/base.html

{% extends 'site_base.html' %}

{% block extra-head %}
<!-- stylesheets -->
<link href="{{ MEDIA_URL }}css/desktop.css" rel="stylesheet" type="text/css" media="screen" />

<!-- JavaScripts -->
<script type="text/javascript" src="{{ MEDIA_URL }}js/jquery.js"></script>
{% endblock %}

Example mobile/base.html

{% extends 'site_base.html' %}

{% block extra-head %}
<!-- stylesheets -->
<link href="{{ MEDIA_URL }}css/mobile.css" rel="stylesheet" type="text/css" media="screen" />
{% endblock %}

{% block sidebar %}{% endblock %}

Please forgive me if the HTML or whatever is incorrect--I butchered the actual templates I use on Code Koala for the examples. There are some neat things you can do in your pages to make them more mobile friendly, such as including something like <meta name="viewport" content="width=device-width; initial-scale=1.0; maximum-scale=1.0; user-scalable=0;" /> in your <head> tag. This is supposed to tell your visitor's browser to not scale your pages to make it all fit on the screen. You can find a lot of other such tips elsewhere on the interwebs, and I'm sure they'll be better explained elsewhere too. You can also find scripts to handle redirecting your visitors to a mobile site and whatnot. Google is your friend.

As for the Django side of things, that should be just about it. If you have other templates you want to customize based on the version of your site that visitors are viewing, simply add those templates to the your_django_project/templates/mobile or your_django_project/templates/desktop directories as necessary. For example, if you have an application called blog, and you want to override the entry_detail.html template for the mobile site, so it doesn't pull in a bunch of unnecessary information to save bandwidth, you could save your modified copy in your_django_project/templates/mobile/blog/entry_detail.html.

With this setup, all you have to do is point your main domain and a subdomain such as m.yourdomain.com to the same Django application, and the middleware will take care of the "heavy lifting". No need for an additional instance of your Django project just for the mobile site. No hackish hiding of elements using CSS. If you find this article useful and decide to use these techniques on your site, please let me know how it works in your environment and if you ran into any snags so I can update the information!

Site-Wide Caching in Django

My last article about caching RSS feeds in a Django project generated a lot of interest. My original goal was to help other people who have tried to cache QuerySet objects and received a funky error message. Many of my visitors offered helpful advice in the comments, making it clear that I was going about caching my feeds the wrong way.

I knew my solution was wrong before I even produced it, but I couldn't get Django's site-wide caching middleware to work in my production environment. Site-wide caching worked wonderfully in my development environment, and I tried all sorts of things to make it work in my production setup. It wasn't until one "Jacob" offered a beautiful pearl of wisdom that things started to make more sense:

This doesn't pertain to feeds, but one rather large gotcha with the cache middleware is that any javascript you are running that plants a cookie will affect the cache key. Google analytics, for instance, has that effect. A workaround is to use a middleware to strip out the offending cookies from the request object before the cache middleware looks at it.

The minute I read that comment, I realized just how logical it was! If Google Analytics, or any other JavaScript used on my site, was setting a cookie, and it changed that cookie on each request, then the caching engine would effectively have a different page to cache for each request! Thank you so much, Jacob, for helping me get past the frustration of not having site-wide caching in my production environment.

How To Setup Site-Wide Caching

While most of this can be gleaned from the official documentation, I will repeat it here in an effort to provide a complete "HOWTO". For further information, hit up the official caching documentation.

The first step is to choose a caching backend for your project. Built-in options include:

To specify which backend you want to use, define the CACHE_BACKEND variable in your settings.py. The definition for each backend is different, so check out the official documentation for details.

Next, install a couple of middleware classes, and pay attention to where the classes are supposed to appear in the list:

  • django.middleware.cache.UpdateCacheMiddleware - This should be the first middleware class in your MIDDLEWARE_CLASSES tuple in your settings.py.
  • django.middleware.cache.FetchFromCacheMiddleware - This should be the last middleware class in your MIDDLEWARE_CLASSES tuple in your settings.py.

Finally, you must define the following variables in your settings.py file:

  • CACHE_MIDDLEWARE_SECONDS - The number of seconds each page should be cached
  • CACHE_MIDDLEWARE_KEY_PREFIX - If the cache is shared across multiple sites using the same Django installation, set this to the name of the site, or some other string that is unique to this Django instance, to prevent key collisions. Use an empty string if you don't care

If you don't use anything like Google Analytics that sets/changes cookies on each request to your site, you should have site-wide caching enabled now. If you only want pages to be cached for users who are not logged in, you may add CACHE_MIDDLEWARE_ANONYMOUS_ONLY = True to your settings.py file--its meaning should be fairly obvious.

If, however, your site-wide caching doesn't appear to work (as it didn't for me for a long time), you can create a special middleware class to strip those dirty cookies from the request, so the caching middleware can do its work.

import re

class StripCookieMiddleware(object):
    """Ganked from http://2ze.us/Io"""

    STRIP_RE = re.compile(r'\b(_[^=]+=.+?(?:; |$))')

    def process_request(self, request):
        cookie = self.STRIP_RE.sub('', request.META.get('HTTP_COOKIE', ''))
        request.META['HTTP_COOKIE'] = cookie

Edit: Thanks to Tal for regex the suggestion!

Once you do that, you need only install the new middleware class. Be sure to install it somewhere between the UpdateCacheMiddleware and FetchFromCacheMiddleware classes, not first or last in the tuple. When all of that is done, your site-wide caching should really work! That is, of course, unless your offending cookies are not found by that STRIP_RE regular expression.

Thanks again to Jacob and "nf", the original author of the middleware class I used to solve all of my problems! Also, I'd like to thank "JaredKuolt" for the django-staticgenerator on his github account. It made me happy for a while as I was working toward real site-wide caching.

Announcing django-ittybitty 0.1.0-pre2

I'd like to take this opportunity to officially announce my latest little side project: django-ittybitty! Some of you out there might not find this to be a useful application, but I hope others will enjoy it.

Many of you are familiar with the URL-shortening sites like http://tinyurl.com/, http://is.gd/, http://cli.gs/, and whole slew of others. These sites are all fine and dandy, right? Wrong! What happens when those sites have downtime and potential visitors to your site never get to your site because the URL-shortening site is down? You lose traffic. That's not good, in case you were unsure about it.

That is why I made this application. It allows you to have short URLs for any and every page on your Django site. No more need to rely on 3rd party servers to translate short URLs to real URLs on your site. So long as your pony-powered site is up and running, your visitors will be able to use URLs generated by this application to get anywhere on your site. All you need to do to make this work is download and install the application, add a middleware class to your MIDDLEWARE_CLASSES, and then use a simple template tag to generate a short URL for any given page.

django-ittybitty will keep track of the number of times a particular "itty bitty URL" has been used to access your site. I suppose some people will find that useful, but it's hardly a true metric for your "most popular" pages.

The algorithm behind this application is very simple, but it can potentially handle around 18,446,744,073,709,551,615 shortened URLs in 64 characters or fewer, neglecting the 'http://www.....' for your site (good luck getting your database to play well with that many records, much less storing them on a server :)).

For more information, please check out the project pages and enjoy:

For those who are interested, here are some code samples for how to use django-ittybitty:

{% extends 'base.html' %}
{% load ittybitty_tags %}

{% block content %}
<a href="{% ittybitty_url %}">Link to this page!</a>
{% endblock %}

or:

{% extends 'base.html' %}
{% load ittybitty_tags %}

{% block content %}
{% ittybitty_url as ittybitty %}
<a href="{{ ittybitty.get_shortcut }}">Link to this page!</a>
{% endblock %}

or:

{% extends 'base.html' %}
{% load ittybitty_tags %}

{% block content %}
{% ittybitty_url as ittybitty %}
{% with ittybitty.get_shortcut as short_url %}
<a href="{{ short_url }}">Link to this page!</a>
<a href="{{ short_url }}">Link to this page again!</a>
<a href="{{ short_url }}">Link to this page one more time!</a>
{% endwith %}
{% endblock %}

Enjoy!

Pluggable Django Apps and Django's Admin

There are a lot of us out there who build these reusable, pluggable Django applications and share them with the world. Some of the applications are more useful than others. Personally, I think my applications tend to fall in the less useful category, but I like to build and release them nonetheless.

The Background

The other day, I received a suggestion from one of the users of django-pendulum for how to make it a little more user-friendly. The problem was that he had not yet configured Pendulum for his current Site. As soon as he tried to access his site after installing django-pendulum, he was greeted with a nasty Django exception page, admonishing him to configure Pendulum for the site. It was pretty dirty and effective, but not very pleasant at all.

This user suggested that, instead of displaying the nasty exception page, I redirect the user to a place where they could in fact configure Pendulum. I thought this was a grand idea, and I set out to make it happen last night.

The Dilemma

That's when I realized I had a problem. django-pendulum is intended to be a reusable Django application. Most of the time it might be safe to assume that everyone uses /admin/ to get to the Django administration utility. However, there's also a good chance that this will not be the prefix used to access the administration utility. What do you do when you hardcode your applications to redirect a user to /admin/ and they use something more secure like /milwaukee/ for their administration utility?

So the question is this: how do you determine what URL will bring a user to the Django administration page for a particular Django model based on the specific site's URLconf setup?

The Solution

It turns out that one solution (not sure if it's safe to say "the solution" necessarily) with Django 1.1+ is to have something like this:

from django.core.urlresolvers import reverse
admin_url = reverse('admin_pendulum_pendulumconfiguration_add')

The named URLconf used in the reverse function should ensure that the appropriate administration URL prefix is used for whatever site your application is installed on.

I did a bit of browsing through the Django documentation, but I've never seen it stated anywhere what the names are for administration URLs actually are. I found out about this little nugget by examining the output of the django.contrib.admin.ModelAdmin.get_urls method (see get_urls(self)). From the looks of it, pretty much any Django model registered with your Django administration will have URLconf items similar to the following:

[<RegexURLPattern admin_pendulum_pendulumconfiguration_changelist ^$>,
 <RegexURLPattern admin_pendulum_pendulumconfiguration_add ^add/$>,
 <RegexURLPattern admin_pendulum_pendulumconfiguration_history ^(.+)/history/$>,
 <RegexURLPattern admin_pendulum_pendulumconfiguration_delete ^(.+)/delete/$>,
 <RegexURLPattern admin_pendulum_pendulumconfiguration_change ^(.+)/$>]

In other words, you get 5 URLconf patterns per model: changelist, add, history, delete, and change. These patterns are named, and they all appear to be prefixed with admin_[app_label]_[model]_. In the case of django-pendulum, the application is called pendulum so [app_label] becomes pendulum. The model in question is called PendulumConfiguration so the [model] becomes pendulumconfiguration. I assume this same pattern will be followed for any registered model. Please correct me if I'm wrong.

More Details

For those of you interested in more details, django-pendulum uses a middleware class to redirect users to the screen where they can configure Pendulum for the site in question (unless it's already been configured). Here's my middleware as of revision 18:

from django.contrib.auth.views import login
from django.contrib.sites.models import Site
from django.core.urlresolvers import reverse
from django.http import HttpResponseRedirect
from django.conf import settings
from pendulum.models import PendulumConfiguration

SITE = Site.objects.get_current()
admin_url = reverse('admin_pendulum_pendulumconfiguration_add')
login_url = getattr(settings, 'LOGIN_URL', '/accounts/login/')

class PendulumMiddleware:
    """
    This middleware ensures that anyone trying to access Pendulum must be
    logged in.  If Pendulum hasn't been configured for the current site, the
    staff users will be redirected to the page to configure it.
    """
    def process_request(self, request):
        try:
            SITE.pendulumconfiguration
        except PendulumConfiguration.DoesNotExist:
            # this will force the user to configure pendulum if they're staff
            if request.user.has_perm('add_pendulumconfiguration') and \
                request.path not in (admin_url, login_url):
                # leave the user a message
                request.user.message_set.create(message='Please configure Pendulum for %s' % SITE)

                return HttpResponseRedirect(admin_url)
        else:
            entry_url = reverse('pendulum-entries')

            if request.path[:len(entry_url)] == entry_url and request.user.is_anonymous():
                if request.POST:
                    return login(request)
                else:
                    return HttpResponseRedirect('%s?next=%s' % (login_url, request.path))

This checks to see if Pendulum has been configured for the site. If not, the middleware will check to see if the current user has permission to add a configuration for Pendulum. If so, they will be redirected to the appropriate configuration screen.

If Pendulum has been configured for the site or the current user does not have permission to add a configuration, the user should be unaffected. If the user is trying to access Pendulum on the site's front-end, the middleware will ensure that they're actually logged in.

As always, suggestions are welcome!

Firebug for !Firefox

Pretty much anyone who's been doing any Web development in the last few years probably prefers to use Firefox because of the incredibly powerful extensions it offers. Among the extensions I hear most Web developers complain about not having in other browsers are Web Developer and Firebug. Several people feel that they could get by with another browser (such as Google Chrome) if it only had Firebug.

Well, my friends, the trusty folks who built Firebug actually offer their amazing product for other browsers! It goes by the name of "Firebug Lite." I'm not sure exactly how long this has been around, but the earliest date I can find suggests that it was released to the public in July of 2008.

I happened upon this utility while perusing Django Snippets the other day. A member by the username of jfw posted a middleware which injects the Firebug Lite utility into a response when you're not using Firefox and when your site is in debug mode. I've found it to be quite useful. I hope you all do too!!