Django Projects

Over the past 6 years, I've built a lot of things with Django. It has treated me very well, and I have very much enjoyed seeing it progress. I got into Django when I helped the company I was working for transition away from a homegrown PHP framework toward something more reliable and flexible. It was very exciting to learn more about Django at a time when the ecosystem was very young.

When I started with Django, there weren't a lot of pluggable apps to fill the void for things like blogs, event calendars, and other useful utilities for the kinds of sites I was building. That has changed quite a bit since then. The ecosystem has evolved and progressed like mad, and it's wonderful. We have so many choices for simple things to very complex things. It's amazing!

Unfortunately, during this whole time period, my development efforts have shifted from creating my own open source projects to share with the world toward more proprietary solutions for my employers. If it's not obvious to you from my blog activity in recent years, I've become very busy with family life and work. I have very little time to give my open source projects the attention they deserve.

For at least 4 years, I've been telling myself that I'd have/make time to revamp all of my projects. To make them usable with what Django is today instead of what it was when I built the projects. Yeah, this time has never showed up. Take a look at the last time I wrote a blog article!

I have decided to disown pretty much all of my open source Django projects. I've basically done this with one or two of the more popular projects already--let someone else take the reigns while I lurk in the background and occasionally comment on an issue here or there. I truly appreciate those who have taken the initiative here. But there are still plenty of projects that people may find useful that need some attention. I'm putting it up to the community to take these projects over if you find them useful so they can get the love and attention they need.

Here is a list of Django projects that anyone is free to assume responsibility for. Most of them are silly and mostly useless now. Some are unpleasant to look at and could use an entire rewrite.

  • django-articles - Blogging engine, which I wrote for my own site many years ago. This one could probably use a full rewrite. [already maintained by John Leith].
  • django-tracking - Visitor tracking to see what page a user is on and their general location based on IP. Update 2015-07-28: Adopted by bashu.
  • django-axes - Track failed login attempts and ban users. This one is already pretty much owned by aclark4life
  • django-reploc - Designed to show representative/retailer locations. I only used it on one site, but it was designed to be generic.
  • django-smileys - To easily replace things like :) and B-) with emoticons or whatever. Pretty lame. Update 2015-07-28: Adopted by bashu.
  • django-pendulum - A very basic time-tracking app. I believe this has already been forked and used for some more interesting commercial projects.
  • django-bibliophile - A way to let your visitors know what you're reading.
  • django-watermark - Some utilities to add watermarks to images. Update 2015-07-28: Adopted by bashu.
  • Any other Django-related project you see at https://github.com/codekoala?tab=repositories or https://bitbucket.org/codekoala/

The fact that I'm giving up these projects does not mean I'm giving up on Django. On the contrary, I'm still using it quite heavily. I'm just doing it in such a way that I can't necessarily post my work for everyone to use. I honestly don't expect much of this disowning effort, since the projects are mostly stale and incompatible with recent versions of Django. But please let me know if you do want to take over one of my projects and care for it.

Command Line Progress Bar With Python

Wow, it's been a very long time since I've posted on my blog. It's amazing how much time twins take up! Now they're almost 6 months old, and things are starting to calm down a bit. I will try to make time to keep my blog updated more regularly now.

Earlier today, a good friend asked me how I would handle displaying a progress bar on the command line in a Python program. I suggested a couple of things, including leaving his already working program alone for the most part and just having a separate thread display the progress bar. I'm not sure exactly why, but he decided to not use a separate thread. It didn't seem like it would be very difficult, so I spent a few minutes mocking up a quick little demo.

This is by no means the most perfect of solutions, but I think it might be useful for others who find themselves in similar circumstances. Please feel free to comment with your suggestions! This is literally my first stab at something like this, and I only spent about 5 minutes hacking it up.

Arduino-Powered Webcam Mount

Earlier this month, I completed yet another journey around the biggest star in our galaxy. Some of my beloved family members thought this would be a good occasion to send me some cash, and I also got a gift card for being plain awesome at work. Even though we really do need a bigger car and whatnot, my wife insisted that I only spend this money on myself and whatever I wanted.

Little did she know the can of worms she just opened up.

I took pretty much all of the money and blew it on stuff for my electronics projects. Up to this point, my projects have all been pretty boring simply because nothing ever moved--it was mostly just lights turning on and off or changing colors. Sure, that's fun, but things really start to get interesting when you actually interact with the physical world. With the birthday money, I was finally able to buy a bunch of servos to begin living out my childhood dream of building robots.

My first project since getting all of my new toys was a motorized webcam mount. My parents bought me a Logitech C910 for my birthday because they were tired of trying to see their grandchildren with the crappy webcam that is built into my laptop. It was a perfect opportunity to use SparkFun's tutorial for some facial tracking (thanks to OpenCV) using their Pan/Tilt Servo Bracket.

It took a little while to get everything setup properly, but SparkFun's tutorial explains perfectly how you can get everything setup if you want to repeat this project.

The problem I had with the SparkFun tutorial, though, is that it basically only gives you a standalone program that does the facial tracking and displays your webcam feed. What good is that? I actually wanted to use this rig to chat with people!! That's when I set out to figure out how to do this.

While the Processing sketch ran absolutely perfect on Windows, it didn't want to work on my Arch Linux system due to some missing dependencies that I didn't know how/care to satisfy. As such, I opted to rewrite the sketch using Python so I could do the facial tracking in Linux.

This is still a work in progress, but here's the current facial tracking program which tells the Arduino where the webcam should be pointing, along with the Arduino sketch.

Now that I could track a face and move my webcam in Linux, I still faced the same problem as before: how can I use my face-tracking, webcam-moving program during a chat with my mom? I had no idea how to accomplish this. I figured I would have to either intercept the webcam feed as it was going to Skype or the Google Talk Plugin, or I'd have to somehow consume the webcam feed and proxy it back out as a V4L2 device that the Google Talk Plugin could then use.

Trying to come up with a way of doing that seemed rather impossible (at least in straight Python), but I eventually stumbled upon a couple little gems.

So the GStreamer tutorial walks you step-by-step through different ways of using a gst-launch utility, and I found this information very useful. I learned that you can use tee to split a webcam feed and do two different things with it. I wondered if it would be possible to split one webcam feed and send it to two other V4L2 devices.

Enter v4l2loopback.

I was able to install this module from Arch's AUR, and using it was super easy (you should be root for this):

modprobe v4l2loopback devices=2

This created two new /dev/video* devices on my system, which happened to be /dev/video4 and /dev/video5 (yeah... been playing with a lot of webcams and whatnot). One device, video4, is for consumption by my face-tracking program. The other, video5, is for VLC, Skype, Google+ Hangouts, etc. After creating those devices, I simply ran the following command as a regular user:

gst-launch-0.10 v4l2src device=/dev/video1 ! \
    'video/x-raw-yuv,width=640,height=480,framerate=30/1' ! \
    tee name=t_vid ! queue ! \
    v4l2sink sync=false device=/dev/video4 t_vid. ! \
    queue ! videorate ! 'video/x-raw-yuv,framerate=30/1' ! \
    v4l2sink device=/dev/video5

There's a whole lot of stuff going on in that command that I honestly do not understand. All I know is that it made it so both my face-tracking Python program AND VLC can consume the same video feed via two different V4L2 devices! A co-worker of mine agreed to have a quick Google+ Hangout with me to test this setup under "real" circumstances (thx man). It worked :D Objective reached!

I had really hoped to find a way to handle this stuff inside Python, but I have to admit that this is a pretty slick setup. A lot of things are still hardcoded, but I do plan on making things a little more generic soon enough.

So here's my little rig (why yes, I did mount it on top of an old Kool-Aid powder thingy lid):

And a video of it in action. Please excuse the subject of the webcam video, I'm not sure where that guy came from or why he's playing with my webcam.

Django-Tracking 0.3.5

I've finally gotten around to looking at a bunch of tickets that had been opened for django-tracking in the past year and a half or so. I feel horrible that it's really taken that long for me to get to them! Every time I got a ticket notification, I told myself, "Okay, I'll work on that this weekend." Many have weekends have passed without any work on any of my projects. I'm going to get better about that!

Anyway, several fixes have gone into the latest version of django-tracking. Some have to do with unicode problems (thanks ramusus!). Others have to do with overall performance, while yet others have to do with overall stability.

The first interesting change in this release is that django-tracking no longer relies on the GeoIP Python API. Instead it's now using django.contrib.gis.utils.GeoIP. I had hoped that this would remove the dependency on the GeoIP C API, but it appears that I was mistaken. Oh well.

Perhaps the biggest improvement in this new release is the use of caching. With caching in place, the middleware classes don't slam the database nearly as badly as they used to. There's still more that could be done with caching to improve performance, but I think what I've got now will be a big help.

Another noteworthy change, in my opinion, is the use of logging. I've sprinkled mildly useful logging messages throughout the code so you can learn when something bad happens that is silently handled. I hope that this will help me improve the quality of the code as it will allow anyone who uses the project (and pays attention to the log messages, of course) to tell me when bad things are happening.

Finally, the packaging code has been updated to be much more simple. Version 0.3.5 has been uploaded to PyPI and is available via pip or easy_install. If you prefer to have the latest copy of the code, the official code repositories are (in order of my personal preference):

I can't wait for your feedback!

GitHub and django-articles

Some of you who prefer to use git for your version control needs and were following the django-articles mirror on GitHub may have noticed some strange activity recently. I noticed today that the GitHub mirror was out of sync with the other mirrors, and I took a bit of time to investigate the problem.

I thought, for some reason, that I might be able to quickly and easily bring it back into sync if I just deleted the repo, recreated it, and pushed my changes to it. That didn't work. This means that all of you who were once following the project there are no longer following it, and I only realized that side effect after I had clicked the delete button. I apologize for this inconvenience.

In the end, it turned out that I had some things misconfigured with git on my box. I have resolved the problems and have brought the mirror back into sync. Please let me know if you run into any problems with it!

New Feature in django-articles: Articles From Email

One of the features that I really like about sites like posterous and tumblr is that they allow you to send email to a special email address and have it be posted as a blog article. This is a feature I've been planning to implement in django-articles pretty much since its inception way back when. I finally got around to working on it.

The latest release of django-articles allows you to configure a mailbox, either IMAP4 or POP3, to periodically check for new emails. A new management command check_for_articles_from_email can be used to process the messages found in the special mailbox. If any emails are found, they will be fetched, parsed, and posted based on your configuration values. Only articles whose sender matches an active user in your Django site will be turned into articles. You can configure the command to mark such articles from email as "inactive" so they don't appear on the site without moderation. The default behavior, actually, is to mark the articles inactive--you must explicitly configure django-articles to automatically mark the articles as active if you want this behavior.

One of the biggest things that you should keep in mind with this new feature, though, is that it does not currently take your attachments into account. In time I plan on implementing this functionality. For now, only the plain text content of your email will be posted. Please see the project's README for more information about this new feature.

Please keep in mind that this is brand new functionality and it's not been very well tested in a wide variety of situations. Right now, it's in the "it works for me" stage. If you find problems with it, please create a ticket or update any similar existing tickets using the ticket tracker on bitbucket.org.

You can install or update django-articles using the following utilities:

  • pip install -U django-articles
  • easy_install -U django-articles
  • hg clone http://bitbucket.org/codekoala/django-articles/ or just hg pull -u if you have already cloned it
  • git clone git://github.com/codekoala/django-articles.git

Enjoy!

P.S. This article was posted via email

Site-Wide Caching in Django

My last article about caching RSS feeds in a Django project generated a lot of interest. My original goal was to help other people who have tried to cache QuerySet objects and received a funky error message. Many of my visitors offered helpful advice in the comments, making it clear that I was going about caching my feeds the wrong way.

I knew my solution was wrong before I even produced it, but I couldn't get Django's site-wide caching middleware to work in my production environment. Site-wide caching worked wonderfully in my development environment, and I tried all sorts of things to make it work in my production setup. It wasn't until one "Jacob" offered a beautiful pearl of wisdom that things started to make more sense:

This doesn't pertain to feeds, but one rather large gotcha with the cache middleware is that any javascript you are running that plants a cookie will affect the cache key. Google analytics, for instance, has that effect. A workaround is to use a middleware to strip out the offending cookies from the request object before the cache middleware looks at it.

The minute I read that comment, I realized just how logical it was! If Google Analytics, or any other JavaScript used on my site, was setting a cookie, and it changed that cookie on each request, then the caching engine would effectively have a different page to cache for each request! Thank you so much, Jacob, for helping me get past the frustration of not having site-wide caching in my production environment.

How To Setup Site-Wide Caching

While most of this can be gleaned from the official documentation, I will repeat it here in an effort to provide a complete "HOWTO". For further information, hit up the official caching documentation.

The first step is to choose a caching backend for your project. Built-in options include:

To specify which backend you want to use, define the CACHE_BACKEND variable in your settings.py. The definition for each backend is different, so check out the official documentation for details.

Next, install a couple of middleware classes, and pay attention to where the classes are supposed to appear in the list:

  • django.middleware.cache.UpdateCacheMiddleware - This should be the first middleware class in your MIDDLEWARE_CLASSES tuple in your settings.py.
  • django.middleware.cache.FetchFromCacheMiddleware - This should be the last middleware class in your MIDDLEWARE_CLASSES tuple in your settings.py.

Finally, you must define the following variables in your settings.py file:

  • CACHE_MIDDLEWARE_SECONDS - The number of seconds each page should be cached
  • CACHE_MIDDLEWARE_KEY_PREFIX - If the cache is shared across multiple sites using the same Django installation, set this to the name of the site, or some other string that is unique to this Django instance, to prevent key collisions. Use an empty string if you don't care

If you don't use anything like Google Analytics that sets/changes cookies on each request to your site, you should have site-wide caching enabled now. If you only want pages to be cached for users who are not logged in, you may add CACHE_MIDDLEWARE_ANONYMOUS_ONLY = True to your settings.py file--its meaning should be fairly obvious.

If, however, your site-wide caching doesn't appear to work (as it didn't for me for a long time), you can create a special middleware class to strip those dirty cookies from the request, so the caching middleware can do its work.

import re

class StripCookieMiddleware(object):
    """Ganked from http://2ze.us/Io"""

    STRIP_RE = re.compile(r'\b(_[^=]+=.+?(?:; |$))')

    def process_request(self, request):
        cookie = self.STRIP_RE.sub('', request.META.get('HTTP_COOKIE', ''))
        request.META['HTTP_COOKIE'] = cookie

Edit: Thanks to Tal for regex the suggestion!

Once you do that, you need only install the new middleware class. Be sure to install it somewhere between the UpdateCacheMiddleware and FetchFromCacheMiddleware classes, not first or last in the tuple. When all of that is done, your site-wide caching should really work! That is, of course, unless your offending cookies are not found by that STRIP_RE regular expression.

Thanks again to Jacob and "nf", the original author of the middleware class I used to solve all of my problems! Also, I'd like to thank "JaredKuolt" for the django-staticgenerator on his github account. It made me happy for a while as I was working toward real site-wide caching.

Google Code + Mercurial = Many Happies

Last night I noticed that Google Code is actually offering the Mercurial project hosting that they promised back in April. I guess it's been around for most of May, but I never saw any news to suggest that it was actually public. As soon as I noticed it, I converted one of my less-known, less-used SVN projects to Mercurial. I'm really liking it.

I need to do a bit more work on this particular project before I announce it to the world, but it's out there, and it's Mercurial powered now babay. I think I will be leaving most of my other projects in SVN so I don't upset all of the other people who actually use them.

Oh, I also noticed that the project quotas were bumped up quite a bit. Now each project seems to get a whopping 1GB of space for free!!! What do you have to say about that, BitBucket/GitHub/Assembla/[insert dirty, rotten free open source project hosting host name here]?!

Hooray for Google Code!