I work on a test automation framework at my day job. It's Django-powered, and there's a lot of neat stuff going on with it. I love building it!

Anyway, yesterday during a meeting, I got an email from a co-worker who seemed to be in a bit of a panic. He wrote that he accidentally deleted the wrong thing, and, being Django on the backend, a nice cascading delete went with it (why he ignored the confirmation page is beyond me). He asked if we had any database backups that we could restore, also curious as to how long it would take.

Well, lucky for him (and me!), I decided very early on while working on the project that I would implement a custom database driver that never actually deletes stuff (mostly for auditing purposes). Instead, it simply marks any record the user asks to delete as inactive, thus hiding it from the UI. Along with this, nightly database backups were put in place.

I'll be quite honest--I had a moment of fear as I considered how long it had been since I really checked that either of these two things were still working as designed. I implemented the database driver before I learned to appreciate unit testing, and I haven't made it to that piece as I've been backfilling my unit test suite (yet). As for the nightly database backups, I had never actually needed to restore one, so for probably the last year I didn't really bother checking a) that they were still being produced or b) that they were valid backups.

Thankfully, both pieces were still working perfectly. All I had to do was undelete a few things from the database, as I haven't made a UI for this. After doing that, I realized that one set of relationships was not handled by the custom driver. To fix this, I just restored the most recent nightly backup to a separate database and extracted just those relationships I was interested in. And it worked!

This is the first time I've really been bitten by a situation like this personally. I'm very pleased that I had the foresight to implement the precautionary measures early on in my project. I've also learned that I should probably keep up with those measures a bit better. I definitely plan to make some changes to help mitigate the potential for bad stuff to happen in the future. But it looks like I have a good foundation to build upon now.

TL;DR: unseen magic and valid database backups FTW.

Over the past 6 years, I've built a lot of things with Django. It has treated me very well, and I have very much enjoyed seeing it progress. I got into Django when I helped the company I was working for transition away from a homegrown PHP framework toward something more reliable and flexible. It was very exciting to learn more about Django at a time when the ecosystem was very young.

When I started with Django, there weren't a lot of pluggable apps to fill the void for things like blogs, event calendars, and other useful utilities for the kinds of sites I was building. That has changed quite a bit since then. The ecosystem has evolved and progressed like mad, and it's wonderful. We have so many choices for simple things to very complex things. It's amazing!

Unfortunately, during this whole time period, my development efforts have shifted from creating my own open source projects to share with the world toward more proprietary solutions for my employers. If it's not obvious to you from my blog activity in recent years, I've become very busy with family life and work. I have very little time to give my open source projects the attention they deserve.

For at least 4 years, I've been telling myself that I'd have/make time to revamp all of my projects. To make them usable with what Django is today instead of what it was when I built the projects. Yeah, this time has never showed up. Take a look at the last time I wrote a blog article!

I have decided to disown pretty much all of my open source Django projects. I've basically done this with one or two of the more popular projects already--let someone else take the reigns while I lurk in the background and occasionally comment on an issue here or there. I truly appreciate those who have taken the initiative here. But there are still plenty of projects that people may find useful that need some attention. I'm putting it up to the community to take these projects over if you find them useful so they can get the love and attention they need.

Here is a list of Django projects that anyone is free to assume responsibility for. Most of them are silly and mostly useless now. Some are unpleasant to look at and could use an entire rewrite.

• django-articles - Blogging engine, which I wrote for my own site many years ago. This one could probably use a full rewrite. [already maintained by John Leith].
• django-tracking - Visitor tracking to see what page a user is on and their general location based on IP. Update 2015-07-28: Adopted by bashu.
• django-axes - Track failed login attempts and ban users. This one is already pretty much owned by aclark4life
• django-reploc - Designed to show representative/retailer locations. I only used it on one site, but it was designed to be generic.
• django-smileys - To easily replace things like :) and B-) with emoticons or whatever. Pretty lame. Update 2015-07-28: Adopted by bashu.
• django-pendulum - A very basic time-tracking app. I believe this has already been forked and used for some more interesting commercial projects.
• django-bibliophile - A way to let your visitors know what you're reading.
• django-watermark - Some utilities to add watermarks to images. Update 2015-07-28: Adopted by bashu.
• Any other Django-related project you see at https://github.com/codekoala?tab=repositories or https://bitbucket.org/codekoala/

The fact that I'm giving up these projects does not mean I'm giving up on Django. On the contrary, I'm still using it quite heavily. I'm just doing it in such a way that I can't necessarily post my work for everyone to use. I honestly don't expect much of this disowning effort, since the projects are mostly stale and incompatible with recent versions of Django. But please let me know if you do want to take over one of my projects and care for it.

Hello again everyone! Soooo much has happened since I last posted on my blog. I figured it was about time to check in and actually be active on my own site again. What follows is just a summary of what has happened in our lives since the beginning of February this year.

Enough About Work

Aside from all of the excitement in my career decisions, things are going very well with the family. We live about 3 hours away from my mom, and we've been out there to visit a few times already. It's really fun to see the kids playing with their grandma! The last time we were out for a visit, for my grandmother's 80th birthday, my son and I took my dad's Rhino for a spin. We got stuck, and it was sooo much fun!

The twins are growing so well too. They're crawling and getting around very well now. Jane has started to stand up on her own, and she tries to take a step every once in a while. Claire prefers to sit, but she loves to wave, clap, and repeat noises that she hears.

My wife is planning on starting up a new website soon, and she keeps taunting me with the possibility of having me build it for her. Yes, taunting.

Okay, Back to Hobbies

My wife also picked up a Dremel Trio for Dad's Day. To get used to it, I made some little wooden signs with the kids' names on them. Being the quasi-perfectionist that I am, I'm not completely satisfied with how they all turned out. I suppose they'll do for a "first attempt" sort of result though!

I've still got various projects in the works with my Arduino and whatnot. A couple of months ago, I finished a project that helps me see where I'm walking when I go down to my mancave at night. The light switches for the basement are all at the stairs, and my setup is on the opposite side of the basement. I typically prefer to have the lights off when I'm on my computer, and it was annoying and horribly inefficient to turn the lights on when entering the basement, go to my computer, then go back to turn the lights off.

To solve that problem, I re-purposed one of my PIR motion sensors and picked up a LED strip from eBay. I have the motion sensor pointing at the entrance to the basement, and the LED strip strung across the ceiling along the path that I take to get to my desk. When the motion sensor detects movement, it fades the LED strip on, continues to power it for a few seconds, and gradually fades them out when it no longer detects movement. It's all very sexy, if I do say so myself.

I've tried to capture videos of the setup, but my cameras all have poor light sensors or something, so it's difficult to really show what it's like. The LED strip illuminates the basement perfectly just long enough for me to get to my desk, but the videos just show a faint outline of my body lurking in the dark. :(

One project that is in the works right now is a desk fan that automatically turns on when the ambient temperature reaches a certain level. The fan's speed will vary depending on the temperature, and there will be an LCD screen to allow simple reporting and configuration of thresholds and whatnot. I'm pretty excited about it, but I want to order a few things off of eBay before I go much further with it.

Obviously, much more had happened in the past months, but this post is long enough already. Things are calming down quite a bit now that we're settled in, so I hope to resume activity on my open source projects as well as this blog.

I've finally gotten around to looking at a bunch of tickets that had been opened for django-tracking in the past year and a half or so. I feel horrible that it's really taken that long for me to get to them! Every time I got a ticket notification, I told myself, "Okay, I'll work on that this weekend." Many have weekends have passed without any work on any of my projects. I'm going to get better about that!

Anyway, several fixes have gone into the latest version of django-tracking. Some have to do with unicode problems (thanks ramusus!). Others have to do with overall performance, while yet others have to do with overall stability.

The first interesting change in this release is that django-tracking no longer relies on the GeoIP Python API. Instead it's now using django.contrib.gis.utils.GeoIP. I had hoped that this would remove the dependency on the GeoIP C API, but it appears that I was mistaken. Oh well.

Perhaps the biggest improvement in this new release is the use of caching. With caching in place, the middleware classes don't slam the database nearly as badly as they used to. There's still more that could be done with caching to improve performance, but I think what I've got now will be a big help.

Another noteworthy change, in my opinion, is the use of logging. I've sprinkled mildly useful logging messages throughout the code so you can learn when something bad happens that is silently handled. I hope that this will help me improve the quality of the code as it will allow anyone who uses the project (and pays attention to the log messages, of course) to tell me when bad things are happening.

Finally, the packaging code has been updated to be much more simple. Version 0.3.5 has been uploaded to PyPI and is available via pip or easy_install. If you prefer to have the latest copy of the code, the official code repositories are (in order of my personal preference):

• http://bitbucket.org/codekoala/django-tracking
• github.com/codekoala/django-tracking
• http://code.google.com/p/django-tracking

I can't wait for your feedback!

I've been working on some neat changes to django-articles recently, and I've just released version 2.1.1. The most noticeable feature in this release is Auto-Tagging. Since I feel like I've described the feature fairly well in the README, I'll just copy/paste that section here.

The auto-tagging feature allows you to easily apply any of your current tags to your articles. When you save an Article object with auto-tagging enabled for that article, django-articles will go through each of your existing tags to see if the entire word appears anywhere in your article's content. If a match is found, that tag will be added to the article.

For example, if you have tags "test" and "art", and you wrote a new auto-tagged Article with the text:

This is a test article.

django-articles would automatically apply the "test" tag to this article, but not the "art" tag. It will only apply the "art" tag automatically when the actual word "art" appears in the content.

Auto-tagging does not remove any tags that are already assigned to an article. This means that you can still add tags the good, old-fashioned way in the Django Admin without losing them. Auto-tagging will only add to an article's existing tags (if needed).

Auto-tagging is enabled for all articles by default. If you want to disable it by default (and enable it on a per-article basis), set ARTICLES_AUTO_TAG to False in your settings.py file.

Auto-Tagging does not attempt to produce any keywords that magically represent the content of your articles. Only existing tags are used!!

I sure had fun programming this little feature. I know it will be particularly useful for my own site.

Another item I'd like to mention about this release: I've finally started using South migrations in this app. This is a move I've been planning to make for quite some time now.

Head on over to http://bitbucket.org/codekoala/django-articles or use pip install -U django-articles (or easy_install django-articles if you must)! Enjoy!

There have been many times when I've been programming, encounter a problem that probably involves a loop of some sort, and I think of two or more possible ways to achieve the same end result. At this point, I usually think about which one will probably be the fastest solution (execution-wise) while still being readable/maintainable. A lot of the time, the essentials of the problem can be tested in a few short lines of code.

A while back, I was perusing some Stack Overflow questions for work, and I stumbled upon what I consider one of the many hidden jewels in Python: the timeit module. Given a bit of code, this little guy will handle executing it in several loops and giving you the best time out of three trials (you can ask it to do more than 3 runs if you want). Once it completes its test, it will offer some very clean and useful output.

For example, today I encountered a piece of code that was making a comma-separated list of an arbitrary number of "%s". The code I saw essentially looked like this:

",".join(["%s"] * 50000)

Even though this code required no optimization, I thought, "Hey, that's neat... I wonder if a list comprehension could possibly be any faster." Here's an example of the contender:

",".join(["%s" for i in xrange(50000)])

I had no idea which would be faster, so timeit to the rescue!! Open up a terminal, type a couple one-line Python commands, and enjoy the results!

$ python -mtimeit 'l = ",".join(["%s"] * 50000)'
1000 loops, best of 3: 1.15 msec per loop
$ python -mtimeit 'l = ",".join(["%s" for i in xrange(50000)])'
100 loops, best of 3: 3.23 msec per loop

Hah, the list comprehension is certainly slower.

Now, for other more in-depth tests of performance, you might consider using the cProfile module. As far as I can tell, simple one-liners can't be tested directly from the command line using cProfile--they apparently need to be in a script. You can use something like:

python -mcProfile script.py

...in such situations. Or you can wrap function calls using cProfile.run():

import cProfile

def function_a():
    # something you want to profile

def function_b():
    # an alternative version of function_a to profile

if __name__ == '__main__':
    cProfile.run('function_a()')
    cProfile.run('function_b()')

I've used this technique for tests that I'd like to have "hard evidence" for in the future. The output of such a cProfile test looks something like this:

3 function calls in 6.860 CPU seconds

Ordered by: standard name

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1    0.000    0.000    6.860    6.860 <string>:1(<module>)
     1    6.860    6.860    6.860    6.860 test_enumerate.py:5(test_enumerate)
     1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

This is useful when your code is calling other functions or methods and you want to find where your bottlenecks are. Hooray for Python!

What profiling techniques do you use?

Yesterday, one of the QA guys at work approached me with a question that turned out to be much more interesting to me than I think he had planned. He's been doing some unit testing using Selenium, exporting his test cases to Python. His question was this: how can I run the same unit tests using multiple browsers and multiple target servers?

I'm pretty sure he expected a simple 3-step answer or something like that. Instead, he got my crazy wide-eyed "ohhh... that's something I want to experiment with!" look. I started rambling on about inheritance, dynamic class creation, and nested for loops. His eyes started to look a little worried. He didn't really appreciate the nerdy lingo that much. I told him to pull up a chair and get comfortable.

Since I already had some other work I needed to pay attention to, I didn't want to spend too much time trying to figure out a good way to solve his problem. After about 20 minutes of devilish chuckles and frantic rustling through Python documentation, I came up with the following code:

from types import ClassType
from selenium import selenium
import unittest

IPS = ['192.168.0.1', '192.168.0.2']
BROWSERS = ['safari', 'chrome']

class SomeUnitTest(object):

    def test_something(self):
        sel = self.selenium
        # test code

def main(base):
    suites = []
    results = unittest.TestResult()

    for iidx, ip in enumerate(IPS):
        for bidx, browser in enumerate(BROWSERS):
            def setUp(self):
                self.verificationErrors = []
                self.selenium = selenium("localhost", 4444, "*%s" % self.browser, "http://%s/" % self.ip)
                self.selenium.start()

            def tearDown(self):
                self.selenium.stop()
                self.assertEqual([], self.verificationErrors)

            ut = ClassType('UT_%i_%i' % (iidx, bidx), (unittest.TestCase, base), {'ip': ip, 'browser': browser})
            ut.setUp = setUp
            ut.tearDown = tearDown

            suites.append(unittest.TestLoader().loadTestsFromTestCase(ut))

    unittest.TestSuite(suites)(results)
    for obj, error in results.errors:
        print 'In: ', obj
        print error

if __name__ == "__main__":
    main(SomeUnitTest)

I know, I know... it's got some dirty rotten tricks in it, and there are probably more efficient ways of doing what I've done. If the code offends you, look up at my previous disclaimer: I had other things I needed to be working on, so I didn't spend much time refining this. One thing I'm almost certain could be done better is not monkey patching the dynamic classes with the setUp and tearDown methods. Also, the output at the end of the test execution could definitely use some love. Oh well. Perhaps another day I'll get around to that.

Basically, you just set the servers you need to test and the browsers you want Selenium to run the tests in. Those are at the top of the script: IPS and BROWSERS. Then a new unittest.TestCase class is created for each combination of IP/server+browser. Finally, each of the test cases is thrown into a TestSuite, and the suite is processed. If there were any errors during the tests, they'll be printed out. We weren't really concerned with printing out other information, but you can certainly make other meaningful feedback appear.

Anyway, I thought that someone out there might very well benefit from my little experiment on my co-worker's question. Feel free to comment on your personal adventures with some variation of the code if you find it useful!

I've spent a little more time lately adding new features to django-articles. There are two major additions in the latest release (2.0.0-pre2).

• Article attachments
• Article statuses

That's right folks! You can finally attach files to your articles. This includes attachments to emails that you send, if you have the articles from email feature properly configured. To prove it, I'm going to attach a file to this article (which I'm posting via email).

Next, I've decided that it's worth allowing the user to specify different statuses for their articles. One of the neat things about this feature is that if you are a super user, you're logged in, and you save an article with a status that is designated as "non-live", you will still be able to see it on the site. This is a way for users to preview their work before making it live. Out of the box, there are only two statuses: draft and finished. You're free to add more statuses if you feel so inclined (they're in the database, not hardcoded).

The article status is still separate from the "is_active" flag when saving an article. Any article that is marked as inactive will not appear on the site regardless of the article's "status".

On a slightly less impressive note (although still important), this release includes some basic unit tests. Most of the tests currently revolve around article statuses and making sure that the appropriate articles appear on the site.

I learned something very interesting today regarding JavaScript. Back in the day, I used to put something like this in my HTML when I wanted to include some JS:

<script language="javascript">
...
</script>

Then I learned that I should be using something like this instead:

<script type="text/javascript">
...
</script>

I've been doing that for years and years now. Turns out I've been wrong all this time. Well, at least for 4 years of that time. I stumbled upon RFC4329 today for whatever reason and noticed that it said the text/javascript mimetype is obsolete. I dug into the RFC a bit and found this:

Various unregistered media types have been used in an ad-hoc fashion
to label and exchange programs written in ECMAScript and JavaScript.
These include:

   +-----------------------------------------------------+
   | text/javascript          | text/ecmascript          |
   | text/javascript1.0       | text/javascript1.1       |
   | text/javascript1.2       | text/javascript1.3       |
   | text/javascript1.4       | text/javascript1.5       |
   | text/jscript             | text/livescript          |
   | text/x-javascript        | text/x-ecmascript        |
   | application/x-javascript | application/x-ecmascript |
   | application/javascript   | application/ecmascript   |
   +-----------------------------------------------------+

Use of the "text" top-level type for this kind of content is known to
be problematic.  This document thus defines text/javascript and text/
ecmascript but marks them as "obsolete".  Use of experimental and
unregistered media types, as listed in part above, is discouraged.
The media types,

   * application/javascript
   * application/ecmascript

which are also defined in this document, are intended for common use
and should be used instead.

So yeah. It's time to go update all of my JavaScript stuff I guess. I thought the rest of you who are/were in the same boat as me might like to know about this...

Some of you who prefer to use git for your version control needs and were following the django-articles mirror on GitHub may have noticed some strange activity recently. I noticed today that the GitHub mirror was out of sync with the other mirrors, and I took a bit of time to investigate the problem.

I thought, for some reason, that I might be able to quickly and easily bring it back into sync if I just deleted the repo, recreated it, and pushed my changes to it. That didn't work. This means that all of you who were once following the project there are no longer following it, and I only realized that side effect after I had clicked the delete button. I apologize for this inconvenience.

In the end, it turned out that I had some things misconfigured with git on my box. I have resolved the problems and have brought the mirror back into sync. Please let me know if you run into any problems with it!