InstArch    Posted:


My blog has obviously been quite inactive the past year. I've started a bunch of new projects and worked on some really interesting stuff in that time. I'm going to try to gradually describe the things I've been playing with here.

One project I started in the summer of 2013 is a personal Arch-based LiveCD. My goal in building this LiveCD was purely personal: I wanted to have a LiveCD with my preferred programs and settings just in case I hosed my main system somehow. I want to have minimal downtime, particularly when I need to keep producing for work. That was the idea behind this project. I called it InstArch, as in "instant Arch".

The build scripts for this project are hosted on bitbucket, while the ISOs I build are hosted on sourceforge. InstArch was recently added to Softpedia, and it has received a bit of interest because of that. Again, the idea behind this project was entirely personal--I'm not trying to make a new distribution or community because I'm dissatisfied with Arch or anything like that. There is some erroneous information about InstArch on Softpedia, but I haven't yet written them to ask them to fix it. Soon enough :)

If you're interested in playing with my live CD, feel free to download it and offer suggestions on the issue tracker. I may or may not implement any suggestions :) I've already had one person email me asking about the default username and password for InstArch. If you also find yourself needing this information:

  • username: inst
  • password: arch

You shouldn't need this information unless you try to use sudo or try to switch desktop sessions.

Here's a video of my live CD in action.

Also note that I haven't built/published a new live CD for several months.


Another part of the InstArch project, which I started looong before the actual LiveCD, was to create my own personal Arch repository. It tracks a bunch of packages that I build from the AUR and other personal Arch packages. Anyone is free to use this repo, and it's one that's built into my live CD.

If you wish to use this repository, add my key:

pacman-key -r 051680AC
pacman-key --lsign-key 051680AC

Then add this to your /etc/pacman.conf:

[instarch]
SigLevel = PackageRequired
Server = http://instarch.codekoala.com/$arch/

Comments

Django Projects    Posted:


Over the past 6 years, I've built a lot of things with Django. It has treated me very well, and I have very much enjoyed seeing it progress. I got into Django when I helped the company I was working for transition away from a homegrown PHP framework toward something more reliable and flexible. It was very exciting to learn more about Django at a time when the ecosystem was very young.

When I started with Django, there weren't a lot of pluggable apps to fill the void for things like blogs, event calendars, and other useful utilities for the kinds of sites I was building. That has changed quite a bit since then. The ecosystem has evolved and progressed like mad, and it's wonderful. We have so many choices for simple things to very complex things. It's amazing!

Unfortunately, during this whole time period, my development efforts have shifted from creating my own open source projects to share with the world toward more proprietary solutions for my employers. If it's not obvious to you from my blog activity in recent years, I've become very busy with family life and work. I have very little time to give my open source projects the attention they deserve.

For at least 4 years, I've been telling myself that I'd have/make time to revamp all of my projects. To make them usable with what Django is today instead of what it was when I built the projects. Yeah, this time has never showed up. Take a look at the last time I wrote a blog article!

I have decided to disown pretty much all of my open source Django projects. I've basically done this with one or two of the more popular projects already--let someone else take the reigns while I lurk in the background and occasionally comment on an issue here or there. I truly appreciate those who have taken the initiative here. But there are still plenty of projects that people may find useful that need some attention. I'm putting it up to the community to take these projects over if you find them useful so they can get the love and attention they need.

Here is a list of Django projects that anyone is free to assume responsibility for. Most of them are silly and mostly useless now. Some are unpleasant to look at and could use an entire rewrite.

The fact that I'm giving up these projects does not mean I'm giving up on Django. On the contrary, I'm still using it quite heavily. I'm just doing it in such a way that I can't necessarily post my work for everyone to use. I honestly don't expect much of this disowning effort, since the projects are mostly stale and incompatible with recent versions of Django. But please let me know if you do want to take over one of my projects and care for it.

Comments

Long Time No See    Posted:


Hello again everyone! Soooo much has happened since I last posted on my blog. I figured it was about time to check in and actually be active on my own site again. What follows is just a summary of what has happened in our lives since the beginning of February this year.

Leaving ScienceLogic

First of all, my wife and I decided toward the end of 2011 that it was time for us to move away from Virginia. For reasons that we did not quite understand yet, we both wanted to move to Utah. I applied for my first Utah-based job opportunity just before Christmas 2011. Several of my friends in the Salt Lake City area were kind enough to get me a few interviews here and there, but none of the opportunities were very serious.

Probably about the time I wrote my last blog post, I was contacted by a recruiter in Boise. I would have loved to move back to Idaho, but my wife would have nothing to do with me if I did that. When I shared this information with the recruiter, he said he had a recruiter friend in the SLC area and that he'd pass my information along to him. Within a day, his friend had me set up with a screening problem for a company just outside of SLC.

I was a little hesitant about that particular opportunity, because it was a Ruby on Rails development shop and an advertising company. However, our timeline was getting smaller and smaller--we had to be out of our apartment by the 1st of April--and I didn't see any other serious opportunities on the horizon. So anyway, I completed the programming problem in both Python and Ruby and had a few video chats with some guys with the company. I guess they liked my work, even though I hadn't touched Ruby in several years.

Sometime in the middle of February, the company extended me an offer letter, which my wife and I considered for a few days before accepting. My last day with ScienceLogic was the 30th of February. My first day with the new company was the 12th of March, so we had a couple of weeks to pack everything up and drive across the country. Packing was ridiculously stressful, but the drive was actually quite enjoyable (my wife wouldn't agree). I drove my Mazda 3 with my 2 year old son in the back, and my wife drove the Dodge Grand Caravan with our 7 month old twins.

The New Job

We arrived in Utah on the 10th of March and immediately fell in love with the little town house we're renting and the surrounding community. It's a really nice area. We spent the first couple of days exploring the area and learning our routes to various locations.

My first week on the new job was interesting. They didn't have much for me to do, and we were all scheduled to go to a local tech conference for the last three days of the week. Very appealing way to begin a new job!

As time went on, I did a bit of work here and there, but most of my time on the job was just spent warming a chair in between requests for things to do. Eventually, I just got fed up with the amount of work I was (or, rather, wasn't) doing. By the beginning of May, I was already looking for another job where I could feel useful.

I got in touch with a guy I worked with for a couple of weeks before he quit working for the company that brought us to Utah (I'm intentionally avoiding the use of the company's name). This guy was only able to stand working for that company for about 3 weeks before he quit and went back to his prior company. He referred me for an interview with his managers, and by the middle of May, I had a new job lined up.

The Better New Job

While I was initially hesitant about the job (test automation), I looked at it as a major step up from what I had been doing since March. That and it cut my commute in half. And they provide excellent hardware. Anyway, I started working for StorageCraft Technology Company at the end of May as a Senior Software Engineer in Test.

My task was to build a framework to make the jobs of the manual testers easier. I had no requirements document to refer to, or any specific guidance other than that. I was simply asked to build something that would make lives easier. StorageCraft had recently hired another test automation developer, and the two of us worked together to come up with a design plan for the framework.

We built a lot of neat things into the framework, gave a couple of demos, and it seems like people are really quite pleased with the direction we've gone. I gave a demo of the (Django) UI just the other day, and my supervisors basically gave me the green light to keep building whatever I wanted to. Since the other test automation guy got the boot for being unreliable, I will get to see many of my plans through exactly the way I want! I'm really excited about that.

Enough About Work

Aside from all of the excitement in my career decisions, things are going very well with the family. We live about 3 hours away from my mom, and we've been out there to visit a few times already. It's really fun to see the kids playing with their grandma! The last time we were out for a visit, for my grandmother's 80th birthday, my son and I took my dad's Rhino for a spin. We got stuck, and it was sooo much fun!

Mudding in the Rhino

The twins are growing so well too. They're crawling and getting around very well now. Jane has started to stand up on her own, and she tries to take a step every once in a while. Claire prefers to sit, but she loves to wave, clap, and repeat noises that she hears.

My wife is planning on starting up a new website soon, and she keeps taunting me with the possibility of having me build it for her. Yes, taunting.

Okay, Back to Hobbies

My wife also picked up a Dremel Trio for Dad's Day. To get used to it, I made some little wooden signs with the kids' names on them. Being the quasi-perfectionist that I am, I'm not completely satisfied with how they all turned out. I suppose they'll do for a "first attempt" sort of result though!

First project with the Dremel Trio

I've still got various projects in the works with my Arduino and whatnot. A couple of months ago, I finished a project that helps me see where I'm walking when I go down to my mancave at night. The light switches for the basement are all at the stairs, and my setup is on the opposite side of the basement. I typically prefer to have the lights off when I'm on my computer, and it was annoying and horribly inefficient to turn the lights on when entering the basement, go to my computer, then go back to turn the lights off.

To solve that problem, I re-purposed one of my PIR motion sensors and picked up a LED strip from eBay. I have the motion sensor pointing at the entrance to the basement, and the LED strip strung across the ceiling along the path that I take to get to my desk. When the motion sensor detects movement, it fades the LED strip on, continues to power it for a few seconds, and gradually fades them out when it no longer detects movement. It's all very sexy, if I do say so myself.

Lazy man's light switch

I've tried to capture videos of the setup, but my cameras all have poor light sensors or something, so it's difficult to really show what it's like. The LED strip illuminates the basement perfectly just long enough for me to get to my desk, but the videos just show a faint outline of my body lurking in the dark. :(

One project that is in the works right now is a desk fan that automatically turns on when the ambient temperature reaches a certain level. The fan's speed will vary depending on the temperature, and there will be an LCD screen to allow simple reporting and configuration of thresholds and whatnot. I'm pretty excited about it, but I want to order a few things off of eBay before I go much further with it.

Obviously, much more had happened in the past months, but this post is long enough already. Things are calming down quite a bit now that we're settled in, so I hope to resume activity on my open source projects as well as this blog.

Comments

Arduino-Powered Webcam Mount    Posted:


Earlier this month, I completed yet another journey around the biggest star in our galaxy. Some of my beloved family members thought this would be a good occasion to send me some cash, and I also got a gift card for being plain awesome at work. Even though we really do need a bigger car and whatnot, my wife insisted that I only spend this money on myself and whatever I wanted.

Little did she know the can of worms she just opened up.

I took pretty much all of the money and blew it on stuff for my electronics projects. Up to this point, my projects have all been pretty boring simply because nothing ever moved--it was mostly just lights turning on and off or changing colors. Sure, that's fun, but things really start to get interesting when you actually interact with the physical world. With the birthday money, I was finally able to buy a bunch of servos to begin living out my childhood dream of building robots.

My first project since getting all of my new toys was a motorized webcam mount. My parents bought me a Logitech C910 for my birthday because they were tired of trying to see their grandchildren with the crappy webcam that is built into my laptop. It was a perfect opportunity to use SparkFun's tutorial for some facial tracking (thanks to OpenCV) using their Pan/Tilt Servo Bracket.

It took a little while to get everything setup properly, but SparkFun's tutorial explains perfectly how you can get everything setup if you want to repeat this project.

The problem I had with the SparkFun tutorial, though, is that it basically only gives you a standalone program that does the facial tracking and displays your webcam feed. What good is that? I actually wanted to use this rig to chat with people!! That's when I set out to figure out how to do this.

While the Processing sketch ran absolutely perfect on Windows, it didn't want to work on my Arch Linux system due to some missing dependencies that I didn't know how/care to satisfy. As such, I opted to rewrite the sketch using Python so I could do the facial tracking in Linux.

This is still a work in progress, but here's the current facial tracking program which tells the Arduino where the webcam should be pointing, along with the Arduino sketch.

Now that I could track a face and move my webcam in Linux, I still faced the same problem as before: how can I use my face-tracking, webcam-moving program during a chat with my mom? I had no idea how to accomplish this. I figured I would have to either intercept the webcam feed as it was going to Skype or the Google Talk Plugin, or I'd have to somehow consume the webcam feed and proxy it back out as a V4L2 device that the Google Talk Plugin could then use.

Trying to come up with a way of doing that seemed rather impossible (at least in straight Python), but I eventually stumbled upon a couple little gems.

So the GStreamer tutorial walks you step-by-step through different ways of using a gst-launch utility, and I found this information very useful. I learned that you can use tee to split a webcam feed and do two different things with it. I wondered if it would be possible to split one webcam feed and send it to two other V4L2 devices.

Enter v4l2loopback.

I was able to install this module from Arch's AUR, and using it was super easy (you should be root for this):

modprobe v4l2loopback devices=2

This created two new /dev/video* devices on my system, which happened to be /dev/video4 and /dev/video5 (yeah... been playing with a lot of webcams and whatnot). One device, video4, is for consumption by my face-tracking program. The other, video5, is for VLC, Skype, Google+ Hangouts, etc. After creating those devices, I simply ran the following command as a regular user:

gst-launch-0.10 v4l2src device=/dev/video1 ! \
    'video/x-raw-yuv,width=640,height=480,framerate=30/1' ! \
    tee name=t_vid ! queue ! \
    v4l2sink sync=false device=/dev/video4 t_vid. ! \
    queue ! videorate ! 'video/x-raw-yuv,framerate=30/1' ! \
    v4l2sink device=/dev/video5

There's a whole lot of stuff going on in that command that I honestly do not understand. All I know is that it made it so both my face-tracking Python program AND VLC can consume the same video feed via two different V4L2 devices! A co-worker of mine agreed to have a quick Google+ Hangout with me to test this setup under "real" circumstances (thx man). It worked :D Objective reached!

I had really hoped to find a way to handle this stuff inside Python, but I have to admit that this is a pretty slick setup. A lot of things are still hardcoded, but I do plan on making things a little more generic soon enough.

So here's my little rig (why yes, I did mount it on top of an old Kool-Aid powder thingy lid):

And a video of it in action. Please excuse the subject of the webcam video, I'm not sure where that guy came from or why he's playing with my webcam.

Comments

Lots Of Happenings    Posted:


Yay! First post in nearly 4 months! I feel kinda bad for leaving my April Fools Day joke on the front page for such a long time, but lucky for me my blog isn't very popular! I could have found myself in a world of hurt.

I'd like to give you all a quick update on what's been going on in my life that somewhat justifies a 4-month window of no blog posts. First of all, we had 25% of our backend development team (1 person, leaving 3 developers) get fired earlier in the year, so the workload at my day job got to be a bit heavier.

Second, my wife and I were pregnant with identical twin girls. It was a relatively high-risk pregnancy, so we spent many days in the hospital for checkups and whatnot. The doctors gave her somewhat strict bed rest orders, and I worked from home since the beginning of July (my job is awesome that way) so I could keep and eye on my wife and help with our soon-to-be two year old son.

Third, we had our identical twin girls this past Tuesday. They arrived at 9:25 and 9:27, and their names are Claire and Jane. Claire weighed 5 lb 3 oz, and Jane weighed 4 lb 14 oz. As expected, they lost a bit of weight at the beginning, but they're starting to gain weight again. My wife and the girls are all doing very well.

My son, on the other hand, is starting to realize that his world is changing quite drastically. We're trying to give him as much attention as we can, but it's definitely not the amount that he's used to. He seems to do very well with his new baby sisters, but there have already been several episodes where he just breaks down. It's sad.

I have a few days of leave and vacation that I'll be taking to help get everything settle at home. The girls are still in the "let's sleep all day and night until we're hungry" sort of phase (which is quite awesome), and my wife is up most of the night with them. That means the only person who doesn't sleep a ton during the day is my son, so we get some good father/son time.

Anyway, on to the nerdy stuff. Since I find myself with a couple hours of downtime here and there, I plan to do a bug-smashing ticket-resolving spree. Just as my blog has sat dormant for months, so have many of my side projects. Now is the time to change that!

Comments

Free Professional Web Development    Posted:


I'm pleased to announce my immediate availability for some freelance Web development projects. Lately, I've been quite swamped with a very awesome project at my day job. Many things had to be put on hold as a result of that project. However, it looks like the worst is now behind me, and I find myself with much more free time than before!

As such, I'm opening the doors to any and all (except full-time, because I love my day job) Web development opportunities. I guarantee absolute satisfaction and reliable, efficient Web sites, and I deliver quickly. All you need to do is provide the requirements for your project in the comments section of this article. I'll pump out at the very least a proof of concept that becomes your property (free of charge), no questions asked.

You will not be disappointed. Use this as an opportunity 1) to get free stuff and 2) to test your requirements-gathering/communicating skills. Let the battle begin!

Comments

PIR Motion Sensor + LCD Screen + Arduino Uno    Posted:


For one reason or another, I've recently had the urge to follow in my father's footsteps and start tinkering with electronics. He's basically a wizard. He has fixed a lot of appliances that others tossed out the door without the slightest bit of investigation. I think he did this with the monitor I had hooked up to my very first computer. He just popped open the case, found the problem, soldered some solution in there, and that monitor worked for probably a decade afterwards. Amazing stuff.

Anyway, I have an itch to create a laser trip wire (don't ask). I took a basic electronics class my freshman year in college, so I am already familiar with resistors, capacitors, ICs, breadboards, soldering, etc. It doesn't seem like a very daunting task to create that trip wire, but I'm starting fresh with electronics (it's been a good 8 or 9 years since I last soldered or anything like that).

One of my co-workers mentioned the Arduino as something to get me started on the trip wire, so I started doing some research here and there. The more I read, the more excited I got. I wanted soo badly to buy all of the junk that I'd need to start tinkering with an Arduino, but I didn't think my wife would appreciate that--especially around Christmas time.

Several of my awesome relatives sent me very generous gift cards for Christmas this year, which finally gave me the opportunity to buy a load of stuff for the Arduino without feeling bad. So I ordered loads of stuff. Most of it is here, some of it is still on its way. My Arduino Uno arrived around noon today, and I haven't been able to stop tinkering! It's really a lot of fun, and stupid easy even if you're not a programmer!

I started with the basic "oooh, blinking LEDs!" sort of projects (or "sketches" in Arduino parlance). Then I moved on to tweak those to work with multiple LEDs. There were a few different scenarios I ran though because two of my LEDs weren't lighting up very well. I guess I either hooked them up wrong or I didn't seat them properly... whatever.

The next project was when I hooked up a PIR (passive infrared) motion sensor ($9.99 at RadioShack) and installed a demo sketch that I found on the Arduino wiki. I wasn't quite sure how to wire everything for the PIR sensor, so I took a look at this Make Magazine video to see one way of setting up the circuit. That one simply lights up an LED and makes some noise. I just made mine light up and LED since I don't have a buzzer (yet).

Next I moved on to the LCD display. The package came with a 20x4 green-on-black backlit LCD display, a 2.2k Ohm resistor, and a set of pins to connect the LCD to my breadboard or whatever. I actually soldered the pins onto the LCD display board (wow, was that ever more difficult than I remember!) so I would have less of a hassle getting all of the contacts working. Getting the LCD to work was pretty easy after that.

Then I took those two projects a bit further. I'm certain others have done this years ago, but I didn't use a sketch that was completely written for me this time so I felt special enough to share :) I added the PIR circuit from before less the LED, merged pieces from both demo programs, and came up with a motion sensor that would flash "INTRUDER ALERT!!" on the LCD screen a few times when triggered. Based on my testing, the sensor works extremely well (once calibrated!!), and it will detect motion well across the open area of my apartment (maybe a bit further than 20 feet)!

I'm quite happy with my work, especially for not having "dealt with" electronics for such a long time. When I tried to share my success with some of my friends, they wanted a video to prove I'm awesome (I guess?). So here it is. Trust me, I know the video is horrible--I speak too quietly (baby sleeping in next room), I stumble over my words (that's just me), and I neglected to even offer you some music to rock out to as I blabber about this stuff.

Here's the program I wrote/modified:

  1  /*
  2   * //////////////////////////////////////////////////
  3   * //making sense of the Parallax PIR sensor's output
  4   * //////////////////////////////////////////////////
  5   *
  6   * Switches a LED according to the state of the sensors output pin. Determines
  7   * the beginning and end of continuous motion sequences.
  8   *
  9   * @author: Kristian Gohlke / krigoo (_) gmail (_) com / http://krx.at
 10   * @author: Josh VanderLinden / codekoala (.) gmail (@) com / http://www.codekoala.com
 11   * @date:   3 Jan 2011
 12   *
 13   * kr1 (cleft) 2006
 14   * Released under a creative commons "Attribution-NonCommercial-ShareAlike 2.0" license
 15   * http://creativecommons.org/licenses/by-nc-sa/2.0/de/
 16   *
 17   *
 18   * The Parallax PIR Sensor is an easy to use digital infrared motion sensor module.
 19   * (http://www.parallax.com/detail.asp?product_id=555-28027)
 20   *
 21   * The sensor's output pin goes to HIGH if motion is present. However, even if
 22   * motion is present it goes to LOW from time to time, which might give the
 23   * impression no motion is present. This program deals with this issue by
 24   * ignoring LOW-phases shorter than a given time, assuming continuous motion is
 25   * present during these phases.
 26   *
 27   */
 28 
 29  #include <LiquidCrystal.h>
 30 
 31  int calibrationTime = 10;   // seconds to calibrate PIR
 32  long unsigned int pause = 5000; // timeout before we "all" motion has ceased
 33  long unsigned int lowIn; // the time when the sensor outputs a low impulse
 34 
 35  boolean lockLow = true;
 36  boolean takeLowTime;
 37 
 38  int flashCnt = 4;  // number of times the LCD will flash when there's motion
 39  int flashDelay = 500; // number of ms to wait while flashing LCD
 40  int pirPin = 7;    // the digital pin connected to the PIR sensor's output
 41  int lcdPin = 13;   // pin connected to LCD
 42  LiquidCrystal lcd(12, 11, 10, 5, 4, 3, 2);
 43 
 44  void setup() {
 45      // Calibrates the PIR
 46 
 47      Serial.begin(9600);
 48      pinMode(pirPin, INPUT);
 49      pinMode(lcdPin, OUTPUT);
 50 
 51      digitalWrite(pirPin, LOW);
 52 
 53      clearLcd();
 54 
 55      // give the sensor some time to calibrate
 56      lcd.setCursor(0,0);
 57      lcd.print("Calibrating...");
 58 
 59      for(int i = 0; i < calibrationTime; i++){
 60          lcd.print(".");
 61          delay(1000);
 62      }
 63 
 64      lcd.print("done");
 65      delay(50);
 66  }
 67 
 68  void clearLcd() {
 69      // Clears the LCD, turns off the backlight
 70      lcd.begin(20, 4);
 71      lcd.clear();
 72      digitalWrite(lcdPin, LOW);
 73  }
 74 
 75  void alertLcd() {
 76      // Turns on the LCD backlight and notifies user of motion
 77      lcd.setCursor(2, 1);
 78      digitalWrite(lcdPin, HIGH);
 79      lcd.print("INTRUDER ALERT!!");
 80      lcd.setCursor(2, 2);
 81      lcd.print("================");
 82  }
 83 
 84  void loop() {
 85      // Main execution loop
 86 
 87      if(digitalRead(pirPin) == HIGH) {
 88          // flash an alert a few times
 89          for (int c = 0; c < flashCnt; c++) {
 90              alertLcd();
 91              delay(flashDelay);
 92              lcd.clear();
 93              delay(flashDelay);
 94          }
 95 
 96          if(lockLow) {
 97              // makes sure we wait for a transition to LOW before any further
 98              // output is made:
 99              lockLow = false;
100              delay(50);
101          }
102          takeLowTime = true;
103      }
104 
105      if (digitalRead(pirPin) == LOW) {
106          clearLcd();
107 
108          if (takeLowTime) {
109              // save the time of the transition from high to LOW
110              // make sure this is only done at the start of a LOW phase
111              lowIn = millis();
112              takeLowTime = false;
113          }
114 
115          // if the sensor is low for more than the given pause,
116          // we assume that no more motion is going to happen
117          if (!lockLow && millis() - lowIn > pause) {
118              // makes sure this block of code is only executed again after
119              // a new motion sequence has been detected
120              lockLow = true;
121              delay(50);
122          }
123      }
124  }

Comments

Django-Tracking 0.3.5    Posted:


I've finally gotten around to looking at a bunch of tickets that had been opened for django-tracking in the past year and a half or so. I feel horrible that it's really taken that long for me to get to them! Every time I got a ticket notification, I told myself, "Okay, I'll work on that this weekend." Many have weekends have passed without any work on any of my projects. I'm going to get better about that!

Anyway, several fixes have gone into the latest version of django-tracking. Some have to do with unicode problems (thanks ramusus!). Others have to do with overall performance, while yet others have to do with overall stability.

The first interesting change in this release is that django-tracking no longer relies on the GeoIP Python API. Instead it's now using django.contrib.gis.utils.GeoIP. I had hoped that this would remove the dependency on the GeoIP C API, but it appears that I was mistaken. Oh well.

Perhaps the biggest improvement in this new release is the use of caching. With caching in place, the middleware classes don't slam the database nearly as badly as they used to. There's still more that could be done with caching to improve performance, but I think what I've got now will be a big help.

Another noteworthy change, in my opinion, is the use of logging. I've sprinkled mildly useful logging messages throughout the code so you can learn when something bad happens that is silently handled. I hope that this will help me improve the quality of the code as it will allow anyone who uses the project (and pays attention to the log messages, of course) to tell me when bad things are happening.

Finally, the packaging code has been updated to be much more simple. Version 0.3.5 has been uploaded to PyPI and is available via pip or easy_install. If you prefer to have the latest copy of the code, the official code repositories are (in order of my personal preference):

I can't wait for your feedback!

Comments

2Ze.us Updates    Posted:


There has been quite a bit of recent activity in my 2ze.us project since I first released it nearly a year ago. My intent was not to become a competitor with bit.ly, is.gd, or anyone else in the URL-shortening arena. I created the site as a way for me to learn more about Google's AppEngine. It didn't take very long to get it up and running, and it seemed to work fairly well.

AppEngine and Extensions

I was able to basically leave the site alone on AppEngine for several months--through about September 2009. In that time, I came up with a Firefox extension to make its use more convenient.

The extension allows you to quickly get a shortened URL for the page you're currently looking at, and a couple of context menu items let you get a short URL for things like specific images on a page. Also included in the extension is a preview for 2ze.us links. The preview can tell you the title and domain of the link's target. It can tell you how much smaller the 2ze.us URL is compared to the full URL. Finally, it displays how many times that particular 2ze.us link has been clicked.

That as all fine and dandy. It was the second Firefox extension I had ever written, and it's still running strong. In June or July of 2009, I started working on a little program to make it easier for me to interact with Twitter the way I wanted to. This was a great opportunity for me to incorporate 2ze.us into the application so any URL I wanted to post to Twitter would automatically be shortened for me, using my own shortener.

Porting to WebFaction And PHP

Anyway, around the end of September 2009, I noticed that there were a lot of problems with 2ze.us. It was slow and sometimes completely unresponsive. Certain URLs would redirect to their full URLs, while others wouldn't. The Firefox extension stopped working nicely. Oh yeah, and AppEngine rolled back to a previous revision of the code without me telling it to. That's when everything just died. It didn't take long for me to decide to migrate my project from AppEngine onto my awesome WebFaction hosting.

At this point, I was faced with a small dilemma: keep the code in Python, or port it to PHP. I opted to port it over to PHP, because I didn't want all of the overhead of a full Django instance for a site that needed to be very zippy. And I was unacquainted with other Python options.

By early October 2009, I had managed to turn the project into a PHP beast, running on Apache. It was a lot more responsive than AppEngine ever let 2ze.us be. There were a few bumps along the road, what with the extension and Twitter client relying on various parts of the site. Eventually it got to a point where I could just let it sit and work.

Chromium Extension

Sometime around the end of December, I decided to write another extension for 2ze.us, only for Google Chrome and Chromium this time. This extension isn't quite as feature-packed as its Firefox brother, but it gets the job done.

Clip2Zeus

Shortly after "completing" the Chromium extension, I had what seemed like a pretty original idea. Who knows if it really is, but I still haven't seen another tool quite like the one that I made as a result of this idea. I thought, "Now, why should I need to install an extension in each Web browser I use on each computer I use? Is there a better way?"

The answer came quickly: a standalone, desktop application. Write one program that handles shortening URLs for you. My laziness told me to make a program that monitors your system clipboard for URLs. If a URL is detected, try to shorten it, and update the clipboard contents in place. Boom. Done. All extensions become useless beyond things like the URL preview (which is very useful, imo).

The next question I asked was, "Do I make it platform-dependent? Should I stick it to the majority of computer users and write my tool for Linux only? For OSX only? For, uh... Windows only?" Again, an easy question to answer. Support them all or don't even bother writing the application.

A week's worth of midnight hacking saw the birth of Clip2Zeus 1.0a. It's a cross-platform compatible desktop application that does exactly what I just mentioned. When it's running and detects a URL on your system clipboard, it will try to shorten it and update it in your clipboard. If you copy a block of text, the application will only modify the URLs in that block of text--meaning the block of text will still be in your clipboard, but it will have shorter URLs.

I use the program every day at work (on OSX). It's been very fun for me to see a short URL any time I copy a nasty URL to my clipboard. Imagine that; I'm a big fan of my own work...

Tornado

Lately, I've noticed that the site was getting kind of slow again. Sometimes it would take several seconds for Clip2Zeus to shorten URLs in my clipboard, when it was normally instantaneous. Every once in a while, Clip2Zeus would completely fail to connect to the website.

One of my friends has asked me a lot of questions about the Tornado framework in the past months. I had read a few things about Tornado when it was open-sourced last year, but I didn't really feel the need to dabble with it. These questions prompted me to tinker a little.

Last night I re-ported 2ze.us to Python, using the Tornado framework this time. So far I'm very impressed with its responsiveness. The framework offers a lot of neat little utilities, and it is very fast (as reported by dozens of other reputable sources).

On top of the speed increase that came with the transition to Tornado, my RAM usage on WebFaction has come down by nearly 100MB. Just by turning off the one Apache-backed website. Now I'm nowhere near my RAM cap! Wahoo!!

Enough rambling. Like I said at the beginning of this article, a lot has been happening with this project in the past year. I didn't even think about all of the time I put into projects related to my simple little side project. Looking back, I'm quite satisfied with how things have unfolded.

Statistics

Here are some simple statistics for 2ze.us. Since March 2009...

  • 5,252 URLs have been shortened using 2ze.us
  • 2ze.us links have been clicked 198,267 times
  • 315,951 URL characters have been turned into 11,532 characters

In April 2009...

  • 217 URLs were shortened
  • 2ze.us links were clicked 617 times

In February 2010...

  • 1,182 URLs were shortened
  • 2ze.us links were clicked 32,830 times

Not too shabby for a side project.

Comments

On Security and Python's Exec    Posted:


A recent project at work has renewed my aversion to Python's exec statement--particularly when you want to use it with arbitrary, untrusted code. The project requirements necessitated the use of exec, so I got to do some interesting experiments with it. I've got a few friends who, until I slapped some sense into them, were seemingly big fans of exec (in Django projects, even...). This article is for them and others in the same boat.

Take this example:

#!/usr/bin/env python

import sys

dirname = '/usr/lib/python2.6/site-packages'

print dirname, 'in path?', (dirname in sys.path)

exec """import sys

dirname = '/usr/lib/python2.6/site-packages'
print 'In exec path?', (dirname in sys.path)

sys.path.remove(dirname)

print 'In exec path?', (dirname in sys.path)"""

print dirname, 'in path?', (dirname in sys.path)

Take a second and examine what the script is doing. Done? Great... So, the script first makes sure that a very critical directory is in my PYTHONPATH: /usr/lib/python2.6/site-packages. This is the directory where all of the awesome Python packages, like PIL, lxml, and dozens of others, reside. This is where Python will look for such packages when I try to import and use them in my programs.

Next, a little Python snippet is executed using exec. Let's say this snippet comes from an untrusted source (a visitor to your website, for example). The snippet removes that very important directory from my PYTHONPATH. It might seem like it's relatively safe to do within an exec--maybe it doesn't change the PYTHONPATH that I was using before the exec?

Wrong. The output of this script on my personal system says it all:

$ python bad.py
/usr/lib/python2.6/site-packages in path? True
In exec path? True
In exec path? False
/usr/lib/python2.6/site-packages in path? False

From this example, we learn that Python code that is executed using exec runs in the same context as the code that uses exec. This is a critical concept to learn.

Some people might say, "Oh, there's an easy way around that. Give exec its own globals dictionary to work with, and all will be well." Wrong again. Here's a modified version of the above script.

#!/usr/bin/env python

import sys

dirname = '/usr/lib/python2.6/site-packages'

print dirname, 'in path?', (dirname in sys.path)

context = {'something': 'This is a special context for the exec'}

exec """import sys

print something
dirname = '/usr/lib/python2.6/site-packages'
print 'In exec path?', (dirname in sys.path)

sys.path.remove(dirname)

print 'In exec path?', (dirname in sys.path)""" in context

print dirname, 'in path?', (dirname in sys.path)

And here's the output:

$ python also_bad.py
/usr/lib/python2.6/site-packages in path? True
This is a special context for the exec
In exec path? True
In exec path? False
/usr/lib/python2.6/site-packages in path? False

How can you get around this glaring risk in the exec statement? One possible solution is to execute the snippet in its own process. Might not be the best way to handle things. Could be the absolute worst solution. But it's a solution, and it works:

#!/usr/bin/env python

import multiprocessing
import sys

def execute_snippet(snippet):
    exec snippet

dirname = '/usr/lib/python2.6/site-packages'

print dirname, 'in path?', (dirname in sys.path)

snippet = """import sys

dirname = '/usr/lib/python2.6/site-packages'
print 'In exec path?', (dirname in sys.path)

sys.path.remove(dirname)

print 'In exec path?', (dirname in sys.path)"""

proc = multiprocessing.Process(target=execute_snippet, args=(snippet,))
proc.start()
proc.join()

print dirname, 'in path?', (dirname in sys.path)

And here comes the output:

$ python better.py
/usr/lib/python2.6/site-packages in path? True
In exec path? True
In exec path? False
/usr/lib/python2.6/site-packages in path? True

So the PYTHONPATH is only affected by the sys.path.remove within the process that executes the snippet using exec. The process that spawns the subprocess is unaffected, and can continue with life, happily importing all of those wonderful packages from the site-packages directory. Yay.

With that said, exec isn't always bad. But my personal point of view is basically, "There is probably a better way." Unfortunately for me, that does not hold up in my current situation, and it might not work for your circumstances too. If no one is forcing you to use exec, you might investigate alternatives in all of that free time you've been wondering what to do with.

Comments