My blog has obviously been quite inactive the past year. I've started a bunch
of new projects and worked on some really interesting stuff in that time. I'm
going to try to gradually describe the things I've been playing with here.
One project I started in the summer of 2013 is a personal Arch-based LiveCD. My
goal in building this LiveCD was purely personal: I wanted to have a LiveCD
with my preferred programs and settings just in case I hosed my main system
somehow. I want to have minimal downtime, particularly when I need to keep
producing for work. That was the idea behind this project. I called it
InstArch, as in "instant Arch".
The build scripts for this project are hosted on bitbucket, while the ISOs I
build are hosted on sourceforge. InstArch was recently added to Softpedia,
and it has received a bit of interest because of that. Again, the idea behind
this project was entirely personal--I'm not trying to make a new distribution
or community because I'm dissatisfied with Arch or anything like that. There is
some erroneous information about InstArch on Softpedia, but I haven't yet
written them to ask them to fix it. Soon enough :)
If you're interested in playing with my live CD, feel free to download it and
offer suggestions on the issue tracker. I may or may not implement any
suggestions :) I've already had one person email me asking about the default
username and password for InstArch. If you also find yourself needing this
information:
username: inst
password: arch
You shouldn't need this information unless you try to use sudo or try to
switch desktop sessions.
Here's a video of my live CD in action.
Also note that I haven't built/published a new live CD for several months.
Another part of the InstArch project, which I started looong before the actual
LiveCD, was to create my own personal Arch repository. It tracks a bunch of
packages that I build from the AUR and other personal Arch packages. Anyone is
free to use this repo, and it's one that's built into my live CD.
Earlier this year, I was approached by the editor of Software Developer's
Journal to write a Python-related article. I was quite flattered by the
opportunity, but, being extremely busy at the time with work and family life, I
was hesitant to agree. However, after much discussion with my wife and other
important people in my life, I decided to go for it.
I had a lot of freedom to choose a topic to write about in the article, along
with a relatively short timeline. I think I had two weeks to write the article
after finally agreeing to do so, and I was supposed to write some 7-10 pages
about my chosen topic.
Having recently been converted to the wonders of test-driven development (TDD),
I decided that should be my topic. Several of my friends were also interested
in getting into TDD, and they were looking for a good, simple way to get their
feet wet. I figured the article would be as good a time as any to write up
something to help my friends along.
I set out with a pretty grand plan for the article, but as the article
progressed, it became obvious that my plan was a bit too grandios for a regular
magazine article. I scaled back my plans a bit and continued working on the
article. I had to scale back again, and I think one more time before I finally
had something that was simple enough to not write a book about.
Well, that didn't exactly turn out as planned either. I ended up writing nearly
40 pages of LibreOffice single-spaced, 12pt Times New Roman worth of TDD stuff.
Granted, a fair portion of the article's length is comprised of code snippets
and command output.
Anyway, I have permission to repost the article here, and I wanted to do so
because I feel that the magazine formatting kinda butchered the formatting I
had in mind for my article (and understandably so). To help keep the formatting
more pristine, I've turned it into a PDF for anyone who's interested in reading
it.
So, without much further ado, here's the article! Feel free to download or
print the PDF as well.
For anyone who still might be visiting my site with any regularity, you might
have noticed some changes around here. For the past several years, I've been
blogging on a Django-based system that I wrote a very long time ago. I wrote it
because, at the time, there weren't many Django-based blogging platforms, and
certainly none of the few were quite as robust as I thought I wanted.
I set out to build my own blogging platform, and I think it worked out fairly
well. As with all things, however, it became obsolete as the ecosystem around
it flourished. I simply didn't have the time to continue maintaining it as I
should have. That's also part of the reason for my lack of activity here these
past couple of years.
Anyway, in an effort to keep this blog alive, I've switched to a much more
simple blogging system known as nikola. It's not
your run of the mill Wordpress clone. No, it's much more simple than that, but
it doesn't sacrifice much of what I had with django-articles. I still get to
write my posts using a format that I enjoy (restructuredtext). I get to write
my posts in an editor that I enjoy (vim). I get to keep my posts in a
"database" that I enjoy (git). I get to deploy using an interface that I enjoy
(the command line). And I don't have to try to keep up with what is happening
in the blogging ecosystem--there are plenty of other people handling that with
nikola for me!
So, you can expect more posts in the coming year. Call it a new year's
resolution.
Wow, it's been a very long time since I've posted on my blog. It's amazing how
much time twins take up! Now they're almost 6 months old, and things are
starting to calm down a bit. I will try to make time to keep my blog updated
more regularly now.
Earlier today, a good friend asked me how I would handle displaying a progress
bar on the command line in a Python program. I suggested a couple of things,
including leaving his already working program alone for the most part and just
having a separate thread display the progress bar. I'm not sure exactly why,
but he decided to not use a separate thread. It didn't seem like it would be
very difficult, so I spent a few minutes mocking up a quick little demo.
This is by no means the most perfect of solutions, but I think it might be
useful for others who find themselves in similar circumstances. Please feel
free to comment with your suggestions! This is literally my first stab at
something like this, and I only spent about 5 minutes hacking it up.
Earlier this month, I completed yet another journey around the biggest star in
our galaxy. Some of my beloved family members thought this would be a good
occasion to send me some cash, and I also got a gift card for being plain
awesome at work. Even though we really do need a bigger car and whatnot, my
wife insisted that I only spend this money on myself and whatever I wanted.
Little did she know the can of worms she just opened up.
I took pretty much all of the money and blew it on stuff for my electronics
projects. Up to this point, my projects have all been pretty boring simply
because nothing ever moved--it was mostly just lights turning on and off or
changing colors. Sure, that's fun, but things really start to get interesting
when you actually interact with the physical world. With the birthday money,
I was finally able to buy a bunch of servos to begin living out my childhood
dream of building robots.
My first project since getting all of my new toys was a motorized webcam mount.
My parents bought me a Logitech C910 for my birthday because they were tired
of trying to see their grandchildren with the crappy webcam that is built into
my laptop. It was a perfect opportunity to use SparkFun's tutorial for some
facial tracking (thanks to OpenCV) using their Pan/Tilt Servo Bracket.
It took a little while to get everything setup properly, but SparkFun's
tutorial explains perfectly how you can get everything setup if you want to
repeat this project.
The problem I had with the SparkFun tutorial, though, is that it basically only
gives you a standalone program that does the facial tracking and displays your
webcam feed. What good is that? I actually wanted to use this rig to chat
with people!! That's when I set out to figure out how to do this.
While the Processing sketch ran absolutely perfect on Windows, it didn't want
to work on my Arch Linux system due to some missing dependencies that I didn't
know how/care to satisfy. As such, I opted to rewrite the sketch using Python
so I could do the facial tracking in Linux.
This is still a work in progress, but here's the current facial tracking
program which tells the Arduino where the webcam should be pointing, along with
the Arduino sketch.
Now that I could track a face and move my webcam in Linux, I still faced the
same problem as before: how can I use my face-tracking, webcam-moving program
during a chat with my mom? I had no idea how to accomplish this. I figured I
would have to either intercept the webcam feed as it was going to Skype or the
Google Talk Plugin, or I'd have to somehow consume the webcam feed and proxy it
back out as a V4L2 device that the Google Talk Plugin could then use.
Trying to come up with a way of doing that seemed rather impossible (at least
in straight Python), but I eventually stumbled upon a couple little gems.
So the GStreamer tutorial walks you step-by-step through different ways of
using a gst-launch utility, and I found this information very useful. I
learned that you can use tee to split a webcam feed and do two different
things with it. I wondered if it would be possible to split one webcam feed
and send it to two other V4L2 devices.
Enter v4l2loopback.
I was able to install this module from Arch's AUR, and using it was super
easy (you should be root for this):
modprobe v4l2loopback devices=2
This created two new /dev/video* devices on my system, which happened to be
/dev/video4 and /dev/video5 (yeah... been playing with a lot of webcams
and whatnot). One device, video4, is for consumption by my face-tracking
program. The other, video5, is for VLC, Skype, Google+ Hangouts, etc.
After creating those devices, I simply ran the following command as a regular user:
There's a whole lot of stuff going on in that command that I honestly do not
understand. All I know is that it made it so both my face-tracking Python
program AND VLC can consume the same video feed via two different V4L2 devices!
A co-worker of mine agreed to have a quick Google+ Hangout with me to test this
setup under "real" circumstances (thx man). It worked :D Objective
reached!
I had really hoped to find a way to handle this stuff inside Python, but I have
to admit that this is a pretty slick setup. A lot of things are still
hardcoded, but I do plan on making things a little more generic soon enough.
So here's my little rig (why yes, I did mount it on top of an old Kool-Aid
powder thingy lid):
And a video of it in action. Please excuse the subject of the webcam video,
I'm not sure where that guy came from or why he's playing with my webcam.
I'm pleased to announce my immediate availability for some freelance Web
development projects. Lately, I've been quite swamped with a very awesome
project at my day job. Many things had to be put on hold as a result of that
project. However, it looks like the worst is now behind me, and I find myself
with much more free time than before!
As such, I'm opening the doors to any and all (except full-time, because I love
my day job) Web development opportunities. I guarantee absolute
satisfaction and reliable, efficient Web sites, and I deliver quickly. All you
need to do is provide the requirements for your project in the comments section
of this article. I'll pump out at the very least a proof of concept that
becomes your property (free of charge), no questions asked.
You will not be disappointed. Use this as an opportunity 1) to get free stuff and 2) to test your requirements-gathering/communicating skills. Let the battle begin!
The activity on my latest blog post has been tremendous! I never expected that
much activity within an hour or two of posting the article.
The aim of this article is to provide an alternative solution to my weighted
sort when you're after increased performance. It might just be useful to those
who came here in search of a way to do weighted sorting in Python. I need to
give a shout out to Jeremy Brown for suggesting this solution. He's so awesome
:P
While the example I posted in my previous article addressed my needs just fine,
it is definitely not the fastest option. A better solution would be to
completely remove the special IDs from the object list altogether and just
place them in front of the list:
importitertoolsimportrandomobject_ids=[random.randint(0,100)foriinrange(20)]special_ids=[random.choice(object_ids)foriinrange(5)]not_special_ids=(iforiinobject_idsifinotinspecial_ids)foriinitertools.chain(special_ids,not_special_ids):# do stuff with each IDpass
This solution is quite different from my weighted sort, as there's no sorting
going on at all, just a simple generator and using itertools to chain two
collections together.
Here's a way you can benchmark see which solution is faster:
$ python weighted_sort.py
24000003 function calls in 18.411 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 18.411 18.411 <string>:1(<module>)
1000000 0.580 0.000 0.580 0.000 copy.py:112(_copy_with_constructor)
1000000 0.791 0.000 1.510 0.000 copy.py:65(copy)
1 1.397 1.397 18.411 18.411 weighted_sort.py:11(sorting)
20000000 8.907 0.000 8.907 0.000 weighted_sort.py:14(<lambda>)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1000000 0.139 0.000 0.139 0.000 {method 'get' of 'dict' objects}
1000000 6.597 0.000 15.503 0.000 {method 'sort' of 'list' objects}
16000003 function calls in 7.381 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 7.381 7.381 <string>:1(<module>)
1 2.744 2.744 7.381 7.381 weighted_sort.py:18(chaining)
16000000 4.636 0.000 4.636 0.000 weighted_sort.py:20(<genexpr>)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
So, you can see that the chaining solution is easily twice as fast as the
sorting solution over 1 million iterations. Both of these solutions work
perfectly well for my purposes, and I will probably end up switching to the
chaining solution sometime in the future.
EDIT After reading lqc's comment on my previous article, I've decided to
update this one with more appropriate benchmarks. The information that lqc has
shared makes the speed of these solutions much closer.
Here's my updated test script:
$ python weighted_sort.py
4000003 function calls in 8.437 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 8.437 8.437 <string>:1(<module>)
1000000 0.558 0.000 0.558 0.000 copy.py:112(_copy_with_constructor)
1000000 0.741 0.000 1.431 0.000 copy.py:65(copy)
1 1.319 1.319 8.437 8.437 weighted_sort.py:11(sorting)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1000000 0.133 0.000 0.133 0.000 {method 'get' of 'dict' objects}
1000000 5.688 0.000 5.688 0.000 {method 'sort' of 'list' objects}
17000003 function calls in 7.545 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 7.545 7.545 <string>:1(<module>)
1 2.818 2.818 7.545 7.545 weighted_sort.py:18(chaining)
17000000 4.726 0.000 4.726 0.000 weighted_sort.py:20(<genexpr>)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
So if you only gain a second over 1 million iterations, I think I prefer the
sort(key=special_ids.__contains__) solution! I hope these two articles will
help you get started on your adventures with handling special objects before
others!
Last night I found myself in need of a simple weighted sort function in Python.
I had a list of integers which represented object IDs in my project. Some of
the objects needed to be processed before the others while iterating over the
list of integers, and I already knew which object IDs those were. The order
the rest of the object IDs were processed didn't matter at all. I just wanted
the special object IDs to arrive at the beginning of the list, and the
remaining object IDs could be in any order.
I was surprised at how simple it was to produce such a weighted sort. Here's
an example of what I did:
Notice that each of the "special" IDs have shifted from their original position
in the object_ids list to be at the beginning of the list after the sort.
The Python documentation for sort says that the key argument "specifies
a function of one argument that is used to extract a comparison key from each
list element." I'm using it to check to see if a given element in the list is
in my special_ids list. If the element is present in the special_ids
list, it will be shifted to the left because of the way the
special_ids.__contains__ works.
In sorting, a value of 1 (or other positive integer) out of a comparison
function generally means "this belongs to the right of the other element." A
value of -1 (or other negative integer) means "this belongs to the left of
the other element." A value of 0 means "these two elements are equal" (for
the purposes of sorting). I'm assuming it works similarly with the key
argument. Please correct me if I'm wrong!
As lqc states in the comments below, the key argument works differently.
It creates a new sequence of values which is then sorted. Before lqc jumped
in, I was using key=int(i in special_ids) * -2 + 1 to do the sorting, which
is pretty dumb. Using key=special_ids.__contains__ is much more
appropriate. Thanks lqc!!
This sort of weighted sort might not be just right for your needs, but
hopefully it will give you a place to start to build your customized weighted
sort!
For one reason or another, I've recently had the urge to follow in my father's
footsteps and start tinkering with electronics. He's basically a wizard. He
has fixed a lot of appliances that others tossed out the door without the
slightest bit of investigation. I think he did this with the monitor I had
hooked up to my very first computer. He just popped open the case, found the
problem, soldered some solution in there, and that monitor worked for probably
a decade afterwards. Amazing stuff.
Anyway, I have an itch to create a laser trip wire (don't ask). I took a basic
electronics class my freshman year in college, so I am already familiar with
resistors, capacitors, ICs, breadboards, soldering, etc. It doesn't seem like
a very daunting task to create that trip wire, but I'm starting fresh with
electronics (it's been a good 8 or 9 years since I last soldered or anything
like that).
One of my co-workers mentioned the Arduino as something to get me started on
the trip wire, so I started doing some research here and there. The more I
read, the more excited I got. I wanted soo badly to buy all of the junk that
I'd need to start tinkering with an Arduino, but I didn't think my wife would
appreciate that--especially around Christmas time.
Several of my awesome relatives sent me very generous gift cards for Christmas
this year, which finally gave me the opportunity to buy a load of stuff for the
Arduino without feeling bad. So I ordered loads of stuff. Most of it is here,
some of it is still on its way. My Arduino Uno arrived around noon today, and
I haven't been able to stop tinkering! It's really a lot of fun, and stupid
easy even if you're not a programmer!
I started with the basic "oooh, blinking LEDs!" sort of projects (or "sketches"
in Arduino parlance). Then I moved on to tweak those to work with multiple
LEDs. There were a few different scenarios I ran though because two of my LEDs
weren't lighting up very well. I guess I either hooked them up wrong or I
didn't seat them properly... whatever.
The next project was when I hooked up a PIR (passive infrared) motion sensor
($9.99 at RadioShack) and installed a demo sketch that I found on the
Arduino wiki. I wasn't quite sure how to wire everything for the PIR sensor,
so I took a look at this Make Magazine video to see one way of setting up
the circuit. That one simply lights up an LED and makes some noise. I just
made mine light up and LED since I don't have a buzzer (yet).
Next I moved on to the LCD. The package came with a 20x4 green-on-black
backlit LCD display, a 2.2k Ohm resistor, and a set of pins to connect the LCD
to my breadboard or whatever. I actually soldered the pins onto the LCD
display board (wow, was that even more difficult than I remember!) so I would
have less of a hassle getting all of the contacts working. Getting the LCD to
work was pretty easy after that.
Then I took those two projects a bit further. I'm certain others have done
this years ago, but I didn't use a sketch that was completely written for me
this time so I felt special enough to share :) I added the PIR circuit from
before less the LED, merged pieces from both demo programs, and came up with a
motion sensor that would flash "INTRUDER ALERT!!" on the LCD screen a few times
when triggered. Based on my testing, the sensor works extremely well (once
calibrated!!), and it will detect motion well across the open area of my
apartment (maybe a bit further than 20 feet)!
I'm quite happy with my work, especially for not having "dealt with"
electronics for such a long time. When I tried to share my success with some
of my friends, they wanted a video to prove I'm awesome (I guess?). So here it
is. Trust me, I know the video is horrible--I speak too quietly (baby sleeping
in next room), I stumble over my words (that's just me), and I neglected to
even offer you some music to rock out to as I blabber about this stuff.
/* * ////////////////////////////////////////////////// * //making sense of the Parallax PIR sensor's output * ////////////////////////////////////////////////// * * Switches a LED according to the state of the sensors output pin. Determines * the beginning and end of continuous motion sequences. * * @author: Kristian Gohlke / krigoo (_) gmail (_) com / http://krx.at * @author: Josh VanderLinden / codekoala (.) gmail (@) com / http://www.codekoala.com * @date: 3 Jan 2011 * * kr1 (cleft) 2006 * Released under a creative commons "Attribution-NonCommercial-ShareAlike 2.0" license * http://creativecommons.org/licenses/by-nc-sa/2.0/de/ * * * The Parallax PIR Sensor is an easy to use digital infrared motion sensor module. * (http://www.parallax.com/detail.asp?product_id=555-28027) * * The sensor's output pin goes to HIGH if motion is present. However, even if * motion is present it goes to LOW from time to time, which might give the * impression no motion is present. This program deals with this issue by * ignoring LOW-phases shorter than a given time, assuming continuous motion is * present during these phases. * */#include <LiquidCrystal.h>intcalibrationTime=10;// seconds to calibrate PIRlongunsignedintpause=5000;// timeout before we "all" motion has ceasedlongunsignedintlowIn;// the time when the sensor outputs a low impulsebooleanlockLow=true;booleantakeLowTime;intflashCnt=4;// number of times the LCD will flash when there's motionintflashDelay=500;// number of ms to wait while flashing LCDintpirPin=7;// the digital pin connected to the PIR sensor's outputintlcdPin=13;// pin connected to LCDLiquidCrystallcd(12,11,10,5,4,3,2);voidsetup(){// Calibrates the PIRSerial.begin(9600);pinMode(pirPin,INPUT);pinMode(lcdPin,OUTPUT);digitalWrite(pirPin,LOW);clearLcd();// give the sensor some time to calibratelcd.setCursor(0,0);lcd.print("Calibrating...");for(inti=0;i<calibrationTime;i++){lcd.print(".");delay(1000);}lcd.print("done");delay(50);}voidclearLcd(){// Clears the LCD, turns off the backlightlcd.begin(20,4);lcd.clear();digitalWrite(lcdPin,LOW);}voidalertLcd(){// Turns on the LCD backlight and notifies user of motionlcd.setCursor(2,1);digitalWrite(lcdPin,HIGH);lcd.print("INTRUDER ALERT!!");lcd.setCursor(2,2);lcd.print("================");}voidloop(){// Main execution loopif(digitalRead(pirPin)==HIGH){// flash an alert a few timesfor(intc=0;c<flashCnt;c++){alertLcd();delay(flashDelay);lcd.clear();delay(flashDelay);}if(lockLow){// makes sure we wait for a transition to LOW before any further// output is made:lockLow=false;delay(50);}takeLowTime=true;}if(digitalRead(pirPin)==LOW){clearLcd();if(takeLowTime){// save the time of the transition from high to LOW// make sure this is only done at the start of a LOW phaselowIn=millis();takeLowTime=false;}// if the sensor is low for more than the given pause,// we assume that no more motion is going to happenif(!lockLow&&millis()-lowIn>pause){// makes sure this block of code is only executed again after// a new motion sequence has been detectedlockLow=true;delay(50);}}}
I've finally gotten around to looking at a bunch of tickets that had been opened for django-tracking in the past year and a half or so. I feel horrible that it's really taken that long for me to get to them! Every time I got a ticket notification, I told myself, "Okay, I'll work on that this weekend." Many have weekends have passed without any work on any of my projects. I'm going to get better about that!
Anyway, several fixes have gone into the latest version of django-tracking. Some have to do with unicode problems (thanks ramusus!). Others have to do with overall performance, while yet others have to do with overall stability.
The first interesting change in this release is that django-tracking no longer relies on the GeoIP Python API. Instead it's now using django.contrib.gis.utils.GeoIP. I had hoped that this would remove the dependency on the GeoIP C API, but it appears that I was mistaken. Oh well.
Perhaps the biggest improvement in this new release is the use of caching. With caching in place, the middleware classes don't slam the database nearly as badly as they used to. There's still more that could be done with caching to improve performance, but I think what I've got now will be a big help.
Another noteworthy change, in my opinion, is the use of logging. I've sprinkled mildly useful logging messages throughout the code so you can learn when something bad happens that is silently handled. I hope that this will help me improve the quality of the code as it will allow anyone who uses the project (and pays attention to the log messages, of course) to tell me when bad things are happening.
Finally, the packaging code has been updated to be much more simple. Version 0.3.5 has been uploaded to PyPI and is available via pip or easy_install. If you prefer to have the latest copy of the code, the official code repositories are (in order of my personal preference):