Archive | February 2010

Basic Parallelisation in Bash

I have recently started writing some quite CPU intensive code and since we have a nice cluster here (without any management software on it) I decided that it would be best for me to take advantage of the number of cores on them. Actually, this works nicely on my desktop which has four cores anyway (and 4GB of RAM). Essentially, I’m lazy and this basically runs my pipeline in parallel for different sources by sending off different jobs to different cores (to the maximum number that you specify)… and then when they finish runs then next few… and so on until they are all done. No longer do I have to worry about waiting for the jobs to finish and wasting time by missing their finishing point. I also no longer have to have lots of [screen] sessions or loads of terminals open… bliss…
Oh here is my basic [Bash] script:

#!/bin/bash
#By Samuel George; http://www.krioma.net
#Original: 27/02/2010
array=(`ls -d */`)
#My scripts run in sub directories, replace with your list of commands to run.
len=${#array[*]}
maxjobs=1
jobnumber=0
#loop over the maximum number of jobs based on the number of files in array
while [ $jobnumber -lt $len ]; do
jobsrunning=0
while [ $jobsrunning -le $maxjobs ]
#start jobs up till maximum and then wait for them to finish before continuing.
do
#go into the directory and run – this is an oddity of my processing
#replace with your own functions
cd “${array[$jobnumber]}”
“run.sh” &
#go back a dir
cd ../
#add to counter so that you know how many jobs are running at once.
jobsrunning=$(( $jobsrunning + 1 ))
#keep a running total, such that the script will loop over all the jobs you want running
jobnumber=$(( $jobnumber + 1 ))
done
wait
#keep a listing of where you are.
echo $jobnumber
done

Stick this in a shell script, chmod 700 file.sh and job done.
There is definitely room for improvement here. For example this code will wait till all of the processes in the inner loop, ideally you’d want it to move on after the first one is finished. Watch this space… well that might not happen since my tasks all take about the same time to finish in.

Monster Truck Madness!

I took my first trip down to the Saddledome here in Calgary yesterday to go watch [Monster Jam]! It was also a chance to experience some “proper white trash”. It was lots of fun watching these huge beasts crush cars and fly through the air. From the pictures its hard to really understand the size of these things… but just look at the people for scale (some pics below).I put a video compilation together of some of the best bits I recorded (if you can’t see it below go to [youtube.com]

Donkey Kong flies through the air…
Monster Jam - donkey kong
The fans favourite, Grave Digger takes on the giant dirty mound…
Monster Jam - Gravedigger
It wasn’t all big trucks crushing things though.. there was a cool freestyle motocross show too:
Monster Jam freestyle motocross
More pics over at .

Dinos vs UBC

I went to watch my first University sport match while I have been in Calgary on Friday. It was the [University of Calgary Dinos] versus the University of British Columbia. Oh it was ice hockey. I’m happy to say the game was good, though due to drinking of beer we did miss the first period… oh well. It was 1-1 when we got there. The final score was 3-1 to the Dinos… go Dinos!
UofC Dinos vs UBC hockey faceoff
This was my second hockey game while I have been here and it was a bit higher pace than the [last one (Canada’s Women’s national team)] with a few big “hits” and a bit of a fight… not a “good” one though…
UofC Dinos vs UBC hockey fight!
A few more pictures over on .

Olympic Celebration in Calgary..

Last Saturday Calgary (well CTV) held a celebration for the start of the Olympic Games over in British Columbia. I have to say this is probably the coldest gig I’ve ever been to. It was fun though.. but still it was darn cold. Not sure if holding an outdoors gig in Calgary in the winter is ever a good idea… it was foogy too – such that you could just make the outline of the Calgary tower:
Is that the tower?
I got there in time to watch some local band called [“the Dudes”] who played a solid set of music. Not sure if they are really my thing but they are defineely worth a look out. Probably the best local band I’ve seen so far. I’ve not seen many though.
"The Dudes" play
Eventually, as it got darker, the Tower broke through the clouds and you could see the Olympic Flame burning brightly over Calgary…
Calgary Tower, olympic flame
The main attraction for me to go down to the event was to see the headline band, Simple Plan. They were definitely worth waiting around in the cold for (well it wasn’t so cold after I had a hot chocolate and then jumped around a bit).
Simple Plan at Olympic Celebration
Simple Plan at Olympic Celebration
.. but by the end of this and when I got on the C-train to go home I couldn’t feel my feet properly.. hmmm. Oh and here is a quick clip of the gig:

More photos from the gig can be found [on flickr].

Playing with gnuplot

The otherday it was about time I tried out [gnuplot] for a change. I used to do all of my scientific ploting (and actually a lot of my analysis) in [R]. These days I tend to use [python+matplotlib]… but I wanted something that would allow me quickly to visualise data and plot functions without much effort… this is where [gnuplot] really does seem to be powerful.
I came across an excellent intro to gnuplot page: [Plotting Data with gnuplot].
Here is an example plot of what you can easily produce in a few seconds. Pretty swish.
gnuplot example
Oh and gnuplot is nicely in the [Ubuntu] repositories… you probably want to get:

sudo apt-get install gnuplot-x11

Turning a PDF into an animated GIF

Today I was asked to take a bunch of PDF files (essentially frames of the animation) into an animated GIF. This is actually a fairly simple task todo with [imagemagick] and its fast too. Simply all you have todo (assuming the PDF, or other image files etc are in order) is:

convert -delay 20 -loop 0 frame*.pdf output.gif

The -delay 20 specifies the time between frames and the -loop 0 tells the image to loop forever.
Quite nice really (the animation is for a conference and so I won’t post it).
The only issue I had is I wanted it to stay on the last frame for a while, but this was simply accomplished by a bit of bash before running the above:

for i in $(seq 28 1 38)
do
cp -r frame27.pdf frame$i.pdf
done

Thus this took frame 27 and copied it in a sequence from 28 to 38. Sweet.
Finally I was asked to put two animations next to each and this can be done by using the append functionality:

for i in $(seq 01 01 38)
do
convert frameset1/frame$i.pdf frameset2/frame$i.pdf +append output$i.pdf
done

This adds the same frame number from each set next to each other in one larger image. Once this is done it is a simple procedure of running the earlier animation script on the newly created pdfs.

Sunset from Science B

My first trip onto the roof of Science B of the University of Calgary was timed nicely with a gorgeous sunset…
View from Science B roof
… here you can see (what I think it is anyway) the ski jump that Eddie the Eagle must have jumped off back in the 1988 games.
I didn’t just go on the roof to look at the sunset, though it was probably worth it just for that, but to also do a bit of astronomy. There was a class on and I tagged along for the start and helped setup the scope (they were investigating the size of craters on the Moon) and took this pic:
The Moon from Calgary
Not a bad setup really..
Telescope and Moon