Canadian Amp

I picked up Beginning Python Visualization: Crafting Visual Transformation Scripts (Books for Professionals by Professionals) on an impulse Amazon buy and got a chance to read it over the weekend. My initial impression is that this would serve as a solid introduction to data visualization in python for people who have a bit of programming experience (no python experience is assumed) and are interested in presenting it in a graphical form. One of the neat thigs about the book is that the first chapter just dives in to a simple case study of visualizing some GPS data. The author first walks you through how to use < a href=”http://pyserial.wiki.sourceforge.net/pySerial”>pySerial to read GPS data from a device and store it in a file, placing an emphasis on creating meaningful file names and directory structures. Once some data has been collected Python’s built in csv reader is used to parse the data and extract position and velocity information which is then used to create a nice GPS track plot including velocity vectors with < a href=”http://matplotlib.sourceforge.net/”>matplotlib. I am undecided on whether I am totally happy with the authors choice of subject for the first chapter case study. On the one hand it is nice to see a really concrete and simple real world example. On the other hand I have to wonder whether reading data from a serial port might be a little intimidating for people who are new to programming, not to mention the fact that not everyone has a GPS receiver. This is a minor quibble since the code is written clearly and is easy to follow along even if you don’t type it out and try it for yourself. Overall the first chapter gives you the “Why” of data visualization with the later chapters focusing on the “How”. Chapter 2 was a nice addition as it focuses on choosing, and setting up, a development platform with a focus on Linux but Windows and Mac are not ignored and it is made clear that all of these programs will work on whichever platform you choose. This part was fairly brief (by neccessity) but covered all of the basics including discussion of choosing an editor (Emacs, vim, Notepad++ and more), prebuilt scientific python packages including the Enthought Python Distribution, IPython and python(x,y). Vaingast also instructs how to get the bedrock scientific and visualization python packages SciPy, matplotlib and the < a href=”http://www.pythonware.com/products/pil/”>Python Imaging Library (PIL). In my experience these packages are going to cover a large percentage of your visualization needs. Chapter 3 provides a quick but good introduction to the Python language for people who have a bit of programming experience. Another great intro to Python of course is Mark Pilgrim’s Dive Into PYTHON. Chapters 4 and 5 cover reading and writing to some common file formats including csv and ini files. Both text and binary files are covered although other formats like XML and Yaml would have been a nice addition. Chapter 6 gives a very nice introduction to matplotlib and whatever isn’t covered here can be found in the extensive online documentation Lots of other important foundations are covered in Chapters 7 and 8 including complex numbers, multi-dimensional arrays and linear algebra functions. There is also coverage of more advanced concepts including interpolation, solving non-linear equations and Fourier transformations (FFT’s). There is some discussion of 2 dimensional data in chapter 9, but I felt the book falls a little bit short in this regard. While contour and filled contour plots are mentioned briefly earlier in the book, there is very little discussion of working with data that is a function of more than one coordinate. In this sense I feel the book falls a bit short since part of the appeal of matplotlib is that is has excellent support for < a href=”http://matplotlib.sourceforge.net/api/pyplot_api.html?highlight=contourf#matplotlib.pyplot.contourf”>contours built in. There is also no discussion of how to use tools like MayaVi and VTK to visualize 3d scientific data which is a shame since this is where visualizing data starts to get really interesting to me. Again this is a minor quibble since this book is aimed at beginners and does not set out to cover everything. In summary I would highly recommend Beginning Python Visualization: Crafting Visual Transformation Scripts (Books for Professionals by Professionals) to anyone looking to get there feet wet with data visualization.

So I just released the first beta of a program called Orbis that does Simple Huckel Molecular Orbital (or SHMO for short) calculations. The beta version is free and valid until June 1st after which it will likely go on sale for around $20 US per license.

Right now there is only a windows version, however I hope to have a Mac version ready in the coming months.

Here is a couple quick videos showing Orbis in action.

Creating a simple benzene ring and then converting it to pyridine by changing one of the atom types.

Orbis displaying the 24 membered molecule Coronene

You can download a trial at www.simplehuckel.com.

I just finished this excellent book: Dreaming in Code: Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software last night and thought I would share it. The book follows the development of Chandler from start to, ummm…not finished over a period of 3 or 4 years. It details all of the trials and tribulations of the group and really just emphasizes just how hard it is to write good software.

Much of what is said in Dreaming in Code will resonate really loudly (read: gives you that sinking feeling in your stomach) with anybody who has ever sat down and tried to write or fix code, particulary in a group setting. The book is written by Scott Rosenberg who founded Salon.com and I think he does a great job of striking a balance between making it technical enough to keep programmers entertained while keeping it accessible to people with little or no software development experience who want a glimpse into the world of software. The latter crowd will definitely gain an appreciation for why many of the programs they use everyday still contain mistakes, crash often or don’t behave the way they feel like it should.

Another bonus for me was that Chandler is written in Python and there is some discussion in Dreaming in Code about the merits of python and why it was chosen for the project.

Anyway I thought it was a great and entertaining read which gave some insight into startups, open source software projects, philosophy, and the joys and immense frustrations of software development. Highly recommended.

I have been building a new project with django, pinax and geodjango. Part of the project involves parsing GPX files created by GPS devices. In the DjangoCon talk (at youtube by Justin Bronn he mentions that GDAL can parse GPX files which would be very helpful for me.

Originally I followed the GeoDjango iinstall instructions However, when I tried to parse a simple gpx file I was present with this error

After a bit of googling I came up with the following solution. First step is to get the expat xml library (including dev) installed like so;

After this you should have a libexpat.h header file and an expat.so located in /usr/include/ and /usr/lib/ respectively.

Now change into the directory that you installed GDAL from during this step of the GeoDjango install process. If you have already installed GDAL then you will need to do a clean install and configure like so (output removed):

After that you should be able to use a gpx file as a datasource in your geodjango project:

Hopefully that will save someone some time in the future as the solution was not immediately obvious to me when I googled around for an answer.

I recently had to geocode a huge number of addresses for a project I'm working on. Here is a simple python script that reads in a bunch of addresses from a text file and then uses the Google maps api to geocode them (get their latitude and longitude).

The format of the address file should be roughly like:

although the Google geocoder is quite forgiving about the format of addresses so you can probably get away with something like

You will also need to get your own google maps api key which you can get over at Google.

Nice and easy. You may also want to check out the geopy package which looks very nice and includes support for distance calculations.

It was just announced that Stephen Hawking will be accepting a postion at the Perimeter Institute for theoretical physics in Waterloo. It appears that Dr. Hawking will not be moving to Waterloo full time but rather making regular stays in Waterloo starting in the summer of '09.

Hawking is likely the most well known modern scientists outside of the scientific community. Largely due to his hugely successful book A Brief History of Time and its follow up The Universe in a Nutshell both of which brought the realms of things such as superstring theory and how the universe evolved to laypersons.

Also, you really know you've made it big in science when you make a cameo on the simpsons.

Hopefully he'll do a couple public lectures that I'll be able to attend.

One of the greatest things about python is how quickly you can develop fully functioning scripts/programs in it. Both from the perspective of the large number of library functions available which make your life easier and also the fact that you don't have to go through a compile step every time you edit your code. Unfortunately this rapid development usually comes at the cost of raw run time speed. Often this isn't a problem, for example if your python script takes 1 second to run when the equivalent program written in c would take 0.2 s to run the 500% difference is practically negligible. Even if the python script takes 10 hours to run where the equivalent c program takes 2 hours this may not be a large problem if you are only going to ever run the script once or twice since the huge amount of time you saved in developing the script in python more than makes up for the difference. The speed deficit of an interpreted language like python really starts to pop up when you have a program that needs to be run repeatedly or requires lots of user interaction and therefore must be fast enough to prevent a user from becoming frustrated.

Luckily, in python there are a number of possibilities that allow developers to take advantage of the rapid development possible in python while still retaining reasonable run times. Probably the two most popular choices are using the python module Psyco and/or rewriting the most computationally intensive functions in c and calling those c functions from your python script. As Jeremy Jones at onlamp.com puts it:

One of the frequently repeated mantras I hear from Python veterans is, "Premature optimization is the root of all evil" (a quote from Donald Knuth, which apparently originated from Tony Hoare). The Pythonic approach to writing code that performs acceptably is:

* Write the code.
* Test it to see whether performance is OK right off the bat.
* If so, you're done.
* If not, optionally try Psyco.
* Profile it and identify the bottlenecks.
* Modify the code until the performance is as fast as you need, rewriting bottlenecks in C (or Pyrex) if necessary.

As an example of this development cycle I wrote a quick python script to produce images of a Julia set using the Python Imaging Library.

Here is the first implementation of this script written purely in python

Ok, that was really quick to write and produces pretty pictures now how quickly does it run?

So roughly a minute. That's okay for producing an image or two every once in a while, but what if I wanted to create a huge amount of images from my own book of fractal images? It's going to take me forever :(

Bar none the easiest way to speed up your code is to try using Psyco. If you're using Ubuntu Psyco is a simple "sudo apt-get install python-psyco" away.

Adding the lines:

to the top of our script gives us a roughly 20% increase in speed:

Not bad but still not nearly as fast as we would like. That means moving on to profile our code to find out where the speed bottleneck is. In this case our program is fairly trivial and we can pretty easily guess before hand that the julia function is burning up most of our time. Turning to the cProfile module we can confirm our guess:

So a huge fraction of time is being spent in the julia function as we expected so the next step we can take is to rewrite this function in c and then call that c function from our python script. This is accomplished in a few steps. First we rewrite our function in c:

And next create a shared library containing our new julia function:

We can then load this library into our python script using the NumPy ctypeslib load_library function. and replace our python implementation of julia() with this:

The only other change we have to make to our script is to cast our variables to ctypes like so:

and when we're done our full script will look like this:

Simple and straightforward, but how much of a gain do we get?

That is a whopping 15 fold decrease in run time for very little programming effort! Of course this is a fairly trivial example, but I hope it gives you an idea of how python can be used for computationally intensive problems without the large decrease in speed normally associated with interpreted languages.

If you enjoyed this post you may want to check out Python Scripting for Computational Science :

Sometime around July 2008 Rogers started the very annoying practice of doing DNS redirects to sponsored search results when you mistyped a url in the search bar or tried to access a non existent website. This may or may not be a violation of Net Neutrality but more than anything it was just annoying to have to retype an address when I mistyped one letter. In some cases this breaks the address bar auto searches of some browsers and their are reports of it killing remote web apps. So just standard Rogers not really paying attention to what it's customers want.

There is a lot of disscussion of this topic at MichaelGeist.ca and most importantly for me was I found a way to avoid this problem in the future. It seems that Rogers recieved a lot of complaints about this and in response quietly provided an alternate dns server which won't do a redirect.

Here is how to set up that DNS server for yourself in Windows XP:
1. Open the Control Panel (Start->Settings->Control Panel)

2. Double click on "Network Connections"

XP Control Panel

3. Right click on the connection used to access your internet (usually Local Area Network)

Network Connections

4. Click on TCP/IP and then click on the Properties button

tcp/ip properties

5. Click on "Use the following DNS Addresses" and enter 64.71.255.202 (this is an alternate Rogers DNS, see the comments below if you want to use Open DNS instead)

Alternate DNS Server

6. Click Ok, then OK and then you're done!

Now when you type www.adsfasdlfkasdlf.com you should no longer be directed to a rogers sponsered search results.

The process is similar on Vista:

Start->Control Panel->View Network Status and Tasks->Manage Network Connections->right click on your connection and then follow from step 3 above.

If you are feeling really ambitious fire off a complaint to :

The Office of the President
Rogers Cablesystems - Rogers Yahoo! HiSpeed Internet
855 York Mills Rd
Don Mills, Ontario
M3B 1Z1

Recently I was tasked with creating a visual representation of a table of scientific data so that a user could explore and visualize their dataset interactively. Visualizing a lot of data at one time can really lead to a much better understanding of what your data looks like and how it behaves as a function of it's parameters. For example, which one of the following representations of temperature difference data provides you with the most intuitive feeling of how the data varies as a function of the positions (x,y)?

Exhibit A - The Spreadsheet:

spread sheet showing data

Exhibit B - A Colour Map:

colour map of temperature data

It's not even close, the colour map instantly shows you where the hottest and coldest regions in space are. We can even take it a step further and get a full 3d representation of the data like so:

3d colour map

Now we are able to get a better idea of how steep the temperature gradients are than we can with the colour map alone.

Okay, so great, now we know how much nicer looking at 3d data is how do you go from your boring spreadsheet to a 3d plot? How about making the data interactive so that we can zoom in, rotate or otherwise manipulate the data?

Well, Excel has very limited 3d plotting capabilities, particulary if your coordinates aren't spaced linearly in space. Ditto for Origin. My next thought was to download some free or very cheap plotting software of which there seems to be a < href="http://www.google.com/search?&q=3d+plotting+software">tonne of options, however my first attempts didn't really give me what I was looking for and most of these products aren't cross platform. Since I move regularly between windows and linux environments this is definitely a must for me. Commercial software such as Matlab and MathCad or Mathematica are all great but expensive so I just decided it would be easier to roll-my-own.

After a bit of searching around, I decided to use a combination of python and the related modules NumPy & SciPy and pyvtk. All three are a snap to install on either windows or linux. The plan is to use python to create a Visual Toolkit file that can be displayed using MayaVi. Yeah, installing all of that might seem like a lot of work, but it truly is pretty simple on either linux or windows and they are all very useful in their own right.

The data I was given was a csv file with three colums (x,y,z) that looked like:

So the first thing that needed to be done is to read the data in, extract all the x and y coordinates and convert the data to a matrix rather than the unordered columns. (There is probably a more elegant way to do this but oh well, it works). Depending on the format that your data is in you may have to manipulate it differently to get it into the correct format

Which gives you a grid that looks like this:

Now all that's left to do is create a vtk file that MayaVi can use. I modified the example from MayaVi guide to create our vtk file.

And we're done. Now we load up MayaVi and go to File->Open->VTK file and open up our freshly created vtk file. When you first open the file nothing will appear on the canvas. We need to add a module in order to be able to see our data. So go to Visualize->Modules->Surface Map and click okay. This will show you a 2D colour map of your data. In order to get a 3d representaion of the data we need to add the WarpScalar filter via Visualize->Filters->WarpScalar. Boom. A full 3d representation of the data that you can zoom in on rotate and much more. Add axes and labels using the Visualize->Modules->Axes option. MayaVi also allows you to take snapshots of the data and export screenshots via File->Save Scene To-> png, bmp etc etc.

mayavi window displaying data

I also was able to package the file into a windows executable using the great bbfreeze which allowed me to distribute the script to computers without python and the other packages installed (except of course mayavi which is needed to display the file).

Dive Into Python is a great introduction to Python if you are unfamiliar with the language. I highly recommend learning the language if you are a scientist who currently uses spreadsheets for doing tedious manipulations of data. It is relatively simple to pick up and can save you a huge amount of time by automating tasks for you.

If you like you can download the visualization script and the sample data.

I just came across an excellent syntax highlighting plugin for WordPress: iG Syntax Hiliter. It has support for a bunch of languages and allows readers to toggle between good looking highlighted html code and plain text (just click on the "PLAIN TEXT" bar) that can easily be cut and pasted into a text editor. Couple examples shown below.

sweet!