After a surprisingly positive reception to my post Writing Idiomatic Python I decided to write an e-book (if you'd like updates on the book's progress, a sign up widget is available below). Having never done so before, I had no prior experience to guide me in how one should go about doing this. While I could have spent a week researching the topic, I decided writing was actually more important and I could figure the rest out as I go. Throughout this process, I've settled on an method for writing and testing the book, which is written entirely in Python (I'll explain below). I've noticed a number of interesting parallels to general project development in Python, hence this post.
I've gotten such a positive response from the blog post that I've decided to turn it into a full-length e-book. See below for more information and to be notified when the e-book is ready.
As someone who evangelizes Python at work, I read a lot of code written by professional programmers new to Python. I've written a good amount of Python code in my time, but I've certainly read far more. The single quickest way to increase maintainability and decrease 'simple' bugs is to strive to write idiomatic Python. Whereas some dynamic languages embrace the idea there being no 'right' way to solve a problem, the Python community generally appreciates the liberal use of 'Pythonic' solutions to problems. 'Pythonic' refers to the principles laid out in 'The Zen of Python' (try typing 'import this' in an interpreter...). One of those principles is
'There should be one-- and preferably only one --obvious way to do it' -from 'The Zen of Python' by Tim Peters
In that vein, I've begun compiling a list of Python idioms that programmers coming from other languages may find helpful. I know there are a ton of things not on here; it's merely a skeleton list that I'll add to over time. If you have a specific idiom you think should be added, let me know in the comments and I'll add it with attribution to the name you use in your comment.
Note: I apologize for the radio silence over the past 3 months. I got married at the end of July and between wedding related stuff and one other reasonably large time sink (which I'll hopefully be able to share here soon), I've been quite busy. My intention going forward is to post a new portion of SO:ASA every two weeks (one week proved to be too sensitive to outliers in my schedule). I'm also going back through all of the comments from the past two months and responding or contacting the author as appropriate. Regardless, hope you enjoy this installment.
In case you missed it, Part One is available here.
The Case of the Idiot Detective
Our end goal in optimization is not to merely increase performance. Rather, it is to do increase performance with proof that our changes were responsible. When you finish your optimization task, you'll likely face the same two questions from whomever you show the new code to:
- How much faster is it?
Your goal is to have hard data that answers those questions for you, and this forms the cornerstone of our approach.
A natural analogy for optimization is detective work. Imagine yourself investigating a robbery at the mansion of your least-favorite wealthy celebrity: You arrive on the scene and are told that, among other things, a priceless piece of art was stolen. Without having even surveyed the scene or collected evidence, you shout out "Professional art thief!" and run back to the station to research every art thief you've ever heard of.
Hopefully, this sounds like a ridiculous way to investigate a crime. Unfortunately, it's the exact method employed by a large number of programmers when tasked with optimization. Instead of doing the appropriate investigation, they make an initial guess based on little more than gut-feeling and programming myths and rush off to fix a portion of the process that is quite likely not the culprit.
Instead, we'll be the dogged investigator who carefully analyzes the crime scene, collects potential evidence, and lets that evidence drive the investigation. When the case has been solved and we present our findings to the jury, the evidence will be damning and overwhelming. Remember, solving the case is not enough. We need to be able to convince the jury that we actually caught the guilty party.
Introducing the S.M.A.R.T Methodology
In our approach to optimization, every step is motivated by data produced from the previous step. All of this data is saved, both to aid in our analysis and to help build the final report proving our changes improved performance.
To optimize software without guesswork and wasted effort, one has to be SMART. In keeping with this, I've named the methodology used for the rest of this paper is the S.M.A.R.T method. The goal is to never encounter a situation where you do some work and then think, "OK, now what?". Data tells us what to work on. The S.M.A.R.T method tells us how to work on it.
Meet Blug. He would like a word with you. You see, he's very sad, insomuch as he is the anthropomorphized representation of a Python script and has no real emotions. Blug has been generating blogs since his assembly. Today, he can frequently be found staring wistfully into the distance, recalling the halcyon days of dynamic blogging engines. Those were good days. Days when he felt useful. Days when he would be instructed to calculate something, give the result, then throw that result on the floor and do it again from scratch. The cycles he used! Blug was gorging on the Internet's bottomless appetite for needlessly repeated calculation.
But it didn't last. Blug began to notice he was asked to calculate less and less. As time wore on, whole quanta would pass where he wouldn't be made to calculate anything. The shame! What was he if not a mindless, calculating automaton? As far as anthropomorphized-robot existential crises go, this was enormous. Before long, he was reduced to a single calculation for each new blog post. And there he sat, pining for the day he could laugh his creepy robot laugh watching CPUs struggle to meet his demands once more.