It took far more effort and time than I ever anticipated, but the Writing Idiomatic Python eBook is finally available! It's in "beta" mode right now, meaning I'm still planning on adding more content over the next month, but if you get it today you'll automatically get all of the updates (and corrections) for free. I really believe that the book will be of use to both those new to Python and those looking to increase their Python mastery.
Behind the scenes
Interestingly, the book has its own automated build and test process, and it's
the most comprehensive I've ever used on a Python project. As the book is
primarily comprised of code samples, regression testing is an absolute must. I'm
using pytest to implement the tests themselves. I
found it a bit more flexible than nose in terms of deciding which
directories/files/functions should be searched for tests. I'm also using
coverage package to make sure all of the code samples are actually being
Since there are two different versions of the book (one for Python 2.7.3+ and one for Python 3.3+), tox is used to test each version of Python against the non-version specific tests plus those specific to version of Python being used. tox is incredibly flexible, which has been vital as my "project" is much different than most other Python projects.
Building the book
After the tests complete, a custom Python
generate script traverses each
directory and process the .py files it finds. Section headings are stored in
__init__.py files, while individual idioms are normal Python files. For each
idiom, the file's docstring represents the idiom's description and analysis (
written in Markdown). This is followed by two functions:
test_idiomatic, which contain the actual sample code.
generate script extracts the code from the two functions just mentioned
as code objects and does a bit of post-processing (stripping out doctest related
docstrings and pytest assertions). The sample code often uses non-existent
functions and classes for illustrative purposes, but these need to exist in
order to test the samples. "Helper" code implements the non-existent classes and
functions in such a way that the sample code both runs and gives sensible
values. The helper code for each idiom resides in that idiom's file and is
stripped out by the
An example idiom
To make the above a bit more clear, here's the full text of a sample file for a
single idiom (in this case named
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
You may notice in the description that some terms are surrounded by a single ` and others
by two. This is used for the purposes of building the index at the end of the
book. Anything with two `'s is both formatted as inline code and marked as an
occurrence of that term for the index. A single \' is similarly formatted but not
indexed (so useless phrases like
has_malformed_email_address don't appear in
the index). The
generate script takes the appropriate action based on which
style of backquote is used.
So after all of these steps are completed, the generate script produces a single
Markdown file that represents the complete text of the book, properly formatted.
This is then run through pandoc with a
custom latex template to produce a '.latex' file. This gets run through
xelatex so that the index may be generated,
makeindex is used to
actually build the index, and
xelatex is run again to produce the
final PDF document.
Of course, none of this infrastructure existed when I started. I had no idea how I was going to write the prose and test the code at the same time. I had no idea how PDF files could be created. I had never used latex in any form. All of the above just gradually grew out of necessity. Looking at it all now, I'm amazed I had the patience to set it all up since none of it is evident in the final product (the book).
You can imagine, then, the pace at which the book has been written. I have a full time job, so work was done in hours stolen from evenings and weekends. My wife has been more supportive of this than I deserve, but is glad that it's quite close to ending. I have a new found respect for those that write technical books for a living. It is a mentally and emotionally draining process.
In the end, though, all that matters is the following: I set out to write a book that newcomers to Python would find helpful, I worked on it whenever I could, and I actually finished it. The last part is the key. I have started and abandoned scores of projects, as I'm sure many of you have. This time, though, I persevered. This time I finished. Even if no one actually buys the book, I still got through the process of writing it.
That's worth quite a lot to me.Posted on by Jeff Knupp