![[Home page]](PythonPowered.gif "Home page"){kind=small}
| Books | Code | Blog | Python | Author | Train | Find | ©M.Lutz |
Below are recent book notes: general comments related to the book, its subject matter, and its goals. The items on this page either span printings or were posted after or just before the first reprint (January 2010). This page and the list below group notes by topic area.
For more book-related topics, see also the book's clarifications pages; in general, I put more technically in-depth notes on that page.
A student in a recent Python class I taught mentioned that a coworker recommended always fetching a sequence's length by using a X.__len__() direct operator-overloading method call instead of the len(X) built-in function call, because the former was supposedly faster than the latter (which seems to make sense on the surface, as the latter essentially invokes the former, through Python's operator-overloading machinary). To test the claim, I coded up a script during a class break which times the two, and shows that the recommendation isn't quite true -- len(X) turns out to be quickest in both Python 2.X and 3.X. Not by much, but the difference can add up over a long-running program's time. Determining why this is true would require deeper analysis, but the Python-language attribute fetch in X.__len__() may add extra time to the method-call form.
All of which underscores the fact that performance claims about Python must generally be backed up by real test numbers instead of intuition, because the language is more general and more optimized than one may think. Trust me; I've either been wrong on such things in the past, or have been made wrong by changes between Python releases. This case also highlights the fact that testing can be tricky, especially across Python lines -- I had to be careful to wrap range() in list() to make the amount of work equal in 2.X and 3.X, because range() is a generator in 3.X only (otherwise, Python 3.1 incurs a big speed hit for lengths, presumably because generators in 3.1 handle this differently then physically stored lists).
For any interested sports fans out there, the custom timing script is available here: lentimer.py. This script's results on a dual-core Windows Vista machine, and under Pythons 2.6, 2.7, 3.1, and 3.2 are available here: lentimer-out.txt. In short, Python 3.X loses to 2.X on this, but by only a very small constant amount. Python 2.7 and 3.2 are also both slightly faster than their predecessors, but probably only because on my machine 2.7 and 3.2 are 64-bit executables, and 2.6 and 3.1 are 32-bit.
If you've read this book, you might also recall that there's a more generic timer() utility function case study in the text which does similar work, on pages 509 to 518 (unfortunately, its code wasn't available when I coded the custom lentimer.py script). For a trace of this generic timer utility in action on this test too, see also this file: lentimer-book.txt.
I've been teaching Python classes to more numerically inclined groups recently -- traders, physicists, and the like -- and some students have requested more examples of code oriented towards basic matrix processing in the core language (not in NumPy, an extension which adds better support for such things and handles large data sets more efficiently). I added the following file to my training materials, but since the class and book overlap, I'm posting it here for book readers as well: matrix-code.py.
Run this file on your own to correlate its output with the code (and be sure to comment-out the one 3.X-only test on Python 2.X). There's similar code in the book already, but this collects and extends the coverage.
I recently wrote a brief description of programming languages for non-programmers (I've been asked this often enough to warrant a canned reply). This is pretty basic material and is probably too simplistic for many readers of this book. If you're looking for a relatively concise definition that also introduces some of the prerequisite terms in this domain, though, you might find this of interest.
Due to its size and limited appeal, this note is located off-page: click here to read. This level of material might not have worked in the book (its early chapters are fairly basic as is, and parts of them already cover some of this note's topics) but feel free to consider it a new virtual sidebar for Chapter 2, or part of a new virtual appendix for true beginners.
[Sep 2011...Feb 2012]
As an afterthought to the three earlier notes that follow this one, people have also wondered how long it can take to learn Python. In short, to learn how to properly and idiomatically use a programming language as complex as today's Python, most people in this book's target audience should probably expect to spend an amount of time equivalent to a college semester. That's roughly how long some readers have reported spending with the book Learning Python 4th Edition, especially readers who are relatively new to programming.
Although books and classes can help optimize the learning process, the required effort is still significant. My own training site puts it this way:
| Learning any programming language as sophisticated as today's Python requires a substantial investment in time and focus, spanning a progression of steps — from initial introduction, to in-depth study, to practical project experience, to full proficiency. Especially for less-experienced students, this can be an ongoing process that spans months or years, depending on required skill level. |
Naturally, this depends in part on your prior experience, and to some extent on how you'll be using the language. People already proficient in object-oriented and functional programming, for example, will find Python less challenging than true beginners starting from scratch. Moreover, basic scripting tasks may require less of an intellectual commitment than full-scale development. To put that another way, there is a substantial distinction between hobbyists and professionals, in both skills and learning requirements (a simple point which goes often overlooked by much of the technical press).
Especially if you're new to the core concepts and paradigms in Python, though, you shouldn't underestimate its learning curve; doing so can lead to frustration both for you, and for later users of your code. With its generators, decorators, closures, Unicode, and even OOP, today's Python is a powerful but substantial topic. While some parts of it can be absorbed quickly or gradually, most of its more advanced tools are pervasive enough in nontrivial Python programs to make them prerequisite to realistic work in the language. The net result is this: for most Python newcomers, its practical learning curve is best estimated not in days or hours, but in weeks for the most experienced and focused professionals, and in months for most others.
Also keep in mind that this estimate includes just the time needed to learn the Python language itself. Software development in general requires even more time to master, of course, just like any other field of engineering. Although Python's syntax and tools may be relatively easy to learn, it can take additional years to learn how to apply them well to real tasks. Even then, the constant change in application domains such as scientific programming or web development makes proficiency an ongoing goal. My applications-focused book Programming Python seems an informal proof of this very point: its presentation of realistically scaled work fills a very large book and requires examples that span multiple chapters, multiple domains, and 5,000 lines of code. Many programs grow much larger still.
This too can depend upon your background, and on where you plan to go in the field. For some, a minimal knowledge of Python might suffice if they will never go any further than calling functions in a precoded library. But consider that full-scale software development is substantial enough to fill a 4-year computer science program at most universities. Advanced degrees which focus on specific topics typically take even longer. Not every task requires an advanced degree, of course, but it makes no sense to expect to find shortcuts to competence in this field alone.
Having spent 15 years teaching Python classes to thousands of students at hundreds of organizations, this misconception is not abstract to me. While not universal, there seemed a palpable trend among some of my clients towards wildly overinflated expectations, no matter how clearly the class's scope was described. In truth, there is very little of value that can be fully mastered in 3 or 4 days time, much less a highly technical skill like software development. Even the best of such short classes can be no more than an introductory first step of a much longer journey. Implying more of a brief training session or similarly shallow resource is probably larcenous; it's also unfortunately common in the training and publishing worlds, and may account in part for some of the underestimates of entry requirements in the software field.
In the end, background, goals, and commitment all come in to play when learning any new subject, Python included. That isn't meant to discourage, but to scale expectations realistically. Python has shown itself to be very much accessible to most newcomers who invest the time and focus required. I encourage anyone interested in learning Python to give it a go; it's still one of the most programmer- and learner-friendly languages in use today. But if you do, you'll probably be more satisfied and successful if you remember that software is not a trivial skill to master, despite what you may have heard.
I revisited this topic in a later article,
Answer Me These Questions Three...
For an earlier related post, see also the section below: Focus, "2.0".
I get a lot of email from people asking if they can learn to program, and whether or not Python is a good choice in this role. Since this is a frequently asked question, below is a recent response on this topic to a Sports Medicine physician in Uruguay who also asked how Python compares to AppleScript, along with a few additional thoughts:
Hello ..., Some of what you're asking is subjective, of course, and I can't give you any absolute answers. Learning to program well in any language requires a substantial amount of focus and effort, and is not for everyone. It depends as much upon the individual as the language and resources. Having said that, I can say that Python still seems relatively simple when compared to larger languages such as C++ and Java, and tends to be more broadly functional and applicable than most domain-specific languages such as AppleScript. I think Python's a good language to get started in for these two reasons -- you can pick up a useful subset quicker, and can move on to learn how to apply it in specific application domains as required over time. Python tends to be useful for just about anything you can do with computers; search the web for links to Python resources for the specific tasks you listed. As an added bonus, Python tends to promote a quality of code that becomes more valuable as your programs become larger. Again, though, much depends on your goals, so your experience might vary arbitrarily. Software in general is a challenging domain, and Python itself has grown substantially advanced features over the years which are not always entirely optional. In fact, the sizes of the current Learning Python and the upcoming Programming Python are testaments to both these points. Though not always obvious to those engaged in simple scripting (or, regrettably, to publishers of books which promise expertise overnight), full-blown software development is as tangible a field as your own, and mastering it requires a similar investment in time. That's not meant to discourage, of course: learning to program can be both fun and rewarding. But you should scale your expectations accordingly. --Mark Lutz (http://learning-python.com)
On a point related to the prior note, people also often ask about prerequisites to learning Python, especially for those new to programming. While this can vary much per individual, the following is another recent reply to a potential reader on this topic.
Hello ..., It helps to have some programming background, of course; but a basic knowledge of math (especially algebra) or logic can often suffice for those new to programming. Programming, after all, is largely just algebra or symbolic logic, with procedural flow and data structures mixed in. Python does introduce some more advanced programming concepts such as object-oriented programming and functional tools which you may or may not be familiar with (and can sometimes seem daunting to beginners on first glance), but these can often be picked up gradually as your skill set evolves. Technical features aside, motivation also tends to matter as much as anything when it comes to learning Python, or programming in general. In the end, a logical approach to problem solving and the ability to focus are the true keys to success in the software field. --Mark Lutz (http://learning-python.com)
Also related to the prior two notes, I've recently fielded multiple reader emails asking about which of my books to get started with. Since this seems to be a common question too, here's a recent reply on this topic, to someone with a prior background in VBA.
> ... In terms of my books, you probably want to check out Learning Python. That's the first in the series, and covers Python language fundamentals. Programming Python is a follow-up second book, which moves on to address applications work -- some of the most common things you can do with Python after you've learned it. Some people seem to want to jump into more advanced Python applications such as GUIs and the Web right away, but I generally recommend learning to walk with Python before trying to run with it. Python is a complex subject in its own right; using it well requires the sort of focused introduction that one can find in Learning Python. Once you've learned Python, what you do with it is up to you, of course (and/or your job description). To help you get started with this second step, Programming Python assumes Learning Python's coverage as prerequisite, and covers how to apply the language in its most common application domains. --Mark Lutz (http://learning-python.com)
The fourth edition of Learning Python is based on Python 2.6 and 3.1, but applies to the entire 2.X and 3.X lines (as noted in the Preface, "3.0" in the book really means "3.X" in general). A new and most likely final release in the 2.X line, Python 2.7, is due to be released shortly.
Per current plans, at least, 2.7 will be the last major 2.X series release, but will have a long maintenance period in which it will continue to be used in production work. After 2.7, new development is to shift to the Python 3.X line. Besides being the last major release in the 2.X line, 2.7 incorporates as backports a handful of features described in the book that were formerly available only in the 3.X line. Among them:
{1, 4, 2, 3, 4}
{c*4 for c in 'spam'}, {c: c*4 for c in 'spam'}
dict.viewkeys(), dict.viewvalues(), dict.viewitems()
'{:,.2f} {}'.format(1234567.891, 'spam')
with X() as x, Y() as y: ...
The book presents these as 3.X topics, but they now apply equally to 2.7. For the full story on 2.7, check out its release notes online, at this page, as well as its "What's New" changes overview document at this page.
When I say that 2.7 is the end of the 2.X line, I mean that it's the last 2.X version that will be officially developed by the current python-dev core developers group (a group that also originally promised a 2.8, and possibly more). Given Python's open source nature, it's not impossible that 2.X will be forked and managed by a different group of developers; in fact, this was recently discussed on the python-dev email list.
Moreover, some in the Python world have come to suggest that it's not an either/or choice: given the size of the existing Python 2.X ecosystem, 2.X and 3.X may both enjoy vibrant user and software bases for years to come, even if they are at least partly distinct. Per this theory, there's room enough in today's Python world for both the established 2.X camp and the 3.X alternative. As evidence, holders of this view point to both the huge amount of 2.X code in use today, as well as the fact that the two lines are similar enough to encourage cross-fertilization.
All speculation aside, the reality today is that although 2.7 is the last 2.X release per 3.X-focused python.org developers, some 2 years after 3.X's initial release 2.X still constitutes the vast majority of the Python user and code bases. As one metric, at this writing 2.X is still downloaded from python.org 3 to 4 times more often than 3.X, at least based on current web download statistics for Windows installers (this ignores other platforms that typically ship with 2.X today, and is probably biased towards new users normally inclined to pick newer versions; other measures such as existing software ratios tend to skew 2.X's dominance even higher).
Or, to borrow a Monty Python line, "I'm not dead yet..."; because it's impossible to know 2.X's future today, stay tuned to the web for developments.
As mentioned in the prior note, the fourth edition of Learning Python is based on Python 2.6 and 3.1. The next version in the 3.X line, Python 3.2, is now in alpha form and should be released by early 2011. Although its feature set is still being hashed out, it introduces a handful of noteable changes to (and potential incompatibilites with) both 2.X and earlier 3.X releases. Among these:
The internal implementation of threading has changed to reduce contention. In particular, the GIL has been changed to use absolute time slices instead of a virtual machine instruction counter. Since threading is not covered in any sort of depth in Learning Python, this has more impact on the book Programming Python 4th Edition.
The way that byte-code files are stored is being changed to avoid the potential for collisions when multiple Python interpreters are installed. In Python 2.6 and 3.1 (and hence in Learning Python), ".pyc" bytecode files are stored in the same directory as the corresponding ".py" source file. In Python 3.2, bytecode files will instead be stored in a "__pycache__" subdirectory created under the package directory, and have names that designate the creating interpreter (e.g., "mymodule.cpython-32.pyc"). This way, source directories are not cluttered with byte-code files, and different Python implementations do not step on each other's work.
Technically, multiple Python interpreters work today in 3.1 and earlier, but this change will avoid requiring each different interpreter to recreate the byte-code file anew when the file's internal "magic" version number does not match that of the importing interpreter (this number is tested in addition to file timestamps to know when a recompile is necessary). The new model allows a single source directory to record cached byte-code for multiple Python versions and implementations, and thus avoids extra start-up time for module recompiles. Note that this does not apply to Python's own standard library, as each version generally installs its own copy of the library.
This change seems a generally good thing, but its downside is that some scripts which use a module's imported file name to infer its source file name may need to be changed to work under 3.2 if coded to be dependent on directory structure. Similarly, makefiles, custom importers, and programs that search for ".pyc" files and simply strip off the last letter to derive the source file's name are also prone to requiring updates for this change.
In terms of module-to-source mapping, though, this is mostly an issue for scripts written for 2.X, and then only if they are not robust. In all 3.X, a module's __file__ normally names its original ".py" source-code file, not its ".pyc" byte-code file, even if the byte-code file was loaded diectly. Hence, most 3.1 programs that derive source names from __file__ should continue to work in 3.2, even if they allow for legacy "pyc" __file__ names too. The __file__ may name the ".pyc" file's path in 2.X, but as long as the 2.X script allows for __file__ to name either a ".py" or ".pyc" as it should, it will likely work as is in 3.2; the 3.X ".py" __file__ will suffice.
As a concrete example, at least one program in the upcoming book Programming Python 4th Edition, the source-code viewer option in PyMalGUI's non-HTML help window, uses a module's __file__ to infer its source file, assuming the pre-3.2 storage model. Since __file__ is still the ".py" in 3.2, this example should work as is (though this is to be verified -- hopefully, this change will not have made the book out of date between its composition and its publication!). For more on this 3.2 change in general, see the PEP description at python.org.
The str/bytes (Unicode text/binary data) string types dichotomy in 3.X is now better supported by Python itself. Ironically, Python's own library did not fully resolve this split. Email content and many Internet support modules, for example, were especially affected (in fact, the email package and CGI uploads are still broken for some use cases in 3.1's library). Python 3.2 will include a number of fixes on this front, though a full rewrite of the email package is still pending.
Again, this change bears primarily upon the applications- and library-focused book Programming Python 4th Ed. In contrast, Learning Python deals primarily with the core language fundamentals of the 3.X string model, not library implications. There has also been a minor change to filesystem Unicode encoding support, which I'll omit here.
For more on 3.2, be sure to see its "What's New" document, available on Python's web site and packaged with the release itself.
[Update Dec-31-10] For additional notes on 3.2 changes, especially on the way its Unicode enhancements impact the applications book Programming Python 4E, see also that book's 3.2 updates note.
I recently finished writing the new 4th Edition of Programming Python--the follow-up book to Learning Python which focuses on applying Python to real domain tasks (systems programming, GUIs, the Web, databases and text, and so on). It should be available sometime this fall. Like the Learning Python update, this was a 6 month project which went through line-by-line to refresh and polish the book in general. Among other things, this edition's examples include a Unicode-aware text editor, and a feature-rich 5K-line desktop email client which supports Internationalized message content and headers. We also trimmed some material along the way, or made it external, to avoid new growth this time around. For instance, 4 prior chapters are gone in the new edition, having been absorbed in part elsewhere or cut altogether.
The main change in this book's new edition, though, is that it has been updated to cover and use Python 3.X only. After going through this update, the good news is that Python 3.X is a viable systems development language today. And as expected, core language changes were relatively straightforward to accommodate. However, the update also underscored that the 3.X Unicode model is more likely to impact programmers than I formerly assumed. Once you enter the realm of realistic programming and the standard libraries it employs, Unicode seems to pop up everywhere--in file content, file names, directory walkers, GUI text displays, network interfaces, Internet protocols, web content, and even in IPC and persistence interfaces such as pipes, DBM files, shelves, and pickles.
In many cases, this requires bringing a new mindset to bear on the notion of data. Essentially, anywhere that you're used to processing "text", you may now be required to amend your thinking to clarify "which kind". For example, any code that opens a text file must now also consider its Unicode encoding if it might ever contain text other than the platform's default. The opening application will have to arrange to know what that encoding may be, or ask its user. Moreover, many contexts and APIs which formerly dealt with "strings" now deal in terms of "byte strings", which cannot be treated as text directly--they print differently, do not support formatting, and don't mix with text strings at all.
As a concrete example, the PyEdit text editor example in this book must take Unicode into account when opening and saving files; when displaying text in a GUI; and when searching files in directories. For opens, it must ask the user for an encoding (suggesting the platform default) if one is not provided by the client application. In addition, it must rely on the GUI toolkit's own support for Unicode text, and its new threaded "grep" directory search tool must also ask for an encoding to apply to all files in the tree before it begins, and skip files which fail to decode--it's not impossible that text files in the tree may be encoded in a variety of formats.
In fact, it's common on Windows to have files in ASCII, UTF-8, and UTF-16 mixed in the same tree (Notepad's "ANSI", "Utf-8", and "Unicode"), and even others in trees that contain content obtained from the Web or email. Opening all these with UTF-8 would trigger exceptions in Python 3.X, and opening all these in binary mode yields encoded text that will likely fail to match a search key string. Technically, to compare at all, we'd still have to decode the bytes read to text, or encode the search key string to bytes, and the two would only match if the encodings used both succeed and agree.
Really, opening in binary mode to read raw byte strings in 3.X mimics the behavior of text files in 2.X, and underscores why forcing programs to deal with Unicode is sometimes a good thing--binary mode avoids decoding exceptions, but probably shouldn't, because the still-encoded text might not work as expected. The names of files searched might fail to decode as well, and the book's PyMailGUI email client is even more Unicode dependent (text parts and headers of messages may all be subject to both MIME and Unicode encodings, and full text must also be decoded for parsing today), but I'll leave this to the book to explain.
Paradoxically, Python 3.X's own libraries are still coming to grips with the implications of Unicode as well, especially in the Internet sector. For instance, the email package in Python 3.1 has significant issues (really, bugs) introduced by 3.X's str/bytes split. By proxy, these issues also impact CGI uploads and perhaps more (in short, the parser still lives in the land of str, but data streams can have mixed text/bytes content).
Some of this is being addressed--Python 3.2 will include a set of patches for the email package to fix many such issues, and a full email package rewrite is in the works, though it may not be entirely backward compatible with the package in 3.1. Other dark Unicode corners lurking in the standard library, however, may require some further honing with time.
There are still some domains in which Unicode can be safely ignored, of course. In numeric programming, for example, Python is typically used for lightweight scripting of nontext libraries, not for full-scale systems development or general programming tasks. However, especially if your work touches files, networks, the Internet, GUIs, databases, or text processing at large, Unicode is probably no longer an optional topic for you in Python 3.X. At least arguably, it shouldn't have been optional before either. Indeed, this book's own email client example was simpler in the prior edition largely because it was too narrow--it handled ASCII messages only, not the full spectrum of Internet message content.
Still, by forcing the issue Python 3.X makes Unicode optional to far fewer than it was in Python 2.X. If you're relatively new to I18N, you should be prepared for a learning curve when you take the leap to 3.X. Given the prominence of Unicode in the software world today, though, you'll probably be glad you did.
Update, October 2012: 3.3 has been released. For more on 3.3 changes, see also the Python 3.3 notes on the Programming Python 4E updates page, and the separate review of 3.3's new Windows launcher not described here.
Update, March 2012: 3.3 is now available in early alpha form, from python.org.
As I write this in November 2011, Python's current 2.X and 3.X releases are 2.7 and 3.2. The next release in the 3.X line, 3.3, is in early development but at present is not scheduled to be released until August 2012. While it's too early to chronicle this next release's changes, a few details are already starting to emerge:
There will surely be more 3.3 news over time, and I may or may not keep this list current. For the full story on 3.3 plans, consult the Python developers resources at python.org, or watch the official story unfold in the 3.3 "What's New" document at docs.python.org/dev/whatsnew/3.3.html.
Below is a (slighly edited) reply to a reader's email about scripts that seemed buggy in both Learning Python and Programming Python 4th Editions. The short story is that the scripts are correct as is, but Python's built-in input() function is broken in Python 3.2.0 (a.k.a. 3.2) when used in a Windows shell console window. This is the only version and context in which input() is broken. It works as it should in all other Python versions; in 3.2.0 outside Windows; and in 3.2.0 on Windows in other interfaces, including the IDLE GUI.
This function is a fundamental Python tool for reading console input, and is widely used by interactive scripts run in shell or command-line contexts. It worked correctly in all Pythons prior to 3.2.0, and has been fixed in later 3.2.X versions since 3.2.1, so this issue's scope is very limited. I never noticed it myself, because I either used 2.X in classrooms (where this function is named raw_input()), or ran 3.2.0 within the IDLE GUI where the problem does not appear. Since this may bite some readers, though, I'm posting the reply here for more details:
...
Apparently, the input() function is broken in Python 3.2.0 (only),
-- it mistakenly returns an extra and bogus '\r' carriage-return
character on Windows in console window mode. The net effect is
to break many console-based scripts run on Windows.
Here's the correct behavior in every other version of Python:
c:\temp>c:\python31\python
Python 3.1.1 (r311:74483, Aug 17 2009, 17:02:12) [MSC v.1500 32 bit (Intel)] on win32
>>> x = input('name:')
name:bob
>>> x
'bob'
And the broken behavior in 3.2.0:
c:\temp>c:\python32\python
Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit (Intel)] on win32
>>> x = input('name:')
name:bob
>>> x
'bob\r'
This will manifest itself in mangled output in any script that prints
input() results in 3.2 when run in a Windows shell, including the PP4E
script you mentioned -- the '\r' in a string returned by input() resets
output to start of line, thereby truncating/erasing text printed earlier.
It will also generally lead to incorrect behavior -- a comparison of the
input() result to 'stop' to break out of a loop, for instance, will never
be True, as the result string is actually 'stop\r'.
TO FIX: either use the IDLE GUI shell window, where this problem
does not appear; use Python 3.1 or 2.X (PP4E uses the former, LP4E uses both);
or download and install the latest version of Python 3.2 -- per the following
bug report, this only appears in 3.2.0 (which was initially called just 3.2),
and is fixed in 3.2.1 and later:
http://bugs.python.org/issue11272
You can also run the input() result through the string rstrip() method
to drop the '\r' (e.g., response.rstrip()), but that seems overkill for
a temporary regression in just one version of Python.
The books' code you mentioned does run correctly and without change on
the test machines and Python versions described in the book, and was
tested exhaustively on these configurations. This is a Python 3.2.0
regression and critical bug which appeared after the book was
published, which I have no control over, which was resolved in 3.2.1,
and which, frankly, frustrates me as much as anyone. Still, it may
merit a post on my errata pages for users who have this specific,
broken version of Python installed.
If this does not resolve your issue, please follow-up with further
details.
Thanks,
--Mark Lutz (http://learning-python.com)
> -----Original Message-----
> From: ...
> To: lutz@learning-python.com
> Subject: PP4E glitch == LP4E code error
> Date: Sat, 17 Dec 2011 12:56:59 -0800
> ...
The behavior of the struct.pack binary data packer tool has changed in Python 3.2 with respect to strings and the "s" type code. It no longer accepts normal str Unicode text strings for this type code, and now allows only bytes, forcing manual encoding of str strings.
This Python 3.2 change impacts examples in the current 4th Editions of all 3 of my books: Learning Python and Programming Python primarily, and one small example in Python Pocket Reference. The books' code still works as shown in the versions of Python which they claim to use (3.0 and 3.1). While the books aren't responsible for changes in Python after their publication, this one exemplifies Python change in general, and warrants a note here.
The original behavior of struct.pack in 3.0 and 3.1, the versions used in the books, allows normal strings, and encodes per UTF8 — an existing behavior which is documented clearly and explicitly in Python 3.1's manuals:
c:\...> c:\python31\python
>>> import struct
>>> data = struct.pack('>i4shf', 2, 'spam', 3, 1.234)
>>> data
b'\x00\x00\x00\x02spam\x00\x03?\x9d\xf3\xb6'
The new behavior in Python 3.2 no longer allows str
strings for the "s" code, but requires bytes strings:
c:\...> c:\python32\python
>>> import struct
>>> data = struct.pack('>i4shf', 2, 'spam', 3, 1.234)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
struct.error: argument for 's' must be a bytes object
This means that str text must be encoded to byte strings manually in 3.2 as
needed — unless you already have a bytes object, you now must always
encode str with bytes() or str.encode(), passing an encoding
name always to the former, and usually to the latter unless your platform's
default sufffices (it's UTF8 on Windows):
c:\...> c:\python32\python
>>> import struct
>>> data = struct.pack('>i4shf', 2, b'spam', 3, 1.234)
>>> data
b'\x00\x00\x00\x02spam\x00\x03?\x9d\xf3\xb6'
>>> data = struct.pack('>i4shf', 2, bytes('spam', 'utf8'), 3, 1.234)
>>> data
b'\x00\x00\x00\x02spam\x00\x03?\x9d\xf3\xb6'
>>> data = struct.pack('>i4shf', 2, 'spam'.encode(), 3, 1.234)
>>> data
b'\x00\x00\x00\x02spam\x00\x03?\x9d\xf3\xb6'
You can read about this change on Python's PEP list:
This tool worked with both str and bytes in 3.1 and 3.0, so this was a case of taking away existing functionality, not fixing a bug — only the second of the following 3.1 usage modes is supported as of 3.2:
c:\...> c:\python31\python
>>> import struct
>>> data = struct.pack('>i4shf', 2, 'spam', 3, 1.234) # removed in 3.2
>>> data
b'\x00\x00\x00\x02spam\x00\x03?\x9d\xf3\xb6'
>>> data = struct.pack('>i4shf', 2, 'spam'.encode(), 3, 1.234) # always worked
>>> data
b'\x00\x00\x00\x02spam\x00\x03?\x9d\xf3\xb6'
This change removes what seemed to some to be an inconsistent automatic
encoding (though a very similar encoding is done for DBM file str keys).
Naturally, though, it comes at the price of breaking existing code,
including book examples used by many as learning tools.
From a large perspective, this sort of non-fix change is common in Python, even long after 3.0's season of arbitrary incompatibility. For better or worse, Python change is often motivated not by actual bugs but by the aesthetic preferences of developers, and often at the expense of end users.
For a similar recent example of developer preferences seeming to rule the day, see the decision to move cgi.escape — a tool very widely used since the mid 90's — to the html module package, renaming it to html.escape to reflect its more general roles beyond CGI scripts:
The original cgi.escape was supposed to issue a warning in 3.2 and be removed altogether in 3.3 (though it's not clear if this plan is being implemented in full). Once executed, this change will impact every CGI scripting program in the book Programming Python, among many others. Minimally, code will need to be changed to import from the different module.In this case, it's not clear to this observer that the aesthetic improvement justifies breaking so much Python web scripting code that has worked for so long and for so many. Incompatible changes initiated by developers in the absence of user input would not be acceptable in commercial compiler development; why should they be so common in open source projects?
But at this point, I've already crossed further into the realm of subjective opinion than I meant to in a book update post; the arguments are too complex to explore here. If you're interested in reading more on the subject of open source development models and politics, I say more on this topic in the second question posed in this article.
There appears to be a bug in Python 3.2 related to the string formatting method for some relatively obscure object types, which impacts just one partial example in the book on Page 768, but may affect Python tools generally. This bug is not present in Python 2.6, 2.7, 3.0, or 3.1. The book itself uses 2.6 and 3.0 primarily and largely applies to 3.1 and 2.7, and its examples surrounding Page 768 all work correctly and as shown on all four of these Pythons. However, one example now fails under 3.2, developed after publication. The simplest fix for this code is to use the "%" expression instead of str.format(). More broadly, this serves as a fair example of both Python change, and debugging in general.
In Python 3.2, some objects fail in the str.format() method, presumably because they do not inherit or correctly use the top-level "object" class's default __format__ method. These objects work correctly in str.format() in 3.1, 2.7, and earlier. Per the reader who reported this issue, they also work in 3.2 with equivalent "%" string formatting expressions, which use a different internal implementation -- perhaps another reason to prefer the expression to the method in some contexts.
In the book, Page 768 notes how that the ListTree recursive class tree attribute lister's __str__ avoids "__X__" built-in names' values to save space. It also mentions that you can see such values by commenting out the test for their names, shows part of their display in Python 2.X, and states the 3.0 shows more such names. Here are the suggested code changes:
### File: testmixin.py
from lister import * # Get lister tool classes
...
class Sub(Super, ListTree): # Mix in ham and a __str__
### File lister.py
class ListTree:
...
def __attrnames(self, obj, indent):
spaces = ' ' * (indent + 4)
result = ''
for attr in sorted(obj.__dict__):
result += spaces + '{0}={1}\n'.format(attr, getattr(obj, attr))
"""
if attr.startswith('__') and attr.endswith('__'):
result += spaces + '{0}=<>\n'.format(attr)
else:
result += spaces + '{0}={1}\n'.format(attr, getattr(obj, attr))
"""
return result
This modified code works correctly under Python 2.7 and 3.1 as shown and described in the book, but fails on 3.2 today (technically, I'm running the third of these on 3.2.0):
C:\...> c:\python27\python testmixin.py
<Instance of Sub, address 25886312:
_ListTree__visited={}
...works...
C:...> c:\python31\python testmixin.py
<Instance of Sub, address 25035120:
_ListTree__visited={}
...works...
C:\...> c:\python32\python testmixin.py
Traceback (most recent call last):
File "testmixin.py", line 21, in <module>
print(X) # Run mixed-in __str__
File "C:\Users\mark\temp\lister.py", line 18, in __str__
self.__listclass(self.__class__, 4))
File "C:\Users\mark\temp\lister.py", line 35, in __listclass
''.join(genabove),
File "C:\Users\mark\temp\lister.py", line 29, in <genexpr>
genabove = (self.__listclass(c, indent+4) for c in aClass.__bases__)
File "C:\Users\mark\temp\lister.py", line 35, in __listclass
''.join(genabove),
File "C:\Users\mark\temp\lister.py", line 29, in <genexpr>
genabove = (self.__listclass(c, indent+4) for c in aClass.__bases__)
File "C:\Users\mark\temp\lister.py", line 34, in __listclass
self.__attrnames(aClass, indent),
File "C:\Users\mark\temp\lister.py", line 42, in __attrnames
result += spaces + '{0}={1}\n'.format(attr, getattr(obj, attr))
TypeError: Type method_descriptor doesn't define __format__
Either use "%" instead of str.format() in 3.2 for this code; use 3.1 instead of 3.2; or make sure the issue is reported in the bug or PEP lists of the python-dev group at python.org (see the prior note), and watch for a fix in a later Python release. The simplest fix, using the "%" formatting expression instead of str.format(), would look like the following, which works in 3.2 and all the other versions mentioned above:
### File lister.py
class ListTree:
...
def __attrnames(self, obj, indent):
....
for attr in sorted(obj.__dict__):
#result += spaces + '{0}={1}\n'.format(attr, getattr(obj, attr))
result += spaces + '%s=%s\n' % (attr, getattr(obj, attr))
....
C:\...> c:\python32\python testmixin.py
<Instance of Sub, address 28722928:
_ListTree__visited={}
...works...
It's possible that the root cause or rationale for this issue is more subtle than described here (and I'll leave searching python-dev resources as suggested exercise), but it seems invalid for any object to not have a default __format__ method in Python 3.X. In any event, this is a regression from 3.1 working code.
After digging into this a few minutes more, I've isolated the objects which trigger the errors in 3.2. To experiment yourself, change the lister's code to catch the 3.2 exceptions:
class ListTree:
...
def __attrnames(self, obj, indent):
....
for attr in sorted(obj.__dict__):
try:
result += spaces + '{}={}\n'.format(attr, getattr(obj, attr))
except:
result += spaces + '**BAD**: ' + attr + '=' + str(getattr(obj, attr)) + '\n'
....
When run now, the new errors in 3.2 occur in the top-level "object" class's
display in the middle of the tree sketch:
............<Class object, address 505269744:
__class__=<class 'type'>
__delattr__=<slot wrapper '__delattr__' of 'object' objects>
__doc__=The most base type
__eq__=<slot wrapper '__eq__' of 'object' objects>
**BAD**: __format__=<method '__format__' of 'object' objects>
__ge__=<slot wrapper '__ge__' of 'object' objects>
__getattribute__=<slot wrapper '__getattribute__' of 'object' objects>
__gt__=<slot wrapper '__gt__' of 'object' objects>
__hash__=<slot wrapper '__hash__' of 'object' objects>
__init__=<slot wrapper '__init__' of 'object' objects>
__le__=<slot wrapper '__le__' of 'object' objects>
__lt__=<slot wrapper '__lt__' of 'object' objects>
__ne__=<slot wrapper '__ne__' of 'object' objects>
__new__=<built-in method __new__ of type object at 0x1E1DCDF0>
**BAD**: __reduce__=<method '__reduce__' of 'object' objects>
**BAD**: __reduce_ex__=<method '__reduce_ex__' of 'object' objects>
__repr__=<slot wrapper '__repr__' of 'object' objects>
__setattr__=<slot wrapper '__setattr__' of 'object' objects>
**BAD**: __sizeof__=<method '__sizeof__' of 'object' objects>
__str__=<slot wrapper '__str__' of 'object' objects>
__subclasshook__=<built-in method __subclasshook__ of type object at0x1E1DCDF0>
............>
Given that, the 3.2 bug can be recreated in simpler terms interactively:
C:\...> c:\python31\python
>>> '{}'.format(object.__reduce__)
"<method '__reduce__' of 'object' objects>"
>>> '{}'.format(object.__sizeof__)
"<method '__sizeof__' of 'object' objects>"
C:\...> c:\python32\python
>>> '{}'.format(object.__reduce__)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Type method_descriptor doesn't define __format__
>>> '{}'.format(object.__sizeof__)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Type method_descriptor doesn't define __format__
At this point, the tale probably leads to the code of Python itself, but I'll
leave further forensics up to the interested reader. For a first entry point
into PEPs to explore, see the following -- a possibly related change to default
object.__format__ behavior, but scheduled for 3.3, and supposedly not
applicable to empty format strings like that in this code:
A reader wrote recently asking about using this book with newer Python releases than those it was based upon (technically, it's based on 3.1/3.0, and 2.6). In general, Python evolves slowly and the book applies to all 2.X and 3.X readers as is. Most readers should expect to browse the "What's New?" documents at some point to brush up on specific items that may have changed, but the fundamentals covered in the book are largely immutable.
Since most of my reply on this may be of general interest, though, and since it summarizes resources for exploring recent Python changes here and elsewhere, below are its important bits:
... The changes since the version of Python this book was based on (3.0, and most of 3.1) have been fairly minor, due in part to a temporary changes moratorium in Python. Really, 3.2 is almost entirely the same language as 3.0; most of the changes between the two appeared in libraries or implementation, not in the language itself, the main topic of this book. That's kept the book largely current. You can still use it as is, even though it's based on 3.0/3.1, as long as you supplement it with information on recent changes. To read about recent Python changes, either see the notes at my book updates pages: -(2.7) lp4e-updates-notes-recent.html#s1 -(3.2) lp4e-updates-notes-recent.html#s1b -(3.2) pp4e-updates.html#notes32 -(3.3) lp4e-updates-notes-recent.html#py33 -(etc) python-changes-2014-plus.html Or read the "What's New?" documents shipped with Python's standard manuals set and available online: -(2.7) http://docs.python.org/whatsnew/2.7.html -(3.1) http://docs.python.org/py3k/whatsnew/3.1.html -(3.2) http://docs.python.org/py3k/whatsnew/3.2.html -(3.3) http://docs.python.org/dev/whatsnew/3.3.html ...
A couple recent reviews of this book that I've been informed of:
I stopped reading book reviews years ago, but still get quite a few comments by email. Over the years, a lot of people have said that they enjoyed my books; some people have said that they didn't; and the vast majority has taken the books for what they are and learned something from them, without feeling compelled to saying anything.
The latter of these groups has purchased 300,000 copies of the books so far, making it roughly three orders of magnitude larger than the number of people who have posted opinions about them. To put that another way, for every 1 comment you might read about my books on a site like amazon.com, there are 1,000 opinions you are not getting. That's 99.9% of the customer base.
Like most authors, I always appreciate reader comments, whether by email, web post, or formal review. Lack of feedback is a much worse omen for writers than poor feedback. For better or worse, though, many opinions on the Internet tend to be skewed towards the subset of people with spare time to post them, and even further towards those in this small subset whose opinions or experiences are negative, and often extreme. The inevitable result is that much of today's Web neither includes nor reflects the opinions of the vast silent majority, and sorely lacks voices of authority.
Books are a personal and subjective experience. I encourage readers to explore them and draw their own conclusions regardless of the opinions of others posted online, both positive and negative.
I recently gave a written interview for the PSF's first newsletter, to be distributed at PyCon 2010. It discusses the new books, Python 3.X, and the software field at large. The newsletter included an abbreviated version, but the full interview is available online at this page.
[Feb 2012] I wrote a more recent article on Python's evolution: Answer me these questions three....
[Preface: I wrote the following almost two years before the "How long will it take to learn Python?" note above; some of it may seem redundant now, but it approaches the topic from a slightly different angle.]
I trust that most readers will understand that the earlier notes about this book's size on this site are intended to be informational, not apologetic. Python is a complex subject, and has grown more so in recent years, and learning to program in Python well requires substantial time and effort. The size of this book reflects that requirement, as well as the common needs of the thousands of students I've taught over the last 15 years.
I don't want to discourage beginners, but while programming can be highly rewarding, it is also much more demanding than many newcomers expect. True competence is a multiyear process, not a two-hour skim. If you're not willing to put in the required time and effort to learn to program well, you probably won't be happy in the software development field in general. On the other hand, if you do invest the effort required to master a tool like Python with an in-depth book like this one, you'll likely find that you are better prepared than many of this field's current practitioners.
There is a broader perspective worth noting here too. Many computer book publishers seem to have been competing in recent years to dumb-down their titles, in order to appeal to a perceived new market demographic. The examples are readily available—books for "Dummies;" books that promise mastery of complex topics "in 24 Hours;" a "Complete Idiot's" guide to just about everything under the sun; even "Manga Guides" to databases, statistics, and physics (no, really).
My own publisher is guilty as much as any with its "Head First" series: books that by design come with all the distraction of a web page, and sometimes seem heavier on cartoons than technical content. O'Reilly's marketing for Head First Programming, for example (a Python-based book which might be very good, despite some of its PR) tries to woo readers by conceding that "your time is too valuable to waste struggling with new concepts", and promising that you won't be asked to read text that "puts you to sleep." (See the full description here; these parts were also later cut-and-paste into the Head First Python page.)
Personally, I find such statements to be horrifying. Books are supposed to elevate their readers, not pander to them. By lowering the intellectual bar this way, we leave readers ill-equipped for the real world, and virtually guarantee the demise of quality in the software development field at large. The larger dumbed-down trend that such statements reflect seems cynical at best; at worst, it underscores a systemic failure of the gatekeepers of human culture and knowledge. Significant achievements require significant effort. Implying otherwise does a great disservice to everyone involved, and just might help to imperil a generation that's already been oversold the idea that focus is optional to success.
Your mileage may vary, of course; there are many types of learners, and no one book can satisfy every possible audience. Indeed, some of the books mentioned earlier might seem a lot more palatable if they were marketed as books for children and teenagers—a legitimate audience in its own right. Apart from such niches, though, there is a core philosophical value at stake here: depth should never be sacrificed in the name of shortcuts to proficiency that serve neither readers nor disciplines. Both parties deserve better from authors of technical books, regardless of the impact on page-count.
Entertainment matters in technical books too (Python is named for a comedy group, after all), but it shouldn't be elevated to the extent that it trumps education. If spending 15 years teaching Python to thousands of students has taught me anything, it's that what beginners need is not necessarily the same as what they want. Complex subjects are complex. It helps no one to sugar-coat them so much as to hide their true nature. That position may not open vast new markets for technical books, of course, but maybe that shouldn't have been the goal in the first place.
For a related opinion, check out the essay by Google's Director of Research, titled "Teach Yourself Programming in Ten Years." Not to diminish the potential consequences of this trend, but it's also encouraging to note that recent reader feedback sides much more with depth than speed. Maybe publishers should give programmers a bit more credit than they sometimes do.
For another case in point on this thread, see the documents pertaining to an initiative promoted by the early Python world in the 1990s known as CP4E—Computer Programming for Everybody. I stumbled onto this again just recently myself. It's a dated but interesting and relevant read, and captures some of the mindset of its time.
In retrospect, this initiative's entire premise seems flawed. Programming is not for everybody, and suggesting that it is does harm both to the field, and to those who hope to enter it. Depending on how you choose to define it, simple scripting tasks may be within the reach of many, but in most cases this qualifies as full-scale programming no more than inflating a car's tires qualifies as automotive engineering. Software development is a professional technical field. It's not a field that can be opened up to novice masses any more than is medicine, law, or physics.
This doesn't mean that beginners are not welcome; software is a very rewarding field for those who are willing to take time to learn it well. But there's a tangible misperception today that programming—unlike any other engineering discipline—is somehow simple enough to pick up overnight. This is arguably absurd, and seems embarrassingly naive in hindsight. As anyone who has actually worked in this field could attest, it's also just plain wrong. Programming is challenging, time-consuming, and based upon complex underlying concepts that take years to master. It may indeed be within the grasp of many, but not without much more time, effort, and motivation than we somehow seem to have become comfortable implying.
And if you're still not buying this, the next time you're in your doctor's office, imagine how you'd react to finding "Surgery for Dummies" or "Teach Yourself Medicine in 24 Hours" on the bookshelf. Software engineering may be a less meaty endeavor, but the same principles apply.
From a broader perspective, it can be argued that the rise of the distraction ecosystem that is today's Internet has been a major factor behind the dumbed-down trend in technical publishing. If you're interested in considering the deeper social implications of the world which us computer folk have wrought, I suggest the following books (among many others), which address the topic in much better depth than I can attempt here:
As documented in these and elsewhere, and despite the fully ungrounded claims of some book marketeers, the actual research done to date shows definitively that distraction does not aid learning; it radically impairs it.
A reader who had just purchased a copy of the 3rd Edition of this book wrote to ask what he would be missing in the 500 additional pages present in the 4th Edition (which grew from roughly 700 to 1200 pages). Given the prevalence of the 3rd Edition, this seems like a general question worth addressing here.
First off, if you're using Python 2.5 or older, and are sure you will be for some time to come, the 3rd Edition is probably okay for getting started. If you care about either Python 2.6 or 3.X, though, or might in the near future, the 4th Edition is a better choice.
The 4th Edition has been updated to cover both Python 2.X and 3.X (technically, 2.6 and 3.1). It is designed to serve readers using either Python line, and be a resource for those transitioning between them. As I mention in this Edition's Preface, the 2.X/3.X split could very well be permanent, given Python's large existing user base (in a world where a company like Google still uses Python 2.4 internally, 2.X could very well endure as long as Fortran77 has). I didn't want this book to leave either group out in the cold.
If you are too new to Python to know which version you must care about, you probably want to start with Python 3.X and the 4th edition of this book, unless you know that you'll be using third-party software that is available for 2.X only. Many such packages still support 2.X only today (including numeric programming libraries and popular web frameworks), but this is changing, and is expected to improve more over time. Python 3.X is the most likely future of Python, despite its current lack of dominance. Even if you are stuck in the 2.X world today, though, the 4th Edition's dual 2.X/3.X coverage allows you to hedge your bets for the future.
As for specific additions, some of this is difficult to reconstruct on a page count basis. Books evolve organically while they are being written, and there isn't a 1-to-1 relationship between pages in the 3rd and 4th Editions. But in terms of what I can quantify, the 500 extra pages in the 4th Edition break down this way:
There are 5 new chapters, 4 of which are in a new "Advanced Topics" part at the end. Here are the chapters added:
The first of these appears in the classes/OOP part, and provides a new and much needed OO tutorial. The last 4 appear in the new advanced topics part which is 189 pages long. The decorators chapter's larger size owes to the fact that it includes some larger case studies that are more satisfying than many of the other examples in the book, which are generally small and narrowly focused (what some would call didactic).
All together, these 5 new chapters add 226 pages as printed. Among these, today I'd classify only the last 3 as truly "optional reading" for many people; the coverage of Unicode strings in Chapter 36 is more widely relevant, and the OO tutorial is core material. That means the 5 new chapters account for 82 pages of new fundamentals material, plus just 144 pages of potentially optional reading at the end.
As mentioned, the 5 new chapters account for 226 of the 500 new pages, roughly half. In other words, even without the new chapters, this book would have still grown to be 1,000 pages long in the new 4th edition (and a bit longer if just 144 new pages are truly optional reading).
The remainder of the added size mostly stems from coverage of new or changed features in Python 2.6 and 3.X, along with the need to cover two incompatible versions of Python in one book. For an overview of what was addressed in the language changes/additions category, see the tables at the end of the draft Preface excerpt which I've posted here: http://learning-python.com/lp4e-preface-preview.html
Beyond specific language changes, covering two versions implies some extra size by itself. I discuss this further in relation to the book's size later on this page. In short, we opted to cover both Python lines instead of requiring readers to buy two books; until the Python world moves beyond its current 2.X/3.X split state, widely used Python books like this one don't really have many other good options.
For instance, Python 3 comes with changed syntax (e.g., raise, except, and print); new tools (the nonlocal statement, set and dictionary comprehensions, and extended sequence unpacking); and subtly different semantics (the new string model, the wider role of generators and iteration, and new-style classes). This is in addition to new tools added to both Python 2.6 and 3.X (class decorators, the new string format method, and so on). All these imply book growth, of course, especially when they must be presented in tandem with 2.X variations to a user community divided between Python lines.
Some minor growth owes to extended coverage for topics already present in the 3rd Edition. This includes new material and examples for tools such as operator overloading, new-style classes, decorators, and iterators and generators. However, most of the growth in this category stems either from new features in 2.6/3.X, or formerly obscure features that have grown to become common practice today.
Someone recently wrote to ask about the story behind the wood rat (no, it's not a mouse) on the cover of this book. I've been asked this many times over the years. Mostly, it's because O'Reilly didn't ask what our cover preference was when the first edition came out a decade ago.
Seriously, their rationale back in 1999 was that wood rats are a common food of pythons. This book would become a common tool of Python programmers, hence the tie-in was born. In retrospect, a parrot or bunny might have been better given the Monty Python namesake, but at the time this choice was left to professional graphic artists to decide, not to authors. I can't speak for other software developers, but given the state of my own drawing skills, this policy was probably for the best...
Older notes: see the first printing notes page
Back to this book's main updates page