Frankenthon!   

Python
Changes
2014+

Latest content revision: November 18, 2020

This page documents and critiques changes made to Python in 2014 and later, after the release of the book Learning Python, 5th Edition. As that book was updated for Pythons 3.3 and 2.7, and the 2.X line is effectively frozen, this page covers Pythons 3.4 and later. Earlier changes are documented in the book, but for brief summaries see its Appendix C, or this site's pages for Python 3.3, 3.2, and 2.7.

There's a wealth of content here for hungry Python readers, but if your time is tight and you're looking for suggested highlights, be sure to catch the intro; the coverage of new formatting tools in 3.6 and 3.5; the coding sagas here and here; and the essays on 3.5+ type hints and coroutines. For the full story, browse this page's contents:

 

Introduction: Why This Page?

Major Changes in Python 3.9 (Oct-2020)

Major Changes in Python 3.8 (Oct-2019)

Major Changes in Python 3.7 (Jun-2018)

  1. General Rehashings and Breakages
  2. StopIteration Busted in Generator Functions

Major Changes in Python 3.6 (Dec-2016)

  1. Yet Another String Formatting Scheme: f'...'
  2. Yet Another String Formatting Scheme: i'...'?
  3. Windows Launcher Hokey Pokey: Defaults
  4. Tk 8.6 Comes to Mac OS Python—DOA
  5. Coding Underscores in Numbers
  6. Etcetera: the Parade Marches On

Major Changes in Python 3.5 (Sep-2015)

  1. Matrix Multiplication: "@"
  2. Bytes String Formatting: "%"
  3. Unpacking "*" Generalizations
  4. Type Hints Standardization [essay]
  5. Coroutines: "async" and "await" [essay]
  6. Faster Directory Scans with "os.scandir()"?
  7. Dropping ".pyo" Bytecode Files
  8. Windows Install-Path Changes
  9. Tk 8.6 and tkinter: PNGs, Dialogs, Colors
  10. Tk 8.6 Regression: PyEdit Thread Crashes
  11. File Loads Seem Faster (TBD)
  12. Docs Broken on Windows, Incomplete
  13. Socket sendall() and smtplib: Timeouts
  14. Windows Installer Drops Support for XP

Major Changes in Python 3.4 (Mar-2014)

  1. New Package Installation Model: pip
  2. Unpacking "*" Generalizations? (to 3.5)
  3. Enumerated Type as a Library Module
  4. Import-Related Library Module Changes
  5. More Windows Launcher Changes, Caveats
  6. Etc: statistics, File Descriptors, asyncio,...

 

Introduction: Why This Page?

The 5th Edition of Learning Python published in mid-2013 has been updated to be current with Pythons 3.3 and 2.7. Especially given its language-foundations tutorial role, this book should address the needs of all Python 3.X and 2.X newcomers for many years to come.

Nevertheless, the inevitable parade of changes that seems inherent in open source projects continues unabated in each new Python release. Many such changes are trivial—and often optional—extensions which will likely see limited use, and may be safely ignored by newcomers until they become familiar with fundamentals that span all Pythons.

The Downside of Change

But not all changes are so benign; in fact, parades can be downright annoying when they disrupt your day. Those downstream from developer cabals have legitimate concerns. To some, many recent Python extensions seem features in search of use cases—new features considered clever by their advocates, but which have little clear relevance to real-world Python programs, and complicate the language unnecessarily. To others, recent Python changes are just plain rude—mutations that break working code with no more justification than personal preference or ego.

This is a substantial downside of Python's dynamic, community-driven development model, which is most glaring to those on the leading edge of new releases, and which the book addresses head-on, especially in its introduction and conclusion (Chapters 1 and 41). As told in the book, apart from the lucky few who are able to stick with a single version for all time, Python extensions and changes have a massive impact on the language's users and ecosystem. They must be:

While the language is still usable for a multitude of tasks, Python's rapid evolution adds additional management work to programmers' already-full plates, and often without clear cause.

Perhaps worst of all, newcomers face the full force of accumulated flux and growth in the latest and greatest release at the time of their induction. Today, the syllabus for new learners includes two disparate lines, with incompatibilities even among the releases of a single line; multiple programming paradigms, with tools advanced enough to challenge experts; and a torrent of feature redundancy, with 4 or 5 ways to achieve some goals—all fruits of Python's shifting story thus far.

In short, Python's constant change has created a software Tower of Babel, in which the very meaning of the language varies per release. This leaves its users with an ongoing task: even after you've mastered the language, new Python mutations become required reading for you if they show up in code you encounter or use, and can become a factor whenever you upgrade to Python versions in which they appear.

Consequently, this page briefly chronicles changes that appeared in Python after the 5th Edition's June 2013 release, as a sort of virtual appendix to the book. Hopefully, this and other resources named here will help readers follow the route of Python change—wherever the parade may march next.

The Value of Criticism

An editorial note up front: because changing a tool used by many comes with accountability, this page also critiques while documenting. Though subjective, its assessments are grounded in technical merit and fair, and reflect the perspective of someone who has watched Python evolve and been one of its foremost proponents since 1992, and who still wishes only the best for its future. Despite what you may have heard, informed criticism is both okay and crucial when its goal is improvement.

Programming language design is innately controversial, and you should weigh for yourself the potential benefits of each change noted here against its impacts on knowledge requirements and usability. At the end of the day, though, we can probably all agree that critical thinking on this front is in Python's best interest. The line between thrashing and evolution may be subjective, but drawing it carefully is as vital to the language's future as any shiny new feature can be.

Wherever you may stand on a given item below, this much seems certain: a bloated system that is in a perpetual state of change will eventually be of more interest to its changers than its prospective users. If this page encourages its readers to think more deeply about such things while learning more about Python, it will have discharged its role in full.

 

Major Changes in Python 3.9 (October 2020)

Per its What's New, Python 3.9 is a minor release with a relatively small set of changes, which is about what you'd expect from a project that now issues a new release each year. It does, however, manage to inject the usual batch of superfluous language convolutions, and furthers the paradigm of thrashing-based—if not troll-driven—engineering. Among the morph:

Erasing the past
A new large-scale effort is afoot to remove the last vestiges of backward compatibility with decades of Python practice. The blurb at that link includes the odd suggestion that you should test in development mode, in order to accommodate incompatible changes in the next version of 3.X. It's official: breaking code in the present is no longer enough to satisfy Python 3.X's developers.
Type-declaration foo
There's additional support for type-declaration syntax in 3.X. You can now write Python code that looks just like that in your favorite statically typed language—and negate most of the reasons for using a dynamically typed language like Python in the first place. (Oh, snap?)
Dictionary "union"
Dictionaries have sprouted | and |= operators purportedly meant to add set union to dictionaries... though the second is just an in-place version of the first; this is just one of many set operations and isn't the same as union at all; and the mod just adds redundant and obscure functionality that is imagined to be better than other redundant and obscure functionality added to 3.X in the past.

Wisdom Deprecated

Perhaps most elaboration worthy among 3.9's changes is the addition of union-like | operators for dictionaries. In truth, this is really just a trivial key-based merge, with an operator syntax borrowed from sets. It's also about as irregular and redundant as it gets; it:

Takeaway: Python 3.X now thrashes so much that it eventually contradicts itself. In this case, there was pushback on multiple grounds, all of which were well reasoned and need no elaboration here—and as usual, were outshouted by the agenda of a few convoluters. The rejection of calls to avoid redundancy with appeals to tenets of the Python Zen, however, merits a callout. Quoting from the change's PEP:

More Than One Way To Do It

  Okay, the Zen doesn't say that there should be Only One Way To Do It. 
  But it does have a prohibition against allowing "more than one way to do it".

Response

  There is no such prohibition.  The "Zen of Python" merely expresses a preference 
  for "only one obvious way":

    There should be one-- and preferably only one --obvious way to do it. [...]

  In practice, this preference for "only one way" is frequently violated in Python. [...]

  We should not be too strict about rejecting useful functionality because it 
  violates "only one way".

And somewhere, angels surely must weep (and Perl folk surely must smirk). Feature bloat kills usability, and past mistakes do not justify new ones. More to the point: needlessly changing a tool used by millions may be rude, but knowingly rejecting past wisdom is juvenile.

Closing Words

This page hopes to continue sparking discourse on the perils of thrashing in engineering projects. But this will likely be its final entry. There will, of course, be a Python 3.10, and a 3.11, and a 3.12. And they will, of course, add redundant functionality, and break existing code, and impose the constantly morphing whims of the few on the many. Popularity implies neither quality nor courtesy, and candidly, there are better things to do in the last quadrant of life than document this stuff.

In the end, the convolution of Python was not a technical story. It was a sociological story, and a human story. If you create a work with qualities positive enough to make it popular, but also encourage it to be changed for a reward paid entirely in the dark currency of ego points, time will inevitably erode the very qualities which made the work popular originally. There's no known name for this law, but it happens nonetheless, and not just in Python. In fact, it's the very definition of open source, whose paradigm of reckless change now permeates the computing world.

Some of this law's consequences are less abstract than others. While Python developers were busy playing in their sandbox, web browsers mandated JavaScript, Android mandated Java, and iOS became so proprietary and closed that it holds almost no interest to generalist developers. Indeed, many of the open-source movement's original ideals of developer freedom are now perilously close to being dismissed as sacrifice on the altar of ego-based churn.

The sage should know when to stop. Python never did. This page does.

Never say never: almost immediately after the preceding review was posted, this site received feedback from readers hoping that this page's updates will continue. It's possible that this page may be back for 3.10 and beyond, though this depends both upon whether there's anything useful to add, and how well the painkillers work...

 

Major Changes in Python 3.8 (October 2019)

Python 3.8 is now in beta. It's scheduled for release in October 2019, and it's documented in full by its What's New. As is now the norm, it also comes with the latest batch of language changes born of opinion-based thrashing. In fact, this page is starting to sound like a broken record (and the changes are starting to sound like flamebait), but a brief review of this version is in order for readers of the book. Among 3.8's lowlights:

Assignment expression
A new and wholly superfluous assignment expression, (x := y), that millions of Python programmers somehow managed to make do without for nearly three decades. Typing x = y separately was never hard, but code with obscurely nested := assignments will almost certainly be brutal.
Function kludge
An ad-hoc syntax extension, def f(x, y=None, /), whose odd / both qualifies as kludge, and forces function arguments to be passed by position only—a wildly special-case role so important that it went unnoticed since the early 1990s (yes, sarcasm intended).
F-string kludge
Another ad-hoc syntax extension, f'{x*y + 15=}', whose weird = invokes an implicit and unexpected evaluation and format, and adds yet another special case to 3.6's f-strings extension described ahead... which was itself fully redundant 3.X morph that mushroomed Python's string-formatting story badly and needlessly. F-strings are now officially ad-hoc squared!
Bytecode folder
A new environment-variable setting, PYTHONPYCACHEPREFIX, which allows the location of the 3.X __pycache__ bytecode folder to vary per program, thereby breaking any tools that assume its formerly fixed location, and further convoluting the ever-changing 3.X module tale.
time.clock() dropped
And the rude removal of the time.clock function, a tool which has been widely used by Python programmers from day one, and whose deletion will break reams of Python code (including some benchmarking examples in the book, and other programs published online).

In short, 3.8 is mostly more of the accumulation-instead-of-design model of language evolution championed by the 3.X line. As usual, some will rush to use the new items in a strange effort to prove that they are somehow smarter than others—thereby sentencing their code to work on 3.8 and later only. As noted repeatedly on this page, you don't have to be one of them; these are optional extensions that most users are better off avoiding.

The Whims of the Few

The pigheaded removal of the widely used time.clock, however, just seems sad. Rather than improving a go-to tool that was supported for some 30 years, it's been deleted altogether in favor of something different. As foreshadowed in the book, this means that very many users of Python 3.8 and later will have to change their code to use alternative tools in the time module that very few opinionated others have now mandated. Friendly not, that, but typical of the Python 3.X culture.

It's easy to see this culture at work for yourself. Well-reasoned objections to the subjective removal of time.clock were indeed registered here, here, and elsewhere; but they were outshouted by the aesthetic preferences of a stubborn and myopic few, whose ego investment in the incompatible change clearly outweighed the consequences for other peoples' code. This is how open source does not work.

In this specific case, the book was able to give work-arounds for future time.clock deprecation (see New Timer Calls in 3.3 on page 655; in short, you must use time.perf_counter wherever time.clock is absent). In general, though, Python 3.X's rate of change far outpaces its learning resources (see the similar fate of imp.reload in 3.7 ahead), and neither documentation nor deprecation protocols qualify as justification; warning people that you're going to be rude doesn't make it okay to be rude.

Keeping It Simple

This writer wishes to reiterate that he still uses and enjoys Python regularly. It remains a great tool for programming in most or all application domains—even some that mutate just as carelessly as Python (per the link, thrashing is both endemic and chronic in today's software world).

But this writer also doesn't use the pointless products of flux that are now cranked out annually, and doesn't recommend that his books' readers use these extensions either. Ego-driven churn may be an inherent downside of open-source projects, but it can also be largely ignored. You can't do much about changes that break existing programs, of course (apart from releasing work-arounds or freezing support levels); optional language extensions, however, are optional.

As always, the best advice is to avoid the extraneous new stuff, and stick to the very large subset of Python that has proved to be so valuable across decades and domains. Your code will be much more widely usable, and your coworkers will be much less inclined to grab the pitchforks.

Postscript: during 3.8's tenure, python.org also ended its support for the still-widely-used Python 2.X—and inserted a rude denigration banner at the top of every page in 2.X users' docs. You can read expanded coverage of this unfriendly move here; alas, 2.X users seem no longer invited to the club.

 

Major Changes in Python 3.7 (June 2018)

As of February 2018, Python 3.7 is in beta and scheduled for release in June 2018. Per its What's New document, this looks to be a relatively minor update in terms of language change (e.g., __getattr__ now works in modules too, but it probably shouldn't, and you'll probably never care). As usual, though, this latest Python continues to rehash its "features," and breaks existing programs in the name of subjective "improvements." This section briefly summarizes both categories, and calls out one of their most crass members.

 

1. General Rehashings and Breakages

In the rehashings department, Python 3.7 is yet again changing its handling of Unicode locales and bytecode files—perennial sources of revisions in recent versions. It's also once more modifying or deprecating portions of its implementations of both type hints and coroutines—convoluted 3.X extensions that have been in a perpetual state of flux since they were introduced a few versions ago (see this and this for background).

In the breakages column, 3.7's changes are numerous and widespread, and require case-by-case analysis. As examples, its standard library changes to the subprocess module's stream defaults and the shuil.rmtree() function's error-handler arguments seem likely to impact programs like those this page's author has written in recent years. For such programs, revalidation and redistribution costs make a 3.7 upgrade impossible.

Reloads Break... Again

As a specific example of a breakage that's directly relevant to the book, Python 3.7 also now issues a deprecation warning when code imports the imp.reload module-reloading tool; you'll soon need to change the first of the following to the second everywhere:

$ python3
Python 3.7.4 (default, Jul 28 2019, 22:33:35)
>>> from imp import reload
__main__:1: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
>>> from importlib import reload
>>>
Sadly, this widely used built-in has now been senselessly moved twice in Python 3.X—from built-in function, to the imp module, and now to the importlib module—breaking all code that relies on formerly documented locations on each hop. This is careless in the extreme, and motivated by nothing more than the personal preferences of those with time to impose their opinions on others' programs.

It's Called Bleeding for a Reason

From the broader view, Python 3.7 is really just the latest installment of the constant churn that is the norm in the 3.X line. Because this is pointless (and just no fun) to document, this page's 3.7 coverage will stop short here. Readers are encouraged to browse Python's What's New documents for more details on 3.7 and later.

Better still, avoid the bleeding-edge's pitfalls altogether by writing code that does not depend on new releases and their ever-evolving feature sets. In truth, the latest Python version is never required to implement useful programs. After all, every Python program written over the last quarter century has managed to work well without Python 3.7's changes. Yours can too.

In the end, new does not necessarily mean better—despite the software field's obsession with change. Someday, perhaps, more programmers will recognize that volatility born of personal agenda is not a valid path forward in engineering domains. Until then, the path that your programs take is still yours to choose.

 

2. StopIteration Busted in Generator Functions

After the preceding summary was written, a reader's errata post in autumn 2019 revealed an insidious Python 3.7 change that merits retroactive and special coverage here. In short, 3.7 also included a change that needlessly and intentionally broke the behavior of exceptions in generator functions, and requires modifications to existing 3.X code. That behavior and its dubious alteration in 3.7 are both subtle, but one partial example in the book fell victim to the thrashing.

Details: In Pythons 3.0 through 3.6, a StopIteration raised anywhere in a generator function suffices to end the generator's value production, because that's just what an explicit or implicit return statement does in a generator. For example, in the following code from a sidebar on page 645 of the book, the next(i) inside the loop's list comprehension will trigger StopIteration when any iterator is exhausted, thereby implicitly terminating the generator function altogether—as intended—through Python 3.6:

>>> def myzip(*args):
...     iters = list(map(iter, args))           # make iters reiterable
...     while iters:                            # guarantee >=1, and force looping
...         res = [next(i) for i in iters]      # any empty? StopIteration == return
...         yield tuple(res)                    # else return result and suspend state                  
...                                             # exit: implied return => StopIteration
>>> list(myzip((1, 2, 3), (4, 5, 6)))
[(1, 4), (2, 5), (3, 6)]
>>> [x for x in myzip((1, 2, 3), (4, 5, 6))]
[(1, 4), (2, 5), (3, 6)]

If such code has to run on 3.7 and later, however, you'll need to change it to catch the StopIteration manually inside the generator, and transform it into an explicit return statement—which just implicitly raises StopIteration again:

>>> def myzip(*args):
...     iters = list(map(iter, args))
...     while iters:
...         try:
...             res = [next(i) for i in iters]
...         except StopIteration:               # StopIteration won't propagate in 3.7+
...             return                          # how generators should exit in 3.7+
...         yield tuple(res)                    # but exit still == return => StopIteration
... 
>>> list(myzip((1, 2, 3), (4, 5, 6)))
[(1, 4), (2, 5), (3, 6)]
>>> [x for x in myzip((1, 2, 3), (4, 5, 6))]
[(1, 4), (2, 5), (3, 6)]

If you don't change such code, the StopIteration raised inside the generator function is covertly changed to a RuntimeError as of 3.7, which won't terminate the generator nicely, but will pass as an uncaught exception causing code failures in most use cases. Here's the impact on the book example's original code when run in 3.7:

>>> list(myzip((1,2,3), (4,5,6)))
Traceback (most recent call last):
...
StopIteration

The above exception was the direct cause of the
following exception:

Traceback (most recent call last):
...
RuntimeError: generator raised StopIteration

Importantly, this change applies to both implicit and explicit uses of StopIteration: any generator function that internally triggers this exception in any way will now likely fail with a RuntimeError. This is a major change to the semantics of generator functions. In essence, such functions in 3.7 and later can no longer finish with a StopIteration, but must instead return or complete the function's code. The following variant, for instance, fails the same way in 3.7:

>>> def myzip(*args):
...     iters = list(map(iter, args))
...     while iters:
...         try:
...             res = [next(i) for i in iters]
...         except StopIteration:
...             raise StopIteration             # this also fails: return required
...         yield tuple(res)                    # even though return raises StopIteration!

And bizarrely, changing the first of the following lines in the code to the second now causes a different exception to be raised in 3.7: despite the semantic equivalence, you'll get a StopIteration for the first but a RuntimeError for the second, and may have to catch both in some contexts to accommodate the new special case:

...             res = [next(i) for i in iters]        # StopIteration
...             res = list(next(i) for i in iters)    # RuntimeError (!)

In other words, 3.7 swaps one subtle and implicit behavior for another subtle and implicit behavior—which also manages to be inconsistent. It's tough to imagine a better example of pointless churn in action.

Worse, the replacement knowingly breaks much existing and formerly valid 3.X code, and as usual reflects the whims of a small handful of people with time to chat about such things online. In this case, developers lamented the need to maintain backward compatibility—and then went ahead and broke it anyhow. As part of their reckless bargain, even code in Python's own standard library which relied on the former 3.6-and-earlier behavior had to be changed to run on 3.7.

Lesson: You can read more about this change at its PEP, find examples of programs it broke with a web search, and draw your own conclusions along the way. Regardless of your take, though, users of Python 3.X should clearly expect that deep-rooted language semantics may change out from under them arbitrarily; with minimal warning and even less user feedback; and according to the opinions of a few people who have no vested interest in your code's success.

Simply put, in the absence of version freezes, your Python programs will probably break eventually through no fault of your own. Buyer, be warned.

 

Major Changes in Python 3.6 (December 2016)