Frankenthon!   

Python
Changes
2014+

Latest content revision: December 28, 2019

This page documents and critiques changes made to Python in 2014 and later, after the release of the book Learning Python, 5th Edition. As that book was updated for Pythons 3.3 and 2.7, and the 2.X line is effectively frozen, this page covers Pythons 3.4 and later. Earlier changes are documented in the book, but for brief summaries see its Appendix C, or this site's pages for Python 3.3, 3.2, and 2.7.

There's a wealth of content here for hungry Python readers, but if your time is tight and you're looking for suggested highlights, be sure to catch the intro; the coverage of new formatting tools in 3.6 and 3.5; the coding sagas here and here; and the essays on 3.5+ type hints and coroutines. For the full story, browse this page's contents:

 

Introduction: Why This Page?

Major Changes in Python 3.8 (Oct-2019)

  1. Assignment, Kludges, and Foo

Major Changes in Python 3.7 (Jun-2018)

  1. General Rehashings and Breakages
  2. StopIteration Busted in Generator Functions

Major Changes in Python 3.6 (Dec-2016)

  1. Yet Another String Formatting Scheme: f'...'
  2. Yet Another String Formatting Scheme: i'...'?
  3. Windows Launcher Hokey Pokey: Defaults
  4. Tk 8.6 Comes to Mac OS Python—DOA
  5. Coding Underscores in Numbers
  6. Etcetera: the Parade Marches On

Major Changes in Python 3.5 (Sep-2015)

  1. Matrix Multiplication: "@"
  2. Bytes String Formatting: "%"
  3. Unpacking "*" Generalizations
  4. Type Hints Standardization [essay]
  5. Coroutines: "async" and "await" [essay]
  6. Faster Directory Scans with "os.scandir()"?
  7. Dropping ".pyo" Bytecode Files
  8. Windows Install-Path Changes
  9. Tk 8.6 and tkinter: PNGs, Dialogs, Colors
  10. Tk 8.6 Regression: PyEdit Thread Crashes
  11. File Loads Seem Faster (TBD)
  12. Docs Broken on Windows, Incomplete
  13. Socket sendall() and smtplib: Timeouts
  14. Windows Installer Drops Support for XP

Major Changes in Python 3.4 (Mar-2014)

  1. New Package Installation Model: pip
  2. Unpacking "*" Generalizations? (to 3.5)
  3. Enumerated Type as a Library Module
  4. Import-Related Library Module Changes
  5. More Windows Launcher Changes, Caveats
  6. Etc: statistics, File Descriptors, asyncio,...

 

Introduction: Why This Page?

The 5th Edition of Learning Python published in mid-2013 has been updated to be current with Pythons 3.3 and 2.7. Especially given its language-foundations tutorial role, this book should address the needs of all Python 3.X and 2.X newcomers for many years to come.

Nevertheless, the inevitable parade of changes that seems inherent in open source projects continues unabated in each new Python release. Many such changes are trivial—and often optional—extensions which will likely see limited use, and may be safely ignored by newcomers until they become familiar with fundamentals that span all Pythons.

The Downside of Change

But not all changes are so benign; in fact, parades can be downright annoying when they disrupt your day. Those downstream from developer cabals have legitimate concerns. To some, many recent Python extensions seem features in search of use cases—new features considered clever by their advocates, but which have little clear relevance to real-world Python programs, and complicate the language unnecessarily. To others, recent Python changes are just plain rude—mutations that break working code with no more justification than personal preference or ego.

This is a substantial downside of Python's dynamic, community-driven development model, which is most glaring to those on the leading edge of new releases, and which the book addresses head-on, especially in its introduction and conclusion (Chapters 1 and 41). As told in the book, apart from the lucky few who are able to stick with a single version for all time, Python extensions and changes have a massive impact on the language's users and ecosystem. They must be:

While the language is still usable for a multitude of tasks, Python's rapid evolution adds additional management work to programmers' already-full plates, and often without clear cause.

Perhaps worst of all, newcomers face the full force of accumulated flux and growth in the latest and greatest release at the time of their induction. Today, the syllabus for new learners includes two disparate lines, with incompatibilities even among the releases of a single line; multiple programming paradigms, with tools advanced enough to challenge experts; and a torrent of feature redundancy, with 4 or 5 ways to achieve some goals—all fruits of Python's shifting story thus far.

In short, Python's constant change has created a software Tower of Babel, in which the very meaning of the language varies per release. This leaves its users with an ongoing task: even after you've mastered the language, new Python mutations become required reading for you if they show up in code you encounter or use, and can become a factor whenever you upgrade to Python versions in which they appear.

Consequently, this page briefly chronicles changes that appeared in Python after the 5th Edition's June 2013 release, as a sort of virtual appendix to the book. Hopefully, this and other resources named here will help readers follow the route of Python change—wherever the parade may march next.

The Value of Criticism

An editorial note up front: because changing a tool used by many comes with accountability, this page also critiques while documenting. Though subjective, its assessments are grounded in technical merit and fair, and reflect the perspective of someone who has watched Python evolve and been one of its foremost proponents since 1992, and who still wishes only the best for its future. Despite what you may have heard, informed criticism is both okay and crucial when its goal is improvement.

Programming language design is innately controversial, and you should weigh for yourself the potential benefits of each change noted here against its impacts on knowledge requirements and usability. At the end of the day, though, we can probably all agree that critical thinking on this front is in Python's best interest. The line between thrashing and evolution may be subjective, but drawing it carefully is as vital to the language's future as any shiny new feature can be.

Wherever you may stand on a given item below, this much is certain: a bloated system that is in a perpetual state of change will eventually be of more interest to its changers than its prospective users. If this page encourages its readers to think more deeply about such things while learning more about Python, it will have discharged its role in full.

 

Major Changes in Python 3.8 (October 2019)

Python 3.8 is now in beta. It's scheduled for release in October 2019, and it's documented in full by its What's New. As is now the norm, it also comes with the latest batch of language changes born of opinion-based thrashing. Indeed, this page is starting to sound like a broken record (and the changes are starting to sound like flamebait), but a brief review of this version is in order for readers of the book.

 

1. Assignment, Kludges, and Foo

Although a comprehensive review is beyond this page's scope, among the 3.8 lowlights are these:

In short, 3.8 is more of the accumulation-instead-of-design model of language evolution championed by the 3.X line. As usual, some will rush to use the new items in a strange effort to prove that they are somehow smarter than others—thereby sentencing their code to work on 3.8 and later only. As noted repeatedly on this page, you don't have to be one of them; these are optional extensions that most users are better off avoiding.

The pigheaded removal of the widely used time.clock, however, just seems sad. Rather than improving what was, it's been deleted altogether in favor of something different. As foreshadowed in the book, this means that very many users of Python 3.8 and later will have to change their code to use alternative tools in the time module that very few opinionated others have now mandated. Friendly not, that, but typical of the Python 3.X culture.

This writer wishes to reiterate that he still uses and enjoys Python regularly. It remains a great tool for programming in most or all application domains—even some that mutate just as carelessly as Python (alas, thrashing is both endemic and chronic in today's software world).

But this writer also doesn't use the pointless products of flux that are now cranked out annually, and doesn't recommend that his books' readers use these extensions either. Ego-driven churn may be an inherent downside of open-source projects, but it can also be ignored.

As always, the best advice is to avoid the extraneous new stuff, and stick to the very large subset of Python that has proved to be so valuable across decades and domains. Your code will be much more widely usable, and your coworkers will be much less inclined to grab the pitchforks.

 

Major Changes in Python 3.7 (June 2018)

As of February 2018, Python 3.7 is in beta and scheduled for release in June 2018. Per its What's New document, this looks to be a relatively minor update in terms of language change (e.g., __getattr__ now works in modules too, but it probably shouldn't, and you'll probably never care). As usual, though, this latest Python continues to rehash its "features," and breaks existing programs in the name of subjective "improvements." This section briefly summarizes both categories, and calls out one of their most crass members.

 

1. General Rehashings and Breakages

In the rehashings department, Python 3.7 is yet again changing its handling of Unicode locales and bytecode files—perennial sources of revisions in recent versions. It's also once more modifying or deprecating portions of its implementations of both type hints and coroutines—convoluted 3.X extensions that have been in a perpetual state of flux since they were introduced a few versions ago (see this and this for background).

In the breakages column, 3.7's changes are numerous and widespread, and require case-by-case analysis. As examples, its standard library changes to the subprocess module's stream defaults and the shuil.rmtree() function's error-handler arguments seem likely to impact programs like those this page's author has written in recent years. For such programs, revalidation and redistribution costs make a 3.7 upgrade impossible.

From the broader view, Python 3.7 is really just the latest installment of the constant churn that is the norm in the 3.X line. Because this is pointless (and just no fun) to document, this page's 3.7 coverage will stop short here. Readers are encouraged to browse Python's What's New documents for more details on 3.7 and later.

Better still, avoid the bleeding-edge's pitfalls altogether by writing code that does not depend on new releases and their ever-evolving feature sets. In truth, the latest Python version is never required to implement useful programs. After all, every Python program written over the last quarter century has managed to work well without Python 3.7's changes. Yours can too.

In the end, new does not necessarily mean better—despite the software field's obsession with change. Someday, perhaps, more programmers will recognize that volatility born of personal agenda is not a valid path forward in engineering domains. Until then, the path that your programs take is still yours to choose.

 

2. StopIteration Busted in Generator Functions

After the preceding summary was written, a reader's errata post in autumn 2019 revealed an insidious Python 3.7 change that merits retroactive and special coverage here. In short, 3.7 also included a change that needlessly and intentionally broke the behavior of exceptions in generator functions, and requires modifications to existing 3.X code. That behavior and its dubious alteration in 3.7 are both subtle, but one partial example in the book fell victim to the thrashing.

Details: In Pythons 3.0 through 3.6, a StopIteration raised anywhere in a generator function suffices to end the generator's value production, because that's just what an explicit or implicit return statement does in a generator. For example, in the following code from a sidebar on page 645 of the book, the next(i) inside the loop's list comprehension will trigger StopIteration when any iterator is exhausted, thereby implicitly terminating the generator function altogether—as intended—through Python 3.6:

>>> def myzip(*args):
...     iters = list(map(iter, args))           # make iters reiterable
...     while iters:                            # guarantee >=1, and force looping
...         res = [next(i) for i in iters]      # any empty? StopIteration == return
...         yield tuple(res)                    # else return result and suspend state                  
...                                             # exit: implied return => StopIteration
>>> list(myzip((1, 2, 3), (4, 5, 6)))
[(1, 4), (2, 5), (3, 6)]
>>> [x for x in myzip((1, 2, 3), (4, 5, 6))]
[(1, 4), (2, 5), (3, 6)]

If such code has to run on 3.7 and later, however, you'll need to change it to catch the StopIteration manually inside the generator, and transform it into an explicit return statement—which just implicitly raises StopIteration again:

>>> def myzip(*args):
...     iters = list(map(iter, args))
...     while iters:
...         try:
...             res = [next(i) for i in iters]
...         except StopIteration:               # StopIteration won't propagate in 3.7+
...             return                          # how generators should exit in 3.7+
...         yield tuple(res)                    # but exit still == return => StopIteration
... 
>>> list(myzip((1, 2, 3), (4, 5, 6)))
[(1, 4), (2, 5), (3, 6)]
>>> [x for x in myzip((1, 2, 3), (4, 5, 6))]
[(1, 4), (2, 5), (3, 6)]

If you don't change such code, the StopIteration raised inside the generator function is covertly changed to a RuntimeError as of 3.7, which won't terminate the generator nicely, but will pass as an uncaught exception causing code failures in most use cases. Here's the impact on the book example's original code when run in 3.7:

>>> list(myzip((1,2,3), (4,5,6)))
Traceback (most recent call last):
...
StopIteration

The above exception was the direct cause of the
following exception:

Traceback (most recent call last):
...
RuntimeError: generator raised StopIteration

Importantly, this change applies to both implicit and explicit uses of StopIteration: any generator function that internally triggers this exception in any way will now likely fail with a RuntimeError. This is a major change to the semantics of generator functions. In essence, such functions in 3.7 and later can no longer finish with a StopIteration, but must instead return or complete the function's code. The following variant, for instance, fails the same way in 3.7:

>>> def myzip(*args):
...     iters = list(map(iter, args))
...     while iters:
...         try:
...             res = [next(i) for i in iters]
...         except StopIteration:
...             raise StopIteration             # this also fails: return required
...         yield tuple(res)                    # even though return raises StopIteration!

And bizarrely, changing the first of the following lines in the code to the second now causes a different exception to be raised in 3.7: despite the semantic equivalence, you'll get a StopIteration for the first but a RuntimeError for the second, and may have to catch both in some contexts to accommodate the new special case:

...             res = [next(i) for i in iters]        # StopIteration
...             res = list(next(i) for i in iters)    # RuntimeError (!)

In other words, 3.7 swaps one subtle, implicit, and inconsistent behavior for another subtle, implicit, and inconsistent behavior. It's tough to imagine a better example of pointless churn in action.

Worse, the replacement knowingly breaks much existing and formerly valid 3.X code, and as usual reflects the whims of a small handful of people with time to chat about such things online. In this case, developers lamented the need to maintain backward compatibility—and then went ahead and broke it anyhow. As part of their reckless bargain, even code in Python's own standard library which relied on the former 3.6-and-earlier behavior had to be changed to run on 3.7.

Lesson: You can read more about this change at its PEP, find examples of programs it broke with a web search, and draw your own conclusions along the way. Regardless of your take, though, users of Python 3.X should clearly expect that deep-rooted language semantics may change out from under them arbitrarily; with minimal warning and even less user feedback; and according to the opinions of a few people who have no vested interest in your code's success. Buyer, be warned.

 

Major Changes in Python 3.6 (December 2016)