![[PP4E cover]](ora-pp4e-large.jpg){kind=small}
|
|
Running PP4E Examples on Python 3.3 and Later |
|---|
Last revised: June 2020
The book Programming Python, 4th Edition (a.k.a. PP4E) was published when Python 3.2 was current. This page summarizes the 4 minor steps required to run this book's major examples with Python versions 3.3 through 3.10 (and likely later), and documents the corresponding changes applied to the current release 1.4 of the book's examples package, created October 15, 2013. In brief:
The first 2 of these steps are needed for Windows only, and the first 3
are incorporated into release 1.4 of the book's examples package automatically. For instance,
with release 1.4 the PyGadgets and PyDemos auto-launchers, as well as the larger
stand-alone programs such as PyMailGUI, PyEdit, and PyCalc will
work on 3.3 through 3.10 out of the box. Still, you'll need to manually install Pillow instead
of PIL for image-processing examples such as PyPhoto, and may need to set your
PY_PYTHON on Windows for general 3.X use.
Note that all these changes are backward-compatible with earlier 3.X releases, so release 1.4 of the book's examples package works with Pythons 3.0 through 3.10, and likely later. Later Pythons may or may not require additional patches; watch for updates to be posted here as impact warrants and time allows.
Also note that most book examples will run unchanged under 3.3 through 3.10; the steps outlined here are mainly required for specific programs only, and mostly for larger graphical programs. That said, you shouldn't be surprised if some examples do require updates not listed here. Change is a constant in software development—an indirect but important lesson in itself.
Before we get to the four specific changes required, this section provides a brief log of developments that have impacted the book's examples, ordered most-recent first.
time.clock has been removed in 3.8, per
this post; replace the book's half-dozen
uses to call time.perf_counter
instead,
and watch for a similar upcoming change to imp.reload described
here.
PY_PYTHON settings (the first bullet above) optional
in some contexts. Unfortunately, this was more convolution than fix, and the
settings are still required in some cases—and always, for Pythons 3.0 through 3.5.
Read the full story here.
email package in the 3.3.3 point release
(only) had a temporary surrogates defect that broke sends of mail with non-ASCII text, but
was fixed in Python 3.3.4 and 3.4.0.
See this note for details.
mailconfig.py at path PP4E\Internet\Email\PyMailGui.
You must use your own servers and accounts for most email, FTP, and other client-side
Internet examples in the book.
For more on Python 3.3 itself, see this review.
As
noted here,
the new Windows launcher automatically installed with Python 3.3 runs scripts
and commands with an installed 2.X by default, unless they give a more specific
release number in #! patterns or command-argument options. This can effectively
break programs meant to be used with 3.X if they're launched with version-agnostic
techniques or have top-level script files with missing or release-ambiguous #! lines.
As it's common practice on Windows to omit a #! line formerly useful only in some
Unix usage modes, this can have broad impacts. Even on Unix, scripts with #! lines
that name a generic "python" resolved by file links are a norm, but may now fail if run by
Python 3.3 on Windows.
To work around this, you can either:
#!...python3 line to the top of every Python 3.X script you run on
Windows—a bizarre expectation that needlessly adds extra work, and
seems just plain rude to people who use Python on Windows.
PY_PYTHON environment variable to 3 to force the 3.3 launcher to
default to the highest-numbered 3.X globally for all programs that don't give a
version explicitly.
PY_PYTHON setting applies to all scripts run on your machine,
it doesn't help much for scripts meant to run immediately with no user configuration
steps—like some of those in the book. In lieu of PY_PYTHON settings, the changes
required in auto-launcher scripts are subtle, and depend on how the launched code is run:
#!...python3 line, so that the entire process is started with 3.X.
os.system(), os.spawnv(), etc.—the
launcher script may or may not require a #!...python3 line for its own code, but must set
PY_PYTHON=3 via os.environ for any version-generic spawned programs (os.environ
settings are inherited by subprocesses).
PY_PYTHON makes both of these changes unnecessary and is recommended
in general, but that obviously violates auto-launchers' goals.
To work even in the absence of PY_PYTHON settings,
release 1.4 of the examples package applies
the required changes to the book's four auto-launcher scripts, plus one utility module that they
leverage; the launchers see regular action as drug-out shortcuts on my Windows desktops (and were
starting points for later standalone releases):
| PyGadgets PP4E\Launch_PyGadgets_bar.pyw | Added #!/usr/bin/python3 to force 3.X execution of itself;
this script runs Launcher.py tools in-process via imports.
|
| PyDemos PP4E\Launch_PyDemos.pyw | Added #!/usr/bin/python3 to force 3.X execution of itself;
this script runs Launcher.py tools in-process via imports.
|
| PyEdit PP4E\Gui\TextEditor\pyedit.pyw | Added #!/usr/bin/python3 line to force 3.X execution of itself;
this script runs the main PyEdit script in-process with exec().
|
| PyMailGUI PP4E\Internet\Email\PyMailGui\ altconfigs\launch_PyMailGui.py | Added os.environ['PY_PYTHON']='3' to force 3.X execution of the PyMailGUI
subprocess spawned with os.system(); a #! line for this script's own execution
is irrelevant, as its code runs on both 2.X and 3.X.
|
| Utility PP4E\Launcher.py | Added both #!/usr/bin/python3 to force 3.X execution of itself (when run standalone),
and os.environ['PY_PYTHON']='3' to force 3.X execution of any subprocess it spawns (see below).
|
Notice the last entry in this table: Launcher.py sets PY_PYTHON=3 in os.environ,
to be inherited by spawned programs. This may be moot for its immediate subprocesses, because this
module runs explicit python ... command lines with os.spawnv() after setting
Python and system paths automatically, thereby cutting out the 3.3 py launcher in the Windows registry;
deeper program descendants, however, may still require this setting. Complex to be sure, but avoidable
if you're willing to require an extra environment setting for using existing 3.X code under 3.3
on Windows, a manual task PP4E's top-level launchers were designed to obviate.
Also keep in mind that this patch applies to the selected auto-launcher scripts only.
When using 3.3 and later, you'll still probably want to manually set your PY_PYTHON
environment variable eventually in order to run other version-agnostic 3.X scripts on Windows
with 3.X instead of 2.X. This includes any book examples not updated by this patch; see the
Advanced system settings in your Control Panel's System entry to set the version default
globally.
Although this patch suffices in the normal case, there are three issues that may impact your Windows launcher experience worth noting here:
It's not impossible that hardcoding a python3 in a #! line for use on Windows may not
be portable to some Unix systems, which would virtually force a manual PY_PYTHON=3
setting on Windows machines with 2.X installs. If this impacts you, you'll have to modify
the patched scripts as needed—set PY_PYTHON, and either use a generic python in #!
lines, or avoid #! lines altogether (per the next bullet, PATH can do similar work with
a python in #! on 3.4). All Unix portability bets are off, of course, if you use a
#! pattern not common on Unix; an abbreviated #!python3 may be a prime suspect.
The Windows launcher was modified in 3.4 to give a PATH variable setting
priority over searching the registry for the highest-numbered Python, when just a generic
python appears in a #! line. This better emulates the /usr/bin/env paradigm on Unix
(and may allow for more portable #! lines in general), but is apparently employed only when
a generic python is used in #! lines—not when #! lines are absent altogether,
the normal case on Windows, and an additional case addressed by PY_PYTHON.
For more background on this change, see Python issue reports:
here and
here.
For unknown reasons, Python 3.3+ Windows MSI installers
can sometimes fail to properly set filename association in the Windows registry for launcher use.
This should automatically associate .py and .pyw file types
(plus bytecode) with the launcher's py.exe and pyw.exe executables, respectively.
However, this has been seen to fail on 2 of 6 Windows machines tested—on 2 machines running Windows Vista and 7,
Python files are incorrectly associated with a former python.exe and pythonw.exe
after installing 3.3;
on 4 other machines running XP, Vista, 7, and 8, the associations are correctly set to the launcher's
programs instead. All these machines had prior Pythons installed, of various versions (3.2 and 2.6
where associations failed; 3.2, 3.1, and 2.5 where they worked).
Without the expected filename associations, the launcher is never invoked for scripts run without explicit py ...
command lines—scripts will instead run under any Python named in the registry, and will
happily ignore any #! lines completely, despite 3.3's best intentions. This was noticeable when 3.2 was
still inexplicably running scripts after a 3.3 install, despite contradictory #! directives; more generally,
you can detect this problem in simpler terms by running a script from the command line as in the following, shown here
on a system with incorrect associations:
c:\PP4E> type t.py #!/usr/bin/python3.3 import sys print(sys.version) c:\PP4E> t.py 3.2.2 (default, Sep 4 2011, 09:07:29) [MSC v.1500 64 bit (AMD64)] c:\PP4E> py t.py 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:06:53) [MSC v.1600 64 bit (AMD64)]
On another failer, this turns out even worse (the middle command here also emulates the effect of a filename icon click):
C:\PP4E> type t.py #!/usr/bin/python3 import sys print(sys.version) C:\PP4E> t.py 2.6 (r26:66721, Oct 2 2008, 11:35:03) [MSC v.1500 32 bit (Intel)] C:\PP4E> py t.py 3.4.0a2 (v3.4.0a2:9265a2168e2c+, Sep 8 2013, 19:41:05) [MSC v.1600 32 bit (Intel)]
If the first command doesn't run the same highest-numbered 3.X Python installed and
print the same version as the second, your associations are probably incorrect. In
this case, the association reported by assoc .py and ftype Python.File
Windows command-line tools seem correct, but are apparently overridden by registry settings,
and link to "python.exe" instead of "Python Launcher for Windows" in the Default Programs GUI.
To fix: open your Default Programs in Control Panel (or similar), and manually
associate all 4 Python file extensions with the launcher's py.exe or pyw.exe executables
(normally installed in C:\Windows), a regrettable and error-prone step, but a one-time
event. This may or may not be a "bug" in the Windows launcher (the registry is notoriously
fragile), but it's certainly an issue; it is especially difficult to justify a mandatory
change that requires extra work and can break existing programs, if that change itself
cannot be relied on to work as advertised.
For more on the 3.3 Windows launcher in general, see the new Appendix B in
Learning Python, 5th Edition,
or its early draft
here. The
often prescribed alternative to environment settings mentioned earlier—adding
a Unix #! line at the top of every script on Windows—seems absurd and extreme,
especially given other launcher issues like those of the former and following sections.
Update: per the updates list above Python 3.6 bifurcated the Windows launcher's defaults to select a Python 3.X in some—but not all—contexts; read the details here. Unfortunately, this muddles the story further.
As
described here,
the new Windows launcher installed with Python 3.3 fails when
a script begins with a Unix #! line that it cannot recognize. Some book
example scripts used a #!/bin/env python pattern that was ignored on Windows
by every Python through 3.2, but is now treated as an error by 3.3. Such
#! forms are valid and even required on some Unix systems, but must be changed
to a launcher-accepted pattern such as #!/usr/bin/python or
#!/usr/bin/env python in order to run under 3.3 on Windows—a platform
where such lines are otherwise irrelevant (yes: !).
Note that this is true even if you use python3 in the #! line or set your
PY_PYTHON=3 per the prior section; an unrecognized #! pattern fails in 3.3
even if it's not version-ambiguous, and even if it's valid on a Unix system.
In release 1.4,
this fix was applied to the #! lines at the top of the following
10 files, changing their /bin/env to /usr/bin/env;
no other code or environment changes are required for this issue:
| PP4E\Launch_PyDemos.pyw (also changed for prior section) |
| PP4E\Launch_PyGadgets_bar.pyw (also changed for prior section) |
| PP4E\LaunchBrowser.pyw |
| PP4E\Internet\Ftp\Mirror\cleanall.py |
| PP4E\Internet\Ftp\Mirror\downloadflat.py |
| PP4E\Internet\Ftp\Mirror\downloadflat_modular.py |
| PP4E\Internet\Ftp\Mirror\ftptools.py |
| PP4E\Internet\Ftp\Mirror\uploadall.py |
| PP4E\Internet\Ftp\Mirror\uploadflat.py |
| PP4E\Internet\Ftp\Mirror\uploadflat_modular.py |
You can locate and edit these files yourself in older example packages
by using the Search/Grep tool in the book's PyEdit GUI example on the
package's PP4E root directory, or by running one of the directory search and
edit/replace utilities presented in the System Programming part of the book
(e.g., see search_all.py and Visitor subclasses later in Chapter 6).
Fixing any additional unrecognized #! patterns lurking in the examples
package is an officially suggested exercise.
For more on the 3.3 Windows launcher in general, see the new Appendix B in Learning Python, 5th Edition, or its early draft here.
As
covered here,
Python 3.3 changed a utility in the standard library's email package which the book's largest example,
PyMailGUI, depends on to format non-ASCII email address names for display. The
patch is simple, and spans just two files:
PP4E\Internet\Email\PyMailGui,
a new file that restores the prior version of the email utility
PP4E\Internet\Email\PyMailGui,
a modified file which runs an import of the patch to affect the change
This is a small bit of "monkey-patching"—replacing part of a module at runtime—which should be avoided in general, but in this case suffices to make the existing code run as is. The changes required to fully support Python's email API changes since the book's publication would be much more involved.
The new and modified file of the patch are automatically included in PyMailGUI's source code directory of the new release 1.4 of the examples package; no other code or environment changes are required.
As mentioned here, the third-party PIL image-processing library used by examples in the book has been subsumed by the Pillow fork—an open source drop-in replacement for PIL which is being actively supported for use in newer Python releases. You'll want to fetch and install Pillow to use most of the book's image examples, including the PyPhoto viewer, which employs image display, thumbnail generation, and resize operations. This page's author has also used Pillow successfully for EXIF photo metadata tag processing.
Pillow is currently
available at this site. For example, all
image-based book examples run fine under on Python 3.3 after installing the following on my
Windows 32- and 64-bit systems, respectively:
See PIL's site for other Pythons and platforms; at this writing there are also Pillow
installers for Python 2.7, and 3.4 through 3.8 (see Pillow's release docs).
Also note that Pillow is not required of all image-display programs: as of Tk 8.6
(included with python.org's
Python installers for Windows since 3.4, and currently
bundled with its installers for Mac OS), Python
No fix was applied, because the example package doesn't ship 3rd-party systems as bundled
items as a rule; you'll want to fetch and install Pillow for your Python and platform, from
its official site above.
tkinter GUIs can display
PNG images without Pillow,
and tkinter has always been able to display GIF
and PPM/PPG images without extensions in any Python. Pillow is still required for
other image types and Python/Tk combinations, as well as for other tasks beyond
display—including the thumbnail and resizing operations implemented by
PyPhoto (now part of standalone PyGadgets).
In the Book Examples Package
![[Home page]](PythonPowered.gif "Home page"){kind=small}