File: thumbspage/

============================================================================== - part of the thumbspage image-gallery builder

From: 'Programming Python, 4th Edition', Author/Copyright M. Lutz, 2010-2020.
License: Provided freely but with no warranty of any kind.

Display all images in a directory as thumbnail-image buttons that display 
full-size images when clicked.  Requires Pillow for JPEGs and thumbnail image 
creation.  To do: add scrolling if too many thumbs for window, and image
scaling if image too big for display (see PyPhoto or thumbspage for both).

This module is now part of thumbspage, which reuses the thumbnail generation 
and other functions here.  This code has evolved much since its PP4E version, 
as described by the major-change sections that follow.  Tags of the form 
"[N.M]" in this file refer to thumbspage release numbers.

2017: Forked version in PyPhoto

PyGadgets' PyPhoto now uses a forked and enhanced version of this module, 
which can store all of a folder's thumbnails in a single pickle file, 
and better detects changes in the source images set:

2018: Integrate most PyPhoto fork enhancements

Modified to skip non-image items in image folder by MIME type, not
exception, and integrate PyPhoto's fix for a Pillow tiff thumbs crash.
Also uses latest "best" downsize filter name in Pillows that have it,
and in script/GUI mode displays a note with PyPhoto/thumbspage links.
PyPhoto's pickle file and change detection are still not used here.

2018: Work around Pillow too-many-open-files bug

This module codes a workaround for a Pillow (PIL) bug that occurs only  
in limited usage contexts for folders having very many images.  In brief, 
Pillow's auto-close of loaded image files does not work as documented, 
which can lead to "Too many open files" errors after many thumbnails 
have been generated.  In thumbspage, this meant that results could reflect 
partial source content for very large folders, though only on Mac OS in 
general.  The best fix is to manually open and close image files here, 
instead of passing filenames to Pillow.  Read on for all the details.
The Bug:
   This bug was seen only on Mac OS when running source code in a shell 
   (specifically, on Mac OS 10.13 and 10.15, and Pillow 4.2.1 and 5.1.0).  
   It does not occur when running a frozen Mac OS app, and does not occur 
   when running source after shell limits are raised with "ulimit -n 9999".
   Mac app and shell limits can differ, but it's wildly convoluted (search 
   the web for "mac os which command controls open file limits").

   This bug has not been seen on Windows, in either command line or frozen
   executable contexts; Windows file limits are likely higher, in the system
   and/or C libraries.  It may also vary by image type: on Macs it cropped 
   up after processing 325 images in one use case, but fewer in others, and 
   handles 470+ in one without error.  You can read more about the bug on
   GitHub and such:

   This bug can also be triggered in PyPhoto, but only for very large folders;
   when running source code in a shell; on platforms with low open-file limits 
   (Macs); and for runs that generate very many thumbs (unlike thumbspage, 
   PyPhoto always caches and reuses thumbs made in prior runs).  Frozen PyPhoto
   apps and executables are generally immune.  By contrast, thumbspage is run 
   via command line only, and must try to support all Pillows on all platforms.

   Subtle bit: opening the PyPhoto app from a shell via "open -a PyGadgets" 
   on a Mac does not trigger the error; this bug is just a source-code issue
   (e.g., "python3 .../pygadgets/_PyPhoto/PIL/ -InitialFolder big").
   It kills the GUI in PyPhoto, but can be avoided by a "ulimit -n 9999".
   PyPhoto note:

The Fix:
   The solution used here is to take control of the file away from Pillow
   altogether.  Instead of passing a filename to, manually open
   an image file; read it into a bytes, and pass it to Pillow wrapped in a 
   BytesIO object; and manually close the file.  This guarantees that files
   are closed.  With this change, thumbspage supports folders of arbitrarily
   many images in all contexts (thousands work, albeit in a fairly large page). 

   Note that the call does not leave files open.  If it ever does,
   a similar fix may be used - instead of passing a filename, either pass a
   manually opened and closed output file, or a BytesIO object whose value
   is extracted and written to a file manually (set its .name, or use Pillow's 
   EXTENSION[ext.lower()] for the format argument).  PyPhoto 
   does similar for storing image data in a pickle file.

Other Workaround Ideas: 
   An explicit image.close() by itself is not enough, and makes the image 
   object unusable.  Although it also works to open with a temp image, .copy()
   to a work image, and explicitly .close() the temp image (as suggested on 
   the web), this makes thumb builds much slower (~5X).  This module could 
   also simply not send the image object back in the result list (it's not 
   used in thumbspage, and is auto-closed when reclaimed), but that won't work
   for programs that need to use the returned image objects (e.g., PyPhoto). 

2018: Auto-rotate images and thumbnails to display right-side up (optionally)

This module now automatically reorients (rotates) any tilted photos and 
their generated thumbnails to be right-side up (top on top), if they have 
valid "Orientation" Exif tags.  This is especially useful for photos shot 
on smartphones, but applies only to images with Exif tags (e.g., JPEG, TIFF) 
from cameras and tools that tag files properly, and must change source-image 
files in place because web browsers will not rotate them when displayed 
in-page by the thumbspage client's viewer pages (see reorientImage() ahead).

Because of its limits and changes, this feature automatically backs up 
image files with ".original" extensions, and both it and its backups can 
be disabled with new function arguments.  If desired or needed, disable 
and manually rotate as preferred.  Reorientation is not automatic or an 
easy option in Pillow (should it be?).  For tests: examples/reorientation.

2019: Remove tkinter GUI dependencies

This module no longer requires the tkinter GUI module, unless its simple 
GUI display is being used as a top-level script.  This should make it 
easier to use thumbspage in limited contexts (e.g., in Android apps).

2020: Convert to RGBA mode before making thumbs, retain rotated JPEG Exifs

To yield better quality thumbnails for downscaled images, some images are 
now converted temporarily and internally to "RGBA" color mode just before 
making thumbnails.  This also benefits some non-downscaled images whose 
thumbs were formerly obfuscated (e.g., some GIFS).  In addition, rotated
JPEG images now retain their Exif tags (per a tip from shrinkpix), with
Orientation and dimension tags updated to reflect the rotation using the 
piexif library now included in thumbspage.  See "[1.7]" here for details.

import os, sys, math 
import mimetypes, io                    # add for file type, buffer [2018]

from PIL import Image                   # <== required for thumbs (always)
from PIL.ExifTags import TAGS           # <== required for orientation tag [2018]

# Feb2019: now run iff __main__, to minimize dependencies (e.g., Android Termux).
# Caution: tkinter has Image too: reimport from PIL to replace, or use obj.attr.
# from tkinter import *                 # for GUI-viewer mode (has own unused Image) 
# from PIL.ImageTk import PhotoImage    # <== required for JPEG display (iff GUI)

# [1.7] use piexif to update Exif tags propagated on JPEG rotation
import piexif    # third party, but shipped with thumbspage

# [1.7] the new color-mode convert() ahead can generate a PIL UserWarning 
# when converting some images "RGB", which appears harmless for this use
# case; silenced here initially, but later made moot by "RGBA" recoding.
# The full warning text is "Palette images with Transparency   expressed 
# in bytes should be converted to RGBA images" (including the 3 spaces), 
# and percolates out of line 888 in in the Pillow version used.
import warnings
warnings.filterwarnings(action='ignore', module="PIL.Image",
                        message="Palette images with Transparency.*") 

# Thumbnail code follows

def printexc(): 
    print('Exception:', sys.exc_info()[1])   # just the instance/message

def imageWideHigh(imgdir, imgname):
    Utility for thumbspage: use PIL to get image dimensions.
    imgpath = os.path.join(imgdir, imgname)
    imgobj =
    size = (imgobj.width, imgobj.height)     # same as .size
    imgobj.close()                           # unless auto closed?
    return size

def isImageFileName(filename):
    Borrowed from PyPhoto: skip non-image files explicitly.
    mimetype = mimetypes.guess_type(filename)[0]                    # (type?, encoding?)
    return mimetype != None and mimetype.split('/')[0] == 'image'   # e.g., 'image/jpeg'

def mimeType(pathorfilename):    
    Get mime type (e.g., 'image/jpeg'); mimetypes module is cryptic. [1.7]
    filename = os.path.os.path.basename(pathorfilename)    # this is optional
    return mimetypes.guess_type(filename)[0]               # ignore encoding

def openImageSafely(imgpath):
    Work around a Pillow image-file auto-close bug, that can lead to 
    too-many-files-open errors in some contexts (most commonly when 
    running source code on Mac OS, due to its low #files ulimit).  The
    fix is to simply take manual control of file opens and saves.  For
    more details, see this module's top docstring.  This does not appear 
    to be required when opening existing thumb files, but is harmless.
    fileobj = open(imgpath, mode='rb')          # was
    filedat =
    fileobj.close()                             # force file to close now
    imgobj =    # file bytes => pillow obj
    return imgobj

def getExifTags(imgobj):
    Collect image-file metadata in a new dict, if any present (else empty).
    This stores each tag value present under its mnemonic string name, by 
    mapping from image tag numeric ids to names, via the PIL.ExifTags table.
    tags = {}
        info = imgobj._getexif()                    # not all have Exif tags
        if info != None:
            for (tagid, value) in info.items():
                decoded = TAGS.get(tagid, tagid)    # map tag's id to name 
                tags[decoded] = value               # use id if not in table
    except Exception as E:
    return tags

def fixFailingExifTags(parsedexif):
    [1.7, Jun-2020] piexif bug temp workaround: correct uncommon Exif tags 
    (e.g., 41729, which is piexif.ExifIFD.SceneType) whose miscoding on some
    devices triggers an exception in piexif's dump() - but not its load(). 
    Other failing tags will still fail in dump(), and skip auto-rotation.

    Until piexif addresses this more broadly, this munges SceneType from
    int to byte and converts another from tuple to bytes*4 if needed (and 
    defensively: stuff happens).  The piexif exceptions formerly crashed 
    thumbspage before it was changed to restore originals.  Bug reports:
        tag 41729 =>
        tag 37121 =>
    parsedExif = parsedexif['Exif']                 # parsed tags: dict of dicts

    # fix SceneType? 1 => b'\x01'
    if 41729 in parsedExif:
        tagval = parsedExif[41729]                  # miscoded on some Galaxy
        if type(tagval) is int:                     # munge from int to byte                
            if 0 <= tagval <= 255:
                parsedExif[41729] = bytes([tagval])
                print('--Note: bad SceneType Exif tag type was corrected')
                del parsedExif[41729]
                print('--Note: bad SceneType Exif tag type was dropped')

    # fix ComponentsConfiguration? (1, 2, 3, 0) => b'\x01\x02\x03\x00'
    if 37121 in parsedExif:    
        tagval = parsedExif[37121]
        if type(tagval) is tuple:
            if (len(tagval) == 4 and 
                all(type(x) is int for x in tagval) and
                all(0 <= x <= 255  for x in tagval)):
                parsedExif[37121] = bytes(tagval)
                print('--Note: bad ComponentsConfiguration Exif tag was corrected')
                del parsedExif[37121]
                print('--Note: bad ComponentsConfiguration Exif tag was dropped')

    # other tag failures cause auto-rotation to be skipped for an image

def getUpdatedExifData(imgstart, imgrotate):
    [1.7] Update the values of Exif orientation and dimension tags in a
    rotated JPEG image to reflect the rotation, before saving to disk.
    Fixing dimensions is not crucial (the rotated copy is meant for display
    in thumbspage galleries only).  But the old orientation tag value would 
    make some other viewers reorient a reoriented image with flipped results
    (this includes underlying web browsers for thumbspage Raw displays), and
    would also trigger spurious thumbspage rerotations on every future run.  

    This requires the third-party piexif library, now shipped with thumbspage,
    because Pillow has no support for Exifs apart from raw bytes data.

    [Jun-2020] piexif bug temp workaround: fixFailingExifTags() now corrects
    some uncommon tags whose miscoding triggers exceptions in piexif.dump().
    origexif ='exif', b'')          # original raw bytes from Pillow
    if not origexif:                                   # piexif bombs if b''
        saveexif = origexif
        parsedexif = piexif.load(origexif)             # parse to dict for changes here

        # piexif work-around: fix tags known to trigger exceptions

        # update tags
        parsedexif['0th'][piexif.ImageIFD.Orientation] = 1
        parsedexif['Exif'][piexif.ExifIFD.PixelXDimension] = imgrotate.width
        parsedexif['Exif'][piexif.ExifIFD.PixelYDimension] = imgrotate.height

        saveexif = piexif.dump(parsedexif)             # back to raw bytes for Pillow
    return saveexif                                    # pass to Pillow save

def reorientImage(imgobj, imgpath, backups=True):
    Rotate the image to be right-side up (top-side-top) if needed, before 
    making its thumb.  This also rotates the source image, and is simply an
    automatic alternative to manually rotating images before making thumbs.

    This might rotate for thumbs only and not save to source images, but
    not all viewers will rotate the source image when opened from the thumb.
    In this code's thumbspage client, specifically, browsers will not rotate
    the image when viewer pages display it in-page (PyPhoto may or may not 
    need to save images too, but uses a forked/custom version of this code). 

    Because this overwrites images, the original is saved to a ".original"
    copy if backups==True when rotated (which happens just once, after which
    no rotation is needed).  Note: this uses transpose(), not rotate() - the 
    latter may change more, subject to Pillow implementation and changes.

    [1.7] This now propagates Exif tags from an original source image to its
    rotated and saved copy, and updates Exif orientation and dimension tags.
    Without propagating, date-taken doesn't appear in image-info dialogs;
    without updating, images are flipped by browsers and thumbspage reruns.

    [1.7, June-2020] This now auto restores the backed-up original if it's
    been saved, but the save of the rotated version fails.  This cropped
    up when piexif.dump() raised bad-tag exceptions in getUpdatedExifData().
    tags = getExifTags(imgobj)                      # not all have Exif tags
    orientation = tags.get('Orientation')           # not all have this tag
    if orientation:

        transforms = {
            # other settings don't make sense here
            1: None,                # top up (normal)
            3: Image.ROTATE_180,    # upside down
            6: Image.ROTATE_270,    # turned right
            8: Image.ROTATE_90      # turned left

        transform = transforms.get(orientation, None)     # not all tag values used
        if transform != None:
            print('--Reorienting tilted image')
            imgstart = imgobj
                imgobj = imgobj.transpose(transform)      # rotate to a new copy
                imgobj = imgstart                         # abandon the mission
                print('--Transform failed: skipped')      # punt: can't auto fix
                # rotate source too, after backup?
                saveimage = True
                if backups:
                    try:                                  # be defensive: stuff fails
                        backup = imgpath + '.original' 
                        os.replace(imgpath, backup)       # rename + overwite prior
                    except:                               # normally rotate just once
                        saveimage = False
                        imgobj = imgstart                 # abandon the mission
                        print('--Source backup failed: skipped')
                if saveimage:
                        # [1.7] propagate JPEG Exifs, update some
                        if mimeType(imgpath) == 'image/jpeg':
                            saveexif = getUpdatedExifData(imgstart, imgobj)
                            saveargs = dict(exif=saveexif)
                            saveargs = {}
              , **saveargs)    # rotate source image too
                        imgobj = imgstart                   # abandon the mission
                        print('--Source rotate failed: skipped')   
                        print('--Rotate manually and rerun if desired')
                        if backups:
                                # [1.7, Jun-2020] pyexif fail, permissions, etc.
                                # restore the original so the build finishes 
                                os.replace(backup, imgpath)
                                # unikely, but no fix: will terminate ahead
                                print('--Restore failed: see .original')

    return imgobj   # new or original

def makeThumbs(imgdir,               # path to the folder of source-image files
               size=(100, 100),      # (x, y) max size for generated thumbnails
               subdir='thumbs',      # path to folder to store thumbnail files
               rotate=True,          # reorient tilted images and their thumbs?
               backups=True):        # save a .original of any images rotated?
    Get thumbnail images for all images in a directory.  For each full image,
    either load and return an existing thumb, or create and save a new thumb.
    Makes thumb folder "subdir" in "imgdir" if it doesn't already exist.
    Returns a list of (image-filename, thumb-image-object), ordered only by 
    underlying platform's directory-listing order: caller may wish to run the
    result through sorted() to unify ordering portably.  Caller can also run 
    os.listdir() on thumbs folder to access/load its thumbs after the call.  

    On bad file types this may raise IOError, or other.  Caveat: could also
    check file timestamps and source-image file existence to detect image 
    changes and deletions requiring new thumbs (see PyPhoto, which does).
    As is, reusing existing thumbs for speed must assume no image changes.
    TBD: skip ".*" Unix hidden files here, so don't appear in galleries?

    [1.7] Downscaling images with shrinkpix yields poor thumbnails for some 
    (e.g., when changed to "P" color mode).  Convert to "RGBA" temporarily 
    here when needed/possible for results as good as those formerly produced 
    for original, unshrunk images.  This also improves thumbs for some unshrunk 
    images too (e.g., GIFs).  See for shrinkpix.

    This conversion is surprisingly complex:
    - It cannot be applied to JPEGs ("RGBA" fails for the later save).
    - It does improve thumbs for some unshrunk GIFs, but cannot be used if 
      they have transparency (else the transparent parts render black).
    - "RGB" mode is less stringent, but renders Mac OS screenshot shadows 
      solid black too, instead of the former graded display (black better
      matches viewer pages, but only if they use default background colors). 
    - "RGB" can issue a spurious PIL warning ignored above (but now moot). 
    - High "quality" in save() didn't help for thumbnails.

    [2018]: Mod to use changes in PyPhoto fork: mimetypes, filter, tiffs.
    [2018]: Mod to avoid too-many-open-files bug in Pillow (see above).
    [2018]: Mod to auto-reorient images+thumbs right-side up if rotate==True.
    [2020]: Mod to propagate and update Exif tag values on rotations.
    [2020]: Mod to use RGB color modes for shrunken images quality.
    thumbdir = os.path.join(imgdir, subdir)
    if not os.path.exists(thumbdir):

    thumbs = []
    for imgfile in os.listdir(imgdir): 
        if not isImageFileName(imgfile):                      # avoid exceptions [2018]
            print('Skipping:', imgfile)

        thumbpath = os.path.join(thumbdir, imgfile)           # thumb name == image name
        if os.path.exists(thumbpath):                     
            # use already-created thumb (assume image same)
            thumbobj = openImageSafely(thumbpath)             # Pillow workaround [2018]
            thumbs.append((imgfile, thumbobj))                # was

            # make new thumb (image same or not)
            print('Making thumbnail:', thumbpath)
            imgpath = os.path.join(imgdir, imgfile)
                # work around Pillow auto-close bug [2018]
                imgobj = openImageSafely(imgpath)             # was            

                # rotate to right-side up, iff needed [2018]
                if rotate:
                    imgobj = reorientImage(imgobj, imgpath, backups)

                # fx for downscaled images and some GIFs per above [1.7]
                if (imgobj.mode != 'RGBA' and
                    mimeType(thumbpath) != 'image/jpeg' and 
                    not (mimeType(thumbpath) == 'image/gif' and 
                    imgobj = imgobj.convert('RGBA')

                if hasattr(Image, 'LANCZOS'):                 # best downsize filter [2018]
                    imgobj.thumbnail(size, Image.LANCZOS)     # but newer Pillows only
                    imgobj.thumbnail(size, Image.ANTIALIAS)   # original filter
                if mimeType(thumbpath) != 'image/tiff':       # mimeType(), not ext [1.7]
                              # type via ext or passed
                    # work around C lib crash: see PyPhoto's code [2018]
          , compression='raw')
                thumbs.append((imgfile, imgobj))              # no need to discard image

            except Exception:                                 # skip ctrl-c, not always IOError
                print('Failed:', imgpath)

    return thumbs    # caller: use this or os.listdir(thumbs folder)

# GUI-mode code follows (used iff __main__, not available to other clients)

if __name__ == '__main__':
    When this file is run directly: make thumbs, show simple image-viewer GUI.
    Feb2019: all GUI code is now nested here, to avoid the tkinter dependency.
    This allows thumbspage to be run in contexts with Pillow but no tkinter 
    (e.g., Termux on Android), but precludes importing and using the GUI class 
    and function below from elsewhere (but that is a very unlikely use case).

    # Use PIL.Image above, not tkinter.Image: reimport from PIL to replace.
    # If not __main__, the earlier imports suffice for the functions above.
    # If __main__, imports here set globals used when functions above are run. 
    # The third import below also replaces tkinter.PhotoImage with PIL's.
    from tkinter import *                 # for GUI-viewer mode
    from PIL import Image                 # <== required for thumbs (post tkinter)
    from PIL.ImageTk import PhotoImage    # <== required for JPEG display (iff GUI)

    class ViewOne(Toplevel):
        Open a single image in a pop-up window when created.  PhotoImage
        object must be saved: Tk images are erased if object is reclaimed.
        def __init__(self, imgdir, imgfile):
            imgpath = os.path.join(imgdir, imgfile)
            imgobj  = PhotoImage(file=imgpath)
            Label(self, image=imgobj).pack()
            print(imgpath, imgobj.width(), imgobj.height())   # size in pixels
            self.savephoto = imgobj                           # keep reference on me

    def viewer(imgdir, kind=Toplevel, cols=None):
        Make thumb-links window for an image directory: one thumb button per image. 
        Use kind=Tk to show in main app window, or Frame container (pack).  imgfile
        differs per loop: must save with a default.  PhotoImage objs must be saved: 
        erased if reclaimed.  Packed row frames (versus grids, fixed-sizes, canvas). 
        win = kind()
        win.title('Viewer: ' + imgdir)
        quit = Button(win, text='Quit', command=win.quit, bg='beige')   # pack first
        quit.pack(fill=X, side=BOTTOM)                                  # so clip last
        thumbs = makeThumbs(imgdir)
        if not cols:
            cols = int(math.ceil(math.sqrt(len(thumbs))))     # fixed or N x N

        savephotos = []
        while thumbs:
            thumbsrow, thumbs = thumbs[:cols], thumbs[cols:]
            row = Frame(win)
            for (imgfile, imgobj) in thumbsrow:
                photo   = PhotoImage(imgobj)
                link    = Button(row, image=photo)
                handler = lambda savefile=imgfile: ViewOne(imgdir, savefile)
                link.pack(side=LEFT, expand=YES)
        return win, savephotos

    Top-level/main logic: make thumbs, open a simple tkinter image-viewer GUI.
    imgdir = (len(sys.argv) > 1 and sys.argv[1]) or 'images'
    main, save = viewer(imgdir, kind=Tk)

    # add other-viewers note [2018]
    popup = Toplevel()
    popup.title('Other Image Viewers')
    note = ('For better viewing options, see:\n\n'
                'PyPhoto - at\n'
                'thumbspage - at')
    label = Label(popup, bg='ivory', text=note, width=60, height=8)
    label.pack(anchor=CENTER, expand=YES, fill=BOTH)
    main.after(1000, popup.lift)   # raise one second after main opens

    main.mainloop()   # open GUI, wait for user events

[Home] Books Programs Blog Python Author Training Search Email ©M.Lutz