"""
================================================================================
scandir_defunct.py:
   Python 3.5+ os.scandir() comparison-phase optimization, no longer used [3.0].

The comparison-phase variant code here, added in 2.2, formerly was some 2X
faster (or better) on Windows and Linux, but also 2X slower on Mac OS X.  This
meant that both variants were maintained and tested redundantly -- a major task.
  
Near the end of the 3.0 project, the non-scandir() variant was optimized to use
os.lstat() stat objects instead of os.path.*() calls, initially for new symlink
support.  This reduced the non-scandir() speed hit on Windows to 50%: runtimes
for a 60k-file archive were 10 seconds with scandir(), and 15 seconds without.

However, a final step of this recoding to avoid all extra stat calls made the
non-scandir() variant as fast as -- or slightly faster than -- this scandir()
variant on Windows too (at ~10 seconds).  This made the code here obsolete.
See ./docetc/miscnotes/3.0-mergeall-comparison-times-Windows.txt for timings.

On Mac OS X, this recoding to eliminate extra stats calls only improved the
non-scandir() performance.  For the same 60k-file archive, runtime fell from
9 seconds for the code here; to 4.5 seconds for the initial non-scandir()
variant; to just 3.8 seconds for the final non-scandir() variant.

See ./docetc/miscnotes/3.0-mergeall-comparison-times.txt for timing results.
These times are slightly better when mergeall is run directly outside the
GUI launcher (for reasons TBD: the launcher's UTF8 stdout encoding or -u?):

  # scandir() version on Mac
  $ py3 ../mergeall.py /MY-STUFF /MY-STUFF -skipcruft -report | grep 'Phase'
  Phase runtime: 8.64699007209856

  # non-scandir() version on Mac
  $ py3 ../mergeall.py /MY-STUFF /MY-STUFF -skipcruft -report | grep 'Phase'
  Phase runtime: 3.5263971380190924

Hence, this file's code will no longer be used, maintained, or tested, but
is retained here indefinitely for reference and historical context only,
and to recreate the relative speed results that led to its demise.

The final lesson here seems to be that os.scandir() can speed programs on
Windows and Linux (though not Mac OS X) which use many os.path.*() calls.
But it is no faster, and may even be slower, than code which uses explicit
os.stat()/lstat() calls and objects.  Given that both the scandir() and stat
schemes require similar changes or special coding, os.scandir() seems no
better, and worse on Mac.  Using stat objects yields code that is arguably
more cluttered visually than os.path.*(), but also a huge performance win.
================================================================================
"""

#
# this code is exec()d instead of imported to avoid recursion
#

# both should be False, except for testing
USETHISCODE = False   # <= change to 1/True to enable comparison variant here
FORCEONMAC  = False   # <= change to 1/True to use on Mac, despite slowness



################################################################################
# COMPARISON PHASE: analyze trees
# Python 3.5 and later version => use faster os.scandir() objects [2.2]
################################################################################


#-------------------------------------------------------------------------------
# Use optimized variant if scandir() available _and_ faster for comparisons:
# Windows:
#    In mergeall, scandir is faster - 5X to 10X before 3.0 optimizations,
#    and 2X faster after: 7 sec vs 14 secs for a large 87G 58k-file archive. 
# Linux:
#    The scandir() variant is 2X faster on Linux post 3.0 optimizations:
#    30 secs vs 60 secs for the same large archive (on a slow test machine).
# Mac OS X:
#    [3.0] BUT the scandir() variant is 3X SLOWER on Mac OS X! - it takes
#    6 secs vs 2 secs for the same 87G 58K-file archive (on a fast machine),
#    with the 3.0 coding optimizations described above.  DON'T USE ON MACS.
#
# UPDATE: these results were seen before the os.lstat() recoding above,
# after which the os.scandir() verison is still 2X *slower* on Macs -- 9~10
# secs vs 4~5 for an archive that grew to 65K files since the prior timings.
# Using python.org's Python 3.5 pm Mac OS X 10.11, only run-time differs:
#
#   /Admin-Mergeall/kingston-savagex256g/feb-2-17$ diff \
#           noopt1--mergeall-date170202-time091326.txt \
#           opt2--mergeall-date170202-time092217.txt 
#   0a1
#   > Using Python 3.5+ os.scandir() optimized variant.
#   4053c4054
#   < Phase runtime: 5.286043012980372
#   ---
#   > Phase runtime: 10.12333482701797
#
# With 3.0 code, the scandir() varient is still 50% to 2X *faster* on Windows.
#
# [3.0] FINAL UPDATE: this scandir() variant is no longer faster on Windows,
# making this code moot; see this file's main dosctring for the full verdict.
#
# Don't use float(sys.version[:3]) >= 3.5; scandir() can be a separate install.
#-------------------------------------------------------------------------------


if not hasattr(os, 'scandir'):          # older Pythons (2.7+): no os.scandir
    try:
        import scandir                  # check for PyPI package install
        os.scandir = scandir.scandir    # store in os, and assume same API
    except:
        pass                            # punt: use Py 3.4- (now all) verions


# [3.0] the non-scandir() version using os.lstat() is now faster everywhere
if (USETHISCODE
    and hasattr(os, 'scandir')                   # skip if not available
    and (not sys.platform.startswith('darwin')   # [3.0] not on Mac: slower!
         or FORCEONMAC)
    ):
    """
    ------------------------------------------------------------------------
    [2.2] Custom comparison code using os.scandir() instead of os.listdir().

    SEE NON-SCANDIR() VERSIONS in mergeall.py for docs not repeated here.

    os.scandir() provides an object API (versus name lists), that can often
    eliminate system calls for attributes such as type, modtime, and size.
    This can speed mergeall's comparison phase by a factor of 5 to 10 on
    Windows.  This speedup is entirely due to os.scandir(), not 3.5 in
    general, and can shave dozens of seconds off total runtimes for larger
    archives (and more on slower devices).

    POSIX systems also run quicker, according to this Python 3.5 change's PEP,
    benchmark, and documentation.  [UPDATE: Not so! => os.scandir() is faster
    on Linux too, but SLOWER on Macs - 2X~3X slower in mergeall; see above.]

    This speeds up the comparison phase only, but this phase always runs and
    can dominate runtimes in typical runs on large archives with light changes.
    The resolution phase was not changed, because it normally visits just a
    handful of file paths (changed items only), and is bound by basic file
    write/delete speed; system calls are minor component of its time.

    Note: diffall.py might benefit from os.scandir() too for path names,
    but it's less likely, as that script doesn't fetch modtimes or sizes,
    and is bound by the time required to fully read all common files.

    Note: scandir() is also available in a scandir module for older Pythons,
    including 2.7, via a PyPI package.  If installed, the code here imports
    and uses it, and so should work with older Pythons too -- not just 3.5+.

    CODING TBD: factor-out differing parts to avoid some redundant code?
    As is, there are two versions to update and test for comparison changes.
    In fact there were -- for 3.0 optimizations, cruft-file skipping, and
    symlinks support.
    But:
      They seem too different in structure to factor well, and factoring
      may slow either or both versions, negating this extension's goal.
    And:
      We need both, because the optimized variant is FASTER on Windows and
      Linux, but SLOWER on Mac.  This speed matters: a single version would
      penalize one or two platforms for the comparison phase, which always
      runs regardless of the number of changes in the trees.  Tentative
      lesson: redundancy is sometimes warranted in the name of optimization.

    [3.0] FINAL STORY: in the end, the non-scandir() grew slightly quicker
    on Windows too, by using stat objects to avoid all os.path.*() calls.
    This made the scandir() 3.5+ optimized variant fully obsolete, and
    removed the coding and testing redundancy.  Consequently, this scandir()
    variant has been moved out of file and kept here for reference only,
    and will no longer be updated or tested in the future.  Good riddance!
    ------------------------------------------------------------------------
    """
    if __name__ == '__main__':  # else prints twice
        trace(0, 'Using Python 3.5+ os.scandir() optimized variant.')

    # this decrufter flavor is used only here now
    from skipcruft import filterCruftDirentrys



    def comparedirs(direntsfrom, direntsto, dirfrom, dirto, uniques):
        """
        [2.2] python 3.5+ custom version using faster os.scandir().
        Doesn't use set difference here, to maintain listing order.
        Note: dirfrom and dirto pathnames still used by resolver phase.
        See mergeall.py's non-scandir() version for docs on this function.
        """
        trace(2, 'dirs:', dirfrom, dirto)

        countcompare.folders += 1
        uniquefrom = [df.name for df in direntsfrom
                          if df.name not in (dt.name for dt in direntsto)]
        uniqueto   = [dt.name for dt in direntsto
                          if dt.name not in (df.name for df in direntsfrom)]
        if uniquefrom:
            uniques['from'].append((uniquefrom, dirfrom, dirto))
        if uniqueto:
            uniques['to'].append((uniqueto, dirfrom, dirto))



    def comparelinks(direntfrom, direntto, dirfrom, dirto, diffs):
        """
        [3.0] python 3.5+ custom version using faster(?) os.scandir().
        See mergeall.py's non-scandir() version for docs on this function.
        """
        name = direntfrom.name 

        # compare link path strs
        linkpathfrom = os.readlink(FWP(direntfrom.path))
        linkpathto   = os.readlink(FWP(direntto.path)) 
        if linkpathfrom != linkpathto:
            diffs.append((name, dirfrom, dirto, 'linkpaths'))



    def comparefiles(direntfrom, direntto, dirfrom, dirto, diffs, dopeek=False, peekmax=10):
        """
        [2.2] python 3.5+ custom version using faster(?) os.scandir().
        Uses stat() objects for sizes+times, perhaps avoiding system calls.
        Note: dirfrom and dirto pathnames still used by resolver phase.
        See mergeall.py's non-scandir() version for docs on this function.
        """
        trace(2, 'files:', direntfrom.path, direntto.path)
        
        def modtimematch(statfrom, statto, allowance=2):      # [1.3] 2 seconds for FAT32
            time1 = int(statfrom.st_mtime)                    # [2.2] os.stat_result object
            time2 = int(statto.st_mtime)
            return time2 >= (time1 - allowance) and time2 <= (time1 + allowance)

        countcompare.files += 1     
        startdiffs = len(diffs)
        
        name     = direntfrom.name                                   # same name in from and to
        statfrom = direntfrom.stat()                                 # call stat() just once
        statto   = direntto.stat()
        
        if not modtimematch(statfrom, statto):                       # try modtime 1st
            diffs.append((name, dirfrom, dirto, 'modtime'))          # the easiest diff

        else:                                                        
            sizefrom = statfrom.st_size                              # try size next
            sizeto   = statto.st_size                                # unlikely case
            if sizefrom != sizeto:
                diffs.append((name, dirfrom, dirto, 'filesize'))
                
            elif dopeek:                                             # try start+stop bytes
                peeksize = min(peekmax, sizefrom // 2)               # scale peek to size/2
                filefrom = open(FWP(direntfrom.path), 'rb')          # sizefrom == sizeto
                fileto   = open(FWP(direntto.path), 'rb')
                if filefrom.read(peeksize) != fileto.read(peeksize):
                    diffs.append((name, dirfrom, dirto, 'startbytes')) 
                else:
                    filefrom.seek(sizefrom - peeksize)
                    fileto.seek(sizeto - peeksize)
                    if filefrom.read(peeksize) != fileto.read(peeksize):
                        diffs.append((name, dirfrom, dirto, 'stopbytes'))
                filefrom.close()
                fileto.close()
                
        return len(diffs) == startdiffs    # true if did not differ, else extends 'diffs'



    def comparetrees(dirfrom, dirto, diffs, uniques, mixes, dopeek, skipcruft, skip=None):
        """
        [2.2] python 3.5+ custom version using faster(?) os.scandir().
        Doesn't use set intersection here, to maintain listing order.
        See mergeall.py's non-scandir() version for docs on this function.
        """
        trace(2, '-' * 20)
        trace(1, 'comparing [%s] [%s]' % (dirfrom, dirto))
        
        def excludeskips(direntsfrom, direntsto, skip):     # [3.0] pre-skipcruft filter 
            if not skip:
                return
            for dirents in (direntsfrom, direntsto):
                matches = [dirent for dirent in dirents if dirent.name == skip]
                if matches:
                    assert len(matches) == 1
                    trace(1, 'excluding', matches[0].path)
                    dirents.remove(matches[0])
            
        # get dir content lists here                        # [2.2] os.DirEntry objects iterator
        direntsfrom = list(os.scandir(FWP(dirfrom)))        # [2.2] need list() for multiple scans! 
        direntsto   = list(os.scandir(FWP(dirto)))          # or pass bytes?--would impact much
        excludeskips(direntsfrom, direntsto, skip)

        # [3.0] filter out system metadata files and folders
        if skipcruft:
            direntsfrom = filterCruftDirentrys(direntsfrom)
            direntsto   = filterCruftDirentrys(direntsto)

        # compare dir file name lists to get uniques 
        comparedirs(direntsfrom, direntsto, dirfrom, dirto, uniques)

        # analyse names in common (same name and case)
        trace(2, 'comparing common names')
        common = [(df, dt) for df in direntsfrom
                               for dt in direntsto if df.name == dt.name]

        # scan common list just once [3.0]
        for (direntfrom, direntto) in common:
            nonlink = dict(follow_symlinks=False)  # [3.0] narrow is() results
            
            # 0) compare linkpaths of links in common [3.0]
            if direntfrom.is_symlink() and direntto.is_symlink():
                comparelinks(direntfrom, direntto, dirfrom, dirto, diffs)

            # 1) compare times/sizes/contents of (non-link) files in common
            elif direntfrom.is_file(**nonlink) and direntto.is_file(**nonlink):
                comparefiles(direntfrom, direntto, dirfrom, dirto, diffs, dopeek)
                               
            # 2) compare (non-link) subdirectories in common via recursion
            elif direntfrom.is_dir(**nonlink) and direntto.is_dir(**nonlink):
                comparetrees(direntfrom.path, direntto.path,
                             diffs, uniques, mixes, dopeek, skipcruft)

            # 3) same name but not both links, files, or dirs (mixed, fifos)
            else:
                mixes.append((direntfrom.name, dirfrom, dirto))