![[Home page]](../PythonPowered.gif "Home page"){kind=small}
| Books | Code | Blog | Python | Author | Train | Find | ©M.Lutz |
File: mergeall-products/unzipped/backup.py
"""
==========================================================================================
backup.py:
automatic backup/restore subsystem (part of the mergeall system [2.0])
Summary:
Make backup copies of all files and directories in the TO directory that will
be destructively replaced or deleted in-place during a mergeall run. These
items' prior versions in the TO tree are saved in the automatically-created
__bkp__ folder at the top of the TO archive, with their full directory paths.
Backups are not synchronized across trees, but are automatically pruned by age
when their number exceeds a limit. This option makes mergeall generally safer,
as unwanted or failed changes can be later undone by restoring backup copies.
2.1 extends this model to support complete rollback of a prior run, by merging
a backup subfolder to an archive root. It augments backups to also record files
added in a __bkp__\subfolder\__added__.txt file, and runs a special merge mode
which does not delete unique items in TO, but does delete all items in TO that
are listed in the backup's __added__.txt file (which is itself removed from TO
after the merge). The net effect restores the TO tree's prior state, as long
as the restore run is made before any additional changes in TO.
Details:
Because a given item name may appear at multiple places in the archive tree,
replaced and deleted items are backed up in folders of this form (the mergeall
log gives a more linear list of changed items):
TO-archive-root\__bkp__\date-time\full-archive-path-less-root\filename
Items in the __bkp__ folder are local to the TO archive copy, and save items changed in
that copy only. Backup folders are not propagated to other archive copies by mergeall:
unlike normal archive items, they are not synchronized across trees. Hence any changes
in __bkp__ folders are not themselves backed up. Backup subfolders in __bkp__ are,
however, automatically pruned by age when their number exceeds a limit (changeable
in the code below). This allows changes from any of multiple mergeall runs to be
selected for restores, if needed.
Any errors while writing a backup copy of an item causes the replacement or deletion
of that item to be skipped, as the operation is unsafe. The item will register as
a difference on the next mergeall run, if not resolved manually.
Note that users can make arbitrary changes to their __bkp__ folders (including deleting
them altogether), as they are not synchronized and are created automatically whenever
needed. For instance, if the default pruning limit results in too much space being
taken by __bkp__, you may delete portions of it freely.
Also note that __bkp__ folders may register differences in diffall.py runs. As each
run has its own uniquely-named subfolders, these will usually be few, and can generally
be ignored. Per-run folders' content differences may be useful, however, to compare and
analyze backup copies from different runs; run diffall on the subfolders themselves.
Purpose:
Though this option makes mergeall runs safer in general, it was primarily intended as an
automatic guard against propagating corrupted files across multiple archive copies, and
an better alternative to manual backup copies or multiple archive stores cycled by date
manually. Merges generally works without issue if all but one archive copy are treated
as read-only; however, unwanted or incorrect changes may spread though archive copies
if changes are made in an archive copy and then synched back to the main copy. In such
cases, the original's versions of files may be quickly lost, if not backed up.
By backing up items to be changed in each TO copy automatically and retaining multiple
runs' backups, bad copies can generally be undone easily by taking the most recent
backup's versions, without falling back to potentially much older full archive copies.
Backup folders also serve as a record of the most recent mergeall runs against a tree.
------------------------------------------------------------------------------------------
More about __bkp__ folders:
Stores __bkp__ folder at top of TO archive so it doesn't become disjoint from it, but
does not include __bkp__ in the mergeall synchronization -- its contents are local to
the TO archive, and not copied to/from other copies automatically. This is so, because:
1) Propagating backups between copies in round-robin trips between three (or more)
devices seems potentially very confusing, and device source would be unclear.
2) Synchronizing would delete copes on TO not also on FROM, thereby limiting TO
from having more than one backup, in one-way propagation use cases.
3) Automatic backup pruning wouldn't be possible, as it violates the principle that
changes in the archive must all be disjoint during a single mergeall run. If __bkp__
was included in the synchronization process, it's not impossible that one of its folders
on the TO archive would be both pruned by the backup system here and deleted due to an
absence in the FROM archive -- regardless of the timing, the second deletion attempt
would fail with an exception.
More about disjoint changes:
More generally, mergeall works at all only because its updates are mutually exclusive,
and disjoint (a given name can appear in just one change category). Its changes are
limited to these items, in this order:
a) Same-named files (replaced in TO)
b) Unique files and directories in TO (deleted from TO)
c) Unique files and directories in FROM (copied to TO)
d) Same-named file/directory type mismatches between TO and FROM (replaced in TO)
Once any item is classified and changed in one of these categories, it is not
further inspected or altered. Hence, each change in the TO tree is limited to
its TO tree location, and cannot be impacted by, or appear as part of, any other
changes category, regardless of the timing of changes made. Order does matter for
renames on case-insensitive machines like Windows, but are sound as long as deletes
occur before adds (see mergeall.py's mergetrees() docstring for more details).
Backups rely on the fact that the change sets are disjoint: a file can be saved
for only one category (a, b, or d), and new copies (c) are never saved and thus
cannot overwrite saves for deletions, even for mixed-case differences on Windows.
Synchronizing the __bkp__ tree between archives would violate this disjointedness rule,
however, because automatic backup subfolder prunes may delete an item also scheduled for
deletion in category (b). This is true even though all differences are detected before
the changes phase begins, and only removing __bkp__ items from the set of backed-up
items does not circumvent this; synchronization changes may still intersect with prunes.
Therefore, top-level __bkp__ folders are removed from the set of synchronized items
in the archives, and an archive copy retains only backups from mergeall runs where
it served the role of TO archive. Backups can be manually copied into normal archive
folders to propagate them to other archive copies if desired, but their management
then becomes a user task; only items in the special-cased __bkp__ folders are
automatically pruned when their items count limit is exceeded.
------------------------------------------------------------------------------------------
General implementation notes:
Caveat: backup paths may be too long in some cases on some platforms. The original file's
full archive path is recreated and required, because the same name may appear in more
than one place in the tree (and filenames created by concatenating path parts seem more
likely to exceed platform length limits).
Does not store a single '.bkp' alongside original: would need to special-case to avoid
.bkp.bkp... accumulation, and a '.bkp' may be a valid to-be-archived user file name.
Uses copy instead of move (rename): need to retain original if code here fails, and also
want to retain the original's mod times in the copy. os.rename also proved unreliable
on Windows, especially across devices.
All errors return with an exception: caller handles. In mergeall, the backup exception
here causes the removal or deletion to be skipped, as the operation is then unsafe.
Backs up files replaced or deleted, but does not backup new files added, as these
would be just redundant copies of new data.
Backup folder are excluded from the synchronization process by removing only top-level
__bkp__ items from os.listdir results in the comparisons procedure. This proved more
reliable than os.rename moves to/from a temporary folder (see unused code ahead).
TBD: should backups be stored outside the archive itself? Originally placed in __bkp__
at root of archive, and synchronized so that prior versions are not stored on just one
device only (which may fail). The downside of this is that, when this option is used,
backups can accumulate quickly, and deletes are removed from the archive's actual folders
but linger on the device taking up space. Could backup replaced files only, but that's
less secure.
RESOLVED: later opted to store __bkp__ in the TO archive for association, but
not synchronize it across other archive copies. This was required, because
prunes violated the disjoint updates rule which is the logical basis of mergeall.
TBD: should backups be pruned? Currently, this is automatic, lest old backups
accumulate. The downside of this is that the user may not want old backups removed.
As an alternative, the end user could be expected to manage backups, and could also be
expected to select the backups folder. This would be flexible, but complicates the
command line and GUI with another directory choice, and deleting backup folders manually
seems a substantial extra admin task.
RESOLVED: backups are stored in the automatically-created __bkp__ at the top of the
TO archives, and are automatically pruned by age after N (default 10) copies have
accumulated. Given that __bkp__ is now local to each archive copy, this policy
allows ample backup retention -- any of the last N runs can be unwound if needed.
Users may also manually delete large backup folders before they are pruned for space.
TBD: __bkp__ folders will register differences in diffall.py runs for their top-level
per-run folders. These can be simply ignored in reports. diffall could skip __bkp__,
like mergeall, but this seems too tight a dependency between the two programs to enforce.
==========================================================================================
"""
###################
# CODE STARTS HERE
###################
from __future__ import print_function # Py 2.X
import os, time, glob, shutil, sys, time, errno, stat, unicodedata
# this module is _largely_ platform-neutral
RunningOnMac = sys.platform.startswith('darwin')
RunningOnWindows = sys.platform.startswith('win')
import mergeall # for copyfile, copytree (from won't work here: recursive)
indent1 = '....' # to distinguish messages here from main mergeall logic
indent2 = '____' # rmdir retries: not just for pruning (also for uniqueto)
# [2.1] from configs file, unless absent or errors
try:
from mergeall_configs import MAXBACKUPS
except:
MAXBACKUPS = 10 # keep up to this many backup folders in each archive copy's __bkp__
#-------------------------------------------------------------
# [2.3] Use UTF8 Unicode encoding for __added__.txt restore
# files in __bkp__, not platform default. These files contain
# filenames which may have arbitrary characters. Python 2.X's
# codecs.open files are always binary mode: must specialize
# line-ends here too. [3.2] ditto for deltas-set __added__.
#-------------------------------------------------------------
ADDENC = 'utf-8'
if sys.version[0] == '3':
unicode_open = open # what 3.X probably should have done?
unicode_linesep = '\n' # 3.X text mode files expand \n as needed
else:
import codecs
unicode_open = codecs.open # 2.X compatibility (or use from...as...)
unicode_linesep = os.linesep # 2.X codecs always binary: no \n expansion
#-------------------------------------------------------------
# Make just 1 subfolder per run, on first file backed up.
# Else may make 1 new subfolder for each second in run!
# Now also used by noteaddition, to get run's subfolder:
# either backupitem or noteaddition may be called first.
#-------------------------------------------------------------
datetimestamp = None
# try to prune just once per run, on first file backed up
pruned = False
# if -quiet, print backups message on first backup only [2.4]
firstbkpmsg = True
# [3.0] fix too-long paths on Windows
from fixlongpaths import FWP
# [3.3] fix Unicode variants in __added__.txt paths
from fixunicodedups import matchUnicodePathnames
def makedirs_ifneeded(dirpath):
"""
--------------------------------------------------------------------------------------
[2.3] Run an os.makedirs() call portably on Python 3.X or 2.X to create any and all
parts of a directory path as needed. Only 3.X has the exists_ok flag to avoid an
exception if a part already exists, and 2.X doesn't have 3.X's detailed exceptions.
Alternative: could also call os.path.exists(dirpath) in an "if", but slightly slower.
Note: Python's os.makedirs() in Lib\os.py is recursive, but probably doesn't need
to be. Because it scans a linear directory path, a simple loop should suffice, and
yield simpler code (alas, "batteries included" means you get what's shipped in the
box). Recursion is really required only for arbitrary shapes, such as folder trees,
and even then can be replaced with explicit stacks (see Learning Python 5E, p555-561).
--------------------------------------------------------------------------------------
"""
if sys.version[0] == '3':
# python 3.X
os.makedirs(FWP(dirpath), exist_ok=True) # 'recursive' mkdir, as needed
else:
# python 2.X
try:
os.makedirs(FWP(dirpath)) # 2.X has no exists_ok
except OSError as E: # 2.X requires errno test
import errno # only need here on exc
if E.errno != errno.EEXIST: raise # reraise all others
def backupitem(pathto, toroot, dobackup, quiet):
"""
----------------------------------------------------------------------------------------
If enabled by command-line (or by proxy via GUI toggle or console reply), make a backup
copy of items (files and directories) in the TO tree before they are destructively
replaced or deleted in-place. This includes a:
1) File or link in the TO tree
that is about to be replaced by a newer same-named file in the FROM tree
2) File or directory in the TO tree
that is about to be deleted due to absence in the FROM tree
3) File, directory, or link in TO
that is about to be replaced due to a dir/file/link type mismatch in the FROM tree
4) File, directory, or link in TO
that is about to be deleted by a -restore run because it's listed in __added__.txt
Unique files in FROM copied to TO are not backed up, as this is not a destructive action,
though they are listed by noteaddition in the backup's __added__.txt file for the run.
"pathto" is where the item to be destroyed resides; "toroot" is the top of the archive in
the TO tree, where the backup will be stored in a __bkp__ subfolder under pathto's tail.
Does not backup if not "dobackup": backups not enabled per user or rollback.py script.
Prints only one general backup log message if "quiet", to mimimize logfile clutter.
Either this or noteaddition may be called first: prune + timestamp if not yet done.
No longer needed, because __bkp__ not synched: "if pathto.startswith(bkproot): return".
[3.0] Use newly-added "strict" arg to copytree() to force it to propagate its first
file exception to here, instead of printing an error message and continuing. We need
to pass the eception to this function's caller, so the item's update is cancelled if
its backup fails. Else, we might delete a TO folder without backing up parts of it!
[3.0] Don't use copytree()'s new "skipcruft" here: we're backing up data already
present in the TO tree.
[3.0] Handle symlinks (to both files or dirs) by passing them to copyfile() just
like normal/simple files; copyfile() tests for links up-front and copies specially.
Symlinks are copied, not followed: backup the link itself, not what it refers to!
[3.2] Now also called to backup __added__.txt items removed from TO by -restore
runs, to allow delta-set changes to be rolled back (and rollbacks to be unrolled...).
----------------------------------------------------------------------------------------
"""
global firstbkpmsg
# do nothing if "-backup" not passed by command, launcher, or rollback.py
if not dobackup:
return # avoid nesting
assert pathto.startswith(toroot) # sanity check: changed file must be in TO tree
todir, tofile = '?', '?' # initialize for early exceptions
try:
# prune old backups the first time through here (or noteaddition)
try:
prunebkpdirs(toroot)
except:
print(indent1 + 'prune failed, but backups and mergeall continued')
print(indent1 + '%s %s' % (sys.exc_info()[0], sys.exc_info()[1]))
# make run's subfolder timestamp first time here (or noteaddition)
datetimestamp = makeruntimestamp()
# verify or create the backup-to path in the TO copy
bkproot = os.path.join(toroot, '__bkp__') # toroot is cmdargs.dirto
todir, tofile = os.path.split(pathto)
archtail = todir[(len(toroot) + len(os.sep)):] # remove prefix=cmdargs.dirto
bkppath = os.path.join(bkproot, datetimestamp, archtail)
makedirs_ifneeded(bkppath) # 'recursive' mkdir, as needed
# copy file or dir over to backup copy
copytopath = os.path.join(bkppath, tofile)
if not quiet:
print(indent1 + 'backing up %s to %s' % (tofile, copytopath))
elif firstbkpmsg:
# [2.4] suppress per-file messages (superfluous?), but indicate backups mode once
allbkpsroot = os.path.join(bkproot, datetimestamp)
print(indent1 + 'backing up all items to %s' % allbkpsroot) # [3.0] not bkppath!
firstbkpmsg = False
if (os.path.isfile(FWP(pathto)) or # [3.0] treat links like files
os.path.islink(FWP(pathto))): # [3.0] fix long windows paths
mergeall.copyfile(pathto, copytopath) # this never catches excs
elif os.path.isdir(FWP(pathto)):
os.mkdir(FWP(copytopath)) # [3.0] fix long Windows paths
mergeall.copytree(pathto, copytopath, strict=True) # [3.0] fail on any except
else:
assert False, ('unknown file type: ' + pathto) # e.g. fifos: punt
except:
print(indent1 + '**Error backing up %s in %s' % (tofile, todir))
raise # reraise: handle in caller - cancel update, as it would be destructive
def makeruntimestamp():
"""
---------------------------------------------------------------------------------------
Make the run's unique timestamp, to be used for its subfolder name in the TO tree's
__bkp__ backups folder. Factored to here, as this may be triggered by either backitem
or noteaddition, either of which may be called first during a mergeall run.
---------------------------------------------------------------------------------------
"""
global datetimestamp
if not datetimestamp:
datetimestamp = time.strftime('date%y%m%d-time%H%M%S') # backup's unique top dir name
return datetimestamp
def prunebkpdirs(toroot, maxbackups=MAXBACKUPS):
"""
---------------------------------------------------------------------------------------
On first backup in session, auto-delete the oldest backup dir(s) in the TO archive
if needed, keeping just the most recent N. Most of this was adapted from the frigcal
GUI's backup system. TBD: this could be left to user, but seems likely to accumulate.
Caller handles any exceptions here: this pre-merge step shouldn't be fatal - proceed.
[3.0] Recoded to skip a failed directory and continue the prune to process other
folders, instead of ending the prune at the first failure. Else, this may miss
folders in the unlikely event that a failure of a more recent backup folder (e.g.,
permissions) prevents the prune from reaching earlier backups later on the list...
which can only ever happen if the #backups has been reduced in the configs file.
Users must address the failure to allow the failing folder to be pruned eventually.
Callers are still notified with an exception, but details will be displayed here.
---------------------------------------------------------------------------------------
"""
global pruned
if pruned:
return
else:
pruned = True
bkproot = os.path.join(toroot, '__bkp__')
if os.path.exists(bkproot):
backuppatt = 'date*-time*'
currbackups = glob.glob(os.path.join(bkproot, backuppatt))
currbackups.sort(reverse=True)
prunes = currbackups[(maxbackups - 1):] # earliest last, via names sort
anyfailed = False
for prunee in prunes: # globs have full paths
print(indent1 + 'pruning', prunee) # normally 0 or 1, unless failed
try:
shutil.rmtree(FWP(prunee, force=True), onerror=rmtreeworkaround)
except Exception:
anyfailed = True
print(indent1 + 'this prunee failed, but pruning continued')
print(indent1 + '%s %s' % (sys.exc_info()[0], sys.exc_info()[1]))
assert not anyfailed, 'Some prunes had errors' # [3.0] notify caller of excs
def noteaddition(pathto, toroot, dobackup):
"""
--------------------------------------------------------------------------------------
[2.1] Log unique FROM items (files and dirs) added to the TO tree in a text file,
with 1 file path (relative to the TO tree's root) per line. The logged items are
stored in file: toroot\__bkp__\subfolder\__added__.txt. This allows additions to be
automatically removed by "-restore" as part of a later run to perform a complete
rollback. The adds file also serves as run documentation, in addition to logfiles.
Either this or backupitem may be called first: prune + timestamp if not yet done.
[3.2] The archtail length calc here can be thrown off by a trailing / or \ in
toroot (a.k.a. cmdargs_dirto); it's now removed in getargs() to avoid the drama.
This happened only when toroot cmdline arg had a trailing slash, and only for
top-level, unnested adds, but truncated the leftmost character in this case.
For items nested in folders, a double '//' left by the comparison-phase walker
spared the calcs here, and backupitem() was made immune by its spilt() call.
This also impacted the new deltas.py, where the issue was first uncovered.
This does not impact diffall.py or cpall.py (they simply treat two // as one).
[3.3] Note only: this doesn't print per-item messages, because the mergeall.py
caller does (for the new copied item recorded here). By contrast, deltas.py now
does print a message in similar code that adds to an __added__.txt file in delta
sets (for future TO removals), because the mergeall.py print code is not run.
--------------------------------------------------------------------------------------
"""
if not dobackup:
return # avoid nesting
try:
# prune old backups the first time through here (or backupitem)
try:
prunebkpdirs(toroot)
except:
print(indent1 + 'prune failed, but note-add and mergeall continued')
print(indent1 + '%s %s' % (sys.exc_info()[0], sys.exc_info()[1]))
# make run's subfolder timestamp first time here (or backupitem)
datetimestamp = makeruntimestamp()
# build and make the adds file's path
addsname = '__added__.txt'
bkproot = os.path.join(toroot, '__bkp__') # toroot is cmdargs.dirto
runroot = os.path.join(bkproot, datetimestamp) # bkp subfolder for this run
addspath = os.path.join(runroot, addsname) # adds file in run's subfolder
makedirs_ifneeded(runroot) # 'recursive' mkdir, as needed
# write copied item's relative path to adds file
# [2.3] use utf8 for filenames, not platform default
archtail = pathto[(len(toroot) + len(os.sep)):] # remove prefix=cmdargs.dirto
addsfile = unicode_open(addspath, encoding=ADDENC, mode='a') # add to end of file
try: # OLD: use default encoding in 3.X
addsfile.write(archtail + unicode_linesep) # 3.X expands \n, 2.X does not
finally:
addsfile.close()
except:
# do NOT reraise: no need to cancel the update, as adds are non-destructive
print('**Error noting add of %s' % pathto)
else:
# don't issue a trace mesage here, as it seems gratuitous in the logs
# print(indent1 + 'noted addition in TO of', archtail)
pass
def trymangle(delpath):
"""
----------------------------------------------------------------------
[3.2] For removeprioradds(): on Windows, mangle nonportable filename
characters to "_" to match other tools' mods. Do so always, instead
of just on errors: there's no way the unmangled form could be stored,
and os.path type tests fail with unmangled names.
Mangling can happen when content is copied from Unix to Windows, by
ziptools default unzips or similar. Mergeall in general assumes that
FROM names are mangled as needed by unzips or other copies before
syncs are run on Windows, but __added__.txt lists evade such transforms.
Hence, this code must mangle items listed in __added__.txt to match the
names under which they are stored on TO on Windows.
Caveats: names may be mangled differently by some tools (e.g., macOS
copies to FAT32/exFAT use Unicode privates), but the unmangled form
clearly won't work. This also risks removing files whose names match
the mangled name only coincidentally, but that seems astronomically
unlikely, and skipping removals seems worse. See ziptools.ziptools.py
and ziptools' _README.html#nomangle for more details; per the latter,
run the included fix-nonportable-filenames.py before transferring
content to skip filename interoperability issues like this altogether,
and always for transfers to Android shared storage not mangled here.
Theory: this use case cannot arise when propagating content with
Mergeall alone (it never mangles), or using intermediate drives with
filesystems or platforms that disallow nonportable names (e.g., Linux
refuses to write such names to FAT32 and exFAT). However, some other
tools mangle names on Windows (e.g., ziptools), and some drive/platform
combos allow but munge names (e.g., macOS munges to and from Unicode
privates on exFAT). In particular, changes propagated indirectly by
Mergeall's deltas.py and ziptools may wind up mangled on only Windows:
comparisons to a proxy drive with unmangled or munged characters may
record unique TO names in __added__.txt in unmangled form, but these
names are mangled on Windows if content was unzipped there by ziptools.
TBD: should mangling also be done for FAT/exFAT write errors on Linux?
ziptools does not mangle on Linux (it skips with error messages), but
other tools may. Really, the filename fixer script should be run in
this use case - unlike macOS (which munges nonportable chars to/from
Unicode privates covertly), Linux mangles will be perpetual diffs.
----------------------------------------------------------------------
"""
startpath = delpath
# drop drive first, so ':' not mangled
drive, delpath = os.path.splitdrive(delpath) # 'c:', '\folder\file'
# split path to parts
delparts = delpath.split(os.path.sep) # only on windows \ here, [0]=''
# illegal chars
nonportables = ' \x00 / \\ | < > ? * : " ' # for filesystems, not platforms
nonportables = nonportables.replace(' ', '') # drop space used for readability
if not any(c in part for part in delparts for c in nonportables):
# none found: mangling won't help
return startpath
else:
# mangle entire path, and report
replacements = {ord(c): '_' for c in nonportables}
mangledparts = [part.translate(replacements) for part in delparts]
mangledpath = os.path.sep.join(mangledparts)
mangledpath = drive + mangledpath
print(indent1 + '--Name mangled: "%s" => "%s"' % (startpath, mangledpath))
return mangledpath
def removeprioradds(fromroot, toroot, dobackup, quiet):
"""
------------------------------------------------------------------------------------
[2.1] Remove items listed in fromroot's __added__.txt file from the toroot
tree, if __added__.txt and the listed files are present, as part of a complete
rollback from backups (or deltas-set apply) in a mergall.py "-restore" run.
This is a pre-merge step: order matters for renames on Windows (the merge
must delete and then add, else delete may remove differently cased names).
Assumes fromroot is a __bkp__ subfolder (or at least has an __added__.txt), but
does not fail if not -- in all cases, ignore exceptions here. The user may have
deleted __added__.txt to back out removals+replacements only (not adds), and may
have created a custom __added__.txt elsewhere in another tree to be merged.
We need to care about closing the file on exceptions; this is now a pre-merge step.
The __bkp__ folder's __added__.txt will be copied over to the TO root by the normal
merge; it's deleted manually later, rather than skipping the name during the merge.
CAVEAT: because noteaddition() records added items using the path syntax of the
platform on which the prior mergeall ran, it's not possible to remove prior
additions on a different platform having incompatible path syntax without editing
either the additions file or the code here. A file added on Windows, for example,
will be noted with "\" in its path, which likely won't work in a restore on Linux.
This could be addressed by always using "/" in additions file paths and running
os.path.normpath() to convert to "\" on Windows only, but this seems a rare use case.
As is, restores with additions should be run on the same platform as the prior merge.
[3.0] UPDATE: this former caveat has been lifted, by converting paths in the
backup's __added__.txt file for the hosting platform's separators. Thus,
backups made on Windows can now be restored on Unix, and vice versa.
[3.2] CAVEAT: this may turn '\' in Unix filenames into path separators, but there
seems no reliable way to know if __added__.txt was created on Unix or Windows.
[3.0] Handle symlinks (to both files or dirs) by passing them to os.remove() just
like normal/simple files; shutil.rmtree() has issues with symlinks at the top;
symlinks are removed, not followed: delete the link itself, not what it refers to!
[3.0] Changed arguably-confusing message format here from "restore removed file:"
to the more consistent "removed added file:" -- it was removed, not restored.
[3.2] Backup deletions here: to support rollbacks of delta sets made by the new
deltas.py, this now always backs up the items it removes from TO, instead of simply
deleting them. For deltas applied with mergeall.py's "-restore -backup", this means
that items which were unique in TO will be put back by a later run with "-restore",
along with other undos. This also makes it possible to roll back true rollbacks
made with "-restore -backup", restoring an archive to its former post-sync state;
this use case is unlikely, but is supported at a small cost in extra backups size.
[3.2] Mangle __added__.txt names on Windows, so they match mangled names that may
have been saved by ziptools extracts for indirect deltas.py syncs; see trymangle().
[3.3] Morph __added__.txt pathnames to the new TO device for delta applies, in
case any component has an equivalent but different decoded Unicode representation.
Also change "removed added..." to generic "removed listed..." for delta-sets usage,
and overloaded "-quiet" to suppress messages for normalized names (here, in paths).
------------------------------------------------------------------------------------
"""
addsname = '__added__.txt'
addspath = os.path.join(fromroot, addsname)
numfilesdel = numdirsdel = 0
if os.path.exists(addspath) and os.path.isfile(addspath):
# [2.3] use utf8, not platform default
addsfile = unicode_open(addspath, encoding=ADDENC) # propagate open() exceptions: cancel merge
try: # OLD: adds file uses default encoding in 3.X
while True: # decodes can fail - catch via while, not for
try:
line = addsfile.readline()
except:
print('**Error: restore cannot read added file name: file retained')
print(sys.exc_info()[0], sys.exc_info()[1])
continue
else:
if not line: break # eof
# [3.0] make restore paths portable
delpath = line.rstrip()
delpath = delpath.replace('/', os.sep).replace('\\', os.sep)
delpath = os.path.join(toroot, delpath)
# [3.2] replace nonportable characters on windows
if RunningOnWindows:
delpath = trymangle(delpath)
# [3.3] morph Unicode in path to the TO device tree
tracer = print if not quiet else lambda *args: None
delpath = matchUnicodePathnames(delpath, tracer)
if os.path.isfile(FWP(delpath)) or os.path.islink(FWP(delpath)): # [3.0] +symlinks
try: # [3.0] longpaths
backupitem(delpath, toroot, dobackup, quiet) # [3.2] backup
os.remove(FWP(delpath))
except:
print('**Error: restore cannot delete file, retained:', delpath)
print(sys.exc_info()[0], sys.exc_info()[1])
else:
numfilesdel += 1
print(indent1 + 'removed listed file:', delpath)
elif os.path.isdir(FWP(delpath)):
try:
backupitem(delpath, toroot, dobackup, quiet) # [3.2] backup
shutil.rmtree(FWP(delpath, force=True), onerror=rmtreeworkaround)
except:
print('**Error: restore cannot delete dir, retained:', delpath)
print(sys.exc_info()[0], sys.exc_info()[1])
else:
numdirsdel += 1
print(indent1 + 'removed listed dir:', delpath)
else:
print('**Error: restore skipped missing or unknown type file:', delpath)
except:
print('**Error during prior adds removal') # others? don't reraise: do merge
print(sys.exc_info()[0], sys.exc_info()[1])
finally:
addsfile.close() # close, except or not (non-CPython)
return (numfilesdel, numdirsdel) # sums: add to merge's delete counts
def dropaddsfile(toroot):
"""
-------------------------------------------------------------------------------
In "-restore" mode, as a post-merge step get rid of the __added__.txt that
the normal merge may have copied over to TO's root. This is a special
case, but it's quicker to drop it forcibly from the root here than to check
for it as a skipped filename at each tree level during the merge (though
the merge's code supports skipping __bkp__ at top, __added__.txt otherwise).
[3.2] This is no longer used: __added__.txt is now skipped during the
comparisons phase, but only at the trees' top levels; see mergeall.py.
-------------------------------------------------------------------------------
"""
addsname = '__added__.txt'
mergedaddspath = os.path.join(toroot, addsname)
if os.path.exists(mergedaddspath):
os.remove(mergedaddspath)
return True
else:
return False # don't adjust merge's counters or print message
def rmtreeworkaround(function, fullpath, exc_info):
"""
---------------------------------------------------------------------------------------
Catch and try to recover from failures in Python's shutil.rmtree() folder
removal tool, which calls this function automatically on all system errors.
---------------------------------------------------------------------------------------
[2.0] PENDING DELETION FAILURES:
On Windows, deletes may be marked as pending, and not finalized atomically,
leaving an item in place after the delete call returns. This can cause
rmtree (shutil or custom) operations to fail with a directory-not-empty
error in rare cases, subject to devices and other activity on the machine.
This seems a shortcoming (bug?) in shutil.rmtree for Windows, and may be
improved in the future. In fact, Python's own test system uses a custom
rmtree with wait loops to avoid the issue. Here, update failures are mostly
harmless (leaving a difference to be resolved on the next run), and rare
(seen on only 1 machine in 1 year's usage), but errors are better avoided.
Short of low-level C API possibilities, the two solutions seem to be to move
(os.rename) to a temp file and delete from there, or fall into a brief wait loop
to watch for the file removal to be finalized. The former is subject to some
os.rename oddness (see ahead), and the latter is used in Python's own test system
for rmtree calls. Adopt the latter here -- this function is a callback on errors
in shutil.rmtree(), and retries the rmdir in a wait-but-bounded loop.
Note that this applies to, and is used by, _both_ backup folder removals here
and general content-tree removals in mergeall.py for unique dirs in the TO
tree; it's here because it was first observed during backup folder removals.
The fix here mimics Python's test system's wait-timing technique of exponentially
increasing delay times up to half a second. Usually 0 or 1, but at most 10,
retries are run, with delays from .001 to .512 seconds (to see how this is
computed, run code [x = 0.001, while x < 1.0: print(x); x *= 2]).
Caveats: Deletes that are only pending seem a curious property for a filesystem,
and this fix feels hackish. But this is harmless (it kicks in only on os.rmdir
failures, adding a minor delay), and there's no budget for further research...
Could watch for not empty (ENOTEMPTY) only, but other errors are not inconceivable.
Related threads (though something more authoritative from Microsoft would be nice):
http://stackoverflow.com/questions/3764072/c-win32-how-to-wait-for-a-pending-delete-to-complete
http://bugs.python.org/issue19811
Note: shutil.rmtree could also be replaced with the following (sans some Unix cruft):
for (root, dirs, files) in os.walk(top, topdown=False):
for name in files:
os.remove(os.path.join(root, name))
for name in dirs:
os.rmdir(os.path.join(root, name))
but this doesn't fix the Windows pending-deletes issue, and may be less robust
and portable than shutil's time-honed alternative.
UPDATE: the retry loop has now been seen to fire in more use cases: both when a
file is truly in use (in which case, the loop and removal ultimately fail), and
when it is not (in which case, the loop generally runs one or two times, and the
removal succeeds). Strange, but true...
---------------------------------------------------------------------------------------
[3.0] READ-ONLY PERMISSION FAILURES (pass):
As a different issue, rmtree operations can also fail due to read-only files in
the tree. To work around this, an onerror handler like the following can be used,
which is portable to both Unix-en and Windows, and works like a Unix "rm -rf":
import stat
def onerror(func, path, exc_info): # this is portable code
if not os.access(path, os.W_OK): # read-only permission?
os.chmod(path, stat.S_IWRITE) # change to allow writes
func(path) # and retry operation
else:
raise
This workaround was also incorporated into the general onerror handler below as
a first step, before attempting the retry loop described above. However, this
code is currently DISABLED (via the False), because it makes no sense to override
read-only permissions in this single context only (what about simple file deletes?),
and the user may have marked an item read-only on purpose to protect it. Instead,
users are expected to fix their read-only permissions and run mergeall again.
It can be argued that permission changes may be okay in this context, because
users understand that mergeall intends to remove items, and read-only items will
leave trash behind if not fixed. Moreover, files on Windows are sometimes marked
read-only without any user action (e.g., camera card copies), causing rmtree to
fail unexpectedly. This argument was rejected in the end, in favor of mergeall's
overarching policy that your data is your property; read-only files should never
be deleted without user intervention, even if this incurs extra manual steps.
---------------------------------------------------------------------------------------
[3.0]..[3.3] FILE-NOT-FOUND ERRORS:
On macOS, a hidden AppleDouble "._xxx" resource-fork file is automatically
deleted with its "xxx" data-fork (real) file. This occasionally causes
exceptions in Python's shutil.rmtree, if the AppleDouble is deleted
automatically _before_ rmtree gets around to deleting it manually. Though
rare, this has been seen to happen both on fast internal SSD and slow USB
flash drives. Skip the exception (and file) here for Mac "._" names only.
A similar coupling can occur on Windows (e.g., for web-site folders and
their HTML files, though this may be an Explorer kludge). As of [3.2], this
case is now handled explicitly for foldrs before the pending-deletion loop.
Should this simply always ignore all file-not-found errors? A file is gone
if it's gone, but file-not-found might be triggered in other contexts that
shouldn't be muted (e.g., long paths).
UPDATE: Windows long paths seem moot here. They can trigger file-not-found
too, but this should never arise in Mergeall. It prefixes all paths by FWP()
to lift the default path-length limit, before they are passed to system calls.
Hence, this context seems impossible, though file-not-found may arise otherwise.
UPDATE: per testing, Windows auto-deletion of folders with their same-named
files appears to happen in Explorer only, and not in the shell or Python.
Conversely, macOS auto-deletion of "._" AppleDouble files with their non-"._"
files IS automatic in all contexts tested, including the shell and Python
(create and view a file on an exFAT or FAT drive to see for yourself). This
seems to be embedded deeply in macOS's libraries, and likely reflects the
odd and proprietary resource/data-fork files split on that platform--a topic
generally best ignored... except in POSIX tree-removal code like this.
UPDATE [3.3]: per later research in the Android Deltas Sync system, Windows
auto-deletes are part of a Windows concept known as Connected Files, which
can be tweaked in the registry; Explorer just happens to recognize this
concept. Still, auto-deletes do NOT happen in either the Windows shell or
Python file ops, so the test here is moot, and has been disabled as of [3.3].
For a fuller exploration, visit the parallel common.rmtreeworkaround() tools
in A-D-S, at https://learning-python.com/android-deltas-sync/common.py.
---------------------------------------------------------------------------------------
"""
"""
# NO: why chmod in this context only?
# try to fix read-only errors first? [3.0]
if (not os.access(fullpath, os.W_OK)
and function in [os.rmdir, os.remove, os.unlink]):
try:
print(indent2 + 'fixing read-only on', fullpath)
os.chmod(fullpath, stat.S_IWRITE)
function(fullpath)
except:
pass # fail: try other workaround below (or not? tbd)
else:
return # okay: this fix worked, proceed with shutil.rmtree
"""
# Windows only, directory deletes only [2.0]
if RunningOnWindows and function == os.rmdir:
"""
# NO: moot per above - disable till use case arises [3.3]
# assume auto-removed with associated file by Windows, or other [3.2]
if exc_info[0] == FileNotFoundError:
msg = '**Note: ignored FileNotFoundError for Windows dir'
print(msg, fullpath)
return # folder deleted with file: proceed
"""
# wait for pending deletes of contents
timeout = 0.001 # nit: need to try iff ENOTEMPTY
while timeout < 1.0: # 10 tries only, increasing delays
print(indent2 + 'retrying rmdir') # set off, but not just for pruning
try:
os.rmdir(fullpath) # rerun the failed delete (post FWP!)
except os.error as exc:
if exc.errno == errno.ENOENT: # no such file (not-empty=ENOTEMPTY)
return # it's now gone: proceed with rmtree
else:
time.sleep(timeout) # wait for a fraction of second (.001=1 msec)
timeout *= 2 # and try again, with longer delay
else:
return # it's now gone: proceed with rmtree
# macOS only, ignore file-not-found for AppleDouble files [3.0]
if exc_info[0] == FileNotFoundError:
if RunningOnMac:
itemname = os.path.basename(fullpath)
if itemname.startswith('._'):
# assume auto-removed with associated file by macOS, or other
print('**Note: ignored FileNotFoundError for AppleDouble', fullpath)
return
elif RunningOnWindows:
# NO: this seems moot - isdir() should be False if FileNotFound [3.3];
# if this once had a purpose in a prior coding, it's been lost to time;
# it may have been for files auto-deleted with folders - that's moot too;
pass
"""
if os.path.isdir(fullpath):
# assume removed by Windows, or other
print('**Note: ignored FileNotFoundError for Windows dir', fullpath)
return # or just ignore all not-found excs on Windows?
"""
raise # all other cases, or wait loop end: reraise exception to kill rmtree caller
'''DEFUNCT
==============================================================================================
THE FOLLOWING FUNCTIONS ARE NO LONGER USED (but retained as examples and lessons)
Instead of the following, restructured mergeall's recursive comparison algorithm to
skip '__bkp__' items in the top-level os.listdir result only.
Testing in Python 3.X showed os.rename to be unreliable. On Windows, it fails when
the directories are on different devices (e.g., a USB stick and C:, possibly due to
differing file systems). It also generated unexplainable permission errors on one
Windows test machine, even when the source and destination were on the same file
system. As recursive copies and deletes are slow, recoded comparisons to skip
the folders in pure Python code instead. The shutil.move call tries os.rename and
falls back on copy+delete too, but it's prone to the same issues seen here.
==============================================================================================
import tempfile, stat
def excludebkpdirs(toroot, fromroot):
"""
---------------------------------------------------------------------------------------
Remove both archive's __bkp__ dirs from consideration, before diffs detection begins.
To avoid complicating and slowing (or rewriting) change detection, simply move
(rename) these out to a temp dir, and restore them after change detection finishes.
They will not register changes, and so won't be propagated to any other archives.
This and restorebkpdirs are coded fairly defensively, as this requires system calls;
os.rename has been seen to fail for a true temp dir on Windows due to permissions
(for no readily apparent reason...), so resort to program's cwd as a fallback option.
On Windows, the destination of os.rename cannot exist, even for dirs; use new subdirs.
mkdtemp adds random 6-character sequences to dir name till unique; add pid to be sure.
Caveat: this may run up against directory path-length limits on some platforms?
---------------------------------------------------------------------------------------
"""
global tempdir, temptobkp, tempfrombkp
tempdir = temptobkp = tempfrombkp = None
exists, join = os.path.exists, os.path.join
try:
tobkp = join(toroot, '__bkp__')
frombkp = join(fromroot, '__bkp__')
if exists(tobkp) or exists(frombkp):
try:
tempdir = tempfile.mkdtemp(prefix='mergeall-', suffix=str(os.getpid()))
if RunningOnWindows:
os.chmod(tempdir, stat.S_IWRITE) # may require force writeable?
open('temp.txt', 'w').write('try rename\n')
try:
os.rename('temp.txt', join(tempdir, 'temp.txt'))
except:
os.remove('temp.txt')
raise # reraise
else:
os.remove(join(tempdir, 'temp.txt'))
except:
print('using cwd as temp dir fallback') # show sys.exc_info()[0,1]?
print(sys.exc_info()[0], sys.exc_info()[1])
tempdir = os.getcwd() # temp unusable; or os.curdir
if exists(tobkp):
print('excluding', tobkp)
temptobkp = join(tempdir, 'to.__bkp__')
os.rename(tobkp, temptobkp) # quick move, not copy
if exists(frombkp):
print('excluding', frombkp)
tempfrombkp = join(tempdir, 'from.__bkp__')
os.rename(frombkp, tempfrombkp) # either can exist or not
except:
print('Cannot move __bkp__ to temp: rerun after manually moving out of archive')
assert False, 'mergeall changes cancelled'
def restorebkpdirs(toroot, fromroot):
"""
---------------------------------------------------------------------------------------
Restore __bkp__ folders from temp dir, after diffs detection, and before changes begin.
See excludebkpdirs above for more details.
---------------------------------------------------------------------------------------
"""
global tempdir, temptobkp, tempfrombkp
join = os.path.join
try:
if temptobkp:
tobkp = join(toroot, '__bkp__')
print('restoring', tobkp)
os.rename(temptobkp, tobkp)
if tempfrombkp:
frombkp = join(fromroot, '__bkp__')
print('restoring', frombkp)
os.rename(tempfrombkp, frombkp)
except:
print('Cannot restore __bkp__ from temp, changes cancelled: restore from %s' % tempdir)
assert False, 'mergeall changes cancelled'
else:
if tempdir != None and tempdir != os.getcwd(): # not if fallback to cwd!
os.rmdir(tempdir)
def isbkpdir(path, archroot):
"""
---------------------------------------------------------------------------------------
Original idea: Call this from mergeall to skip a TO or FROM __bkp__ path during diffs
detection phase. Because these are skipped, they won't trigger any changes in the
changes phase. This is required to avoid including __bkp__ in synchronization (see top
docstring). Later replaced with os.rename moves, which was later replaced with
comparison recoding.
---------------------------------------------------------------------------------------
"""
bkproot = os.path.join(archroot, '__bkp__')
return os.path.normpath(path).startswith(os.path.normpath(bkproot)) # equate / and \; case?
# that is...
#return path[(len(archroot) + len(os.sep)):] == '__bkp__'
DEFUNCT'''