|Version:||1.1, November 26, 2021 (changes)|
|License:||Provided freely, but with no warranties of any kind|
|Author:||© M. Lutz (learning-python.com), 2021|
|Install:||Fetch the download, unzip its code, see ahead|
|Requires:||Python 3.5 or later, Mergeall 3.2 or later|
|Runs on:||macOS, Windows, Linux, Android|
This package was born of frustration with the twin removals of POSIX USB access in Android 11, and microSD-card support in Samsung's flagship phones. Those removals crippled content-processing tools, as covered here, here, and here.
This usage guide leaves the drama behind, and focuses instead on how to use the work-around embodied in this package's scripts to propagate and sync content from PCs to phones running Android 11 and later. As we'll see, this work-around is partly manual but straightforward. It may also be permanent; while the future is unknowable, the Android 11 speed and access limitations we'll remedy here are still present in Android 12.
This doc begins with a brief overview, followed by a quick start, additional usage details, and general notes. If you'd like a headfirst preview, try this package's screenshots, examples, and code.
This package provides Python scripts which copy content from a PC to an Android device initially; sync the Android device's copy later for changes made on the PC; and optionally verify or export the Android device's content copy to the PC. These scripts are run as pairs on PC and phone, and automate steps that would be tedious and error prone to run manually.
In this guide, "PC" means any macOS, Windows, or Linux computer, and "Android device" means any Android phone or tablet—though "phone" includes tablets here for brevity, and this package is primarily of interest to users of Android 11 and later because it addresses USB restrictions that debuted in 11 (see Scope ahead for more info). In brief, the scripts here:
The net effect allows POSIX programs like Mergeall to be used for content management on Android 11 and later, and propagates content from PC to phone safely and easily. The next section details the process that achieves this.
To sync changes from PC to phone, this system leverages a proxy drive, which takes the place of the phone for change detection: as long as the proxy's content copy is always the same as the phone's, PC-to-proxy changes double as PC-to-phone changes, and can be computed independently of the phone. The proxy's role is similar to an intermediate drive in earlier Mergeall models, and partly replaces now-defunct microSD cards. Under Android 11, however, the proxy's content copy cannot be merged to the phone directly by USB with Mergeall.
To address this loss of USB access, this system also employs a new Mergeall
This script detects differences between FROM and TO as usual, but instead of applying
FROM changes to TO immediately like a normal sync, it saves the changes separately to
a folder, in the same format as Mergeall backups. This allows changes (a.k.a. deltas)
to be archived as a unit, copied between devices, and applied to TO later with a
-restore run. Because delta sets are almost always very small,
these tasks are almost always very quick.
By combining the new script's deferred changes with the proxy's forgery, this system is able to apply changes to the phone transitively: changes are collected by comparing the PC's content copy to the proxy's; transferred from the proxy to the phone manually with a file explorer; and finally applied to the phone itself. The net effect updates the phone for changes on the PC indirectly, without requiring POSIX USB access on the phone.
Beyond orchestrating the new script and proxy, this system also ensures that content is zipped and unzipped to evade interoperability perils in filesystems and platforms along the way. Depending on the target of content propagations, this can avoid loss of symlinks, permissions, folder modtimes (modification times), and nonportable filenames, all of which may otherwise be silently munged in raw transfers. Zipping also minimizes the role of Android file explorers, which are generally slow and frequently fail for raw copies of large folders.
This may sound like a lot of work (and it is, compared to the direct two-merge process before Android 11), but it's still well worth the effort for people who don't want to store their private digital property on clouds, where it can be arbitrarily misused by hosting companies.
The good news is that this package's scripts automate most of the details behind these indirect syncs. In short, you will simply:
In the following, you'll generally run the setup tasks and initial copy just once, repeat the sync steps any time you wish to propagate changes from PC to phone, and verify occasionally. Important: script pairs are meant to be run in sequence; in syncs, for example, the phone-side script must be run before the pc-side script is run again. See install details ahead for specific version requirements and more options.
pkg install python
On both PC and phone:
Dedicate an external USB "proxy" drive to host a content copy that will serve as a stand-in for your phone in later syncs. This copy should never be changed except by this system's scripts, else change detection won't work and you may have to rerun an initial copy.
The drive should be large enough to hold your content and more—twice as large for verifications and initial copies, but a smaller increment for later change syncs (see space requirements ahead). exFAT is recommended for the proxy for its larger file sizes and friendlier modtimes, but any filesystem supported by both your PC and phone will do.
This section goes into more detail about the steps outlined in the preceding Quick Start. It describes setup tasks and interaction, and covers enough Android and script-launch fundamentals to get you started.
Download and unzip both this
and the source-code package of
to any folder on your PC and phone (e.g., the usual
suffices on Android). This system requires Mergeall 3.2 or later, for its new
script and included
ziptools 1.3 or later is
required, but is shipped with Mergeall 3.2 and need not be installed separately;
you can find more info at ziptools' separate source
This system also requires and works with any Python 3.5 or later. Python is available in Android Python apps, and may be installed for PCs from here if not already present. See the lists below for more on installing Android-app Pythons, and the online guide for PC install tips. Python 3.5 hails from 2015, and most PC distributions and Android apps are already well past it (e.g., the apps listed below use 3.10 and 3.9 at this writing).
To unzip packages, see PC and Android file explorers for initial options; you can use ziptools to unzip install packages safely and quickly, but not until you've unzipped it. Also note that you do not need to install Termux if you will be using another app to run phone-side scripts (e.g., Pydroid 3), but Termux is recommended if you have no other preference. The following is a usage-essentials rundown for host apps on Android:
pkg install pythonto be run in its console to install its Python.
termux-setup-storageto be run in its console if you will be using shared storage for your content. Read the details here and here, and stay tuned for storage-options coverage ahead. Termux may lose this permission shortly after it is installed on Android 11; to fix, toggle its storage permissions in Settings/Apps off and on (yes, really).
Acquire wakelockin Notifications, plus an equivalent console command, to ensure that a long-running script will continue even if your phone's screen goes blank. This is more important for initial copies than later syncs, because deltas are normally small. Either way, remember to later release a manually acquired wakelock to avoid battery drain.
For more on Termux and other on-phone options, watch for upcoming launch details.
Update: Shims Wakelock The "shim" scripts provided for use with the Termux home-screen widget on your phone now automatically acquire and release an Android wakelock around script runs, to ensure that the scripts continue even if your phone screen goes blank. Hence, you don't have to manage the lock manually. See the shims and their README here for more info, and see ahead for more on the Termux widget; it's optional but recommended for on-phone scripts.
The initial-copy scripts here temporarily require twice as much total space as your content collection itself on your PC, proxy drive, and phone. To copy a 100G content collection, for example, you'll need 200G on all three devices during the copy, but just 100G later. This stems from the scripts' use of zipfiles to avoid interoperability issues; an extra zipfile copy is passed between devices to avoid data loss.
If your devices don't have enough space, see the later notes on proxy and phone space for manual work-arounds. In short, you can emulate this system's steps with command lines, or skip some altogether. PCs require twice content's size during initial copies too, but are less likely to be as tight on space as phones and proxies.
By contrast, the sync-changes scripts here require only enough extra space for one delta set, which is normally very small and transfers quickly. In other words, once you get an initial copy onto your phone one way or another, later syncs with this system's scripts are unlikely to run into space limits.
In all cases, this system's scripts automatically delete both zipfiles and folders on PC and phone no longer needed after script completion. This includes temporary content and deltas folders, as well as their zipfiles. Exception: zipfiles for both the initial copy and deltas set on the proxy drive cannot be automatically deleted, because these scripts cannot access USB on the phone. You can freely delete these zipfiles from your proxy to save space, after their content has been propagated to your phone. Caution: don't delete the proxy's content-copy folder; it's needed for later syncs.
If your proxy drive is formatted as FAT32, you may also bump up against that filesystem's 4G file-size limit during initial copies of content; consider reformatting your proxy drive to use exFAT if it's supported by your phone or an add-on app, and see ahead for more details.
Update: Verify Space When used, the 1.1 verify scripts described ahead also temporarily require twice the space of your content on your phone, proxy drive, and PC, for an extra content zip on the first two, and an extra content unzip on the PC. They're also a sparingly used special case and fully optional.
Edit both config_pc.py and config_phone.py to set the folder paths to code and content on your PC and phone, respectively, and change run options, optionally. See these files' top-of-file documentation and comments for directions, and the examples folder for config-file examples to emulate. Config-file settings are straightforward and are checked for errors before scripts start, but here are a few pointers on this step to help you avoid mistakes:
=assignments. The settings' names should not be changed.
/sdcardis usually the root (leftmost) part of internal-storage paths on your phone (regardless of what file explorers call it), and PC and proxy roots vary per platform (e.g.,
D:\). For more info, see your file explorer, the pathname primer, and the examples.
The config files are also technically Python code, but their settings are few and simple. For readers new to this language, here are five essential syntax tips:
#anywhere starts an ignored comment
Falsestart with an uppercase letter
The following list provides additional details and advanced usage tips for config files:
\) in strings as the start of an escape sequence (e.g.,
\nmeans line-break). Since this character is also used as a path separator on Windows, it requires special handling in Python strings: the
r'...'prevents Python from treating
\as an escape character; doubling up with
\\is an escape that stands for one backslash; and forward slashes (
/) work in place of backslashes in Windows paths too.
See the Windows
example for all
three codings in action.
Forward slashes in Unix paths require no such special handling.
Embedded quotes must be escaped everywhere (e.g.,
but are enormously rare.
~in a path string is replaced with the path to the home folder associated with your user account, and
%var%on Windows) are replaced with the value of the named environment variable.
This works on all platforms: a leading
~ is replaced with
/Users/you on macOS,
C:\Users\you on Windows,
/home/you on Linux, and
/data/data/com.termux/files/home on Android Termux.
Variable replacement is handy for Linux proxy-drive paths that depend
on your user name: use
For finer details, see the underlying Python tools
and the examples
'D:\\', using the Python doubled-up escape scheme described above). If the slash is omitted (e.g.,
'D:'), the path is relative to the current working directory on the named drive, which is not what you'll usually want. For more background on this, see this Microsoft article. Scripts here don't automatically add a
\if omitted, because this would preclude drive-relative paths in contexts where they are useful.
See also the "z-pc-froms" captures in the screenshots gallery starting here for more path examples. Coverage for specific configuration settings appears ahead in this guide, in the topics with which settings are associated; search in your browser to find more about a given setting here.
In the phone config file, you'll need to choose one of three on-phone storage categories to host your content. In brief:
/sdcard) can be both viewed and changed by most apps, and is never automatically deleted on app uninstall; but it may run much slower than the other two storages in Android 11—perhaps 2X to 100X slower, depending on usage. This space generally includes everything under its path that's not part of the next bullet.
/sdcard/Android/data/appname) is quicker, and can be viewed and changed by many apps; but it is unsupported by some apps, and may be automatically deleted when the owning app is uninstalled. This space looks like it's part of shared storage, but it's implemented (i.e., mounted) differently.
/data/data/appname) is also fast, and supports Unix metadata best; but it can be accessed only by the owning app, and may also be automatically deleted when the owning app is uninstalled. This space's sole-app role largely disqualifies it for general content stores, and most of this doc's coverage.
Three disclaimers up front. First, due to Android fragmentation, these
category's paths may vary across versions and vendors (e.g.,
/storage/emulated/0); for a deeper dive, try the
Second, the appname part in these paths varies too; it's
And third, due to a now-ancient SD-card legacy, some Android docs confusingly call
app-specific "external" and app-private "internal," but both usually reside
in your phone today; they're just managed differently.
Also note that everything in this section and doc applies to unrooted devices; while phone rooting may loosen some constraints, it's not possible on many phones, and not desirable for many users (see this package's genesis for more details). Small print aside, the choice of storage host for your content pivots on four dimensions, some of which may be more important for your use cases than others:
In the plus column, most Android 11 file explorers and the apps they open allow you to view and change your content in both shared storage and any app's app-specific storage. This may require one or more permission dialogs, but is broadly supported. Less positively, app-specific storage may be a read-only resource if Unix permissions are propagated, and app-specific support varies by explorer and viewer (e.g., Fx can currently explore shared but not app-specific storage on 11, and Music Folder Player bears the same limitation for audio files).
More rigidly, Termux and Python apps cannot access other apps' app-specific storage in Android 11 today, so your choice of content host may be binding. If you run this system's scripts in Termux, for example, you can store content in either shared storage or Termux's own app-specific storage, but programs run in Pydroid 3 won't be able to access the latter. Conversely, because Pydroid 3 is required to run tkinter GUIs, their code and data must be in either shared storage or Pydroid 3's app-specific storage, but Termux is locked out of the latter.
For the gritty details on Python-related apps, see the captured results for
As shown, these apps today can access shared storage (including
and their own app storages only, though this depends on both which
Androids apps target, and the ever-changing rules of Android itself.
As an example, unzipping and deleting a 200G/180k-file collection in app-specific storage with Termux took just 18 and 10 minutes, respectively; but the same tasks in shared storage checked in at an arguably unusable 1 hour and 4 hours, respectively (and the shared-storage delete failed altogether in two file-explorer apps). Indeed, large-folder deletion is so slow in Android 11's shared storage that a phone reset may be quicker.
That said, not all shared-storage operations are so slow, and typical content access may be adequately fast for your purposes. Moreover, this package's use of delta sets naturally speeds on-phone syncs, because it avoids full on-phone content comparisons: smaller delta-set compares offset slower shared-storage access.
In fact, per practice so far, media access in shared storage is generally quick, and
the phone portion of syncs for numerous changes with shared storage usually takes
no more than 2 minutes—including time for the manual zipfile copy.
Still, your syncs may naturally vary, and auto-deletion prunes of large
in shared storage can be so slow that they seem to hang. Prune sloth was eventually
sidestepped with config presets described ahead,
but check script status in logfiles if pauses are
long, and use app storage if waits are intolerable.
This content-loss risk alone is nearly enough to disqualify app storage. It's also not entirely abstract; the Mergeall dev team itself accidentally erased gigabytes of content this way while setting up apps to test Termux's widget.
Special case: shared storage does not emulate FAT32's "local time" for file modtimes, and hence is immune to FAT32 skew on DST rollovers and time-zone changes described here. This matters because modtimes in on-phone storage must still differ with those in a deltas zip for changes to be applied. On multiple devices, shared, app-specific, and app-private modtimes all agreed with each other and PCs after changing dates and zones—and still did after an actual DST rollover. If modtimes ever skew anyhow, run the fixer script on deltas, or try app storage.
So where should you put your stuff on your phone, then? As guidelines—and barring a future Android fix for shared-storage throttling which would make that storage a clearer winner:
If pressed for a single recommendation, shared storage's cross-app accessibility make it the better choice for content-creation tasks such as programming, and its permanence is a clear win for valued content. App-specific storage, however, remains a faster and viable contender for basic content views, and app-private storage can still take the title in single-app roles; just don't delete the app!
You can run this system's scripts by typing command lines which name them in consoles (a.k.a. terminals), or by running their files directly with icon clicks, shortcuts, and IDEs (edit/run GUIs). IDEs work because these scripts are coded in Python, and icon clicks and shortcuts work because the scripts are, for ease of use, configured by settings in files you'll generally edit once, not by command-line arguments passed on each run. Here are the basics of each of these run modes:
chmod +x script.pycommand, and ensuring that they are on your system search path (or using
./script.pyto skip the search).
Command-line novices tip:
depending on where you unzipped this package, its scripts
may not be in the folder you start at in your console tool.
If the scripts are elsewhere, either add their location's path
to the front of their names in the commands
cd path to go to their folder before
will likely be
on your phone, and similar on PCs.
ForceKeyToCloseconfig, but this is automatic for Windows clicks.
macOS and Linux run Python files by clicks too; see their file explorers for tips, or simply click a Python file to try. In brief: on macOS, right click a file in Finder and pick Get Info to associate a Python opener program globally, and do the same to access a file's Properties and associate openers in Nautilus (a.k.a. Files) on Linux.
On Android, you can use a file explorer to make home-screen links that open phone-side scripts in an IDE app, ready to run in just two taps. For instance, the Total Commander app can make such links to open scripts in Pydroid 3 on all Androids (but be sure to unclick its "Parameters: content://url" checkbox in later Androids), and other explorer/IDE combos may offer similar options. See Pydroid 3 shortcuts at work in screenshots here, here, and here.
Also on Android, a Termux
add-on provides a
home-screen widget which can run a Python console script like those in this
package in a single tap.
Because this widget is so handy, this system's phone-side scripts support it with
an absolute path in their shebang (
lines, along with a
setting to require a key
press before the widget erases the console (like Windows icon clicks).
If the shebang fails in other Android apps, edit scripts to use the line just
below it, or simply use an explicit "python3" in console
See the Termux widget in action in screenshots
get the widget on
and see other Termux
Termux-widget tip: see the included
shim scripts for an easy way to run this
system's main scripts, without having to copy the main scripts to the widget's folder,
change the prior paragraph's config, or manually acquire and release the
wakelock. Simply copy the shims to your
~/.shortcuts folder in Termux, edit their
runee paths if
needed, and launch scripts by taps in the widget.
See the README for details.
Constraint: shortcut or not, it doesn't help to open this system's scripts in an IDE app on your phone, unless you host your content in storage which that app can access. As we saw earlier, this means either slow shared storage or that app's own specific or private storage, and the choice can limit scope in Android 11. You can't use Pydroid 3 to open scripts in Termux's app-specific storage—and vice versa. This is a catch-22 inherent in Android 11, and a nearly compelling argument for using slower shared storage.
For demos of script interaction, see the run logs in this package's examples folder, as well as the Android screenshot. For instance, here are typical sync sessions on a PC and a phone. These scripts explain their actions along the way and are straightforward to use, but the following covers a few salient points:
< fileinput-stream or
<<EOFhere-document syntax. See other resources for pointers. Tip: at "y or n" (yes or no) prompts, pressing enter/return alone is the same as inputting "n" (no)—as is any reply that isn't "y" itself.
Falsein either or both config files. Alternatively, pause key-press inputs can be automated, per the preceding bullet.
Falsein the config files (these times may be more useful for long-running initial copies than quicker later syncs). Total elapsed runtime—all steps' time + all user time—is always displayed at script exit.
Tip: most of this package's screenshots have the luxury of being captured on a large Fold3 screen, but you'll probably want to run its scripts in landscape orientation on smaller phones to avoid line wraps in messages. The narrower Note20 Ultra used in this shot demos the idea.
Step logfiles give the full details of a run.
They show up in the folders whose pathnames you assigned to
LOGS in your PC and phone config-file settings.
Here are example logfile-folder listings from a
PC and a
Read the logfiles there later, or monitor a long-running step's progress
with a command like the following in a console run in a different window,
logfilename is the latest item in your
tail -f LOGS/logfilenameThis command works in all Unix-like shells, including Termux on your phone. In Termux, you can both open a new session to tail logfiles and switch sessions by swiping in from the top left.
Besides saving step output to logfiles, this system also displays
the last few lines of each step's output in the run's console
to give you a quick summary of results. If this summary looks amiss,
open the logfile in
LOGS for the complete story.
For a broad sample of run logfiles,
see the demos'
in this package's examples folder (here's a typical
Tech note: logfiles are written in unbuffered mode, so their output
shows up right away to give script status: the end of the log is
the most-recent message written.
On your phone, you will be instructed to manually copy a zipfile from the proxy drive's root to a specific phone folder, during the initial copy, and later syncs. Verify runs ask you to copy a zipfile from your phone to the proxy instead.
When prompted, copy the named zipfile using any Android file-explorer app which has permission to access both a USB drive, and the storage location to which or from which you are copying. Most explorers will suffice, and will work well because this is a single-file copy, but see the Android-explorer coverage ahead for tips.
This copy must be a manual step, because it cannot be run by command line in Termux (or similar). Like all other POSIX programs, Termux no longer has direct access to USB in Android 11, per this screenshot and coverage. This is the only manual step in initial copies and later syncs and verifications.
Note that file explorers'
abstract the root folder of internal
storage and USB drives in different ways. The root path of your internal
/sdcard, for instance, may be simply called "Internal Memory,"
"Main storage," or similar, and an attached USB drive at
/mnt/media_rw/ID uses nicknames too.
Extrapolate from path to explorer label as needed.
is really the root of your per-user space in internal storage. Rooted phones
have access to the higher and broader device storage root,
but they also can access USB drives from POSIX programs at
and so have no compelling reason to run this package.
As preset, Unix file and folder permissions are not propagated to the destination phone, because they have no effect in shared storage, and can make it impossible to change or delete content in app-specific storage. The latter may reflect an Android bug, but is crucial for content hosted in app-specific storage.
Not all Android 11 devices retain propagated Unix permissions in app-specific storage. Per ziptools' research and demo noted earlier, the Z Fold3 does, but the Pixel 4a does not. Where supported, however, propagated Unix permissions typically make it impossible to update or remove content in app-specific storage in any file explorer or app, other than the storage's owner. With permissions propagation, for example, content stored by this system in Termux's app-specific storage can be viewed by other apps, but can be modified only by Termux. Allowing Android to stamp items with default permissions instead enables updates and deletions from any approved app.
Hence, this package's phone scripts by default omit ziptools'
flag in on-phone extracts, to prevent propagation of Unix permissions. To force
propagation in contexts where it makes
True in the phone config file.
This may be useful for content hosted in app-private storage; since
content there can never be accessed by any app except the owner, cross-app
updates are irrelevant. This config is moot for shared storage, because
permission propagations there are always ignored by Android.
Note that permissions are always retained in zipfiles created on Unix PCs; the config setting prevents copying them to phone storage types which support them (but perhaps shouldn't). Also note that this system assumes phones are view-only devices in general, but does not assume that on-phone content can never be changed; program data files, for example, may still require write access.
Update: Verify Permissions As a minor twist, the verify scripts added in 1.1 always propagate Unix permissions from phone to PC—though these will generally be defaults when mirrored from both shared and app-specific storage (and largely pointless on Windows PCs).
Except when running on Windows natively, this system by default offers to automatically run Mergeall's fixer tool for nonportable filenames on the PC, as a first step. Where relevant, this step is crucial to content interoperability, though its usage story depends on whether you run the PC-side scripts on Unix or Windows:
If you don't run this step, nonportable filenames that work on Unix may be mangled (i.e. changed) when propagated elsewhere, and their files may be discarded in full by some tools. In this package specifically: such files are skipped with an error message when transferring them to Android 11 shared storage, but saved intact to Android 11 app-specific and app-private storage, as well as earlier Androids' shared storage. This reflects the underlying ziptools system.
The fate of nonportably named files on proxy drives, however, is PC and drive dependent. Such files fail on Linux when written to FAT32 and exFAT proxies. By contrast, such names are saved on such proxies without error on macOS due to that platform's munging to and from Unicode privates, but may fail in later network back syncs. On both platforms, nonportable names may also leak out to Android in this system's initial-copy and deltas zips, and fail there. For examples of the effects, see the shared-storage and Linux demos here and here.
Though recommended in general for all these reasons,
if this step proves irrelevant to your
use cases, it can be skipped at its prompt, and its offer can be disabled
altogether by setting
False in the
PC configs file.
win32) in Python's
sys.platform. Non-native Windows tools (e.g., Cygwin's own Python) may yield nonportable names, but do not register as Windows this way, and so will invoke the fixer step to adjust filenames.
The filename behavior of non-native Windows tools is subtle.
Cygwin, for example, allows nonportable
names to be created via name
much like macOS. Its own Python reverses
the munge to nonportable characters, but registers as
Windows' Python run in Cygwin register as Windows, but does not map munge
characters back to nonportables, so files do not fail.
Linux munges nonportables just like Cygwin, and registers as
For demos of Cygwin and WSL in action, see the examples
Hence, the name-fixer step is always skipped in Windows-native
contexts, for which the
CheckFilenames config setting
is ignored. Cygwin and WSL users can change this setting to skip
the step as desired.
For more on the name-fixer step in general, see the comments at the top of Mergeall's fixer tool, and ziptools' related coverage of nonportable filenames here.
Related note: Unix filenames should generally be run through the name-fixer script anytime they are copied to Windows—including in manual zips created outside this system. Else, the unzip may discard files or handle their names unexpectedly (e.g., backslashes in Unix filenames generate subfolders on Windows in ziptools, but are less harmfully replaced with an underscore in the fixer). See the Windows example for a rare case where this matters, and the Windows unzips coverage ahead for related info. Avoiding nonportable filenames in the first place may be their best work-around.
script supports Mergeall's
-report option to
preview differences without saving them, though using the same
has the same effect. This preview is
run initially by this system's PC-side sync
True in the
PC config file, and you confirm the run interactively.
This config setting is preset to
True because previews are generally advised.
Use them as desired to inspect deltas before writing them to disk.
This may, for example, avoid unintended and time-consuming
full copies at DST rollovers when using FAT32 proxy drives (see Mergeall
There is no direct way to verify the on-phone content copy against the proxy or PC copies, because POSIX code like Python cannot access USB drives normally on Android 11, and phone access by MTP and network servers is not fully usable. The phone might be compared to the proxy after syncs by manually copying full zipped content from proxy to phone, or vice versa; this is too slow to automate here, but might serve as an occasional and manual option (tip: zip content first on phone or proxy using a tool like ziptools—or see the update ahead for a precoded solution that does).
As a partial measure, setting
in the PC config file
runs mergeall and/or diffall
to verify the proxy's content against the PC's in both initial copies
and later syncs. The mergeall run compares by structure and modtimes,
and diffall performs a full byte-for-byte comparison.
The config setting is preset to
True because verifications
are generally recommended, but you are prompted to confirm each verify
step independently because they add time to the run unevenly. A mergeall
verify is usually quick, but a more-complete diffall can be slow for larger
Note that it's normal and harmless to have one diffall diff: a unique item
for the copies' top-level
Also note that symlinks on Linux cannot be written to FAT32 or exFAT drives,
and so will always register as PC-side diffs. This is an innate limitation
of these filesystems on Linux—which macOS avoids only by forging a
symlink stub file which passes for an actual link on macOS alone, and will
still yield diffs elsewhere.
For an example of the consequences on Linux, see its
The only external-drive symlink option on Linux is other filesystems.
Update: Phone Verifications As of version 1.1 of this system, it is now possible to verify the phone's content copy against the PC's, by running the new verify scripts. For a quick look, see them in action in the new example and screenshots. These scripts switch dataflow order—content moves from phone to PC—but config-file settings work unchanged on both phone and PC.
This process is optional and meant to be used only occasionally, because it is relatively slow and temporarily requires twice the space of your content on phone, proxy, and PC for an extra content copy or unzip. As chronicled by the new example's readme, for example, verifying an on-phone content folder of some 200G space and 160K files required an extra 200G everywhere (400G total capacity), and took roughly one hour (50 minutes on phone and 15 on PC). Especially if your phone hosts a throwaway copy, such costs may be unwarranted.
Nevertheless, if you do wish to prove that the phone's copy matches that on PC, the verify scripts zip full content on your phone; prompt you to manually copy the zipfile to your USB proxy; and then unzip the phone's content on your PC and compare it to the PC's copy with Mergeall and diffall. As noted, the former compares content copies by modtimes and structure; the latter is a byte-for-byte compare which is more complete but slower, and an optional step in verify scripts.
Though too slow for frequent use, this suffices to validate phone content: you should see differences only for normal skew (e.g., backup folders) and items that cannot be stored on your phone (e.g., symlinks in shared storage). Just as importantly, the verify scripts can also be used to export phone content to PC—simply opt to retain the unzipped phone content at the end of the scripts, and sync it to the PC's copy with Mergeall or adopt it in full with a move. We'll revisit this role as an alternative to back syncs ahead.
Depending on your use case, you may wish to edit the z_postcopy_hook_phone.py script, to add custom steps required after to-phone copies are finished. This file is run at the end of both the initial-copy and synch-changes phone scripts. It's shipped with no commands, but any Python code you add to it will be run after all to-phone content propagations.
As an example, you might expand this script if you're keeping your full content in Termux's app-specific storage for speed, and need to copy specific folders to shared storage so they can be accessed by apps with limited permissions—including tkinter GUIs run in Pydroid 3. This hook may also be useful to copy media files to shared storage for constrained players.
This won't help, however, for copying website content to browsers' app-specific storage so it can be viewed locally; Termux cannot access other apps' app-specific storage in Android 11, so a file explorer with either app-specific permissions or local-view heroics is required. Some explorers, for example, now spawn HTTP servers to circumvent Android access limitations.
If ever needed, changes made by applying a deltas set with this package can
be fully rolled back (i.e., undone), by manually running another
mergeall.py -restore command in the host platform's console,
passing the backups set saved in the content copy's top-level
as the command's FROM source.
This restore command can be applied to both the proxy and on-phone content copies, to restore their state before the latest sync. It takes the following general form, where TO is the path to the root folder of your proxy or phone content copy, and backup is the latest date/time-labeled backup folder:
python3 mergeall.py TO/__bkp__/backup TO -restore -auto -skipcruft
See Mergeall's core
docs, as well as
main docstring for more on running restore commands.
This package's scripts do their part, by using Mergeall's
flag when applying delta sets during syncs—apart from the major optimization
exception added late in the 1.0 project, and called out by the following update.
Update: Backup Configs
-backup option automatically prunes old
backup folders. Because these deletions can be horrifically slow in
shared storage in Android 11 (5 to 10 minutes for
under 2G was not uncommon), backups in this package's scripts can now be configured
on or off in both the PC and phone config files.
True in the PC file to enable backups
for proxy-copy changes. Do the same in the phone file to enable backups for
phone-copy changes. When enabled, Mergeall's
-backup flag will be used
by sync scripts when applying delta sets, to enable later rollbacks.
Important: as preset, backups are enabled for the proxy drive, but disabled for the phone to avoid slow shared-storage prunes. If left disabled, you cannot perform a later rollback of phone-sync changes, and must recopy content if it's ever corrupted by a sync. However, the presets make syncs much faster; phone copies are often view only and disposable; and disabling backups saves space on any device. The speed boost is moot if you opt to use app storage for content on your phone, but shared storage has usability advantages which are compelling enough to support it by default.
Android 11 file-explorer apps allow you to access your on-phone content, but vary widely in both metadata support and storage access. Some may be updated further by the time you read this note, and Android is nothing if not convoluted. But as pointers for users of this package: among the ten apps tested at this writing (October 2021), and on two unrooted Android 11 devices tested (a Pixel 4a and Z Fold3):
The nature of this support varies too. For example, some explorers including
Samsung and Solid may require an app restart to notice USB; paths used to open files
on USB drives are
/mnt/media_rw on Solid but
URIs on others; Total Commander uses an unusual link path to open app-specific
files and doesn't support Copy To;
and tapping a USB drive in Notifications may also open a vendor-default browser.
Some Android explorers have also been seen to hang or fail on large copies and
deletes, though this may reflect temporary bugs.
Naturally, USB and app-specific content access depends on both file explorers and the apps they invoke to open files, but a useful survey of the latter is outside this doc's scope. For definitive results, always test the latest versions of explorers and apps you wish to use on your own devices. Related details: see also the coverage of exFAT-driver apps for USB proxies ahead; the peek at website local-view support in explorers earlier; and the parallel coverage of explorers off page.
Finally, it's worth adding a general caution for the sake of your content. Though perhaps obvious, this is a content-propagation system. It will change content on your proxy drive and phone by design. Some of its changes can be undone, but this depends on usage and configuration. To avoid unpleasant surprises, you should read this guide and its examples before using this system, and would do well to experiment with the system on non-critical data initially before using it on valued material.
Although this system is tested, open source, and used by its maker regularly, any content propagation tool can fail due to variables beyond its control, including errors in usage, hardware, and software. By using this system, you both accept responsibility for all its actions, and benefit freely from all its capabilities.
This section closes with some notes about the system's design. Some of it is probably of more interest to developers than users, but is also provides some additional context and caveats along the way.
See the overview page for more on this system's scripts, and this folder for the unzipped content of this package. If you are new to Mergeall, the best starting points are its main web page and user guide. For more on the ziptools system used throughout these scripts, start at its web page or user guide. For this system's rationale in general, try this and this.
This section summarizes this system's versions. The date at the top of this page always gives its most recent release. If you find an issue in this system, please report it via the "Input" link in the toolbar below. Software improves best with use and feedback.
This system's scripts are
Python translations and extensions
originals retained in the
folder. The Bash originals work, but the Python recoding is better
for portability of the PC-side scripts to Windows: they may be run in
normal Windows shells like Command Prompt and PowerShell, without having
Moreover, Python scripts can also be run on PCs by icon clicks and IDEs as covered
Cygwin and WSL can be used on Windows too, but have tangible seams which make them less user friendly (e.g., newline rules and drive mounts). Bash might suffice for phone-side scripts since commands can be run in Termux, but config files must be in Python or Bash (not both), and Python enables launches by Android IDE apps and shortcuts, and better supports code evolution.
A downside of Python in this context is that its abstractions used for portability may obscure the command lines it runs; see the Bash originals for a direct (if cryptic and Unix-only) command tour.
This system does most of its work by leveraging the Mergeall and ziptools systems as a software stack. ziptools is shipped both stand-alone and as a tool embedded in the Mergeall package. Scripts here use Mergeall's copy, though the stand-alone version is also available here. You can enable ziptools stand-alone installs in common.py, but this is not required. The ziptools included with Mergeall 3.2 or later suffices here, and the same maker publishes ziptools, Mergeall, and this package; the three are kept in sync and help drive development in each other.
To report real elapsed runtimes, these scripts now use Python timer
calls which are immune to complexities in shell timing tools.
The scripts' original versions used the Bash
to display step runtimes, but this command ends timing prematurely on
control-Z to background a step, and an alternative
survives control-Z but isn't present on phone in Termux and doesn't
respect Bash settings as well. Python avoids these issues, but as a
general rule, open another console to tail log files during runs,
per the logfiles usage coverage above.
This system zips content for transfer to minimize the risks of data loss and copy failures. Else, some copy destinations drop folder modtimes, symlinks, and permissions, and may discard or covertly rename files with nonportable names. Worse, Android file explorers run very slowly for raw folder copies, and may even fail to finish. Without naming names, multiple apps fully hung while testing copies of large folders, and some suffered disk unmounts (see also the earlier explorer coverage).
Zipping content neutralizes all these factors in full. As a bonus, zips are also generally much faster to transfer than individual files on every device. Unzips can outpace raw copies too, because the former opens just one input file: see the initial-copy script's cpall note.
A cross-platform deltas create/apply GUI was rejected for this project for a number of reasons. For one, in the simplest use case with no zips, syncs would require two runs in the GUI on PC (create deltas, apply deltas), and forgetting the second would leave the proxy different than the phone. For another, zips are nearly required per the preceding note, and this makes for numerous steps (four on both PC and phone; see the scripts). Automating steps and zips in scripts is far less error prone.
This system won't work with FAT32 proxy drives if its content or deltas zipfiles are larger than 4G (this is a FAT32 limitation). This limit is unlikely to be broached by syncs, because their deltas zips are normally small. It could, however, be a factor for initial copies of content collections larger than 4G, as they zip content in full.
If your zipfiles are that large, your best option is to reformat the proxy drive to use exFAT, or another more-functional filesystem supported by both your PC and your phone. exFAT is generally your most interoperable choice for USB storage devices. It does not have FAT32's 4G limit, avoids FAT32's modtime perils that can derail proxy syncs, and is broadly usable on both PCs and most recent Android phones. While exFAT support on phones has been vendor dependent in the past, its recent adoption by the underlying Linux kernel should make it standard on Android soon.
If you're unsure of your devices' exFAT support, try your user docs or the web, or simply format a drive and test. Where unavailable as a built in, you may still be able to use exFAT on your phone by installing an app. Such apps do not mount USB drives in ways accessible to POSIX programs, and may require specific file explorers, but can suffice for the manual zipfile copies employed by this system's scripts. On an unrooted Android 11 Pixel 4a, for example, the Paragon app here and here enables exFAT access in Total Commander, X-plore, and FX, albeit with occasional remounts.
If exFAT (or other filesystem) is not an option in your use case, your last resort is to use raw copies in file explorers. This still requires enough space on the proxy to hold a copy of your content, and regrettably adds Android file-explorer variables to the mix, but it may avoid FAT32's 4G limit for individual files. Moreover, you may still be able to use this system's sync scripts for later updates because their zips are usually small.
As coded, the initial-copy-part1-pc.py script temporarily requires twice as much total space on your proxy drive as your content collection, to store both a content copy and an uncompressed content zip. For example, a 100G content collection temporarily requires 200G on the proxy, but just 100G later. This is necessary because stingier alternatives have too many steps to automate.
If your proxy doesn't have enough space, simply run this script's steps manually, but move the zip to your phone before copying content to the proxy (see the Bash original for more-direct command examples). The upside is that this space requirement is temporary (the zip may be manually removed from the proxy after the to-phone transfer), and applies only to the initial content copy. Later syncs require just enough extra space for normally small delta sets.
Special case: the 1.1 verify scripts described earlier temporarily require twice the space of your content on your proxy for an extra zipped copy (and similar on your phone and PC). They're also entirely optional.
As coded, the initial-copy-part2-phone.py script temporarily requires twice as much total space on your phone as your content collection, to store both an uncompressed content zip and a content copy. For example, a 100G content collection temporarily requires 200G on the phone, but just 100G later. This is necessary to avoid issues in Android file explorers, which have been seen to fail for direct copies of larger content collections.
If your phone doesn't have enough space, manually copy unzipped content from proxy to phone by file explorer, and run this script's other steps manually as needed (see the Bash original for more-direct command examples). This script deletes the zip from the phone automatically after unzipping, and its size requirement also applies only to the initial content copy. Space requirements for later syncs are much more modest.
Special case: the 1.1 verify scripts described earlier temporarily require twice the space of your content on your phone for an extra zipped copy (and similar on your proxy and PC). They're also entirely optional.
It's atypical but not impossible that the proxy drive's content copy may differ slightly from that on the phone for which it stands in, because the phone's copy reflects zipfiles created on the PC, not the proxy itself, and the proxy and phone may support esoteric content differently.
Exotica like symlinks and Unix permissions, for example, may differ across both proxy filesystems and Android storage types, and the disposition of nonportable filenames is platform specific. We've touched on some of these topics earlier (e.g., here, here, and here), but this note takes another look from the perspective of content skew.
First and foremost, it's important to note that most use cases don't need to care. Proxy and phone copies will never differ for normal content (files and folders) and its modtimes. Moreover, even when deviations arise, they may be acceptable; may be irrelevant as long as the proxy suffices for syncs to PC; and may be easily avoided in general. If a content component doesn't work on a drive or Android, you can opt to simply not propagate it.
If your use case ever does need to care, however, here's a quick rundown of expected differences when twilight-zone content is moved from PC platforms to drives and phones:
-nomangle. Windows has symlinks too, but their requirements for admin permission and the NTFS filesystem make them even scarcer than on Unix. The Unix-like Cygwin and WSL can produce skew in Windows much like Linux; see their earlier coverage.
The upside here is that components which may differ between proxy and phone are generally rare and trivial, and can be controlled by users. They also cannot be better supported in any event. Zipping on the proxy instead of the PC might avoid some skew, but it would also slow initial copies and syncs, and is unlikely to offer any advantages for notoriously nonportable content like symlinks and Unix-only filenames.
More generally, the content copy on the phone may grow out of sync if it is ever changed on the phone: because it will never again be synced to the full copy on PC and proxy after initial creation, on-phone changes may endure until another full copy. This is an inherent downside in using only incremental delta-set syncs, but necessitated by Android 11's removal of POSIX USB access. On the other hand, this is acceptable if the phone's copy is used as a view-only resource—a current assumption of this system discussed ahead.
Update: Checking for Skew As of version 1.1 of this system, you can now check for differences between on-PC and on-phone content copies, by running the verify scripts introduced earlier. While this is a slow process and not a direct sync, it can be used periodically to detect unintended skew between phone and PC. This can also be used to export phone content in general per ahead.
This system is primarily meant for Android 11 and later. It works to sync devices running Android 10 and earlier too, but may be overkill there. Prior to 11, you can simply run two direct Mergealls by USB: from PC to external drive, and from external drive to phone—and using either Mergeall's command lines or GUI for both. This is arguably more user friendly, as no manual deltas-set copy is required along the way. See Mergeall's original docs for its simpler usage on earlier and less restrictive Androids.
In principle, this system could also be used to sync other platforms besides phones. For example, once a deltas set is created, it could be applied to a heterogeneous set of devices whose content copies are all kept in sync with the proxy drive. In this mode, a destination would not necessarily be a "phone" and not necessarily be Android. Phone config files would reflect platform-specific paths on each sync target, and the phone-sync script would simply be run on each target device in turn—whether they are phones or not.
This usage mode works today too, but this system is not explicitly designed for it, because this mode inflexibly requires all other devices to be synced at the same time (while deltas are valid), and other platforms can use direct and simpler merges. Syncing to a mix of macOS, Windows, Linux, and Android-10-or-earlier devices, for instance, can use a basic Mergeall run on each as before, both whenever convenient, and with either command line or GUI. Only Android 11 requires the indirect deltas-based syncs performed by this system.
Nevertheless, syncing multiple devices with a precomputed deltas set might be quicker than basic merges, because it avoids an initial comparison phase on each target. Moreover, a device mix that includes Android 11 must create deltas for Android 11 anyhow, making them available for use on other devices. This system awaits broader usage feedback to determine whether generalizing its scope makes sense.
Update: Android 12 In late 2021, Android 12 is now known to have the same shared-storage-speed and USB-access limitations as Android 11. The bad news in this is that these constraints seem poised to be either long-lived or permanent. The good news is that this package's scripts work around them on Android 12 too.
Update: Usage Feedback In practice so far, running this package's scripts on earlier phones seems as easy as Mergeall's GUI (especially when the Termux home-screen widget is used for launches), and is noticeably faster because it skips full comparisons. These scripts now also seem simpler and safer than pulling and syncing a microSD card on phones that still have one; in fact, they may be a better option even if cards reemerge on future Folds. Still, Mergeall's GUI remains useful on older Androids for impromptu syncs, enables direct content verification that indirect schemes cannot, and supports broader usage modes of the sort discussed in the next section.
This system generally assumes that the phone (i.e., sync destination) is used as a view-only device. While changes on the phone might be synced back to the PC with a similar indirect scheme, supporting this would require substantial script retooling, if possible at all.
Such back syncs are made complex—if not implausible—by the same lack of direct POSIX USB access on Android 11 which was this system's genesis. Because there is no way to compare on-phone content to the proxy drive itself, either:
All of these would be prohibitively expensive in time or space for large data sets, unless full content transfers or duplicate copies could be avoided. Although it's possible to imagine options that might reduce overheads (e.g., computing deltas from separately maintained doppelgänger trees with modtimes but zero-length files), this may again require religiously syncing multiple devices at the same time, and could complicate and slow content syncs radically or impractically—and all for the sake of Android 11 alone. This may rule out back syncs themselves, but the following update provides an alternative.
Update: Phone Exports As of version 1.1 of this system, it is now possible to export phone content to your PC, by running the verify scripts introduced earlier. These scripts use zipfiles to copy phone content to PC for comparison, and were designed to verify phone content. They can also be used for phone exports, however, by opting to retain the phone's unzipped content on your PC at the end of the verify process. Then, simply sync the phone's content to the PC's with a normal Mergeall run on your PC, or adopt the phone's content copy in full as your new on-PC copy.
This is a slow process, and does not qualify as a back sync per se: it's a full copy from phone to PC, with an optional sync on the PC. Nevertheless, its automated zipfiles speed the copy, and it may suffice to propagate changes from phone to PC occasionally. If you'll be using only your phone on a trip, for example, you can sync from PC to phone before leaving, make arbitrary changes on your phone while away, and use the verify scripts to propagate changes back to your PC when you return. This is nowhere near as nice as direct Mergealls, but will have to pass as usable on Android 11 and later. Which bring us to this guide's conclusion.
This doc has largely resisted editorializing, but in closing, a few words of perspective are in order.
This system is the culmination of a content-management odyssey on Android which has spanned four years and versions—from Oreo, whose working modtimes first enabled syncs, through 11, whose by-design USB lockdowns and shared-storage throttling made this task much more difficult. In particular, Android 11's removal of USB access for POSIX programs convolutes content management badly, and remains an affront to usability and interoperability. And Samsung's removal of microSD cards only seals this door tighter.
This may be disappointing, but should not be wholly unexpected. Android is not desktop Linux, despite the shared heritage and code. It bridles devices broadly with SELinux mods; throttles programs aggressively based in part on screen interaction; and walls up storage into opaque content prisons, which align with cloud-storage agendas, and grow ever tighter. Equating such devices to PCs seems the stuff of deceptive marketing and quixotic fancy.
Together with the ethics of Android's owner, this has naturally spawned a culture in which customer exploitation passes as a norm. Indeed, smartphones seem crafted to provide their makers with points of control—whether this means monopolizing access to media; tracking users for advertising; extorting revenue from developers; locking down product repairs; or harvesting personal information stored in clouds. While some maker motives remain matters of speculation, these devices are clearly not as customer focused as they might be.
Even so, this package's scripts provide a path to maintaining content on these devices for those determined enough to try, and manage, at least for their intended roles, to subvert whatever Android-storage intrigues or gaffes may be. The work-around is not trivial, but cloud-based alternatives are innately perilous. Trusting valued content to profit-driven companies with clear patterns of invading privacy and violating civil liberties will, for many, never be a viable option.
At the end of the day, keeping your stuff on your phone is getting harder, but it's still possible. And it's still worth the extra steps to keep the bad guys from turning your personal life into their revenue stream. Rooting through your stuff without your permission is just as creepy in the digital realm as it is in the real one. But you can still say no.