![[image]](_etc/screenshots/android11-termux-widget-edit.jpg)
Preface: as of May 2023, the system described on this page has been superseded by the newer PC-Phone USB Sync—a standalone backup/sync app that runs on both Android 8-and-later devices, as well as Windows, macOS, and Linux PCs. Read about and fetch the new app at its separate website, and tap the button below to learn how this app improves on prior solutions.
On Android 11 and later, this new app uses the All files access permission to regain access to both shared storage and removable (e.g., USB) drives. This circumvents the loss of file-path/POSIX access to USB drives in later Androids. Shared storage is still slower in recent Androids, but the new app supports content hosting in faster app-specific storage, and UFS 4.0 chips promise to speed storage in general. As a self-contained app, the new program also eliminates the Termux and Pydroid 3 app dependencies of this and other former solutions, and provides a much more seamless and graphical Android experience.
On PCs, the new program is delivered as a standalone app or executable to avoid extra installs and mesh with the host platform's paradigms. And on every platform, this app is still an all-Python system—despite Android's Java biases. It embeds and uses the portable Mergeall system to process content; codes its GUI in Python with the portable Kivy toolkit; builds apps and executables with the buildozer and PyInstaller tools; and talks to Java with pyjnius only when it must. In fact, this is still largely Mergeall, with a new and improved GUI launcher that also provides access to USB drives on recent Androids.
Like all the earlier Android/PC content-sync solutions worked out at this site over the last five years, the new app is also private by design: its use of USB drives for folder syncs both boosts speed, and ensures that your content remains your property—not someone else's point of control. This app is available as full and trial versions on the Play store for Android, and as free downloads for PCs. Please visit the new app's website for more screenshots, more info, and download links.
| Summary: | Sync content folders to Android and more by USB |
| Version: | 1.2, January 6, 2023 (changes) |
| License: | Provided freely, but with no warranties of any kind |
| Author: | © M. Lutz (learning-python.com), 2021-2023 |
| Install: | Fetch the download, unzip its code, see ahead |
| Requires: | Python 3.5 or later, Mergeall 3.3 or later |
| Runs on: | Linux, macOS, Windows, Android |
This package was born of frustration with the twin removals of POSIX USB access in Android 11, and microSD-card support in Samsung's flagship phones. Those removals crippled content-processing tools, as covered here, here, and here.
This usage guide leaves the drama behind, and focuses on how to use the work-around automated by this package's scripts to propagate and sync content folders from PCs to phones running Android 11 and later. As we'll see, this work-around requires just USB ports and a portable drive; uses the Mergeall and ziptools systems as nested tools; is partly manual but simple and fast; and can be used to sync PCs and older Androids too.
This work-around also replaces prior solutions, and may be permanent. While the future is unknowable, the Android 11 speed and access limitations we'll remedy here are still present in Androids 12 and 13, the former of which poses additional hurdles of its own. Using Android devices for content storage and other PC roles is an ongoing battle, but it's still worth the effort to many discriminating users.
This doc begins with a short overview, followed by a quick start, in-depth usage details, general notes, and a brief conclusion. If you'd like a headfirst preview, try this package's screenshots, examples, and code.
This package provides Python scripts which:
In typical usage, you'll copy to your phone once, sync whenever you wish to update your phone, and verify or export your phone occasionally when it's useful. The combined effect keeps your content the same on both your PC and phone.
The content which these scripts sync is an entire folder, including all its files and nested subfolders of any depth, and can be any folder you wish to sync. The scripts that make this work are run as pairs on PC and phone, and automate steps that would be tedious and error prone to run manually. In brief, the scripts here:
Ultimately, this system is just a director: it employs Mergeall to create and apply sets of changes; ziptools to bundle those changes for transmission; and you the user to copy those bundles when needed. The combination enables POSIX-based content management by USB on Android 11 and later, and propagates changes between your PC and phone safely and easily. The next section details the process that achieves this.
From a closer perspective, this system's solution relies on three components:
To sync changes from PC to phone, this system leverages a USB drive known as the proxy because it takes the place of the phone during change detections run on the PC. As long as the proxy's content copy is always the same as the phone's, PC-to-proxy changes double as PC-to-phone changes, and can be computed independently of the phone. The proxy's role partly mimics now-defunct microSD cards, and is similar to an intermediate drive in earlier Mergeall models. Under Android 11, however, the proxy's content copy cannot be merged to the phone directly by USB with Mergeall.
To address this loss of USB access, this system also employs a new Mergeall script,
deltas.py.
This script detects differences between FROM and TO as usual, but instead of applying
FROM changes to TO immediately like a normal sync, it saves the changes separately to
a folder, in the same format as Mergeall backups. This allows changes (a.k.a.
deltas) to be saved as a unit, copied between devices, and applied to TO later
with a -restore run of Mergeall's main
mergeall.py.
The same batch of changes can be applied this way to both proxy and phone to
keep them in sync; because delta sets are almost always very small, these tasks are almost
always very quick.
On top of this, this system also ensures that content is zipped and unzipped to evade interoperability perils in filesystems and platforms along the way. The portable ziptools handles the bundling on every device. Depending on the target of content propagations, this can avoid loss of symlinks, permissions, folder modification times (a.k.a. modtimes), and nonportable filenames, all of which may otherwise be silently munged in raw transfers. Zipping also minimizes the role of Android file explorers, which are generally slow, and have even been seen to fail for copies of large unzipped folders.
By combining the proxy's phone forgery, the deltas script's deferred changes, and zipfiles' content protection, this system is able to sync phones despite Android 11's closures. To propagate changes from PC to phone, changes are simply:
This may sound like a lot of work—and it is, compared to the direct two-merge process before Android 11—but it's still well worth the effort for people who don't want to store their private digital property on clouds, where it can be arbitrarily misused by hosting companies. If you've read this far, that category probably includes you.
Perhaps the best news here is that this package's scripts automate most of the details behind these indirect syncs. In short, you will simply:
Once you're set up, syncs require no manual steps, apart from starting a script on your PC and phone, and copying a single zipfile on your phone when prompted—a process you'll likely find to be just as easy as running Mergeall's GUI on PCs and older Androids.
For an alternative take on this system's operation, see the parallel off-page coverage. Here, the rest of this doc provides a brief reference guide to this process, followed by more-complete details.
(In this doc, boxes like this are used for sidebars, supplements, and updates.)
Throughout this guide and its scripts, "PC" means any macOS, Windows, or Linux computer, but "phone" and "Android device" can be anything. Though initially designed to sync changes from PCs to phones and tablets running Android 11 and later, these scripts can also be used to sync to earlier Androids, other PCs, and any combination of these. As of 1.2, changes can even be backsynced from "phone" to PC with exports, which reverses the flow of changes, and allows you to edit content on your phone itself and propagate the updates to PC and proxy.
Hence, these scripts are more general than may be implied in this doc, and when you see "phone" both here and in the scripts, keep in mind that it's a generic term for the destination of most syncs, and not necessarily a phone at all. There's more on this generality in Scope ahead. That said, this package is likely of more interest to users of Android 11 and later, because it addresses those systems' removal of functionality, and POSIX USB access and microSD cards may still provide additional options on PCs and older Androids. But tech moves on; let's do the same.
In the following, you'll generally run the setup tasks and initial copy just once, repeat the sync steps any time you wish to propagate changes from PC to phone, and verify or export phone content occasionally. See the full install details ahead for specific version requirements and more options. All the software you'll install below is free and open source.
As a first setup step, dedicate an external USB "proxy" drive to host a content copy that will serve as a stand-in for your phone in later syncs run on your PC. The proxy is used to push changes to (and optionally from) your phone, but also automatically doubles as a backup copy for your PC's content.
The proxy drive must have enough free space to hold your content and more. Specifically, initial copies, verifications, and exports require twice as much space as your content. By contrast, later syncs require just a small increment beyond your content's size, because they store changes only; this increment's size varies per sync. See the complete coverage of space requirements ahead.
For the proxy, any USB storage device and filesystem supported by both your PC and phone will work with this package's scripts. As guidelines, though, portable SSDs (like these) are recommended for both speed and reliability, and the exFAT filesystem is recommended for its larger file sizes, sync-friendly modtimes, and broad availability on Android phones today.
Important: to keep the proxy's content copy in sync, it should be changed only by this system's scripts, else change detection won't work and you may have to rerun an initial copy or run a backsync. In particular, do not run a direct Mergeall between PC and proxy, because this system's scripts won't find any proxy differences to propagate to the phone later (details). Similarly, to keep the phone's content copy in sync, do not change it directly, unless you plan to export it back to your PC and proxy (details).
Next, install the Python programming language, if not already present, to run this package's scripts on both PC and phone. You can find install pointers for PCs here, though it's generally a simple click-and-go. On phones, tap the downloaded APK file to install.
On your PC:
On your phone:
pkg install python
termux-setup-storage
Finally, install this package's scripts and the Mergeall system they use, and configure the paths to your content's top-level folder on your PC, proxy, and phone. The required path edits are described in both the configuration files listed below and the section ahead.
On your PC:
On your phone:
You are now ready to run scripts to propagate content and changes per the following steps.
In the next four sections, the script files named in commands are located at the top level of the unzipped scripts package. They may be run by the commands shown, or by icon clicks on PCs, home-screen widget taps on phones, and more; see ahead for full coverage, especially if you're new to running scripts. Once started, you'll interact with a script in its console window (ahead), and use a file explorer to copy a zipfile on your phone when prompted (ahead).
Important: be sure to run these scripts as pairs, in the sequence shown. In syncs, for example, the phone-side script must be run second, and before the pc-side script is run again. If you don't follow the sequences here, your phone may grow out of sync with your PC, and require a recopy or backsync.
After setup, you're able to copy and sync your content. Initial copies duplicate the PC's content in full on both proxy and phone. They are generally run just once; later changes are propagated by syncs which are much faster than full copies.
Run the following steps to copy content from PC to phone initially (see the prior sidebar for more on these scripts):
python3 initial-copy-part1-pc.py
python3 initial-copy-part2-phone.py
Syncs record just changes made to the PC since the last sync, and propagate them to both proxy and phone. They may be run anytime you wish to make your phone's content match your PC's.
Run the following steps to propagate changes from PC to phone (see above for more on these scripts):
python3 sync-changes-part1-pc.py
python3 sync-changes-part2-phone.py
Verifications copy the phone's content in full, and compare it to the PC's copy. They are optional, but provide an integrity check which you can run on demand to validate your phone's content.
Run the following steps to verify phone content to PC (see above for more on these scripts):
python3 x-verify-phone-part1-phone.py
python3 x-verify-phone-part2-pc.py
Exports copy the phone's content in full, and propagate its changes to the PC's and proxy's copies. They are optional, but useful as a backsync after content has been changed on your phone alone, and can also export phone content in full.
Run the following steps to export phone changes to PC and proxy (see above for more on these scripts):
python3 x-export-phone-part1-phone.py
python3 x-export-phone-part2-pc.py
This section goes into more detail about the steps outlined in the preceding Quick Start. It describes setup tasks and interaction, and covers enough Android and script-launch fundamentals to get you started.
This system's scripts work on most macOS, Windows, and Linux PCs in use today. On phones, they work on all Androids 8 (Oreo) and later, which includes every Android since 2017 (earlier versions didn't support file modtime comparisons). At present, the scripts have been used successfully and most often on Androids 10 through 13 (including 12L), but are expected to work on 14 and later too. For recent info, see the coverage ahead for Androids 12 and 13, as well as the Androids log.
To run these scripts, you'll install this package, Mergeall, and Python, per the following notes:
Download folder
suffices and is suggested on Android).
This system requires Mergeall 3.3 or later, for its
deltas.py
script, included
ziptools,
and Unicode filename normalization.
ziptools 1.3 or later is also
required, but is shipped with Mergeall 3.3 and need not be installed separately;
you can read more about it at ziptools' separate source
here.
To unzip packages after you've downloaded them, see your PC and Android file explorers for initial options. You can use ziptools to unzip packages portably, safely, and quickly, but not until you've unzipped it.
Termux
pkg install python to be run in its console to
install its Python.
termux-setup-storage to be run in its console
if you will be using shared storage for your content. Read the details
here and
here,
and stay tuned for storage-options coverage
ahead.
Termux may lose this permission shortly after it is installed on some
Android 11s; to fix, toggle its storage permissions in Settings/Apps
off and on (yes,
really).
Acquire wakelock in Notifications, plus an
equivalent console
command,
to ensure that a long-running script will continue even if your phone's
screen goes
blank
(where supported).
This is more important for initial copies than later syncs, because
deltas are normally small.
Either way, release a manually acquired wakelock
to avoid battery drain—or see the next bullet.
termux-change-repo and see the
issue and
fixes
if installs fail due to repository issues.
Pydroid 3
For more on Termux and other on-phone options, watch for upcoming launch details, including their update on the new Termux how-to. First, the following updates box covers extra topics related to running this package's scripts.
As of December 2021, this package's minimum Mergeall version requirement has been bumped up from 3.2 to 3.3. Mergeall 3.2 has the requisite deltas.py and ziptools, but 3.3 adds Unicode normalization for filename matches, which is most important when applying delta sets like those leveraged by this package. Without the 3.3 enhancement, some rare use cases may generate duplicates for items whose names are the same but use different Unicode code points on source and destination devices.
For more details, see Mergeall's coverage
here
and
here.
Mergeall 3.3's -quiet flag is also now used everywhere here
to silence normalization messages; grep for this flag in
scripts if it ever proves undesirable.
As of fall 2021, the "shim" scripts provided for use with the Termux home-screen widget on your phone now automatically acquire and release an Android partial wakelock around script runs, to ensure that the scripts continue even if your phone screen goes blank due to timeout or power-button press (on phones that still respect the wakelock, at least). Hence, you don't have to manage the lock manually in Notifications. See the shims and their README here for more info, and see ahead for more on the Termux widget; it's optional but recommended for on-phone scripts.
Update: Backgrounding Shim Calls for Android 12+
In version 1.2,
Termux-widget shims
now run their calls to acquire and release the
wakelock in the background, to avoid pauses on some Android devices.
A shell & is used to allow the rest of the shims' code
to continue in parallel.
When run normally, these calls each return immediately on an Android-10
Note20 Ultra; take 2–3 seconds on an Android-12 Fold3; and can hang for an
unreasonable 10 seconds on an Android-12L Fold4 when run from a widget (only).
Running them in the background locks and unlocks correctly, without making
shims or users wait.
To disable automatic locking, change variable auto_wakelock in
the shims and lock manually in Notifications (environment variables
don't work because widgets run in a limited context).
Nit: it's possible to subvert 1.2's wakelock backgrounding by starting and stopping multiple widget shims in rapid succession (their lock and unlock calls may overlap arbitrarily); and it's easy to trounce auto-locking in general by running multiple shims at the same time (they share a single app-global lock). So don't do that. Those are not normal—or even valid—use cases, and, barring a Termux fix, the only alternatives today are manual locking, or two 10-second pauses on some phones. 1.2's automatic locks work the best they can, and do well in realistic usage.
For more info and status, see the GitHub bug report filed by this project. As noted there, phone restarts reduce the wait times, but only temporarily; 12L's 10-second pauses come back in full shortly thereafter. As also noted there and ahead, Android 13 (and/or Samsung's UI 5) later reduced the wait times to Android-12 levels, but not enough to make backgrounding optional, and Termux upgrades didn't help. Related mod: Version 1.2 also automates shim setup tasks with helper scripts; see 1.2's release notes.
Android 12’s “Phantom” Killer versus Termux
As noted earlier, Android 12 has not lifted the Android 11 constraints that motivated this package. In addition, Termux, and possibly Pydroid 3, may require extra steps to work around the new and draconian "phantom" process killer in Android 12 and later. In brief, Android 12 bizarrely starts killing child processes after a hard limit of 32 has been reached. This limit is for the entire phone, not per app. Because nearly everything Termux does relies on child processes, its shell and the programs it runs—including the Python scripts in this package—may be terminated arbitrarily by the culling. In the context of content syncs, this is obviously a Bad Thing.
That said, the risk of this peril may vary by phone and usage. A Z Fold3 upgraded
to Android 12 and used to test this package, for example, has not triggered a single
Termux kill in roughly one year of continuous use, though it also does not run antivirus
software, X11 Windows
managers,
or similarly process-intensive tools. If process culling may be an issue for you, you
can avoid the kills with adb commands in Android 12, or a new
Developer-options setting to appear in Android 12L and 13. Specifically, the following
stops the reaping in Android 12 (see also
this search and the coverage
ahead if you're new to
adb, and the update below for a
better alternative in 12L):
adb shell "/system/bin/device_config set_sync_disabled_for_tests persistent; \
/system/bin/device_config put activity_manager max_phantom_processes 2147483647"
And the following Developer feature flag is scheduled to appear soon in Android 12L and
13 to subvert the issue altogether; this will presumably show up as a toggle or feature
flag in the "Developer options" menu of Settings (the one activated by the silly
tapping on the phone's build number), and may even pop up in some Android 12 updates,
though all such prospects naturally depend on phone vendors:
settings_enable_monitor_phantom_procs
For the full story on this still-evolving issue and its work-arounds, see the off-site
pages
here,
here,
here, and
here.
The above fixes draw from these sources, especially the
second,
which also goes into more detail than this doc can about risks of the first
adb command, including possible reboot loops on upgrades.
In an ideal world, software would not be silently changed in ways that cripple
devices and negatively impact millions of users. In the world of Android, it's a norm;
a hastily coded work-around—though not a reversal—will have to do as a
happy ending.
Update: Disabling Process Kills in Android 12L+
In fall 2022, Android 12L did not ship with the new
Developer setting as promised, but it does allow disabling the phantom-process
killer with the following adb command, run on a connected device as
usual:
adb shell settings put global settings_enable_monitor_phantom_procs false
This command's effect persists after being run just once, and supersedes others recommended for Android 12 earlier. As an example, 2022's Samsung Fold4 ships with an Android 12L that has neither feature flag nor Developer toggle for this, but the above command reportedly suffices (e.g., reddit; as before, this package's own Fold4 test machine has yet to see a single process kill, so this command has not been required or used).
A Developer toggle is still supposed to appear in Android 13 at
this writing, but vendor support may vary, this story line changes regularly,
and user processes really shouldn't be silently killed in the first place.
The upside here is that Termux may be inconvenienced, but is not broken.
The process-killer's impact varies per phone usage; may not affect
you at all; and can be mitigated with either adb
commands available today, or a switch that will hopefully show up somewhere
soon on a phone near you.
Postscript: at the start of 2023, Android 13 arrived to a Pixel 4a and
Galaxy Fold4 in this package's testing stable without the Developer toggle.
Use the preceding adb command instead if your Termux processes are killed
on phones like these.
Android 13 Locks Down App-Specific Storage?
Per early reports, Android 13 may lock down app-specific storage, such that is cannot be accessed by file-explorer apps. See the full coverage in Android Storage ahead. This may wind up being a non-issue, and there are simple work-arounds for this package's users. But Android releases have now become an annual rite of access closures—and developer cringes.
The initial-copy scripts here temporarily require twice as much total space as your content collection itself on your PC, proxy drive, and phone. To copy a 100G content collection, for example, you'll need 200G on all three devices during the copy, but just 100G later. This stems from the scripts' use of zipfiles to avoid interoperability issues; an extra zipfile copy is passed between devices to avoid data loss.
If your devices don't have enough space, see the later notes on proxy and phone space for manual work-arounds. In short, you can emulate this system's steps with command lines, or skip some altogether. PCs require twice content's size during initial copies too, but are less likely to be as tight on space as phones and proxies.
By contrast, the sync-changes scripts here require only enough extra space for one delta set, which is normally very small and transfers quickly. In other words, once you get an initial copy onto your phone one way or another, later syncs with this system's scripts are unlikely to run into space limits.
In all cases, this system's scripts automatically delete both zipfiles and folders on PC and phone no longer needed after script completion. This includes temporary content and deltas folders, as well as their zipfiles. Exception: zipfiles for both the initial copy and deltas set on the proxy drive cannot be automatically deleted, because these scripts cannot access USB on the phone. You can freely delete these zipfiles from your proxy to save space, after their content has been propagated to your phone. Caution: don't delete the proxy's content-copy folder; it's needed for later syncs.
If your proxy drive is formatted as FAT32, you may also bump up against that filesystem's 4G file-size limit during initial copies of content; consider reformatting your proxy drive to use exFAT if it's supported by your phone or an add-on app, and see ahead for more details.
Space Requirements in 1.1 Verifies
When used, the 1.1 verify scripts described ahead also temporarily require twice the space of your content on your phone, proxy drive, and PC, for an extra content zip on the first two, and an extra content unzip on the PC. They're also a sparingly used special case and fully optional.
Space Requirements in 1.2 Exports
As of 1.2, the same rule applies to the new export scripts described ahead: your phone, proxy, and PC must all have enough space to hold an extra copy of your content. This is an unavoidable consequence of Android 11's removal of POSIX USB access.
Before you can use this package, you must edit its config files to specify paths to your content's top-level folders on your PC, proxy, and phone. These settings are usually made just once, and used by all of this system's scripts. You'll also need to choose an Android storage type to host content on your phone, per the section that follows. First, this section covers how to edit configs in general.
To configure this system, edit both:
Both of these files reside at the top-level of the unzipped install package. Edit them according to the instructions provided in their top-of-file documentation strings. In both, you'll set folder paths to code and content, and optionally change run options. See these files' documentation and "#" comments for complete details, and the examples folder for config-file examples to emulate.
Config-file settings are straightforward and are checked for errors before scripts start, but here are a few pointers on this step to help you avoid mistakes:
= assignments.
The settings' names should not be changed.
/sdcard is usually the
root (leftmost) part of internal-storage paths on your phone (regardless
of what file explorers call it), and PC and proxy roots vary per platform
(e.g., /home/you and D:\).
For more info, see your file explorer, the pathname primers
here
and
here,
and the examples.
The config files are also technically Python code, but their settings are few and simple. For readers new to this language, here are five essential syntax tips:
# anywhere starts an ignored comment
= in assignments
True and False start with an uppercase letter
'...')
r'...' or '..\\..'
or '../..' for backslashes
The following list provides additional details and advanced usage tips for config files:
\) in strings as the start of an escape sequence
(e.g., \n means line-break).
Since this character is also used as a path separator on Windows,
it requires special handling in Python strings: the r'...'
prevents Python from treating \ as an escape character; doubling
up with \\ is an escape that stands for one backslash; and forward
slashes (/) work in place of backslashes in Windows paths too.
See the Windows
example for all
three codings in action.
Forward slashes in Unix paths require no such special handling.
Embedded quotes must be escaped everywhere (e.g., '..\'..')
but are enormously rare.
~ in a path string is replaced with
the path to the home folder associated with your user account, and
$var and ${var}
(plus %var% on Windows) are replaced with the
value of the environment variable named var.
This works on all platforms: a leading ~ is replaced with
/Users/you on macOS,
C:\Users\you on Windows,
/home/you on Linux, and
/data/data/com.termux/files/home on Android Termux.
Variable replacement is handy for Linux proxy-drive paths that depend
on your user name: use /media/$USER/drivename.
For finer details, see the underlying Python tools
here
and
here,
and the examples
here,
here, and
here.
'D:\\', using the Python
doubled-up escape scheme described above).
If the slash is omitted (e.g., 'D:'), the path is relative to the
current working directory on the named drive, which is not what you'll usually want.
For more background on this, see
this
Microsoft article. Scripts here don't automatically add a \ if
omitted, because this would preclude drive-relative paths in contexts
where they are useful.
See also the "z-pc-froms" captures in the screenshots gallery starting here for more path examples. Coverage for specific configuration settings appears ahead in this guide, in the topics with which settings are associated; search in your browser to find more about a given setting here.
Tip: you can sync multiple folders separately by editing the config files before each run, but this may grow tedious. To avoid frequent edits, store all your synched content in a single folder, and use this folder's paths in your configs; because this package syncs subfolders nested arbitrarily deep in the folder root, you can structure a single content folder as richly as you wish.
Beginning with release 1.2, you can find the original versions of
the PC and phone config files shipped, in the _etc folder
configs-originals. They're
stored there to serve as a restart fallback, in case your edits go awry.
When editing the phone configuration file, you'll need to choose Android storage folders—and hence storage types—for both your content itself, and temporary zipfiles copied to and from your phone. The zipfiles folder should generally be as fast as possible to speed scripts, but need not support anything but file-explorer copies from/to USB. This section focuses on the content folder, because the access it supports will determine the ways in which you can use your on-phone content.
In brief, you can choose from three storage types, and corresponding folder paths, for both content and zipfiles:
/sdcard (or /storage/emulated/0)
This storage type can be both viewed and changed by most apps, and is never automatically deleted on app uninstall; but it may run much slower than the other two storages in Android 11—perhaps 2X to 100X slower, depending on usage. This space generally includes everything under its path that's not part of the next bullet.
/sdcard/Android/data/appname
This storage type is quicker, and can be viewed and changed by many apps; but it is unsupported by some apps, and may be automatically deleted when the owning app is uninstalled. This space looks like it's part of shared storage, but it's implemented (i.e., mounted) differently.
/data/data/appname
This storage type is also fast, and supports Unix metadata best; but it can be accessed normally by the owning app only, and may also be automatically deleted when the owning app is uninstalled. This space's sole-app role may disqualify it for general content stores, but see the SAF update ahead for possible exceptions.
Three disclaimers up front. First, due to Android fragmentation, these
categories' paths may vary across versions and vendors (e.g., /sdcard also
goes by /storage/emulated/0); for a deeper dive, try the
pathname
primer.
Second, the appname part in these paths varies too; it's
com.termux for
Termux, and
ru.iiec.pydroid3 for
Pydroid 3.
And third, due to a now-ancient SD-card legacy, some Android docs confusingly call
app-specific "external" and app-private "internal," but both usually reside
in your phone today; they're just managed differently.
Also note that everything in this section and doc applies to unrooted devices; while phone rooting may loosen some constraints, it's not possible on many phones, and not desirable for many users (see this package's genesis for more on rooting). Moreover, this doc deals only with unremovable (internal) storage, because it's the only on-phone option on the Fold3; for the similar rules on removable (external) storage like microSD cards, see the older package's docs or test on your device.
All that small print aside, the choice of storage host for your content pivots on four dimensions, some of which may be more important for your use cases than others:
In the plus column, most Android 11 file explorers and the apps they open allow you to view and change your content in both shared storage and any app's app-specific storage. This may require one or more permission dialogs, but is broadly supported. Less positively, app-specific storage may be a read-only resource if Unix permissions are propagated, and app-specific support varies by explorer and viewer (e.g., Fx can currently explore shared but not app-specific storage on 11, and Music Folder Player bears the same limitation for audio files).
More rigidly, Termux and Python apps cannot access other apps' app-specific storage in Android 11 today, so your choice of content host may be binding. If you run this system's scripts in Termux, for example, you can store content in either shared storage or Termux's own app-specific storage, but programs run in Pydroid 3 won't be able to access the latter. Conversely, because Pydroid 3 is required to run tkinter GUIs, their code and data must be in either shared storage or Pydroid 3's app-specific storage, but Termux is locked out of the latter.
For the gritty details on Python-related apps, see the captured results for
Pydroid 3 and
Termux.
As shown, these apps today can access shared storage
and their own app storages only, though in general this depends on app permission choices,
the version of Android that apps target, and the ever-changing rules of Android itself.
Special case: both Termux and Pydroid 3 can run scripts saved in Download,
but this is technically part of shared storage—and subject to the next section's trade-off.
As an example, unzipping and deleting a 200G/180k-file collection in app-specific storage with Termux took just 18 and 10 minutes, respectively; but the same tasks in shared storage checked in at an arguably unusable 1 hour and 4 hours, respectively (and the shared-storage delete failed altogether in two file-explorer apps). Indeed, large-folder deletion is so slow in Android 11's shared storage that a phone reset may be quicker.
That said, not all shared-storage operations are so slow, and typical content access may be adequately fast for your purposes. Moreover, this package's use of delta sets naturally speeds on-phone syncs, because it avoids full on-phone content comparisons: smaller delta-set compares offset slower shared-storage access.
In fact, per practice so far, media access in shared storage is generally quick, and
the phone portion of syncs for numerous changes with shared storage usually takes
no more than 2 minutes—including time for the manual zipfile copy.
Still, your syncs may naturally vary, and auto-deletion prunes of large
__bkp__
folders
in shared storage can be so slow that they seem to hang. Prune sloth was eventually
sidestepped with config presets described ahead,
but check script status in logfiles if pauses are
long, and use app storages if waits are intolerable—unless, that is, the next
section's trade-off outweighs the delays.
Though irrelevant to temporary zipfiles, this risk alone is nearly enough to disqualify app storages for content. It's also not entirely abstract: the Mergeall dev team itself accidentally erased gigabytes of content this way while setting up apps to test Termux's widget. While that's not exactly a minor downside, uninstalls are naturally up to you, and the subtler trade-offs of the next and final section may decide the game for some.
Special case: happily, shared storage does not emulate FAT32's "local time" for file modtimes, and hence is immune to FAT32 skew on DST rollovers and time-zone changes described here. This matters because modtimes in on-phone storage must still differ with those in a deltas zip for changes to be applied. On multiple devices, shared, app-specific, and app-private modtimes all agreed with each other and PCs after changing dates and zones—and still did after an actual DST rollover. If modtimes ever skew anyhow in some future Android, run the fixer script on deltas, or give app storages a second look.
So where should you put your stuff on your phone, then? There is no single answer to this question, and your choices may need to evolve with Android over time.
But in general, shared storage's cross-app accessibility make it the better choice for content-creation tasks such as programming, and its permanence is a clear win for storing valued content. App-specific storage, however, remains both a faster option for temporary zipfile folders and a viable contender for content, and app-private storage can still take the title in single-app roles or when accessed by specially equipped explorers per the upcoming update; just don't delete the app!
As guidelines—and barring a future Android fix for shared-storage throttling which would make that storage a clearer winner:
You can code an all-app-specific scheme in the phone config
file with settings like this
(remember to replace com.termux with ru.iiec.pydroid3
if you're using Pydroid 3 instead of Termux to run this package's scripts):
FROM = '/sdcard/Android/data/com.termux' # zipfiles folder TO = '/sdcard/Android/data/com.termux' # content folder
You can code an all-shared storage scheme in the phone config
file with settings like this
(be sure to use /storage/emulated/0
instead of its /sdcard shorthand on phones that don't have the latter):
FROM = '/sdcard' # zipfiles folder TO = '/sdcard' # content folder
You can code a hybrid scheme in the phone config file using one of the following alternatives (and see the upcoming update for more on the requirements for app-private storage used by the second option here):
FROM = '/sdcard/Android/data/com.termux' # app-specific for zips speed TO = '/sdcard' # shared for content access/safety FROM = '/data/data/com.termux/files/home' # app-private for zips speed TO = '/sdcard' # shared for content access/safety
Footnotes: other combinations are possible, though some are less plausible than others. An all-app-private scheme works too, for example, but suffers from content-access limitations detailed in the first update ahead. Storage choices are also subject to Android mutation. Android 13, for instance, may lock explorers out of app-specific storage per the second update ahead; and even shared storage access relies on Python-friendly apps retaining opt-outs of storage closures in Android 11 and later (backdated API levels and temporary reprieves suffice, but are tenuous and limiting). The schemes above, however, best jive with reality today.
For more on how all this translates to this package's settings, be sure to also consult the docstring at the top of the phone config file. Here, the following updates box explores advanced but useful storage-related topics.
Using Termux App-Private Storage in File Explorers
In 2022, it's still true that app-private storage is inaccessible to other apps per Android's normal permission rules. Because Android file-explorer apps are locked out from accessing this storage type too, its support for both zipfile copies and content storage in this package is limited.
Nevertheless, Android rules regularly come with exceptions. Today, a handful of Android file explorers do allow you to access Termux's app-private home folder from outside the Termux app, by using Termux's Storage Access Framework (SAF) interface instead of traditional POSIX techniques. This requires an SAF-capable explorer and is not entirely seamless, but provides fully functional access. SAF-capable explorers also support USB-drive access, which makes app-private storage a candidate for zipfiles, content, or both—the FROM and TO in the phone config file.
Details: this somewhat new SAF access mode is described tersely on Termux's
wiki near the bottom of
this page.
In testing on Android 12L, four out of eleven file-explorer apps support this SAF
mode—
Fx, stock
AOSP (a.k.a. Files),
Material Files,
and
MiXplorer.
All four provide full read/write/open utility for content stored in the Termux home folder
(known as
Per the Termux wiki, the
AOSP explorer can even be started automatically from inside Termux by
command line; this could be run by this package's scripts to automate
explorer launches, but dictates explorer choice too rigidly:
To dig deeper into app-private usage with this package, see the new
demo, as well as the
new file-explorer screenshots with notes starting
here.
Disclaimer: it was impossible to test all Android explorers, so be sure to test others'
SAF support as desired.
It's important to keep in mind that this isn't normal (POSIX) file access—SAF
is a proprietary Android-only model, that provides a sort of virtual folder accessed
by
As another example, there is only one way to view a
local web page
in app-private storage, among the
four SAF-capable explorers and four browsers tested. MiXplorer
launching Firefox works well by spawning a local web server,
but the
support
for local web-page
views in non-SAF browsers like Cx and Solid is absent in all
other SAF options tested.
SAF explorers impose additional constraints (e.g., home-screen folder shortcuts)
we'll skip here for space.
In short, you should expect some limits when accessing content in app-private storage
using SAF-capable explorers. Most of these limits don't apply to occasional zipfile copies,
and app-private can work well for this package's zipfiles folder—the FROM in the phone
config file. For content at TO, though, this storage
type is difficult to recommend.
All that said, being able to access Termux's home folder outside the Termux app
makes this app-private storage much more useful than it would be if it was
limited to Termux alone. App-private storage will still be silently removed on
Termux uninstall (you've already been
warned!), but shares app-specific storage's
speed, and betters it on Unix metadata support; see above
for more on its tradeoffs.
Alternative: though more complex, it's also possible to use Termux's
app-private storage in an all-Termux environment which runs X-Windows GUI file
explorers and other tools. See the first-level details
here, and more
in the next update. This won't work for zipfile copies because Termux provides
no USB access (that's the genesis of this package, after all), but it might suffice
for content. This scheme's content-usage options also seem likely to be more limited
than the universe of Android apps, though your needs may vary.
Segue: neither MTP nor adb have access to app-private storage, so they
won't help with this update's mission, but they may play a role in that
of the next—which also explains what they are.
Android 13 Hides App-Specific Storage from File Explorers?
Per early
reports
in 2022, Android 13 may try to lock down the
app-specific
But per the buzz, the only reason
app-specific storage was accessible to file explorers at all since Android
11 was a loophole (really, bug) that Android (really Google) has now opted to
close—despite valid use cases, user outcry, and the fact that users have
already granted these apps All-files-access
permission.
This matters here because app-specific storage is faster than
shared storage and more broadly accessible than app-private storage, but it cannot
be used even for temporary zipfile copies if it's off-limits to file explorers.
Still, there are also early
reports
that file explorers may have already found other ways
to provide this access in Android 13 (yes, with another loophole),
so this may wind up being a non-issue—as
was the case for multiple levels of access in 11 (spoiler: see the
update ahead).
But if the closure ever
becomes permanent, there are multiple alternatives for running this package
on your phone, some of which call out this package's choice points in general.
Here are the options, ordered by simplest and most suggested first:
Using app-private for just zipfiles, however, is a viable work-around to
app-specific demise. This role still relies on SAF access, but you
would need to use supporting Android file explorers for occasional zipfile copies only,
and hence would avoid this scheme's tradeoffs for content.
See above for config-file
code that implements this hybrid mode.
MTP is not without
hiccups,
and doesn't help with using content on your phone itself. But assuming it can still access
app-specific storage in future Androids, it's a simple way to copy zipfiles to/from
your phone, and provides an alternative to Android file-explorer apps in this role.
The USB proxy drive need only be used on the PC in this mode.
In fact, MTP might be used instead of Android explorers for zip copies in
general, and this may render zip copies on USB drives unnecessary.
This mode was rejected early by this project, because MTP has
weak support outside
Windows;
shows disturbing glitches on some
platforms;
doesn't provide access to app-private storage on unrooted
phones;
won't help for syncing a deltas set to another
PC;
and can be noticeably slower than Android explorer
copies.
More fundamentally, MTP seems a lot more to ask of casual users than on-phone apps.
While it may suffice as a work-around for Android closures,
this package's recommended usage was chosen for portability, reliability, and ease.
Android explorers are still advised—while they last.
Assuming adb retains its app-specific access, though, its
Among its downsides, adb, like MTP, doesn't normally have access to app-private folders,
so it applies to app-specific and shared storages only
(see the demo).
Most prominently, though, adb commands are even further above the pay grade of casual
users than MTP. This likely disqualifies it as a general work-around,
but your pay grade may vary.
While this Android 13 change looks at first glance as potentially
dire as Android 11's
closure of POSIX USB
access
and Android 12's "phantom" process
killer,
app-specific storage already had access and longevity issues that made it
a weak choice for content; file explorers seem likely to come to the rescue again;
there are already simple work-arounds in the list above; and Android's future is about
as unpredictable as it can be, so this will for now have to remain a
"stay tuned."
Update: Explorers Save the Day in Android 13
At the start of 2023, roughly half of the ten Android file-explorer apps
tested on a Pixel and Fold4 do provide access to
Hence, you can still use app-specific storage with this package on Android 13
for zipfiles, content, or both.
You must use a file explorer that supports app-specific access on this Android,
but Cx, MiXplorer, Total Commander, and others do already, and more explorers
will likely do the same shortly.
The non-app-specific alternatives listed above—shared for all, app private
for zips, and the rest—work on 13 too, but are not required there.
At least that's the verdict for Android 13. The explorers' work-around of pushing the
permissions dialog to an app subfolder relies on a coding loophole, and seems at odds
with Android's crippling, and arguably totalitarian,
security dogma. Whether the loophole survives in Android 14+ remains TBD,
and the alternatives above remain fallbacks for a more-closed possible future.
Curiosity: per user feedback, the Solid explorer supports app-specific
folders in Android 13 too—if you simply make a bookmark to them on 12 before upgrading to 13.
Alas, Android's app-specific lockdown seems about as well thought out as a Death Star
that can be laid low by a single hit to an exhaust
vent...
Also on 13: Android 13 still has the shared-storage-sloth and USB-access
issues that first cropped up on 11; see the Fold4's
results.
In the plus column, 12L's 10-second
wakelock delays
seem to have been cut to 3–4 seconds on 13,
though that still warrants backgrounding to avoid pauses.
Rare Unicode-Variants Android Bug—and Fix
Per findings of the underlying
Mergeall system,
shared storage in Android 11 is also sometimes unable to store files
having non-ASCII Unicode filenames recorded in the composed (e.g., NFC)
code-point format.
This is a complex issue, whose basis is covered by Mergeall 3.3's new
module
and release
note.
It also appears to be a bug in Android 11 shared storage of unclear scope:
it does not occur in app-specific storage; was isolated on just one Samsung device;
may occur only after storing multiple name forms; and has not produced any
reports of impacted use cases.
But if your non-ASCII filenames fail in copies or syncs to shared storage,
see the included
converter script for a work-around and more info
on the bug. This script is not required for Mergeall 3.3's automatic name normalization
noted earlier, and may be a temporary measure. Between
this bug, another involving nonportably named folders, and its egregious
deletion sloth described above, Android 11 shared storage seems prime for
repairs.
~ in the shell, and
/data/data/com.termux/files/home formally). All four also support USB
drive access, which may similarly use SAF if the drive is not mounted for POSIX access.
am start -a android.intent.action.VIEW -d "content://com.android.externalstorage.documents/root/primary"
content:// URIs which bypasses Android permissions—but the ruse
suffices for most uses. It's also important to note that this SAF-based access mode
comes with limits, tradeoffs and glitches:
/sdcard/Android/data folder described
above,
such that it can no longer be accessed by Android file-explorer apps.
This affects this package only if you use
app-specific storage for content or zipfiles, and is preliminary
and to-be-verified info which should be taken with the usual grain of salt.
push and pull
commands
would work for zipfile copies just as well as MTP, and provide another way to
perform this task without Android explorers.
Also like MTP, adb would not require access to USB on Android:
simply copy zips directly from/to the proxy drive attached to the PC.
/sdcard/Android/data/appname
app-specific folders on Android 13. To get around 13's lockdown of the
data folder, these explorers simply list this folder's contents and
launch the permissions dialog one level lower—for the target app's subfolder,
appname. For example, selecting Termux's com.termux
subfolder and granting access provides the same utility as before.
You can run this system's scripts by typing command lines which name them in consoles (a.k.a. terminals), or by running their files directly with icon clicks, shortcuts, and IDEs (edit/run GUIs). IDEs work because these scripts are coded in Python, and icon clicks and shortcuts work because the scripts are, for ease of use, configured by settings in files you'll generally edit once, not by command-line arguments passed on each run. Here are the basics of each of these run modes:
chmod +x script.py command,
and ensuring that they are on your system search path (or using
./script.py to skip the search).
Command-line novices tip:
depending on where you unzipped this package, its scripts
may not be in the folder you start at in your console tool.
If the scripts are elsewhere, either add their location's path
to the front of their names in the commands
above
(e.g., python3 path/script.py),
or simply cd path to go to their folder before
running commands. path
will likely be /sdcard/Download/android-deltas-sync
on your phone, and similar on PCs.
ForceKeyToClose config,
but this is automatic for Windows clicks.
macOS and Linux run Python files by clicks too; see their file explorers for tips, or simply click a Python file to try. In brief: on macOS, right click a file in Finder and pick Get Info to associate a Python opener program globally, and do the same to access a file's Properties and associate openers in Nautilus (a.k.a. Files) on Linux.
On Android, you can use a file explorer to make home-screen links that open phone-side scripts in an IDE app, ready to run in just two taps. For instance, the Total Commander app can make such links to open scripts in Pydroid 3 on all Androids (but be sure to unclick its "Parameters: content://url" checkbox in later Androids), and other explorer/IDE combos may offer similar options. See Pydroid 3 shortcuts at work in screenshots here, here, and here.
Because this widget is so handy, this system's phone-side scripts support it with
an absolute path in their shebang (#!)
lines, along with a
ForceKeyToClose config
setting to require a key
press before the widget erases the console (like Windows icon clicks).
If the shebang fails in other Android apps, edit scripts to use the line just
below it, or simply use an explicit "python3" in console
commands.
Better yet, use the Termux-widget
shim scripts included in this package to
run its scripts easily, without having to type command lines, copy the main scripts
to the widget's folder, change the prior paragraph's config, or manually acquire
and release the
wakelock. Simply copy the shims to your
~/.shortcuts folder in Termux, edit their runee paths if
needed, and launch scripts by taps in the widget.
For full usage details on the widget and shim scripts, see the
README and the
update at the end of this section.
Constraint: shortcut or not, it doesn't help to open this system's scripts in an IDE app on your phone, unless you host your content in storage which that app can access. As we saw earlier, this means either slow shared storage or that app's own specific or private storage, and the choice can limit scope in Android 11. You can't use Pydroid 3 to open scripts in Termux's app-specific storage—and vice versa. This is a catch-22 inherent in Android 11, and a nearly compelling argument for using slower shared storage.
There is now a file in this package's Termux-widget folder which provides a comprehensive guide to installing, configuring, and using both Termux and Termux:Widget. Please consult the new how-to for more info omitted here. The how-to also gives instructions for setting up custom widget icons, using images like those now provided in this package. Per the how-to, helper scripts automate both shim and icon copies in 1.2.
After launching a script, you will interact with it in the script's console window using your physical or on-screen keyboard. For demos of script interaction, see the run logs in this package's examples folder, as well as the Android screenshot. For instance, here are typical sync sessions on a PC and a phone. These scripts explain their actions along the way and are straightforward to use, but the following covers a few salient points:
< file input-stream or
<<EOF here-document syntax. See other resources for
pointers.
Tip: at "y or n" (yes or no) prompts, pressing enter/return alone is the same as
inputting "n" (no)—as is any reply that isn't "y" itself.
PauseSteps to False
in either or both config files.
Alternatively, pause key-press inputs can be automated, per the preceding bullet.
ShowRuntimes to False
in the config files (these times may be more useful for long-running initial
copies than quicker later syncs).
Total elapsed runtime—all steps' time + all user time—is
always displayed at script exit.
Tip: most of this package's screenshots have the luxury of being captured on a large Fold3 screen, but you'll probably want to run its scripts in landscape orientation on smaller phones to avoid line wraps in messages. The narrower Note20 Ultra used in this shot demos the idea.
Step logfiles give the full details of a run.
They show up in the folders whose pathnames you assigned to
LOGS in your PC and phone config-file settings.
Here are example logfile-folder listings from a
PC and a
phone.
Read the logfiles there later, or monitor a long-running step's progress
with a command like the following in a console run in a different window,
where logfilename is the latest item in your LOGS
folder:
tail -f LOGS/logfilenameThis command works in all Unix-like shells, including Termux on your phone. In Termux, you can both open a new session to tail logfiles and switch sessions by swiping in from the top left.
Besides saving step output to logfiles, this system also displays
the last few lines of each step's output in the run's console
window,
to give you a quick summary of results. If this summary looks amiss,
open the logfile in LOGS for the complete story.
For a broad sample of run logfiles,
see the demos' logs subfolders
in this package's examples folder (here's a typical
log).
Tech note: logfiles are written in unbuffered mode, so their output
shows up right away to give script status: the end of the log is
the most-recent message written.
On your phone, you will be instructed to manually copy a zipfile from the proxy drive's root to a specific phone folder, during the initial copy, and later syncs. Verify and export runs ask you to copy a zipfile from your phone to the proxy instead.
When prompted, copy the named zipfile using any Android file-explorer app which has permission to access both a USB drive, and the storage location to which or from which you are copying. Most explorers will suffice, and will work well because this is a single-file copy, but see the Android-explorer coverage ahead for tips.
This copy must be a manual step, because it cannot be run by command line in Termux (or similar). Like all other POSIX programs, Termux no longer has direct access to USB in Android 11 and later, per this screenshot and coverage. On the upside, this copy is simple, and is the only manual step in any of this system's scripts.
Note that file explorers'
GUIs
abstract the root folder of internal
storage and USB drives in different ways. The root path of your internal
storage at /sdcard, for instance, may be simply called "Internal Memory,"
"Main storage," or similar, and an attached USB drive at
/mnt/media_rw/ID uses nicknames too.
Extrapolate from path to explorer label as needed.
Fine point: /sdcard (a.k.a. /storage/emulated/0)
is really the root of your per-user space in internal storage. Rooted
phones
have access to the higher and broader device storage root, /,
but they also can access USB drives from POSIX programs at /mnt/media_rw,
and hence may have no compelling reason to run this package.
As preset, Unix file and folder permissions are not propagated to the
destination phone, because they have no effect in shared storage, and can make
it impossible to change or delete content in app-specific storage. The latter
may reflect an Android bug, but is crucial for content hosted in app-specific
storage.
Be sure to change the setting UnixPermissions
to True when using this system with app-private phone
storage,
which better supports permissions propagation.
Details: not all Android 11 devices retain propagated Unix permissions in app-specific storage. Per ziptools' research and demo noted earlier, the Z Fold3 does, but the Pixel 4a does not. Where supported, however, propagated Unix permissions typically make it impossible to update or remove content in app-specific storage in any file explorer or app, other than the storage's owner. With permissions propagation, for example, content stored by this system in Termux's app-specific storage can be viewed by other apps, but can be modified only by Termux. Allowing Android to stamp items with default permissions instead enables updates and deletions from any approved app.
Hence, this package's phone scripts by default omit ziptools' -permissions
flag in on-phone extracts, to prevent propagation of Unix permissions. To force
propagation in contexts where it makes
sense, set UnixPermissions
to True in the phone config file.
This may be useful for content hosted in app-private storage, where
direct cross-app updates are largely disallowed and irrelevant, and where
you'll probably want to retain Unix permissions in general. This config is
moot for shared storage, because permission propagations there are always
ignored by Android.
Note that permissions are always retained in zipfiles created on Unix PCs; the config setting prevents copying them to phone storage types which support them (but perhaps shouldn't). Also note that this system assumes phones are view-only devices in general (sans exports), but does not assume that on-phone content can never be changed; program data files, for example, may still require write access.
Permissions Policy in 1.1 Verifies
As a minor twist, the verify scripts added in 1.1
always propagate Unix permissions from phone to PC—though these will
generally be
defaults when mirrored from both shared and app-specific storage (and are
largely pointless on Windows PCs in any event).
Because of this, there is no
UnixPermissions setting in the PC config file.
Permissions Policy in 1.2 Exports
This same policy is adopted by the export scripts added in 1.2: backsyncs always propagate Unix permissions from phones to PCs, subject to the PC platform's permissions support (again, Windows is mostly indifferent). This may change permissions on PCs when backsyncing from Android storage types that do not support permissions well (specifically: shared and some app-specific storage), but this occurs only for files changed on the phone, and is an unavoidable interoperability obstacle.
Except when running on Windows natively, this system by default offers to automatically run Mergeall's fixer tool for nonportable filenames on the PC, as a first step. Where relevant, this step is crucial to content interoperability, though its usage story depends on whether you run the PC-side scripts on Unix or Windows:
If you don't run this step, nonportable filenames that work on Unix may be mangled (i.e. changed) when propagated elsewhere, and their files may be discarded in full by some tools. In this package specifically: such files are skipped with an error message when transferring them to Android 11 shared storage, but saved intact to Android 11 app-specific and app-private storage, as well as earlier Androids' shared storage. This reflects the underlying ziptools system.
The fate of nonportably named files on proxy drives, however, is PC and drive dependent. Such files fail on Linux when written to FAT32 and exFAT proxies. By contrast, such names are saved on such proxies without error on macOS due to that platform's munging to and from Unicode privates, but may fail in later network backsyncs. On both platforms, nonportable names may also leak out to Android in this system's initial-copy and deltas zips, and fail there. For examples of the effects, see the shared-storage and Linux demos here and here.
Though recommended in general for all these reasons,
if this step proves irrelevant to your
use cases, it can be skipped at its prompt, and its offer can be disabled
altogether by setting CheckFilenames to False in the
PC configs file. See also the update at this section's
end: the filename fixer is now also run for names exported from Android, because
some of its storage types follow Unix's flexibility.
win32)
in Python's sys.platform. Non-native Windows tools (e.g., Cygwin's
own Python) may yield nonportable names, but do not register as Windows this way,
and so will invoke the fixer step to adjust filenames.
The filename behavior of non-native Windows tools is subtle.
Cygwin, for example, allows nonportable
names to be created via name
munging
much like macOS. Its own Python reverses
the munge to nonportable characters, but registers as cygwin.
Windows' Python run in Cygwin register as Windows, but does not map munge
characters back to nonportables, so files do not fail.
WSL
Linux munges nonportables just like Cygwin, and registers as linux.
For demos of Cygwin and WSL in action, see the examples
here.
Hence, the name-fixer step is always skipped in Windows-native
contexts, for which the CheckFilenames config setting
is ignored. Cygwin and WSL users can change this setting to skip
the step as desired.
For more on the name-fixer step in general, see the comments at the top of Mergeall's fixer tool, and ziptools' related coverage of nonportable filenames here.
Related note: Unix filenames should generally be run through the name-fixer script anytime they are copied to Windows—including in manual zips created outside this system. Else, the unzip may discard files or handle their names unexpectedly (e.g., backslashes in Unix filenames generate subfolders on Windows in ziptools, but are less harmfully replaced with an underscore in the fixer). See the Windows example for a rare case where this matters, and the Windows unzips coverage ahead for related info. Avoiding nonportable filenames in the first place may be their best work-around.
Filenames Option in 1.2 Exports
As of version 1.2, CheckFilenames has also been added to the phone
configs file for phone exports
and preset to True, because it's possible to create nonportable filenames
in some Android app storages which conform to Unix naming
conventions. App-private folders, for example,
allow nearly arbitrary
names.
As preset, unless the "phone" is really a Windows
PC,
the export scripts offer to fix these names first, as they may otherwise
fail if exported to Windows, as well as some external drive's filesystems (post
PC-side unzip).
You can change this setting to False to skip the test if you either will
export only to Unix, or are using an Android storage type that disallows nonportable
names. Shared storage on Android 11 and later generally prohibits such
names.
The Mergeall
deltas.py
script supports Mergeall's -report option to
preview differences without saving them, though using the same
switch with
mergeall.py
has the same effect. This preview is
run initially by this system's PC-side sync
script, if PreviewSyncs is
True in the
PC config file, and you confirm the run interactively.
This config setting is preset to True because previews are generally advised.
Use them as desired to inspect deltas before writing them to disk.
This may, for example, avoid unintended and time-consuming
full copies at DST rollovers when using FAT32 proxy drives (see Mergeall
docs).
There is no direct way to verify the on-phone content copy against the proxy or PC copies, because POSIX code like Python cannot access USB drives normally on Android 11, and phone access by MTP and network servers is not fully usable. The phone might be compared to the proxy after syncs by manually copying full zipped content from proxy to phone, or vice versa; this is too slow to automate here, but might serve as an occasional and manual option (tip: zip content first on phone or proxy using a tool like ziptools—or see the update ahead for a precoded solution that does).
As a partial measure, setting
VerifyProxy to True
in the PC config file
runs mergeall and/or diffall
to verify the proxy's content against the PC's in both initial copies
and later syncs. The mergeall run compares by structure and modtimes,
and diffall performs a full byte-for-byte comparison.
The config setting is preset to True because verifications
are generally recommended, but you are prompted to confirm each verify
step independently because they add time to the run unevenly. A mergeall
verify is usually quick, but a more-complete diffall can be slow for larger
content collections.
Note that it's normal and harmless to have one diffall diff: a unique item
for the copies' top-level __bkp__ backup
folders.
Also note that symlinks on Linux cannot be written to FAT32 or exFAT drives,
and so will always register as PC-side diffs. This is an innate limitation
of these filesystems on Linux—which macOS avoids only by forging a
symlink stub file which passes for an actual link on macOS alone, and will
still yield diffs elsewhere.
For an example of the consequences on Linux, see its
demos.
The only external-drive symlink option on Linux is other filesystems.
As of version 1.1 of this system, it is now possible to verify the phone's content copy against the PC's, by running the new verify scripts. For a quick look, see them in action in the new example and screenshots. These scripts switch dataflow order—content moves from phone to PC—but config-file settings work unchanged on both phone and PC.
This process is optional and meant to be used only occasionally, because it is relatively slow and temporarily requires twice the space of your content on phone, proxy, and PC for an extra content copy or unzip. As chronicled by the new example's readme, for example, verifying an on-phone content folder of some 200G space and 160K files required an extra 200G everywhere (400G total capacity), and took roughly one hour (50 minutes on phone and 15 on PC). Especially if your phone hosts a throwaway copy, such costs may be unwarranted.
Nevertheless, if you do wish to prove that the phone's copy matches that on PC, the verify scripts will zip full content on your phone; prompt you to manually copy the zipfile to your USB proxy; and then unzip the phone's content on your PC and compare it to the PC's copy with Mergeall and diffall. As noted, the former compares content copies by modtimes and structure; the latter is a byte-for-byte compare which is more complete but slower, and an optional step in verify scripts.
Though too slow for frequent use, this suffices to validate phone content: you should see differences only for normal skew (e.g., backup folders) and items that cannot be stored on your phone (e.g., symlinks in shared storage). Just as importantly, the verify scripts can also be used to export phone content to PC—simply opt to retain the unzipped phone content at the end of the scripts, sync it to the PC's copy with Mergeall or adopt it in full with a move, and sync the proxy to the PC as needed. We'll revisit this role as an alternative to backsyncs ahead—and meet the newer 1.2 export scripts which do backsyncs better.
Depending on your use case, you may wish to edit the z_postcopy_hook_phone.py script, to add custom steps required after to-phone copies are finished. This file is run at the end of both the initial-copy and synch-changes phone scripts. It's shipped with no commands, but any Python code you add to it will be run after all to-phone content propagations.
As an example, you might expand this script if you're keeping your full content in Termux's app-specific storage for speed, and need to copy specific folders to shared storage so they can be accessed by apps with limited permissions—including tkinter GUIs run in Pydroid 3. This hook may also be useful to copy media files to shared storage for constrained players.
This won't help, however, for copying website content to browsers' app-specific storage so it can be viewed locally; Termux cannot access other apps' app-specific storage in Android 11, so a file explorer with either app-specific permissions or local-view heroics is required. Some explorers, for example, now spawn HTTP servers to circumvent Android access limitations.
If ever needed, changes made by applying a deltas set with this package can
be fully rolled back (i.e., undone), by manually running another
mergeall.py -restore command in the host platform's console,
passing the backups set saved in the content copy's top-level __bkp__
folder
as the command's FROM source.
This restore command can be applied to both the proxy and on-phone content copies, to restore their state before the latest sync. It takes the following general form, where TO is the path to the root folder of your proxy or phone content copy, and backup is the latest date/time-labeled backup folder:
python3 mergeall.py TO/__bkp__/backup TO -restore -auto -skipcruft
See Mergeall's core
docs, as well as
"ROLLBACKS" in
deltas.py's
main docstring for more on running restore commands.
This package's scripts do their part, by using Mergeall's -backup
flag when applying delta sets during syncs—apart from the major optimization
exception added late in the 1.0 project, and called out by the following update.
Configuring Backups in Syncs
The -backup option automatically prunes old __bkp__
backup folders. Because these deletions can be horrifically slow in
shared storage in Android 11 (5 to 10 minutes for
under 2G was not uncommon), backups in this package's scripts can now be configured
on or off in both the PC and phone config files.
Set BackupChanges to True in the PC file to enable backups
for proxy-copy changes. Do the same in the phone file to enable backups for
phone-copy changes. When enabled, Mergeall's -backup flag will be used
by sync scripts when applying delta sets, to enable later rollbacks.
When disable, -backup is not used and rollbacks are not possible.
Important: as preset, backups are enabled for the proxy drive, but disabled for the phone to avoid slow shared-storage prunes. If left disabled, you cannot perform a later rollback of phone-sync changes, and must recopy content if it's ever corrupted by a sync. However, the presets make syncs much faster; phone copies are often view only and disposable; and disabling backups saves space on any device. The speed boost is moot if you opt to use app storages for content on your phone, but shared storage has usability advantages which are compelling enough to support it by default with presets.
Backups in 1.2 Exports
As of 1.2, the new export scripts use the
BackupChanges setting in the PC config file,
when applying changes to both PC and proxy in backsync mode. Hence, you'll
generally want this setting to be True so you can rollback
both normal syncs to the proxy, as well as backsyncs to both proxy and PC.
For more on export-script backsyncs, see ahead.
Exports don't mod the phone, so its backups setting is irrelevant in this mode.
This package's phone-side scripts prompt you to copy a zipfile with a file explorer on your phone. For this copy, you can use any explorer that allows you to access both your USB drive, and the storage type you've chosen to host copied zipfiles. You'll also want to choose an explorer which can access the storage type that hosts your on-phone content in general. This section provides pointers on your options.
All Android file-explorer apps allow you to access on-phone content, but their support varies widely when it comes to both metadata and storage types. Some may be updated further by the time you read this note, and Android is nothing if not convoluted. But as pointers for users of this package: among the ten apps tested at this writing (October 2021), and on two unrooted Android 11 devices tested (a Pixel 4a and Z Fold3):
The nature of this support varies too. For example, some explorers including
Samsung and Solid may require an app restart to notice USB; paths used to open files
on USB drives are /mnt/media_rw on Solid but content://
URIs on others; Total Commander uses an unusual link path to open app-specific
files and doesn't support Copy To;
and tapping a USB drive in Notifications may also open a vendor-default browser.
Some Android explorers have also been seen to hang, fail, or crawl on large copies
and deletes, though this may reflect temporary bugs in either apps or the Android
frameworks they use.
Naturally, USB and app-specific content access depends on both file explorers and the apps they invoke to open files, but a useful survey of the latter is outside this doc's scope. For definitive results, always test the latest versions of explorers and apps you wish to use on your own devices. Related details: see also the coverage of exFAT-driver apps for USB proxies ahead; the peek at website local-view support in explorers earlier; and the parallel coverage of explorers off page.
File Explorers 2022
A year on in Oct-2022, most Android file-explorer apps use the liberal All Files Access permission to provide content broadly in Android 11 and later; this more or less provides the same utility as older Androids. Less positively, USB and app-specific storage sometimes seem slow in explorers; this may reflect the speed of Android frameworks used to access these mediums. In at least one explorer's built-in image viewer, for example, photos in app-specific storage render more slowly than the same in shared storage, despite the latter's sloth in Termux. Because this story changes too rapidly to document, be sure to test your explorers for up-to-date results.
File Explorers and Termux App-Private Storage
Also in 2022, this guide was updated with info on using Termux app-private storage in Android file-explorer apps that support the Storage Access Framework; see the new and related coverage above.
File Explorers and App-Specific Storage in Android 13
And also in 2022, there is some chance that fie explorers may be cut off from access to app-specific storage folders in Android 13. This is early info, and seems at least partly moot at this writing; see the related coverage and alternatives above.
Finally, it's worth adding a general caution for the sake of your content. Though perhaps obvious, this is a content-propagation system. Initial copies and syncs will change content on your proxy drive and phone by design, and backsyncs (a.k.a. exports) will change content on your PC and proxy and may harm nonportable items.
Some of this system's changes can be undone, but this depends on usage and configuration. To avoid unpleasant surprises, you should read this guide and its examples before using this system, and would do well to experiment with the system on non-critical data initially before using it on valued material.
Although this system is tested, open source, and used by its maker regularly, any content propagation tool can fail due to variables beyond its control, including errors in usage, hardware, and software. By using this system, you both accept responsibility for all its actions, and benefit freely from all its capabilities.
This section closes with some notes about the system's design. Some of it is probably of more interest to developers than users, but is also provides some additional context and caveats along the way.
See the overview page for more on this system's scripts, and this folder for the unzipped content of this package. If you are new to Mergeall, the best starting points are its main web page and user guide. For more on the ziptools system used throughout these scripts, start at its web page or user guide. For this system's rationale in general, try this and this.
This section summarizes this system's versions. The date at the top of this page always gives its most recent release, and is updated whenever actual code or functionality is changed. If you find an issue in this system, please report it via the "Input" link in the toolbar below. Software improves best with use and feedback.
Roles: use exports if you'll be changing content on your phone directly. Their full copy of phone content may make them too slow for frequent phone-to-PC backsyncs of larger content collections, but they may suffice for smaller content, and might be useful to occasionally transfer changes in arbitrarily sized content back to your PC (e.g., after a trip with your phone sans PC). Though designed for backsync change propagation, these scripts may also be used to export phone content in full, if the target folder is empty on PC, proxy, or both.
Caution: phone exports work well for normal files and folders, but have the potential to overwrite nonportable items like symlinks on your PC. For details, see the coverage above, and the caution in the part-1 export script. Symlinks morph to simple stub files when unzipped in Android storage types that do not support them, and will come back as such to PCs during exports. Hence, such nonportable items may not survive roundtrips to and from a phone, and are generally better omitted from content which will be exported back to a PC.
Symlink tips: an included utility
script can be used to locate
all your symlinks;
app-private storage retains symlinks on phones where
usable;
and BackupChanges=True in the PC config
file ensures that any symlinks overwritten by
exports will be saved in your PC content's __bkp__ folder for
tactical restores.
The utility script is not run automatically, because symlinks are rarer than
nonportable filenames and cannot be fixed.
Other lossiness: though less harmful, Unix permissions may also be overwritten by exports from on-phone shared or (some) app-specific storages which do not support them well. This happens only for files changed on the phone alone, and is an unavoidable interoperability dead end. Read more details here and here.
.nomedia file was added
to prevent Android galleries from assimilating this package's screenshots
_etc/
as a fallback courtesy
Update 1:
version 1.2 was republished in early Oct-2022 with a fix for a rare
Windows-only issue: during folder removals, paths too long for Windows'
default limits are now prefixed with \\?\ to make them work correctly.
See rmtreeworkaround in common.py
as well as the example
results
for more on the fix; the underlying
Mergeall
and
ziptools
systems already do this
globally
(and this package must be able to delete the folders which ziptools creates).
Also new in this update: an on-demand utility
script that reports
potentially nonportable
symlinks;
upgrades for Termux-widget
installers, including a new master
script;
start/final content in the 1.2
example;
screenshot Notes;
and more on shims and
symlinks.
Update 2: the Oct-18-2022 anniversary release of 1.2 rebranded this package as "Android Deltas Sync," replacing its prior name "Android Deltas Scripts." This better reflects its purpose and scope; after a year of polishing, it's no longer a skunk-works project for developers, and is potentially useful to people who don't normally run scripts but need a simple USB sync tool for their Android phones. The renaming is nearly total; examples were updated globally with this to avoid confusion, but you may still see the old "Scripts" in a handful of screenshots that cannot be recreated a year on. This seems a minor sacrifice for a more useful app name.
Updates 3+: version 1.2 was repackaged in Nov-2022, Dec-2022, and Jan-2023 with doc changes that call out Termux app-private storage usage in SAF-capable explorers, as well as incoming signals on Android 13's partial lockdown of app-specific storage. The former includes a new demo; both add new screenshots starting here and here; and the new coverage justified restyling this guide for readability. In addition, a minor coding change shuts down scripts immediately and gracefully with a message on rare failures in Python runs (e.g., unzips). No other code or functionality was changed in these releases.
Update 1: version 1.1 was rereleased in late Dec-2021,
with a minor Unicode
usage note in this doc;
regenerated example logfiles
for the now-required Mergeall 3.3;
and a new example that demos PC sync
targets.
No core functionality was changed, though the Mergeall 3.3
-quiet flag is now used here to silence possibly numerous
Unicode-normalization messages in all runs of
mergeall.py,
deltas.py,
and
diffall.py,
and a new and rarely required utility script was added:
convertunicode.py.
Updates 2+: as minor 1.1 point releases, additional doc- and example-only edits were applied in Jan-2022; a new Termux how-to and widget info were added in Feb-2022; examples were updated for recent changes in Mergeall 3.3's console output in Mar-2022; export notes and content-rescue tips were added in May-2022; and the Quick Start was reworked for clarity and the Android 12 tips were updated in Sep-2022. No program code or behavior was modified in any of these point releases.
Updates: 1.0's final Nov-2021 release included mods to print logfiles in unbuffered mode for better status info; make backups configurable and disabled by default on phones to sidestep shared-storage sloth for backup prunes; remove an existing deltas folder during phone syncs; and automatically acquire and release the wakelock in Termux-widget shims.
This system's scripts are
Python translations and extensions
of same-named
Bash
originals retained in the _etc/bash-version
folder. The Bash originals work, but the Python recoding is better
for portability of the PC-side scripts to Windows: they may be run in
normal Windows shells like Command Prompt and PowerShell, without having
to install
Cygwin or
WSL.
Moreover, Python scripts can also be run on PCs by icon clicks and IDEs as covered
earlier.
Cygwin and WSL can be used on Windows too, but have tangible seams which make them less user friendly (e.g., newline rules and drive mounts). Bash might suffice for phone-side scripts since commands can be run in Termux, but config files must be in Python or Bash (not both), and Python enables launches by Android IDE apps and shortcuts, and better supports code evolution.
A downside of Python in this context is that its abstractions used for portability may obscure the command lines it runs; see the Bash originals for a direct (if cryptic and Unix-only) command tour.
This system does most of its work by leveraging the Mergeall and ziptools systems as a software stack. ziptools is shipped both stand-alone and as a tool embedded in the Mergeall package. Scripts here use Mergeall's copy, though the stand-alone version is also available here. You can enable ziptools stand-alone installs in common.py, but this is not required. The ziptools included with Mergeall 3.2 or later suffices here, and the same maker publishes ziptools, Mergeall, and this package; the three are kept in sync and help drive development in each other.
To report real elapsed runtimes, these scripts now use Python timer
calls which are immune to complexities in shell timing tools.
The scripts' original versions used the Bash time command
to display step runtimes, but this command ends timing prematurely on
control-Z to background a step, and an alternative /usr/bin/time
survives control-Z but isn't present on phone in Termux and doesn't
respect Bash settings as well. Python avoids these issues, but as a
general rule, open another console to tail log files during runs,
per the logfiles usage coverage above.
This system zips content for transfer to minimize the risks of data loss and copy failures. Else, some copy destinations drop folder modtimes, symlinks, and permissions, and may discard or covertly rename files with nonportable names. Worse, Android file explorers run very slowly for raw folder copies, and may even fail to finish. Without naming names, multiple apps fully hung while testing copies of large folders, and some suffered disk unmounts (see also the earlier explorer coverage).
Zipping content neutralizes all these factors in full. As a bonus, zips are also generally much faster to transfer than individual files on every device. Unzips can outpace raw copies too, because the former opens just one input file: see the initial-copy script's cpall note for metrics.
A Year of Zips
After syncing with this package for a full year in October 2022, zipfiles have proved to be a big win over raw copies. In fact, even if POSIX USB access were magically restored in Android, this system would still likely use zipfiles for to-and-from phone transfers as it does today. The only change would be to automate the manual zipfile copy steps on the phone. Zipping content eliminates entire categories of interoperability pitfalls—no matter who does the copying.
A cross-platform deltas create/apply GUI was rejected for this project for a number of reasons. For one, in the simplest use case with no zips, syncs would require two runs in the GUI on PC (create deltas, apply deltas), and forgetting the second would leave the proxy different than the phone. For another, zips are nearly required per the preceding note, and this makes for numerous steps—four or more on both PC and phone for syncs:
See the scripts for full fidelity. Even with a GUI, that's a lot to ask users to complete as a set, especially when skipping a step might harm on-phone content. Automating steps and zips in scripts is far less error prone, and no more demanding of users in practice.
This system won't work with FAT32 proxy drives if its content or deltas zipfiles are larger than 4G (this is a FAT32 limitation). This limit is unlikely to be broached by syncs, because their deltas zips are normally small. It could, however, be a factor for initial copies of content collections larger than 4G, as they zip content in full.
If your zipfiles are that large, your best option is to reformat the proxy drive to use exFAT, or another more-functional filesystem supported by both your PC and your phone. exFAT is generally your most interoperable choice for USB storage devices. It does not have FAT32's 4G limit, avoids FAT32's modtime perils that can derail proxy syncs, and is broadly usable on both PCs and most recent Android phones. While exFAT support on phones has been vendor dependent in the past, its recent adoption by the underlying Linux kernel should make it standard on Android soon.
If you're unsure of your devices' exFAT support, try your user docs or the web, or simply format a drive and test. Where unavailable as a built in, you may still be able to use exFAT on your phone by installing an app. Such apps do not mount USB drives in ways accessible to POSIX programs, and may require specific file explorers, but can suffice for the manual zipfile copies employed by this system's scripts. On an unrooted Android 11 Pixel 4a, for example, the Paragon app here and here enables exFAT access in Total Commander, X-plore, and FX, albeit with occasional remounts.
If exFAT (or other filesystem) is not an option in your use case, your last resort is to use raw copies in file explorers. This still requires enough space on the proxy to hold a copy of your content, and regrettably adds Android file-explorer variables to the mix, but it may avoid FAT32's 4G limit for individual files. Moreover, you may still be able to use this system's sync scripts for later updates because their zips are usually small.
As coded, the initial-copy-part1-pc.py script temporarily requires twice as much total space on your proxy drive as your content collection, to store both a content copy and an uncompressed content zip. For example, a 100G content collection temporarily requires 200G on the proxy, but just 100G later. This is necessary because stingier alternatives have too many steps to automate.
If your proxy doesn't have enough space, simply run this script's steps manually, but move the zip to your phone before copying content to the proxy (see the Bash original for more-direct command examples). The upside is that this space requirement is temporary (the zip may be manually removed from the proxy after the to-phone transfer), and applies only to the initial content copy. Later syncs require just enough extra space for normally small delta sets.
Special case: the 1.1 verify scripts and 1.2 export scripts described here and here temporarily require twice the space of your content on your proxy for an extra zipped copy (and similar on your phone and PC). They're also entirely optional.
As coded, the initial-copy-part2-phone.py script temporarily requires twice as much total space on your phone as your content collection, to store both an uncompressed content zip and a content copy. For example, a 100G content collection temporarily requires 200G on the phone, but just 100G later. This is necessary to avoid issues in Android file explorers, which have been seen to fail for direct copies of larger content collections.
If your phone doesn't have enough space, manually copy unzipped content from proxy to phone by file explorer, and run this script's other steps manually as needed (see the Bash original for more-direct command examples). This script deletes the zip from the phone automatically after unzipping, and its size requirement also applies only to the initial content copy. Space requirements for later syncs are much more modest.
Special case: the 1.1 verify scripts and 1.2 export scripts described here and here temporarily require twice the space of your content on your phone for an extra zipped copy (and similar on your proxy and PC). They're also entirely optional.
It's atypical but not impossible that the proxy drive's content copy may differ slightly from that on the phone for which it stands in, because the phone's copy reflects zipfiles created on the PC, not the proxy itself, and the proxy and phone may support esoteric content differently.
Exotica like symlinks and Unix permissions, for example, may differ across both proxy filesystems and Android storage types, and the disposition of nonportable filenames is platform specific. We've touched on some of these topics earlier (e.g., here, here, and here), but this note takes another look from the perspective of content skew.
First and foremost, it's important to note that most use cases don't need to care. Proxy and phone copies will never differ for normal content (files and folders) and its modtimes. Moreover, even when deviations arise, they may be acceptable; may be irrelevant as long as the proxy suffices for syncs to PC; and may be easily avoided in general. If a content component doesn't work on a drive or Android, you can opt to simply not propagate it.
If your use case ever does need to care, however, here's a quick rundown of expected differences when twilight-zone content is moved from PC platforms to drives and phones:
-nomangle. Windows has symlinks too, but their
requirements
for special permissions or modes
and the NTFS filesystem make them even scarcer than on Unix. The Unix-like Cygwin and
WSL
can produce skew in Windows much like
Linux; see their earlier coverage.
The upside here is that components which may differ between proxy and phone are generally rare and trivial, and can be controlled by users. They also cannot be better supported in any event. Zipping on the proxy instead of the PC might avoid some skew, but it would also slow initial copies and syncs, and is unlikely to offer any advantages for notoriously nonportable content like symlinks and Unix-only filenames.
More generally, the content copy on the phone may grow out of sync if it is ever changed on the phone: because it will never again be synced to the full copy on PC and proxy after initial creation, on-phone changes may endure until another full copy. This is an inherent downside in using only incremental delta-set syncs, but necessitated by Android 11's removal of POSIX USB access. On the other hand, this is acceptable if the phone's copy is used as a view-only resource—a current assumption of this system discussed ahead.
Checking for Skew in 1.1
As of version 1.1 of this system, you can now check for differences between on-PC and on-phone content copies, by running the verify scripts introduced earlier. While this is a slow process and not a direct sync, it can be used periodically to detect unintended skew between phone and PC. This can also be used to export phone content in general per ahead.
Though it seems a well-kept secret, Windows 11 requires no admin permission to create symlinks; you simply need to enable Developer Mode in Settings => Privacy & security => For developers. Once you do, symlinks can be made on NTFS drives from both the shell and Python, and in both native Windows and WSL. Some of this evolved in Windows 10, but the rules were substantially stricter in the past.
More importantly here, after you enable this mode, the ziptools system underlying this package happily recreates Unix symlinks on Windows, and even adjusts their path separators as needed to make this work. There's more on ziptools' symlink support on Windows here. While this is good news if you sync to a Windows PC as though it were a phone (a trick we'll meet in the next section), this package's main focus is syncing content to Android—where symlinks support is still sketchy.
This system is primarily meant for Android 11 and later. It works to sync devices running Android 10 and earlier too, but may be overkill there. Prior to 11, you can simply run two direct Mergealls by USB: from PC to external drive, and from external drive to phone—and using either Mergeall's command lines or GUI for both. This is arguably more user friendly, as no manual deltas-set copy is required along the way. See Mergeall's original docs for its simpler usage on earlier and less restrictive Androids.
In principle, this system could also be used to sync other platforms besides phones. For example, once a deltas set is created, it could be applied to a heterogeneous set of devices whose content copies are all kept in sync with the proxy drive. In this mode, a destination would not necessarily be a "phone" and not necessarily be Android. Phone config files would reflect platform-specific paths on each sync target, and the phone-sync script would simply be run on each target device in turn—whether they are phones or not.
This usage mode works today too, but this system is not explicitly designed for it, because this mode inflexibly requires all other devices to be synced at the same time (while deltas are valid), and other platforms can use direct and simpler merges. Syncing to a mix of macOS, Windows, Linux, and Android-10-or-earlier devices, for instance, can use a basic Mergeall run on each as before, both whenever convenient, and with either command line or GUI. Only Android 11 requires the indirect deltas-based syncs performed by this system.
Nevertheless, syncing multiple devices with a precomputed deltas set might be quicker than basic merges, because it avoids an initial comparison phase on each target. Moreover, a device mix that includes Android 11 must create deltas for Android 11 anyhow, making them available for use on other devices. This system awaits broader usage feedback to determine whether generalizing its scope makes sense.
There is now a complete example that shows how to sync a deltas set to a Windows PC with this package—see the new demo. This usage mode confusingly calls the PC a "phone" in its prompts, but otherwise works well: after storing PC paths in the phone config file on your PC, you'll run the phone-sync scripts on your PC, trade Mergeall-GUI paths selection for a file-explorer zipfile copy, and skip a full comparison step in syncs. It's roughly as simple as a direct Mergeall, and often quicker.
This mode remains an afterthought, and user input on its value is welcome. With it, however, this package can be used to propagate the same deltas set to all the devices in your gadget collection—Android 11 and later, earlier Androids, and PCs masquerading as phones. Despite original scope, this has already begun to prove useful in practice.
Android 12, 12L, 13, and later
While this system was developed in response to constraints added in Android 11, it runs just as well on later versions too, albeit with additional plot twists. Here's a brief summary of the story so far:
Feedback from the Trenches
In practice so far, running this package's scripts on earlier phones seems as easy as Mergeall's GUI (especially when the Termux home-screen widget is used for launches), and is noticeably faster because it skips full comparisons. These scripts now also seem simpler and safer than pulling and syncing a microSD card on phones that still have one; in fact, they may be a better option even if cards reemerge on future Folds. Still, Mergeall's GUI remains useful on older Androids for impromptu syncs, enables direct content verification that indirect schemes cannot, and supports broader usage modes of the sort discussed in the next section.
This system generally assumes that the phone (i.e., sync destination) is used as a view-only device, because normal syncs are unidirectional, flowing from PC to phone only. While changes on the phone might be synced back to the PC with a similar indirect scheme, supporting such backsyncs would require substantial script retooling, if possible at all.
Backsyncs from phone to PC are made complex—if not implausible—by the same lack of direct POSIX USB access on Android 11 which was this system's genesis. Because there is no way to compare on-phone content to the proxy drive itself, either:
All of these would be prohibitively expensive in time or space for large data sets, unless full content transfers or duplicate copies could be avoided. Although it's possible to imagine options that might reduce overheads (e.g., computing deltas from separately maintained doppelgänger trees with modtimes but zero-length files), this may again require religiously syncing multiple devices at the same time, and could complicate and slow content syncs radically or impractically—and all for the sake of Android 11 alone. This may rule out backsyncs themselves, but the following updates provide usable alternatives.
As of version 1.2, this system now provides dedicated scripts that export on-phone changes to both PC and proxy, and qualify as an improved backsync alternative. See the release note for complete details. These new export scripts are still relatively slow for larger content collections because they must copy your phone's content in full, and they require space for the extra content copy on your phone, proxy, and PC. Still, they automate the task of propagating on-phone changes back to your PC; may work well for smaller content and occasional use; and fully supersede the former verify-based export and backsync schemes described in the following note.
Phone Exports in 1.1
As of version 1.1 of this system, it is now possible to export phone content in full to your PC, by running the verify scripts as described earlier. These scripts use zipfiles to copy phone content to PC for comparison, and were designed to verify phone content: run the first of the pair on your phone and the second on your PC, and inspect the comparison results on your PC to prove your phone copy's validity.
In addition, though, these new scripts can also be used to export content from phone to PC, by opting to retain the phone's unzipped content on your PC at the end of the verify process. Then, simply sync the phone's content to the PC's with a normal Mergeall run on your PC, or adopt the phone's content copy verbatim as your new on-PC copy with a move or rename.
Important: if you export content from your phone this way, keep in mind that the verify scripts do not update the proxy to match the phone automatically, because this is unneeded in verification roles, and the content copies on the phone and proxy are normally the same when using this system's scripts. If your phone's copy has been changed directly, though, and now differs from the proxy, you must also be sure to sync the proxy to match the PC's new content exported from the phone, so that all three copies—phone, proxy, and PC—will be the same. This requires an extra and manual Mergeall step described in the PC script.
Exporting phone content this way is a slow process, and does not qualify as a backsync per se: it's a full copy from phone to PC, plus optional syncs on the PC and proxy. Nevertheless, its automated zipfiles speed the copy, and it may suffice to propagate changes from phone to PC occasionally. If you'll be using only your phone on a trip, for example, you can sync from PC to phone before leaving, make arbitrary changes on your phone while away, and use the verify scripts to propagate changes back to your PC when you return. This is nowhere near as nice as direct Mergealls, but will have to pass as usable on Android 11 and later sans USB access. Which bring us to this guide's conclusion.
This doc has largely resisted editorializing, but in closing, a few words of perspective seem in order.
This system is the culmination of a content-management odyssey on Android which has spanned four years and versions—from Oreo, whose working modtimes first enabled syncs, through 11, whose by-design USB lockdowns and shared-storage throttling made this task much more difficult. In particular, Android 11's removal of USB access for POSIX programs undermines content management badly, and remains an affront to usability and interoperability. And Samsung's removal of microSD cards only seals this door tighter.
This may be disappointing, but should not be wholly unexpected. Android is not desktop Linux, despite the shared heritage and code. Unlike its freedom-focused cousin, Android:
Together with the ethics of Android's owner, this motif has naturally spawned a culture in which customer exploitation passes as a norm. The platform's pay-or-else advertising model, for example, smacks more of ransomware than commerce. Indeed, smartphones in general seem crafted explicitly to provide their makers with dark points of control—whether this means:
Even so, this package's scripts provide a path to maintaining content on these devices for those determined enough to try, and manage, at least for their intended roles, to subvert whatever Android-storage intrigues or gaffes may be. The work-around is not trivial, but cloud-based alternatives are innately perilous. Trusting valued content to profit-driven companies with clear patterns of invading privacy and scorning ethics will, for many, never be a viable option.
At the end of the day, keeping your stuff on your phone is getting harder, but it's still possible. And it's still worth the extra steps to keep the bad guys from turning your personal life into their covert revenue stream. Rooting through your property without your permission is just as creepy in the digital realm as it is in the real one, and it's sad that the computer field has embraced this as a valid business practice.
At least for now, though, you can still say no.
![[Python Logo]](_etc/PythonPowered.gif "Host website"){kind=small}
|
