[image]

Android Deltas Scripts

Version:  1.1, November 26, 2021 (changes)
License:  Provided freely, but with no warranties of any kind
Author:  © M. Lutz (learning-python.com), 2021
Install:  Fetch the download, unzip its code, see ahead
Requires:  Python 3.5 or later, Mergeall 3.2 or later
Runs on: macOS, Windows, Linux, Android

This package was born of frustration with the twin removals of POSIX USB access in Android 11, and microSD-card support in Samsung's flagship phones. Those removals crippled content-processing tools, as covered here, here, and here.

This usage guide leaves the drama behind, and focuses instead on how to use the work-around embodied in this package's scripts to propagate and sync content from PCs to phones running Android 11 and later. As we'll see, this work-around is partly manual but straightforward. It may also be permanent; while the future is unknowable, the Android 11 speed and access limitations we'll remedy here are still present in Android 12.

This doc begins with a brief overview, followed by a quick start, additional usage details, and general notes. If you'd like a headfirst preview, try this package's screenshots, examples, and code.

Overview

This package provides Python scripts which copy content from a PC to an Android device initially; sync the Android device's copy later for changes made on the PC; and optionally verify or export the Android device's content copy to the PC. These scripts are run as pairs on PC and phone, and automate steps that would be tedious and error prone to run manually.

In this guide, "PC" means any macOS, Windows, or Linux computer, and "Android device" means any Android phone or tablet—though "phone" includes tablets here for brevity, and this package is primarily of interest to users of Android 11 and later because it addresses USB restrictions that debuted in 11 (see Scope ahead for more info). In brief, the scripts here:

The net effect allows POSIX programs like Mergeall to be used for content management on Android 11 and later, and propagates content from PC to phone safely and easily. The next section details the process that achieves this.

How the Work-Around Works

To sync changes from PC to phone, this system leverages a proxy drive, which takes the place of the phone for change detection: as long as the proxy's content copy is always the same as the phone's, PC-to-proxy changes double as PC-to-phone changes, and can be computed independently of the phone. The proxy's role is similar to an intermediate drive in earlier Mergeall models, and partly replaces now-defunct microSD cards. Under Android 11, however, the proxy's content copy cannot be merged to the phone directly by USB with Mergeall.

To address this loss of USB access, this system also employs a new Mergeall script, deltas.py. This script detects differences between FROM and TO as usual, but instead of applying FROM changes to TO immediately like a normal sync, it saves the changes separately to a folder, in the same format as Mergeall backups. This allows changes (a.k.a. deltas) to be archived as a unit, copied between devices, and applied to TO later with a Mergeall -restore run. Because delta sets are almost always very small, these tasks are almost always very quick.

By combining the new script's deferred changes with the proxy's forgery, this system is able to apply changes to the phone transitively: changes are collected by comparing the PC's content copy to the proxy's; transferred from the proxy to the phone manually with a file explorer; and finally applied to the phone itself. The net effect updates the phone for changes on the PC indirectly, without requiring POSIX USB access on the phone.

Beyond orchestrating the new script and proxy, this system also ensures that content is zipped and unzipped to evade interoperability perils in filesystems and platforms along the way. Depending on the target of content propagations, this can avoid loss of symlinks, permissions, folder modtimes (modification times), and nonportable filenames, all of which may otherwise be silently munged in raw transfers. Zipping also minimizes the role of Android file explorers, which are generally slow and frequently fail for raw copies of large folders.

This may sound like a lot of work (and it is, compared to the direct two-merge process before Android 11), but it's still well worth the effort for people who don't want to store their private digital property on clouds, where it can be arbitrarily misused by hosting companies.

How You'll Use the System

The good news is that this package's scripts automate most of the details behind these indirect syncs. In short, you will simply:

  1. Setup your PC and phone to run Python, Mergeall, and these scripts
  2. Pick a USB drive whose content copy will be used as proxy for your phone
  3. Run scripts here to copy content to your proxy and phone initially
  4. Run scripts here to sync changes from PC to phone via proxy
  5. Optionally run scripts here to verify or export on-phone content
The rest of this doc provides a brief reference guide to this process, followed by more-complete details.

Quick Start

In the following, you'll generally run the setup tasks and initial copy just once, repeat the sync steps any time you wish to propagate changes from PC to phone, and verify occasionally. Important: script pairs are meant to be run in sequence; in syncs, for example, the phone-side script must be run before the pc-side script is run again. See install details ahead for specific version requirements and more options.

Setup: Install Python

On PC:

  1. Install Python 3.X if not already present, from python.org

On Phone:

  1. Install the free Termux app, from F-Droid (preferred) or Google Play
  2. Install Python 3.X in Termux, with command pkg install python
  3. See ahead for more on-phone setup pointers

Setup: Install Mergeall and Scripts

On both PC and phone:

  1. Download Mergeall's source-code package by clicking here, and unzip
  2. Download this script package by clicking here, and unzip
  3. Edit this package's config_pc.py on your PC, and config_phone.py on your phone

Setup: Choose a Proxy Drive

Dedicate an external USB "proxy" drive to host a content copy that will serve as a stand-in for your phone in later syncs. This copy should never be changed except by this system's scripts, else change detection won't work and you may have to rerun an initial copy.

The drive should be large enough to hold your content and more—twice as large for verifications and initial copies, but a smaller increment for later change syncs (see space requirements ahead). exFAT is recommended for the proxy for its larger file sizes and friendlier modtimes, but any filesystem supported by both your PC and phone will do.

To Copy Content to Your Phone

  1. Plug your proxy drive into your PC by USB, and on your PC run this command or its file:
    python3 initial-copy-part1-pc.py
  2. Plug your proxy drive into your phone by USB, and on your phone run this command or its file:
    python3 initial-copy-part2-phone.py

To Sync Changes to Your Phone

  1. Plug your proxy drive into your PC by USB, and on your PC run this command or its file:
    python3 sync-changes-part1-pc.py
  2. Plug your proxy drive into your phone by USB, and on your phone run this command or its file:
    python3 sync-changes-part2-phone.py

To Verify or Export Phone Content

  1. Plug your proxy drive into your phone by USB, and on your phone run this command or its file:
    python3 verify-phone-part1-phone.py
  2. Plug your proxy drive into your PC by USB, and on your PC run this command or its file:
    python3 verify-phone-part2-pc.py

Usage Details

This section goes into more detail about the steps outlined in the preceding Quick Start. It describes setup tasks and interaction, and covers enough Android and script-launch fundamentals to get you started.

Installs and Versions

Download and unzip both this package and the source-code package of Mergeall, to any folder on your PC and phone (e.g., the usual Download folder suffices on Android). This system requires Mergeall 3.2 or later, for its new deltas.py script and included ziptools. ziptools 1.3 or later is required, but is shipped with Mergeall 3.2 and need not be installed separately; you can find more info at ziptools' separate source here.

This system also requires and works with any Python 3.5 or later. Python is available in Android Python apps, and may be installed for PCs from here if not already present. See the lists below for more on installing Android-app Pythons, and the online guide for PC install tips. Python 3.5 hails from 2015, and most PC distributions and Android apps are already well past it (e.g., the apps listed below use 3.10 and 3.9 at this writing).

To unzip packages, see PC and Android file explorers for initial options; you can use ziptools to unzip install packages safely and quickly, but not until you've unzipped it. Also note that you do not need to install Termux if you will be using another app to run phone-side scripts (e.g., Pydroid 3), but Termux is recommended if you have no other preference. The following is a usage-essentials rundown for host apps on Android:

Termux

Pydroid 3

For more on Termux and other on-phone options, watch for upcoming launch details.

Update: Shims Wakelock The "shim" scripts provided for use with the Termux home-screen widget on your phone now automatically acquire and release an Android wakelock around script runs, to ensure that the scripts continue even if your phone screen goes blank. Hence, you don't have to manage the lock manually. See the shims and their README here for more info, and see ahead for more on the Termux widget; it's optional but recommended for on-phone scripts.

Space Requirements

The initial-copy scripts here temporarily require twice as much total space as your content collection itself on your PC, proxy drive, and phone. To copy a 100G content collection, for example, you'll need 200G on all three devices during the copy, but just 100G later. This stems from the scripts' use of zipfiles to avoid interoperability issues; an extra zipfile copy is passed between devices to avoid data loss.

If your devices don't have enough space, see the later notes on proxy and phone space for manual work-arounds. In short, you can emulate this system's steps with command lines, or skip some altogether. PCs require twice content's size during initial copies too, but are less likely to be as tight on space as phones and proxies.

By contrast, the sync-changes scripts here require only enough extra space for one delta set, which is normally very small and transfers quickly. In other words, once you get an initial copy onto your phone one way or another, later syncs with this system's scripts are unlikely to run into space limits.

In all cases, this system's scripts automatically delete both zipfiles and folders on PC and phone no longer needed after script completion. This includes temporary content and deltas folders, as well as their zipfiles. Exception: zipfiles for both the initial copy and deltas set on the proxy drive cannot be automatically deleted, because these scripts cannot access USB on the phone. You can freely delete these zipfiles from your proxy to save space, after their content has been propagated to your phone. Caution: don't delete the proxy's content-copy folder; it's needed for later syncs.

If your proxy drive is formatted as FAT32, you may also bump up against that filesystem's 4G file-size limit during initial copies of content; consider reformatting your proxy drive to use exFAT if it's supported by your phone or an add-on app, and see ahead for more details.

Update: Verify Space When used, the 1.1 verify scripts described ahead also temporarily require twice the space of your content on your phone, proxy drive, and PC, for an extra content zip on the first two, and an extra content unzip on the PC. They're also a sparingly used special case and fully optional.

Editing Configs

Edit both config_pc.py and config_phone.py to set the folder paths to code and content on your PC and phone, respectively, and change run options, optionally. See these files' top-of-file documentation and comments for directions, and the examples folder for config-file examples to emulate. Config-file settings are straightforward and are checked for errors before scripts start, but here are a few pointers on this step to help you avoid mistakes:

The config files are also technically Python code, but their settings are few and simple. For readers new to this language, here are five essential syntax tips:

The following list provides additional details and advanced usage tips for config files:

Backslashes in Windows paths
Python normally treats a backslash (\) in strings as the start of an escape sequence (e.g., \n means line-break). Since this character is also used as a path separator on Windows, it requires special handling in Python strings: the r'...' prevents Python from treating \ as an escape character; doubling up with \\ is an escape that stands for one backslash; and forward slashes (/) work in place of backslashes in Windows paths too.

See the Windows example for all three codings in action. Forward slashes in Unix paths require no such special handling. Embedded quotes must be escaped everywhere (e.g., '..\'..') but are enormously rare.

Home and environment path expansions
As a convenience, this system automatically expands home-folder and environment-variable references in your path settings like a Unix shell. For example, ~ in a path string is replaced with the path to the home folder associated with your user account, and $var and ${var} (plus %var% on Windows) are replaced with the value of the named environment variable.

This works on all platforms: a leading ~ is replaced with /Users/you on macOS, C:\Users\you on Windows, /home/you on Linux, and /data/data/com.termux/files/home on Android Termux. Variable replacement is handy for Linux proxy-drive paths that depend on your user name: use /media/$USER/drivename. For finer details, see the underlying Python tools here and here, and the examples here, here, and here.

Windows drive-relative paths
On Windows, paths should generally include a backslash after the drive letter to make them absolute (e.g., 'D:\\', using the Python doubled-up escape scheme described above). If the slash is omitted (e.g., 'D:'), the path is relative to the current working directory on the named drive, which is not what you'll usually want. For more background on this, see this Microsoft article. Scripts here don't automatically add a \ if omitted, because this would preclude drive-relative paths in contexts where they are useful.
Zipfiles are stored in proxy roots
Finally, you can locate the proxy drive's content copy anywhere on the proxy drive, but the zipfiles to be copied to or from the phone are always stored in the proxy drive's root folder (i.e., unnested) for easy access. In more tangible terms, this means that the PC config file's TO setting can be any arbitrarily nested proxy folder; its drive-root prefix is automatically determined and used for zipfile saves.

See also the "z-pc-froms" captures in the screenshots gallery starting here for more path examples. Coverage for specific configuration settings appears ahead in this guide, in the topics with which settings are associated; search in your browser to find more about a given setting here.

Android Storage

In the phone config file, you'll need to choose one of three on-phone storage categories to host your content. In brief:

Three disclaimers up front. First, due to Android fragmentation, these category's paths may vary across versions and vendors (e.g., /sdcard also goes by /storage/emulated/0); for a deeper dive, try the pathname primer. Second, the appname part in these paths varies too; it's com.termux for Termux, and ru.iiec.pydroid3 for Pydroid 3. And third, due to a now-ancient SD-card legacy, some Android docs confusingly call app-specific "external" and app-private "internal," but both usually reside in your phone today; they're just managed differently.

Also note that everything in this section and doc applies to unrooted devices; while phone rooting may loosen some constraints, it's not possible on many phones, and not desirable for many users (see this package's genesis for more details). Small print aside, the choice of storage host for your content pivots on four dimensions, some of which may be more important for your use cases than others:

Access
Android storage is more complex than that on PCs, and reflects the platform's meandering march towards tighter security. For this package's purposes, the Android app you'll use to run the phone-side scripts can generally store content in both shared storage and its own specific and private storages, but accessing the content later is more constrained. You'll need to choose a medium that supports the ways you'll access your content.

In the plus column, most Android 11 file explorers and the apps they open allow you to view and change your content in both shared storage and any app's app-specific storage. This may require one or more permission dialogs, but is broadly supported. Less positively, app-specific storage may be a read-only resource if Unix permissions are propagated, and app-specific support varies by explorer and viewer (e.g., Fx can currently explore shared but not app-specific storage on 11, and Music Folder Player bears the same limitation for audio files).

More rigidly, Termux and Python apps cannot access other apps' app-specific storage in Android 11 today, so your choice of content host may be binding. If you run this system's scripts in Termux, for example, you can store content in either shared storage or Termux's own app-specific storage, but programs run in Pydroid 3 won't be able to access the latter. Conversely, because Pydroid 3 is required to run tkinter GUIs, their code and data must be in either shared storage or Pydroid 3's app-specific storage, but Termux is locked out of the latter.

For the gritty details on Python-related apps, see the captured results for Termux and Pydroid 3. As shown, these apps today can access shared storage (including Download) and their own app storages only, though this depends on both which Androids apps target, and the ever-changing rules of Android itself.

Speed
Despite such app-specific constraints, shared storage may not be the best choice for larger content collections, because it has been throttled to the point of being almost impractically slow for some use cases in Android 11. The slowdowns vary per task, but are often noticeable, and sometimes glaring.

As an example, unzipping and deleting a 200G/180k-file collection in app-specific storage with Termux took just 18 and 10 minutes, respectively; but the same tasks in shared storage checked in at an arguably unusable 1 hour and 4 hours, respectively (and the shared-storage delete failed altogether in two file-explorer apps). Indeed, large-folder deletion is so slow in Android 11's shared storage that a phone reset may be quicker.

That said, not all shared-storage operations are so slow, and typical content access may be adequately fast for your purposes. Moreover, this package's use of delta sets naturally speeds on-phone syncs, because it avoids full on-phone content comparisons: smaller delta-set compares offset slower shared-storage access.

In fact, per practice so far, media access in shared storage is generally quick, and the phone portion of syncs for numerous changes with shared storage usually takes no more than 2 minutes—including time for the manual zipfile copy. Still, your syncs may naturally vary, and auto-deletion prunes of large __bkp__ folders in shared storage can be so slow that they seem to hang. Prune sloth was eventually sidestepped with config presets described ahead, but check script status in logfiles if pauses are long, and use app storage if waits are intolerable.

Longevity
Only shared storage is guaranteed to survive app uninstalls. Content you place in both app-specific and app-private storage is by default automatically deleted when the owning app is uninstalled. Apps can opt to avoid these auto-deletions with a prompt in recent Androids, but most apps do not use this new feature. If you store your content in Termux's app-specific storage, for example, it will be silently removed when Termux is uninstalled. The same goes for Pydroid 3.

This content-loss risk alone is nearly enough to disqualify app storage. It's also not entirely abstract; the Mergeall dev team itself accidentally erased gigabytes of content this way while setting up apps to test Termux's widget.

Metadata
Beyond all this, power users should also note that support for metadata and filenames varies across Android 11 storage types, and even per vendor. Per ziptools research, symlinks are supported only in app-private storage; nonportable filenames must be changed to be saved in Android 11 shared storage alone; and Unix permissions support varies by vendor but is always absent in shared storage. If you must care, see the fine points in ziptools' coverage here and here, its demos here and here, and this package's permissions policies ahead.

Special case: shared storage does not emulate FAT32's "local time" for file modtimes, and hence is immune to FAT32 skew on DST rollovers and time-zone changes described here. This matters because modtimes in on-phone storage must still differ with those in a deltas zip for changes to be applied. On multiple devices, shared, app-specific, and app-private modtimes all agreed with each other and PCs after changing dates and zones—and still did after an actual DST rollover. If modtimes ever skew anyhow, run the fixer script on deltas, or try app storage.

So where should you put your stuff on your phone, then? As guidelines—and barring a future Android fix for shared-storage throttling which would make that storage a clearer winner:

If pressed for a single recommendation, shared storage's cross-app accessibility make it the better choice for content-creation tasks such as programming, and its permanence is a clear win for valued content. App-specific storage, however, remains a faster and viable contender for basic content views, and app-private storage can still take the title in single-app roles; just don't delete the app!

Launching Scripts

You can run this system's scripts by typing command lines which name them in consoles (a.k.a. terminals), or by running their files directly with icon clicks, shortcuts, and IDEs (edit/run GUIs). IDEs work because these scripts are coded in Python, and icon clicks and shortcuts work because the scripts are, for ease of use, configured by settings in files you'll generally edit once, not by command-line arguments passed on each run. Here are the basics of each of these run modes:

Command lines
To run by command line, type the Quick Start's Python commands: In some of these, you may be able to omit the commands' "python3" or may have to use "python" instead, and in basic Windows consoles you should use "py -3" instead of "python3." Try "python3" first, and the others as needed. You can also run the scripts' files directly and skip the "python3" in both Unix on PCs and Termux on phones, by giving them executable permissions with a chmod +x script.py command, and ensuring that they are on your system search path (or using ./script.py to skip the search).

Command-line novices tip: depending on where you unzipped this package, its scripts may not be in the folder you start at in your console tool. If the scripts are elsewhere, either add their location's path to the front of their names in the commands above (e.g., python3 path/script.py), or simply cd path to go to their folder before running commands. path will likely be /sdcard/Download/android-deltas-scripts on your phone, and similar on PCs.

Icon clicks
On most platforms, you can also run the PC-side scripts by simply clicking (or tapping) their filename icons in a file explorer, without having to open a console and type a command line. On Windows, for instance, a click (or tap) runs a Python script, and the script ensures that the console window opened for the run endures after script exit so you can see its output; press enter/return to dismiss it. You can also force a script's console to stay open until a key press with the ForceKeyToClose config, but this is automatic for Windows clicks.

macOS and Linux run Python files by clicks too; see their file explorers for tips, or simply click a Python file to try. In brief: on macOS, right click a file in Finder and pick Get Info to associate a Python opener program globally, and do the same to access a file's Properties and associate openers in Nautilus (a.k.a. Files) on Linux.

Python IDEs
Thanks to their later Python translation, you can also run this system's scripts directly and without a command line from most Python IDEs, including IDLE and PyEdit on PCs, and Pydroid 3 and QPython on Android. Open a script's source file in your IDE and run it there per your IDE's normal actions. This really isn't any easier than running a command line in a console if you have to navigate to the file first, but the next section's pointers can make both options quicker.
Shortcuts
You can save time with UI shortcuts that run scripts immediately or open them in the prior bullet's IDEs. Shortcuts of some form are available on every platform on which this system runs. PCs support desktop shortcuts broadly, and Android supports home-screen shortcuts in multiple flavors. On PCs, for example, paste desktop shortcuts on Windows, drag aliases to the desktop on macOS, and see pointers here for techniques on Linux. Depending on your filename associations, scripts either open in IDEs or run in a single click.

On Android, you can use a file explorer to make home-screen links that open phone-side scripts in an IDE app, ready to run in just two taps. For instance, the Total Commander app can make such links to open scripts in Pydroid 3 on all Androids (but be sure to unclick its "Parameters: content://url" checkbox in later Androids), and other explorer/IDE combos may offer similar options. See Pydroid 3 shortcuts at work in screenshots here, here, and here.

Also on Android, a Termux add-on provides a home-screen widget which can run a Python console script like those in this package in a single tap. Because this widget is so handy, this system's phone-side scripts support it with an absolute path in their shebang (#!) lines, along with a ForceKeyToClose config setting to require a key press before the widget erases the console (like Windows icon clicks). If the shebang fails in other Android apps, edit scripts to use the line just below it, or simply use an explicit "python3" in console commands.

See the Termux widget in action in screenshots here, here, and here; get the widget on F-Droid or Google Play; and see other Termux add-ons and tools. Termux-widget tip: see the included shim scripts for an easy way to run this system's main scripts, without having to copy the main scripts to the widget's folder, change the prior paragraph's config, or manually acquire and release the wakelock. Simply copy the shims to your ~/.shortcuts folder in Termux, edit their runee paths if needed, and launch scripts by taps in the widget. See the README for details.

Constraint: shortcut or not, it doesn't help to open this system's scripts in an IDE app on your phone, unless you host your content in storage which that app can access. As we saw earlier, this means either slow shared storage or that app's own specific or private storage, and the choice can limit scope in Android 11. You can't use Pydroid 3 to open scripts in Termux's app-specific storage—and vice versa. This is a catch-22 inherent in Android 11, and a nearly compelling argument for using slower shared storage.

Script Interaction

For demos of script interaction, see the run logs in this package's examples folder, as well as the Android screenshot. For instance, here are typical sync sessions on a PC and a phone. These scripts explain their actions along the way and are straightforward to use, but the following covers a few salient points:

Inputs
In addition to their configuration files, these scripts request and read input interactively in the console. As usual with command-line programs, these inputs can be automated by using shell < file input-stream or <<EOF here-document syntax. See other resources for pointers. Tip: at "y or n" (yes or no) prompts, pressing enter/return alone is the same as inputting "n" (no)—as is any reply that isn't "y" itself.
Pauses
When running scripts, press the enter/return key to continue when prompted about a new step starting. To avoid having to press a key this way between steps, set PauseSteps to False in either or both config files. Alternatively, pause key-press inputs can be automated, per the preceding bullet.
Runtimes
Step runtimes are displayed in the console after each step completes, unless you set ShowRuntimes to False in the config files (these times may be more useful for long-running initial copies than quicker later syncs). Total elapsed runtime—all steps' time + all user time—is always displayed at script exit.
Aborts
A control-C at any pre-step pause or other prompt kills the running script safely, but may leave zipfiles and partial progress behind; see console messages and the killed script itself for details on progress before the abort. Reruns suffice to clean up temporary items left by aborted scripts, and are generally safe because deltas are not applied to the proxy until they are zipped and copied to it. If an emergency reset is ever necessary, see the rollbacks coverage ahead.

Tip: most of this package's screenshots have the luxury of being captured on a large Fold3 screen, but you'll probably want to run its scripts in landscape orientation on smaller phones to avoid line wraps in messages. The narrower Note20 Ultra used in this shot demos the idea.

Step Logfiles

Step logfiles give the full details of a run. They show up in the folders whose pathnames you assigned to LOGS in your PC and phone config-file settings. Here are example logfile-folder listings from a PC and a phone. Read the logfiles there later, or monitor a long-running step's progress with a command like the following in a console run in a different window, where logfilename is the latest item in your LOGS folder:

tail -f LOGS/logfilename
This command works in all Unix-like shells, including Termux on your phone. In Termux, you can both open a new session to tail logfiles and switch sessions by swiping in from the top left.

Besides saving step output to logfiles, this system also displays the last few lines of each step's output in the run's console window, to give you a quick summary of results. If this summary looks amiss, open the logfile in LOGS for the complete story. For a broad sample of run logfiles, see the demos' logs subfolders in this package's examples folder (here's a typical log). Tech note: logfiles are written in unbuffered mode, so their output shows up right away to give script status: the end of the log is the most-recent message written.

Manual Copies

On your phone, you will be instructed to manually copy a zipfile from the proxy drive's root to a specific phone folder, during the initial copy, and later syncs. Verify runs ask you to copy a zipfile from your phone to the proxy instead.

When prompted, copy the named zipfile using any Android file-explorer app which has permission to access both a USB drive, and the storage location to which or from which you are copying. Most explorers will suffice, and will work well because this is a single-file copy, but see the Android-explorer coverage ahead for tips.

This copy must be a manual step, because it cannot be run by command line in Termux (or similar). Like all other POSIX programs, Termux no longer has direct access to USB in Android 11, per this screenshot and coverage. This is the only manual step in initial copies and later syncs and verifications.

Note that file explorers' GUIs abstract the root folder of internal storage and USB drives in different ways. The root path of your internal storage at /sdcard, for instance, may be simply called "Internal Memory," "Main storage," or similar, and an attached USB drive at /mnt/media_rw/ID uses nicknames too. Extrapolate from path to explorer label as needed.

Fine point: /sdcard (a.k.a. /storage/emulated/0) is really the root of your per-user space in internal storage. Rooted phones have access to the higher and broader device storage root, /, but they also can access USB drives from POSIX programs at /mnt/media_rw, and so have no compelling reason to run this package.

Unix Permissions

As preset, Unix file and folder permissions are not propagated to the destination phone, because they have no effect in shared storage, and can make it impossible to change or delete content in app-specific storage. The latter may reflect an Android bug, but is crucial for content hosted in app-specific storage.

Not all Android 11 devices retain propagated Unix permissions in app-specific storage. Per ziptools' research and demo noted earlier, the Z Fold3 does, but the Pixel 4a does not. Where supported, however, propagated Unix permissions typically make it impossible to update or remove content in app-specific storage in any file explorer or app, other than the storage's owner. With permissions propagation, for example, content stored by this system in Termux's app-specific storage can be viewed by other apps, but can be modified only by Termux. Allowing Android to stamp items with default permissions instead enables updates and deletions from any approved app.

Hence, this package's phone scripts by default omit ziptools' -permissions flag in on-phone extracts, to prevent propagation of Unix permissions. To force propagation in contexts where it makes sense, set UnixPermissions to True in the phone config file. This may be useful for content hosted in app-private storage; since content there can never be accessed by any app except the owner, cross-app updates are irrelevant. This config is moot for shared storage, because permission propagations there are always ignored by Android.

Note that permissions are always retained in zipfiles created on Unix PCs; the config setting prevents copying them to phone storage types which support them (but perhaps shouldn't). Also note that this system assumes phones are view-only devices in general, but does not assume that on-phone content can never be changed; program data files, for example, may still require write access.

Update: Verify Permissions As a minor twist, the verify scripts added in 1.1 always propagate Unix permissions from phone to PC—though these will generally be defaults when mirrored from both shared and app-specific storage (and largely pointless on Windows PCs).

Nonportable Filenames

Except when running on Windows natively, this system by default offers to automatically run Mergeall's fixer tool for nonportable filenames on the PC, as a first step. Where relevant, this step is crucial to content interoperability, though its usage story depends on whether you run the PC-side scripts on Unix or Windows:

On Unix
The fixer-tool run is strongly recommended when transferring content or changes from Unix (i.e., macOS or Linux), to either Windows, some Android's shared storage, or proxy drives using FAT32 or exFAT. It's also generally a good idea when burning content to BDR and other optical drives, but these are outside this system's scope.

If you don't run this step, nonportable filenames that work on Unix may be mangled (i.e. changed) when propagated elsewhere, and their files may be discarded in full by some tools. In this package specifically: such files are skipped with an error message when transferring them to Android 11 shared storage, but saved intact to Android 11 app-specific and app-private storage, as well as earlier Androids' shared storage. This reflects the underlying ziptools system.

The fate of nonportably named files on proxy drives, however, is PC and drive dependent. Such files fail on Linux when written to FAT32 and exFAT proxies. By contrast, such names are saved on such proxies without error on macOS due to that platform's munging to and from Unicode privates, but may fail in later network back syncs. On both platforms, nonportable names may also leak out to Android in this system's initial-copy and deltas zips, and fail there. For examples of the effects, see the shared-storage and Linux demos here and here.

Though recommended in general for all these reasons, if this step proves irrelevant to your use cases, it can be skipped at its prompt, and its offer can be disabled altogether by setting CheckFilenames to False in the PC configs file.

On Windows
The fixer-script step is never run when using Windows native tools, because it's pointless in this context: Windows disallows nonportable filenames globally (it is the platform that makes them nonportable, after all), so there are no names to fix. Windows native here means that these scripts are run by a Python that registers as Windows (i.e., win32) in Python's sys.platform. Non-native Windows tools (e.g., Cygwin's own Python) may yield nonportable names, but do not register as Windows this way, and so will invoke the fixer step to adjust filenames.

The filename behavior of non-native Windows tools is subtle. Cygwin, for example, allows nonportable names to be created via name munging much like macOS. Its own Python reverses the munge to nonportable characters, but registers as cygwin. Windows' Python run in Cygwin register as Windows, but does not map munge characters back to nonportables, so files do not fail. WSL Linux munges nonportables just like Cygwin, and registers as linux. For demos of Cygwin and WSL in action, see the examples here.

Hence, the name-fixer step is always skipped in Windows-native contexts, for which the CheckFilenames config setting is ignored. Cygwin and WSL users can change this setting to skip the step as desired.

For more on the name-fixer step in general, see the comments at the top of Mergeall's fixer tool, and ziptools' related coverage of nonportable filenames here.

Related note: Unix filenames should generally be run through the name-fixer script anytime they are copied to Windows—including in manual zips created outside this system. Else, the unzip may discard files or handle their names unexpectedly (e.g., backslashes in Unix filenames generate subfolders on Windows in ziptools, but are less harmfully replaced with an underscore in the fixer). See the Windows example for a rare case where this matters, and the Windows unzips coverage ahead for related info. Avoiding nonportable filenames in the first place may be their best work-around.

Sync Previews

The Mergeall deltas.py script supports Mergeall's -report option to preview differences without saving them, though using the same switch with mergeall.py has the same effect. This preview is run initially by this system's PC-side sync script, if PreviewSyncs is True in the PC config file, and you confirm the run interactively.

This config setting is preset to True because previews are generally advised. Use them as desired to inspect deltas before writing them to disk. This may, for example, avoid unintended and time-consuming full copies at DST rollovers when using FAT32 proxy drives (see Mergeall docs).

Sync Verifications

There is no direct way to verify the on-phone content copy against the proxy or PC copies, because POSIX code like Python cannot access USB drives normally on Android 11, and phone access by MTP and network servers is not fully usable. The phone might be compared to the proxy after syncs by manually copying full zipped content from proxy to phone, or vice versa; this is too slow to automate here, but might serve as an occasional and manual option (tip: zip content first on phone or proxy using a tool like ziptools—or see the update ahead for a precoded solution that does).

As a partial measure, setting VerifyProxy to True in the PC config file runs mergeall and/or diffall to verify the proxy's content against the PC's in both initial copies and later syncs. The mergeall run compares by structure and modtimes, and diffall performs a full byte-for-byte comparison. The config setting is preset to True because verifications are generally recommended, but you are prompted to confirm each verify step independently because they add time to the run unevenly. A mergeall verify is usually quick, but a more-complete diffall can be slow for larger content collections.

Note that it's normal and harmless to have one diffall diff: a unique item for the copies' top-level __bkp__ backup folders. Also note that symlinks on Linux cannot be written to FAT32 or exFAT drives, and so will always register as PC-side diffs. This is an innate limitation of these filesystems on Linux—which macOS avoids only by forging a symlink stub file which passes for an actual link on macOS alone, and will still yield diffs elsewhere. For an example of the consequences on Linux, see its demos. The only external-drive symlink option on Linux is other filesystems.

Update: Phone Verifications As of version 1.1 of this system, it is now possible to verify the phone's content copy against the PC's, by running the new verify scripts. For a quick look, see them in action in the new example and screenshots. These scripts switch dataflow order—content moves from phone to PC—but config-file settings work unchanged on both phone and PC.

This process is optional and meant to be used only occasionally, because it is relatively slow and temporarily requires twice the space of your content on phone, proxy, and PC for an extra content copy or unzip. As chronicled by the new example's readme, for example, verifying an on-phone content folder of some 200G space and 160K files required an extra 200G everywhere (400G total capacity), and took roughly one hour (50 minutes on phone and 15 on PC). Especially if your phone hosts a throwaway copy, such costs may be unwarranted.

Nevertheless, if you do wish to prove that the phone's copy matches that on PC, the verify scripts zip full content on your phone; prompt you to manually copy the zipfile to your USB proxy; and then unzip the phone's content on your PC and compare it to the PC's copy with Mergeall and diffall. As noted, the former compares content copies by modtimes and structure; the latter is a byte-for-byte compare which is more complete but slower, and an optional step in verify scripts.

Though too slow for frequent use, this suffices to validate phone content: you should see differences only for normal skew (e.g., backup folders) and items that cannot be stored on your phone (e.g., symlinks in shared storage). Just as importantly, the verify scripts can also be used to export phone content to PC—simply opt to retain the unzipped phone content at the end of the scripts, and sync it to the PC's copy with Mergeall or adopt it in full with a move. We'll revisit this role as an alternative to back syncs ahead.

Finalization Hook

Depending on your use case, you may wish to edit the z_postcopy_hook_phone.py script, to add custom steps required after to-phone copies are finished. This file is run at the end of both the initial-copy and synch-changes phone scripts. It's shipped with no commands, but any Python code you add to it will be run after all to-phone content propagations.

As an example, you might expand this script if you're keeping your full content in Termux's app-specific storage for speed, and need to copy specific folders to shared storage so they can be accessed by apps with limited permissions—including tkinter GUIs run in Pydroid 3. This hook may also be useful to copy media files to shared storage for constrained players.

This won't help, however, for copying website content to browsers' app-specific storage so it can be viewed locally; Termux cannot access other apps' app-specific storage in Android 11, so a file explorer with either app-specific permissions or local-view heroics is required. Some explorers, for example, now spawn HTTP servers to circumvent Android access limitations.

Sync Rollbacks

If ever needed, changes made by applying a deltas set with this package can be fully rolled back (i.e., undone), by manually running another mergeall.py -restore command in the host platform's console, passing the backups set saved in the content copy's top-level __bkp__ folder as the command's FROM source.

This restore command can be applied to both the proxy and on-phone content copies, to restore their state before the latest sync. It takes the following general form, where TO is the path to the root folder of your proxy or phone content copy, and backup is the latest date/time-labeled backup folder:

python3 mergeall.py TO/__bkp__/backup TO -restore -auto -skipcruft

See Mergeall's core docs, as well as "ROLLBACKS" in deltas.py's main docstring for more on running restore commands. This package's scripts do their part, by using Mergeall's -backup flag when applying delta sets during syncs—apart from the major optimization exception added late in the 1.0 project, and called out by the following update.

Update: Backup Configs

The -backup option automatically prunes old __bkp__ backup folders. Because these deletions can be horrifically slow in shared storage in Android 11 (5 to 10 minutes for under 2G was not uncommon), backups in this package's scripts can now be configured on or off in both the PC and phone config files. Set BackupChanges to True in the PC file to enable backups for proxy-copy changes. Do the same in the phone file to enable backups for phone-copy changes. When enabled, Mergeall's -backup flag will be used by sync scripts when applying delta sets, to enable later rollbacks.

Important: as preset, backups are enabled for the proxy drive, but disabled for the phone to avoid slow shared-storage prunes. If left disabled, you cannot perform a later rollback of phone-sync changes, and must recopy content if it's ever corrupted by a sync. However, the presets make syncs much faster; phone copies are often view only and disposable; and disabling backups saves space on any device. The speed boost is moot if you opt to use app storage for content on your phone, but shared storage has usability advantages which are compelling enough to support it by default.

Android File Explorers

Android 11 file-explorer apps allow you to access your on-phone content, but vary widely in both metadata support and storage access. Some may be updated further by the time you read this note, and Android is nothing if not convoluted. But as pointers for users of this package: among the ten apps tested at this writing (October 2021), and on two unrooted Android 11 devices tested (a Pixel 4a and Z Fold3):

The nature of this support varies too. For example, some explorers including Samsung and Solid may require an app restart to notice USB; paths used to open files on USB drives are /mnt/media_rw on Solid but content:// URIs on others; Total Commander uses an unusual link path to open app-specific files and doesn't support Copy To; and tapping a USB drive in Notifications may also open a vendor-default browser. Some Android explorers have also been seen to hang or fail on large copies and deletes, though this may reflect temporary bugs.

Naturally, USB and app-specific content access depends on both file explorers and the apps they invoke to open files, but a useful survey of the latter is outside this doc's scope. For definitive results, always test the latest versions of explorers and apps you wish to use on your own devices. Related details: see also the coverage of exFAT-driver apps for USB proxies ahead; the peek at website local-view support in explorers earlier; and the parallel coverage of explorers off page.

Usage Caution

Finally, it's worth adding a general caution for the sake of your content. Though perhaps obvious, this is a content-propagation system. It will change content on your proxy drive and phone by design. Some of its changes can be undone, but this depends on usage and configuration. To avoid unpleasant surprises, you should read this guide and its examples before using this system, and would do well to experiment with the system on non-critical data initially before using it on valued material.

Although this system is tested, open source, and used by its maker regularly, any content propagation tool can fail due to variables beyond its control, including errors in usage, hardware, and software. By using this system, you both accept responsibility for all its actions, and benefit freely from all its capabilities.

General Notes

This section closes with some notes about the system's design. Some of it is probably of more interest to developers than users, but is also provides some additional context and caveats along the way.

Background Docs

See the overview page for more on this system's scripts, and this folder for the unzipped content of this package. If you are new to Mergeall, the best starting points are its main web page and user guide. For more on the ziptools system used throughout these scripts, start at its web page or user guide. For this system's rationale in general, try this and this.

Release History

This section summarizes this system's versions. The date at the top of this page always gives its most recent release. If you find an issue in this system, please report it via the "Input" link in the toolbar below. Software improves best with use and feedback.

Version 1.1: November 26, 2021
Version 1.1 adds new verify scripts, which can be used to both verify the on-phone content copy, and export this copy to the PC as an alternative to back syncs. There is also a new Termux-widget shim for the on-phone verify script. Because version 1.1 does not change any existing 1.0 functionality, all prior examples and screenshots are still valid.
Version 1.0: October 18, 2021
Version 1.0 was a fully functional system, repackaged for minor changes and fixes after its initial Oct-18-2021 publication. Its final Nov-12-2021 release included mods to print logfiles in unbuffered mode for better status info; make backups configurable and disabled by default on phones to sidestep shared-storage sloth for backup prunes; remove an existing deltas folder during phone syncs; and automatically acquire and release the wakelock in Termux-widget shims.

Python Translation

This system's scripts are Python translations and extensions of same-named Bash originals retained in the _etc/bash-version folder. The Bash originals work, but the Python recoding is better for portability of the PC-side scripts to Windows: they may be run in normal Windows shells like Command Prompt and PowerShell, without having to install Cygwin or WSL. Moreover, Python scripts can also be run on PCs by icon clicks and IDEs as covered earlier.

Cygwin and WSL can be used on Windows too, but have tangible seams which make them less user friendly (e.g., newline rules and drive mounts). Bash might suffice for phone-side scripts since commands can be run in Termux, but config files must be in Python or Bash (not both), and Python enables launches by Android IDE apps and shortcuts, and better supports code evolution.

A downside of Python in this context is that its abstractions used for portability may obscure the command lines it runs; see the Bash originals for a direct (if cryptic and Unix-only) command tour.

ziptools Version

This system does most of its work by leveraging the Mergeall and ziptools systems as a software stack. ziptools is shipped both stand-alone and as a tool embedded in the Mergeall package. Scripts here use Mergeall's copy, though the stand-alone version is also available here. You can enable ziptools stand-alone installs in common.py, but this is not required. The ziptools included with Mergeall 3.2 or later suffices here, and the same maker publishes ziptools, Mergeall, and this package; the three are kept in sync and help drive development in each other.

Times and Logs

To report real elapsed runtimes, these scripts now use Python timer calls which are immune to complexities in shell timing tools. The scripts' original versions used the Bash time command to display step runtimes, but this command ends timing prematurely on control-Z to background a step, and an alternative /usr/bin/time survives control-Z but isn't present on phone in Termux and doesn't respect Bash settings as well. Python avoids these issues, but as a general rule, open another console to tail log files during runs, per the logfiles usage coverage above.

Why Zips

This system zips content for transfer to minimize the risks of data loss and copy failures. Else, some copy destinations drop folder modtimes, symlinks, and permissions, and may discard or covertly rename files with nonportable names. Worse, Android file explorers run very slowly for raw folder copies, and may even fail to finish. Without naming names, multiple apps fully hung while testing copies of large folders, and some suffered disk unmounts (see also the earlier explorer coverage).

Zipping content neutralizes all these factors in full. As a bonus, zips are also generally much faster to transfer than individual files on every device. Unzips can outpace raw copies too, because the former opens just one input file: see the initial-copy script's cpall note.

Why Not a GUI

A cross-platform deltas create/apply GUI was rejected for this project for a number of reasons. For one, in the simplest use case with no zips, syncs would require two runs in the GUI on PC (create deltas, apply deltas), and forgetting the second would leave the proxy different than the phone. For another, zips are nearly required per the preceding note, and this makes for numerous steps (four on both PC and phone; see the scripts). Automating steps and zips in scripts is far less error prone.

Size Caveat: FAT32

This system won't work with FAT32 proxy drives if its content or deltas zipfiles are larger than 4G (this is a FAT32 limitation). This limit is unlikely to be broached by syncs, because their deltas zips are normally small. It could, however, be a factor for initial copies of content collections larger than 4G, as they zip content in full.

If your zipfiles are that large, your best option is to reformat the proxy drive to use exFAT, or another more-functional filesystem supported by both your PC and your phone. exFAT is generally your most interoperable choice for USB storage devices. It does not have FAT32's 4G limit, avoids FAT32's modtime perils that can derail proxy syncs, and is broadly usable on both PCs and most recent Android phones. While exFAT support on phones has been vendor dependent in the past, its recent adoption by the underlying Linux kernel should make it standard on Android soon.

If you're unsure of your devices' exFAT support, try your user docs or the web, or simply format a drive and test. Where unavailable as a built in, you may still be able to use exFAT on your phone by installing an app. Such apps do not mount USB drives in ways accessible to POSIX programs, and may require specific file explorers, but can suffice for the manual zipfile copies employed by this system's scripts. On an unrooted Android 11 Pixel 4a, for example, the Paragon app here and here enables exFAT access in Total Commander, X-plore, and FX, albeit with occasional remounts.

If exFAT (or other filesystem) is not an option in your use case, your last resort is to use raw copies in file explorers. This still requires enough space on the proxy to hold a copy of your content, and regrettably adds Android file-explorer variables to the mix, but it may avoid FAT32's 4G limit for individual files. Moreover, you may still be able to use this system's sync scripts for later updates because their zips are usually small.

Size Caveat: Proxy

As coded, the initial-copy-part1-pc.py script temporarily requires twice as much total space on your proxy drive as your content collection, to store both a content copy and an uncompressed content zip. For example, a 100G content collection temporarily requires 200G on the proxy, but just 100G later. This is necessary because stingier alternatives have too many steps to automate.

If your proxy doesn't have enough space, simply run this script's steps manually, but move the zip to your phone before copying content to the proxy (see the Bash original for more-direct command examples). The upside is that this space requirement is temporary (the zip may be manually removed from the proxy after the to-phone transfer), and applies only to the initial content copy. Later syncs require just enough extra space for normally small delta sets.

Special case: the 1.1 verify scripts described earlier temporarily require twice the space of your content on your proxy for an extra zipped copy (and similar on your phone and PC). They're also entirely optional.

Size Caveat: Phone

As coded, the initial-copy-part2-phone.py script temporarily requires twice as much total space on your phone as your content collection, to store both an uncompressed content zip and a content copy. For example, a 100G content collection temporarily requires 200G on the phone, but just 100G later. This is necessary to avoid issues in Android file explorers, which have been seen to fail for direct copies of larger content collections.

If your phone doesn't have enough space, manually copy unzipped content from proxy to phone by file explorer, and run this script's other steps manually as needed (see the Bash original for more-direct command examples). This script deletes the zip from the phone automatically after unzipping, and its size requirement also applies only to the initial content copy. Space requirements for later syncs are much more modest.

Special case: the 1.1 verify scripts described earlier temporarily require twice the space of your content on your phone for an extra zipped copy (and similar on your proxy and PC). They're also entirely optional.

Skew Caveat: Proxy vs Phone

It's atypical but not impossible that the proxy drive's content copy may differ slightly from that on the phone for which it stands in, because the phone's copy reflects zipfiles created on the PC, not the proxy itself, and the proxy and phone may support esoteric content differently.

Exotica like symlinks and Unix permissions, for example, may differ across both proxy filesystems and Android storage types, and the disposition of nonportable filenames is platform specific. We've touched on some of these topics earlier (e.g., here, here, and here), but this note takes another look from the perspective of content skew.

First and foremost, it's important to note that most use cases don't need to care. Proxy and phone copies will never differ for normal content (files and folders) and its modtimes. Moreover, even when deviations arise, they may be acceptable; may be irrelevant as long as the proxy suffices for syncs to PC; and may be easily avoided in general. If a content component doesn't work on a drive or Android, you can opt to simply not propagate it.

If your use case ever does need to care, however, here's a quick rundown of expected differences when twilight-zone content is moved from PC platforms to drives and phones:

The upside here is that components which may differ between proxy and phone are generally rare and trivial, and can be controlled by users. They also cannot be better supported in any event. Zipping on the proxy instead of the PC might avoid some skew, but it would also slow initial copies and syncs, and is unlikely to offer any advantages for notoriously nonportable content like symlinks and Unix-only filenames.

More generally, the content copy on the phone may grow out of sync if it is ever changed on the phone: because it will never again be synced to the full copy on PC and proxy after initial creation, on-phone changes may endure until another full copy. This is an inherent downside in using only incremental delta-set syncs, but necessitated by Android 11's removal of POSIX USB access. On the other hand, this is acceptable if the phone's copy is used as a view-only resource—a current assumption of this system discussed ahead.

Update: Checking for Skew As of version 1.1 of this system, you can now check for differences between on-PC and on-phone content copies, by running the verify scripts introduced earlier. While this is a slow process and not a direct sync, it can be used periodically to detect unintended skew between phone and PC. This can also be used to export phone content in general per ahead.

Scope: Other Platforms

This system is primarily meant for Android 11 and later. It works to sync devices running Android 10 and earlier too, but may be overkill there. Prior to 11, you can simply run two direct Mergealls by USB: from PC to external drive, and from external drive to phone—and using either Mergeall's command lines or GUI for both. This is arguably more user friendly, as no manual deltas-set copy is required along the way. See Mergeall's original docs for its simpler usage on earlier and less restrictive Androids.

In principle, this system could also be used to sync other platforms besides phones. For example, once a deltas set is created, it could be applied to a heterogeneous set of devices whose content copies are all kept in sync with the proxy drive. In this mode, a destination would not necessarily be a "phone" and not necessarily be Android. Phone config files would reflect platform-specific paths on each sync target, and the phone-sync script would simply be run on each target device in turn—whether they are phones or not.

This usage mode works today too, but this system is not explicitly designed for it, because this mode inflexibly requires all other devices to be synced at the same time (while deltas are valid), and other platforms can use direct and simpler merges. Syncing to a mix of macOS, Windows, Linux, and Android-10-or-earlier devices, for instance, can use a basic Mergeall run on each as before, both whenever convenient, and with either command line or GUI. Only Android 11 requires the indirect deltas-based syncs performed by this system.

Nevertheless, syncing multiple devices with a precomputed deltas set might be quicker than basic merges, because it avoids an initial comparison phase on each target. Moreover, a device mix that includes Android 11 must create deltas for Android 11 anyhow, making them available for use on other devices. This system awaits broader usage feedback to determine whether generalizing its scope makes sense.

Update: Android 12 In late 2021, Android 12 is now known to have the same shared-storage-speed and USB-access limitations as Android 11. The bad news in this is that these constraints seem poised to be either long-lived or permanent. The good news is that this package's scripts work around them on Android 12 too.

Update: Usage Feedback In practice so far, running this package's scripts on earlier phones seems as easy as Mergeall's GUI (especially when the Termux home-screen widget is used for launches), and is noticeably faster because it skips full comparisons. These scripts now also seem simpler and safer than pulling and syncing a microSD card on phones that still have one; in fact, they may be a better option even if cards reemerge on future Folds. Still, Mergeall's GUI remains useful on older Androids for impromptu syncs, enables direct content verification that indirect schemes cannot, and supports broader usage modes of the sort discussed in the next section.

Scope: Back Syncs

This system generally assumes that the phone (i.e., sync destination) is used as a view-only device. While changes on the phone might be synced back to the PC with a similar indirect scheme, supporting this would require substantial script retooling, if possible at all.

Such back syncs are made complex—if not implausible—by the same lack of direct POSIX USB access on Android 11 which was this system's genesis. Because there is no way to compare on-phone content to the proxy drive itself, either:

All of these would be prohibitively expensive in time or space for large data sets, unless full content transfers or duplicate copies could be avoided. Although it's possible to imagine options that might reduce overheads (e.g., computing deltas from separately maintained doppelgänger trees with modtimes but zero-length files), this may again require religiously syncing multiple devices at the same time, and could complicate and slow content syncs radically or impractically—and all for the sake of Android 11 alone. This may rule out back syncs themselves, but the following update provides an alternative.

Update: Phone Exports As of version 1.1 of this system, it is now possible to export phone content to your PC, by running the verify scripts introduced earlier. These scripts use zipfiles to copy phone content to PC for comparison, and were designed to verify phone content. They can also be used for phone exports, however, by opting to retain the phone's unzipped content on your PC at the end of the verify process. Then, simply sync the phone's content to the PC's with a normal Mergeall run on your PC, or adopt the phone's content copy in full as your new on-PC copy.

This is a slow process, and does not qualify as a back sync per se: it's a full copy from phone to PC, with an optional sync on the PC. Nevertheless, its automated zipfiles speed the copy, and it may suffice to propagate changes from phone to PC occasionally. If you'll be using only your phone on a trip, for example, you can sync from PC to phone before leaving, make arbitrary changes on your phone while away, and use the verify scripts to propagate changes back to your PC when you return. This is nowhere near as nice as direct Mergealls, but will have to pass as usable on Android 11 and later. Which bring us to this guide's conclusion.

Closing Thoughts

This doc has largely resisted editorializing, but in closing, a few words of perspective are in order.

This system is the culmination of a content-management odyssey on Android which has spanned four years and versions—from Oreo, whose working modtimes first enabled syncs, through 11, whose by-design USB lockdowns and shared-storage throttling made this task much more difficult. In particular, Android 11's removal of USB access for POSIX programs convolutes content management badly, and remains an affront to usability and interoperability. And Samsung's removal of microSD cards only seals this door tighter.

This may be disappointing, but should not be wholly unexpected. Android is not desktop Linux, despite the shared heritage and code. It bridles devices broadly with SELinux mods; throttles programs aggressively based in part on screen interaction; and walls up storage into opaque content prisons, which align with cloud-storage agendas, and grow ever tighter. Equating such devices to PCs seems the stuff of deceptive marketing and quixotic fancy.

Together with the ethics of Android's owner, this has naturally spawned a culture in which customer exploitation passes as a norm. Indeed, smartphones seem crafted to provide their makers with points of control—whether this means monopolizing access to media; tracking users for advertising; extorting revenue from developers; locking down product repairs; or harvesting personal information stored in clouds. While some maker motives remain matters of speculation, these devices are clearly not as customer focused as they might be.

Even so, this package's scripts provide a path to maintaining content on these devices for those determined enough to try, and manage, at least for their intended roles, to subvert whatever Android-storage intrigues or gaffes may be. The work-around is not trivial, but cloud-based alternatives are innately perilous. Trusting valued content to profit-driven companies with clear patterns of invading privacy and violating civil liberties will, for many, never be a viable option.

At the end of the day, keeping your stuff on your phone is getting harder, but it's still possible. And it's still worth the extra steps to keep the bad guys from turning your personal life into their revenue stream. Rooting through your stuff without your permission is just as creepy in the digital realm as it is in the real one. But you can still say no.

 

[Python Logo] Top Code Fold3 Android11 Android Mergeall Mergeall Ziptools Input ©M.Lutz