![[Python Logo]](_etc/PythonPowered.gif){kind=small}
|
This is an early draft of the usage guide, as it was before this system's scripts were translated to Python. It's no longer maintained, and is retained only because it may be relevant to the Bash-script originals in this folder. The main folder above hosts the official guide, which has moved on substantially from the draft here; please see it for up-to-date usage info.
| Version: | (Draft, September 29, 2021) |
| License: | Provided freely, but with no warranties of any kind |
| Author: | © M. Lutz (learning-python.com) 2021 |
| Install: | Unzip the download, browse code, see ahead |
| Start: | Run with Bash, on macOS/Windows/Linux and Android |
| Web: | See this package's rationale here and here |
This doc's package was born of frustration with the twin removals of POSIX USB access in Android 11, and microSD-card support in Samsung's flagship phones. Those removals crippled content-processing tools, as covered here and here. This doc leaves the drama behind, and focuses instead on how to use the work-around embodied in this package's scripts. It starts with a brief overview, followed by a quick start, additional usage details, and general notes.
This package provides Bash scripts that automate steps used to copy content from a PC to an Android device initially, and sync the Android device's copy later for changes made on the PC. PC here means any macOS, Windows, or Linux computer, and Android device means any Android phone or tablet—though "phone" includes tablets here for brevity, and this package is primarily of interest to users of Android 11 and later (see Scope ahead).
In brief, the scripts here:
The net effect allows POSIX programs like Mergeall to be used for content management on Android 11 and later, and propagates content from PC to phone safely and easily. The next section explains the process that achieves this.
To sync changes from PC to phone, this system leverages a proxy drive, which takes the place of the phone for change detection: as long as the proxy's content copy is always the same as the phone's, PC-to-proxy changes double as PC-to-phone changes, and can be computed independently of the phone. The proxy's role is similar to an intermediate drive in earlier Mergeall models, and partly replaces now-defunct microSD cards. Under Android 11, however, the proxy's content copy cannot be merged to the phone directly by USB with Mergeall.
To address this loss of USB access, this system also employs a new Mergeall
script,
deltas.py.
This script detects differences between FROM and TO as usual, but instead of applying
FROM changes to TO immediately like a normal sync, it saves the changes separately to
a folder, in the same format as Mergeall backups. This allows changes to be both
archived as a unit, and applied to TO later with a Mergeall -restore run.
Because change sets are almost always very small, both tasks are almost always very quick.
By combining the new script's deferred changes with the proxy's forgery, this system is able to apply changes to the phone transitively: changes are collected by comparing the PC's content copy to the proxy's; transferred from the proxy to the phone manually with a file explorer; and finally applied to the phone itself. The net effect updates the phone for changes on the PC indirectly, without requiring POSIX USB access on the phone.
Beyond orchestrating the new script and proxy, this system also ensures that content is zipped and unzipped to evade interoperability perils in filesystems and platforms along the way. Depending on the target of content propagations, this can avoid loss of symlinks, permissions, folder modtimes, and nonportable filenames, all of which may otherwise be silently munged in raw transfers. Zipping also minimizes the role of Android file explorers, which are generally slow, and frequently fail for larger copies.
This may sound like a lot of work (and it is, compared to the direct two-merge process before Android 11), but it's still well worth the effort for people who don't want to store their private digital property on clouds, where it can be arbitrarily misused by hosting companies.
The good news is that this package's scripts automate most of the details behind these indirect syncs. In short, you will simply:
In the following, you'll generally run the setup tasks and initial copy just once, and repeat the sync steps any time you wish to propagate changes from PC to phone. Skip steps you've already performed, but see Installs ahead for specific version requirements.
On PC:
On Phone:
pkg install python
On both PC and phone:
Dedicate an external USB "proxy" drive to host a content copy that will serve as a stand-in for your phone in later syncs. This copy should never be changed except by this system's scripts, else change detection won't work and you may have to rerun an initial copy.
The drive should be large enough to hold your content and more—twice as large for initial copies, but a smaller increment for later change syncs (see space requirements ahead). exFAT is recommended for the proxy for its larger file sizes and friendlier modtimes, but any filesystem supported by both your PC and phone will do.
bash initial-copy-part1-pc.sh
bash initial-copy-part2-phone.sh
bash sync-changes-part1-pc.sh
bash sync-changes-part2-phone.sh
Download suffices on Android).
This system requires Mergeall 3.2 or later, for its
deltas.py
script and included ziptools. ziptools 1.3 or later is also
required, but is shipped with Mergeall 3.2; it can be found
separately at learning-python.com
here.
This system works with any Python 3.X; install Python only
if not already present, and see the
web
for PC install tips. See also PC and Android file explorers
for unzip options; you can use ziptools to unzip install packages,
but not until you've unzipped it.
These files are technically Bash code, but the settings are few and simple. Two syntax tips: use single quotes ('...') for paths with spaces and other non-alphanumeric oddities, but don't add a space around the "=" in assignments.
/sdcard) is usable by most apps and
not auto-deleted on app uninstall, but may run slower than the
other two storages in Android 11 (perhaps 4x to 100x, depending on
usage).
/sdcard/Android/data/com.termux/)
is quicker and can be accessed by many apps, but may be automatically deleted
when the owning app is
uninstalled.
/data/data/com.termux) is also
fast, but can be accessed only by the owning app, and may also be automatically
deleted when owning the app is uninstalled. [Edit: see also SAF explorer access.]
Depending on where you unzipped this package, its scripts
may not be in the folder you start at in your command tool.
If so, add their path to the front of their names in the commands
above (e.g., bash path/script.sh),
or cd to their folder first.
This copy must be a manual step, because it cannot be run by command line in Termux. Like all other POSIX programs, Termux no longer has direct access to USB in Android 11, per this coverage. This is the only manual step in initial copies and later syncs.
PauseSteps=n in either or both config
files.
A control-C at a pre-step pause's prompt kills the running
script, but may leave temporary zipfiles behind; see the killed
script for details on its progress.
tail -f $L/filename
at a console run in a different window. On your phone in Termux, you can
open a new session to tail logfiles and switch sessions by
swiping in from the top left. Don't control-Z to background a step in
the run window; it may terminate timing.
As a partial measure, setting
VerifyProxy=y in the PC config file
runs mergeall and diffall
to verify the proxy's content against the PC's in both initial copies
and later syncs. You are prompted to confirm this step, because it adds
time to the run.
-report option to
preview differences without saving them, though using the same
switch with
mergeall.py
has the same effect. This preview is
run initially by the syncs PC
script, if PreviewSyncs=y in the
PC config file.
Use these previews as desired to inspect deltas before writing them to disk; this may, for example, avoid full copies at DST rollovers for FAT32 proxy drives (see Mergeall docs). You are prompted to confirm this step, because it adds time to the run.
CheckFilenames=n in the
PC configs file.
mergeall.py -restore command, passing the backups set saved in
the archive's __bkp__ folder. See Mergeall's
docs, as well as
"ROLLBACKS" in
deltas.py's
main docstring for usage details.
As an example, you might expand this script if you're keeping your full content in Termux's app-specific storage for speed, and need to copy specific folders to shared storage so it can be accessed by apps with limited permissions—including tkinter GUIs run in Pydroid 3. This won't help, however, for copying website content to browsers' app-specific storage so it can be viewed locally; Termux cannot access other apps' app-specific storage in Android 11, so a file explorer with broader permissions is required.
time command to display step run
times, but this command ends timing prematurely on control-Z
to background a step. An alternative /usr/bin/time survives
control-Z, but isn't present on phone in Termux, and doesn't
respect Bash settings as well. Both timers work with &
backgrounding, but this is unusable in this serial system.
As a general rule: open another console to tail log files during runs,
per Logfiles above.
In the end, Android 11's removal of USB access for POSIX programs convolutes this story badly and remains an affront to usability and interoperability, while Samsung's removal of microSD cards seals the door tighter. Even so, this package's scripts manage to subvert whatever storage agendas may be.
|
|