|Version:||June 14, 2019|
|Author:||© M. Lutz, learning-python.com|
and fetch this package|
View this guide's media|
|View the related tkinter doc|
This document describes how to use Mergeall to sync content on Android devices. It primarily focuses on running command lines in the Termux app, and more briefly explores a Mergeall GUI alternative on this platform. This coverage is provided freely, but with no warranties of any kind. Findings here reflect testing on Samsung Galaxy devices running Android 7, 8, and 9 (Nougat, Oreo, and Pie). Given the wide variety of products and rapid—really, supersonic—rate of change in the Android world, applicability to other devices and Androids cannot be guaranteed. Mergeall itself changes content by design: see its usage cautions before using the approaches outlined here.
If you're able to meet its requirements below, Mergeall on Android provides incremental backups and changes propagation for content stored on your smartphone. With a few simple commands (or a fledgling GUI), you can sync content on your phone's internal storage or removable SD card both to and from an attached USB drive, and can do so right on your phone itself. The net effect turns your phone into a portable storage device for arbitrarily large content collections, without requiring card removal or device rooting, and without the cost, speed, and privacy negatives of cloud storage.
that hosts the guide you are reading is a supplement to the Mergeall system.
It comes with precoded helper scripts
that you can run directly on your phone from the Termux app's prompt. Though not
required, these scripts are designed to simplify Mergeall's command-lines mode on
Android, and minimize the risk of user errors. To make edits on smartphones easier,
these scripts also use common settings in the included
which you'll modify for your drives' IDs and folder names. Besides their role in
scripts, these settings provide quick command-line access to your folders in general.
For Mergeall users who break out in hives at the thought of using command lines, this guide's Appendix B also includes coverage of using Mergeall's GUI in the Pydroid 3 app, and this package includes patch files needed to make the GUI work. While this approach comes with code changes, operational issues, and pay-or-else advertising that make it a secondary choice here, some users may prefer a GUI even if it entails extra drama. If you wish to explore this option, be sure to read the Termux coverage in the main part of this guide first, as most of it applies to Pydroid 3 GUIs too.
Android's history and variability make documentation challenging.
In this guide and its scripts, the terms "SD" and "USB" are
used to mean a removable microSD card, and anything that works on
the USB port, respectively, and the terms "external" and "removable"
are used interchangeably for a drive that can be detached from
your phone (i.e., SD or USB).
Your device also has nonremovable storage called "internal" here,
though Android often refers to this as "external" and makes it
available in a folder named
/sdcard. And if you don't
have a removable card, your phone instead has a nonremovable
"external" partition on your phone's internal drive set aside
for storage—and confusion.
In the name of generality (and sanity), this package's scripts use the more function-related "data" to mean the content copy kept on your phone (e.g., on internal storage or removable SD card), and "plug" to refer to the content copy on your attached USB drive (e.g., on flashdrive or portable SSD); extrapolate as needed for your use cases. The term "phone" crops up informally here too, but this guide applies to Android devices of all types.
And to any grammarians in this document's audience: some of the closed compounds used here, including "logfile," "zipfile," and "fullscreen" may rankle a few purists, but they just make sense (the compounds, that is, not the purists).
The Mergeall system, originally developed for PCs, can be used unchanged on Android to synchronize on-phone content to and from a USB drive, without rooting your phone, and without requiring SD card removals—an arguably amazing feat in such a limited form factor. To make this work, though, you'll have to meet a few requirements up front. In short, Android has multiple issues and permission rules that require a custom approach; to use Mergeall on this platform, you:
The sections that follow go into these requirements in more detail. Later sections cover setup for these requirements, and usage of this package's helper scripts. The positive take here is that Mergeall on Android may be constrained, but it does work. While some users may prefer pulling their SD card and syncing on a PC, this is delicate work in the extreme, and isn't possible on all devices. With a USB drive, a little up-front work, and either simple command lines or GUI, Mergeall brings a more PC-like experience to your on-phone content management.
The first requirement is an absolute: Mergeall works on Android devices, but only if they are running 2017's Android 8 (a.k.a. Oreo, and sometimes just O) or later. Earlier Androids have a showstopper bug which renders content-sync tools like Mergeall unusable. You can still use Mergeall to manage your phone's content prior to Oreo, but not on Android itself: remove your card and sync it on your PC.
The short story on the bug is that earlier Androids are unable to copy file modification times to either FAT32 and exFAT drives or internal storage on non-rooted phones: file content can be copied correctly, but copied files are always stamped with the time at which the copy was made, not that of the source. This is lethal to content-sync tools like Mergeall, which rely on file timestamps for comparison speed. Without timestamp copies, on-phone content copies cannot be compared to other copies accurately, and quick content syncs are impossible.
This issue is well-known, long-lived, device-wide, and not specific to
Mergeall, Python, or Termux. In fact, a shell
cp -p copy
command fails to copy times with an error before Oreo too—see the Nougat
It stems from Android's former FUSE-based filesystem, which required root
access to update file times but not content, and was eventually replaced
with Samsung's SDCardFS. For more background, try
the bug report
and searches like
As an example of its wide swath, this bug also kills calendar-file backup copies
in the Frigcal program, which makes a cameo appearance in
prior to Oreo, many programs were impossible on smartphones.
Though likely unintentional, it's unfortunate that the world's most-used mobile operating system didn't support non-cloud content syncing until 2017 (and still doesn't for many prior-version users). The good news is that this bug is now fixed, making cloud-free storage on Android finally practical for non-trivial content—and just in time for the larger-storage devices beginning to appear.
This section was rewritten February 2019 for new findings, and updated April 2019 for Samsung's reply.
Due to a current Samsung Android bug regarding modification times recorded by the exFAT filesystem, removable drives used with Mergeall on Android should generally use FAT32 instead of exFAT for interoperability with other platforms. In short, the modification times of files created on exFAT drives are known to be incorrectly skewed as follows when using Android Nougat and Oreo (and later Pie) on multiple Samsung devices:
Both skews were observed in and are relative to the US Pacific time zone, which is normally 8 hours behind the UTC base. These are likely symptoms of the same exFAT bug in Samsung Android, and may eventually be fixed in its future versions, but they can wreak havoc with Mergeall file comparisons today.
It's important to note up front that, due to testing constraints, it's unknown whether this flaw lies only in Samsung software or is more widespread. exFAT is not part of Android's AOSP core-functionality stack (described here and here) which all phone makers augment with code from various sources (described here and here), and the issue is labeled "Android" in this document for brevity alone. Before making filesystem choices, you'll want to compare exFAT-drive file times on your own PC and Android devices to see if this defect impacts you. Even if limited to Samsung devices, though, this bug's scope is large.
It's also worth noting that you may be able to use exFAT in some contexts anyhow. It's now supported on most Androids; the first bullet above applies only to content created on Mac OS; this may be a Samsung-only issue; and exFAT-drive files may in some use cases squeak by as incorrect-but-good-enough for Mergeall comparisons because their mistakes are consistent. Moreover, users who will only ever backup on-phone content to a USB drive are exempt from exFAT PC interoperability concerns. Until this bug is fixed, though, it's dire enough to recommend FAT32 as the more reliable choice for your Mergeall-on-Android drives.
If you're willing to take that recommendation on faith, feel free to skip ahead to the next requirement now. For the curious and skeptical, though, the following traces the bug's forensics—beginning at the scene of the crime.
The first symptom of this exFAT bug is easiest to diagnose: in testing on three recent Samsung devices (a J7, an S8+, and multiple Note9s), exFAT file timestamps created on some PC platforms are interpreted differently on Samsung Android than they are on any other platform tested. That is, the same file on the same exFAT drive reports a different modification time on Android alone.
In a comprehensive test conducted in early 2019 in the Pacific time zone:
In other words, this test's results make the flaw appear to be a double bug: the Mac OS PC platform records exFAT timestamps in such a way that triggers an incorrect interpretation on the Android mobile platform. Files created on exFAT drives by Mac OS (only) report incorrect times on Android (only).
No, really. For a more graphic look at the problem, here is a folder of files created on Mac OS High Sierra, as it's seen on Mac OS El Capitan, Windows 10, and Android Oreo ("Today" on Mac OS means 1/15/2019, and 1/16 was still tomorrow when this test was run). As you can see, the file times are wrong by 16 hours on Android alone—files created around 8:20 AM report their times near 12:20 AM the next day.
Conversely, files created by this test on Android report the same and correct times on Mac OS, Windows, and Android. That makes this seem a one-way bug: it impacts just exFAT files created on Mac OS and viewed on Android, and not vice versa. You'd also never notice it if you stick to PCs (Windows times are correct on Mac OS and vice-versa), or limit your scope to just Windows and Android (Windows times are correct on Android and vice-versa). As the next section will explain, though, the latter deduction falls apart in the face of fresh evidence.
But wait, it gets worse. The second symptom of Android exFAT anomaly is subtler and took more digging to unearth: in addition to the prior section's +16-hour Mac OS content skew, Samsung Android devices may also write file modification times incorrectly on exFAT drives. In later tests run on the same Samsung Android devices, files saved to internal storage and FAT32 drives recorded modification times correctly, but some saves to an exFAT USB drive erroneously recorded times that are interpreted as 16 hours behind the actual save times on every platform except Samsung Android itself.
No, really. The facts in this category are bizarre and even appear to be random. The incorrect times saved by Android are always incorrect on both Mac OS and Windows, but not all times saved by Android are incorrect (which is why this effect was uncovered later), and those that are wrong seem to depend on program and/or context (e.g., one program wrote new files' times correctly but existing files' times skewed—sometimes).
This makes this skew difficult to trigger, but easy to spot when it appears. As a brief example, a folder of files on an exFAT USB drive which began like this was updated and extended by assorted programs on Android around 11:30 AM on 2/27/2019 (again, in the Pacific time zone), yielding the following scenes on Android, Windows, and Mac OS; as you'll notice, some of the Android-written timestamps are wrong by -16 hours on the latter two.
That is, it's not just reads, and it's not just Mac OS: the whole conception of time in whatever software does exFAT on Samsung Android disagrees badly with others. This is really just the other side of a single bug's coin, and this guide doesn't have time or space to document it in more detail (and it's really not this document's job to do so in any event). But there's clearly something quite wrong with exFAT on Samsung Android today.
Gory details aside, this is another showstopper for timestamp-based programs like Mergeall: because exFAT files don't process timestamps correctly on Android, such files are generally unusable across any platform mix that includes Android. These files' incorrect times may work on Android itself—Mergeall doesn't care that times are correct, only that they differ when files are changed (i.e., consistently incorrect is good enough). However, the time skews can still cause problems when using exFAT drives in at least three noticeably devious ways:
Just as insidious, there seems no way to compensate for the Android time skews automatically. For example, because exFAT drives may contain content created on multiple platforms, just some of your files, those saved on Mac OS, will report incorrect times on Android, and programs like Mergeall cannot tell which differences are valid and which reflect +16-hour Mac OS skew. The -16-hour write skew will be present on all PC platforms, and its nearly random nature seems even more impossible to work around.
Happily, FAT32 drives both interpret and record times correctly, and are thus immune to this bug. For instance, here's what the same file saved on Mac OS looks like on Mac OS and Windows; on Android, this same file's time is skewed on an exFAT drive, but correct on a FAT32 drive. This means FAT32 provides a path for using Mergeall on Android. Less happily, this precludes mixed FAT32/exFAT usage: because only exFAT drives skew times, their files won't compare correctly with the same files on FAT32 drives (as for others) and spurious differences and recopies will ensue.
In sum, this defect renders exFAT generally unusable on Samsung (and perhaps other) Android devices, and makes it impossible to recommend exFAT for Mergeall on this platform today.
Ultimately, this is a bug in Samsung or Android code that should be fixed. Although file times may have been broken on Android in the past, exFAT files today still report and record incorrect modification times on the Samsung Android platform alone, and neither file sources nor program contexts justify further deferring a repair. Barring a future Samsung or Android fix, though, your safest bet is to use FAT32 for all the removable drives you will sync with Mergeall on Android for both accuracy and interoperability.
This is regrettable, given exFAT's support for larger files and compatibility with other filesystems' times. Per its well-known limitations, FAT32 doesn't support files larger than 4G, and you may need to manually adjust your FAT32 drives' modification times twice a year to keep them in sync with non-FAT drives after daylight-savings-time (DST) rollovers. The internal storage in both your PC and phone, for example, is usually a non-FAT drive whose UTC times clash with FAT's local times. See the FAT times fixer tool covered ahead for a simple way to make the adjustment; if you use FAT32, you'll run this tool on your intermediate USB drive (to match your PC and phone), and possibly on a removable SD card (to match your intermediate drive).
The upside is that, thanks to Microsoft patent constraints, FAT32 is still much more widely usable today than exFAT. More importantly, its timestamps are implemented more consistently and interoperably; for Mergeall on Android, this is a killer asset.
The exFAT epilog:
After this section was written, the exFAT bug it
describes was verified to still exist on Samsung devices running Android Pie (9),
in addition to Nougat and Oreo.
Despite this longevity, the web is woefully silent about this bug—which is
hardly surprising, given that timestamps couldn't be copied on
Android at all until 2017's Oreo, which made the platform unusable
for an entire class of software. It's not impossible
that Linux mount options may be involved (see this
for leads), but it is impossible to use—or even explore—them
on non-rooted phones.
looks at a seemingly unrelated issue, but hints at pathology in Mac OS's
exFAT implementation in general.
For its part, the Mergeall development team has dutifully posted this bug on both Samsung and Android websites, as well as in the Samsung+ app (the US equivalent of Samsung Members at the time), but doesn't presently hold much hope for a fix. The Android website has already dismissed the bug as an OEM-specific "won't fix" (with no apparent interest in forwarding it to the market-leading OEM); there's been no substantive reply from Samsung on either its website or its app in a month and a half; and most support forums in the smartphone domain seem more likely to suggest restarting your phone and counting to 10.
Update: shortly after the preceding paragraph was written, Samsung replied on the Samsung Members app to acknowledge an issue regarding the valid bit for time-zone offset not being set in their exFAT implementation (background details here and here). This makes exFAT files' times correct when created and used only on a phone, but fails to accommodate creation and usage on other platforms, and explains this section's peculiar empirical findings. The reply also promised to try to improve the implementation, but cautioned that a fix may take some time due to dependencies on the current policy. While it's too early to file this under "happy ending," it now seems reasonable to look forward to a future repair; exFAT drives will be much more useful on Samsung devices when those drives are interoperable with PCs.
On Android, Mergeall's Tk-based GUI works marginally with caveats, but its command-line mode works fully and qualifier free. To use this mode, you simply need to use Mergeall's source code, and type command lines to launch it. Although there are multiple ways to run Mergeall's Python source code on Android, the Termux app provides a full-featured Linux experience that seems best suited to the Mergeall task. Neither Termux nor command lines are required for Mergeall on Android, but they're suggested for most users.
In more detail, Mergeall's apps and executables for PCs (Mac OS, Windows, and Linux) do not apply to Android, and its reliance on the Tk library (and Python 3.X's tkinter interface to it) means that using its GUI's source code on this platform entails major tradeoffs. Although there is some new support for using Tk in Python programs run on Android today, it comes with rude advertising, substantial glitches, and extra steps that make it difficult to recommend for Mergeall syncs. See the notebox ahead for more on this story; it's evolving, but at this writing Tk may still be a nonstarter for many Mergeall users on Android.
Luckily, Mergeall is built with a decoupled architecture that allows its main code to be run separately from its GUI. To use Mergeall's command-line mode on your phone, you'll need to:
We'll walk through all these steps in more detail later in this guide. This is obviously more work than installing and opening a standalone app, but it's the easiest workable option for using Mergeall on Android today. On the bright side, this isn't nearly as difficult as it may sound. This package's helper scripts simplify the required command lines; the necessary installs are really just a few minutes of easy up-front work; and Termux provides a complete and general-purpose Linux command-line environment where Mergeall's source code is fully at home. As we'll learn later, a content sync is as simple as entering this at the Termux prompt:
This is roughly just a dozen taps, with the tab completion stressed repeatedly in this guide (and can be even faster if saved to a file). Although Mergeall's GUI comes with major downsides on Android today, its helper-script commands should still be easy enough to appease even the most chronically command-line phobic.
But if you really hate command lines: For an alternative approach, check out Appendix B's coverage of running Mergeall's GUI with the new tkinter support in the Pydroid 3 app's IDE. There, you'll learn more about using Python tkinter GUIs on Android, and sample a leading alternative to Termux for running Python code on this platform. Because this alternative comes with significant tradeoffs—and shares most of the requirements and setup tasks covered here for Termux—it's been relegated to the supplemental-reading section of this guide. Despite its rough edges, the GUI does work in Pydroid 3, and you may wind up preferring it (and this author may too); but this guide's Android topics still apply, and its suggestions are for a user base at large.
In addition to installing Termux, you must run its utility script
termux-setup-storage to create Termux's writeable app-specific
folders, and allow it to update files in internal storage and access removable
drives more broadly.
See this page
for more on the Termux setup command. Without it, Termux—and the
Python scripts like Mergeall that it runs—cannot read or update
most of the folders on your phone's internal storage or removable drives.
This makes content-processing programs like Mergeall unusable in Termux
until you run this command.
To illustrate the Termux permissions story, let's run some Python code within the Termux app. To do this yourself you'd need to install Termux and Python per ahead, but there's no reason to do so yet; this is just a demo. This also assumes some familiarity with pathnames (jump ahead to this section's notebox if you need more background); assumes basic Python knowledge (but you'll probably get the gist of this simple code either way); and uses Termux 0.65 (if your mileage varies in the future, it's probably due to a new Termux, a new Android, or both).
Before the Termux setup command is run, your storage options are severely
limited: on non-rooted phones, you can read and update files in the fully
private Termux app folder
/data/data/com.termux (the one holding
~ home), but not much more.
In Termux's Python, with internal
/sdcard, a removable drive named
/storage/8BB9-1202, and tracebacks trimmed for space here:
>>> open('/data/data/com.termux/xxx.txt', 'w').write('1234') 4 >>> open('/sdcard/xxx.txt', 'w').write('1234') PermissionError: [Errno 13] Permission denied: '/sdcard/xxx.txt' >>> >>> open('/storage/8BB9-1202/xxx.txt', 'w').write('1234') PermissionError: [Errno 13] Permission denied: '/storage/8BB9-1202/xxx.txt' >>> >>> open('/storage/8BB9-1202/Android/data/com.termux/xxx.txt', 'w').write('1234') FileNotFoundError: [Errno 2] No such file or directory: '/storage/8BB9-1202/Android/data/com.termux/xxx.txt'This code shows most writes (i.e., updates) failing, but read accesses at all these locations work the same way at this point. As you can see, there's not much accessible before the Termux setup command is run.
The Termux app-private folder at
/data is the standout exception.
It supports changes out of the box, but is largely useless for Mergeall, because this
folder cannot be accessed outside the Termux app on non-rooted devices. This
includes file explorers, photo viewers, and text editor apps, and disqualifies
this folder for most Mergeall roles. Because you really don't want to store your
content in a location that can be accessed by just one app (and a command-line-only
one at that), Termux's scope is very narrow when first installed.
Technically, at this point Termux can also read and update its nested
Android/data/com.termux app-specific folders like the one in the last
command above, provided you make them manually on either removable drives or internal
storage first. But you would have to do so in another app or on a PC: because
Termux cannot access the containing folders yet, it cannot build the required
folder paths under them. To do more, we have to move on.
After running the Termux setup command, your storage prospects improve radically. You—and hence Python scripts like Mergeall—have update access to all of internal storage, plus Termux's nested app-specific folders on removable drives:
>>> open('/data/data/com.termux/xxx.txt', 'w').write('1234') 4 >>> open('/sdcard/xxx.txt', 'w').write('1234') 4 >>> open('/storage/8BB9-1202/xxx.txt', 'w').write('1234') PermissionError: [Errno 13] Permission denied: '/storage/8BB9-1202/xxx.txt' >>> >>> open('/storage/8BB9-1202/Android/data/com.termux/xxx.txt', 'w').write('1234') 4
Read access for these folders is essentially the same as the write access shown,
except that the failing folder here can be read without error too. That is,
after augmenting its permissions with
and the programs like Mergeall that it runs:
/data), though it's not very useful
/sdcard), though its space may be limited
/storage/xxxx-xxxx), like SD cards and USB drives
As noted, Termux's setup command creates its app-specific folders, like the one used in the last command run above, but you can create these folders manually too (and we will in the setup part of this guide). Running the setup command creates these nested app folders if they don't already exist; this ensures that Termux, and the scripts it runs, have a place to make changes on removable drives.
For a more comprehensive look at accessible paths which we'll skip in this guide, see:
For more on Android pathnames:
By now, you may be feeling a bit confused
by folder pathnames if you haven't done much with command lines in the past. For a
gentler introduction to the subject, see this guide's Appendix A for
extra background that we'll omit here. Like much technical documentation, this
guide has to walk a thin line between alienating newcomers, and boring experts.
Update: though uncertain at this writing, Android Q may impose the same access constraints on internal storage that KitKat imposed on removable media, and invalidate some of this section's coverage in the process. See Appendix C for more on the possible consequences for Mergeall—and every other program that saves shared data on the Android platform.
After you run the Termux permissions command of the prior section, Mergeall is able to both read and change content anywhere in internal storage, and can read content anywhere on an external (i.e., removable) drive. To allow Mergeall to make changes on external drives, though, your content must be nested in the drives' Termux app-specific folders. In Mergeall-speak, this nesting is not necessary if an external drive will only ever be the from copy in synchronizations, but it is required if the drive will ever play the role of to.
In more-concrete terms, the Mergeall content folders that you will only read on your phone can be located anywhere, but those you will change (i.e., update) must reside at either of the following Android locations:
Android/data/com.termuxfolder of your external drives'
You can use the first of these for on-phone content copies if you have enough space, but must use the second for content on SD cards and USB drives that you wish to update. Since this is an arguably bizarre constraint, let's take a closer look at its rules.
First, the simple case: if you're lucky enough to have a phone with enough free space to store your content, internal storage is the easiest approach for on-phone storage, and recommended over SD cards. Internal storage:
There's more on internal storage's speed for Mergeall-specific tasks
Internal storage is also growing larger in recent phones, and is the only option
for on-phone storage if your device doesn't have a removable card.
To store your Mergeall content in internal storage, simply copy it to a
folder like the following on your phone;
/sdcard really means internal
storage, and the
MY-STUFF folder name here can be any name you wish:
This seems simple—and even PC-like—but internal storage isn't enough by itself. For one thing, even if your on-phone content is in internal storage, you'll still need to sync to and/or from an external USB drive in most use cases. For another, if your phone's available internal storage space isn't large enough for your content (and many today are not), you'll need to store your on-phone content on your device's SD card instead. Since both USB drives and SD cards require nested storage to support updates, let's turn to this model next.
Removable storage is generally slower, but it's required if your phone's internal space
is too small, and is still the ultimate in content control.
To store your changeable Mergeall content on external (i.e., removable) storage—including
USB drives and your phone's removable SD card—simply copy it to a
folder nested in the drive's Termux app-specific folder like this;
MY-STUFF means your stuff's folder:
Android uses IDs like this path's
25C9-1405 to uniquely identify
external drives. Your drives' IDs will naturally vary too, and are displayed in popular
Android file explorers (including
More unusual here: on both SD cards and USB drives, content folders like this path's
MY-STUFF must be nested six-levels deep in the drives' Termux app-specific
folder as shown, if you wish to update them with Mergeall on your phone.
This differs from internal storage, where anything under the
/sdcard root works.
Termux's explanation for why this is so is told on pages here and here. This restriction on updating removable drives came online in Android KitKat—to the dismay of both users and developers—and has seen multiple later mutations, including app-initiated run-time requests for storage permissions. Termux, and all other Python apps tested, however, remain limited to their own app-specific folders for updates to SD cards and USB drives.
This may seem an odd requirement given that internal storage is wide open to changes, but it cannot be lifted in the context of Termux command lines today, and isn't as grievous in practice as it may sound. For instance, you can still access your content normally in a file explorer (as captured in this SD-card screen shot), and a home-screen shortcut (like this one) provides quick access to your nested content folder; see your file explorer to set one up.
Keep in mind that no nesting is required for content stored in your
phone's internal storage (i.e., in
/sdcard), because it's freely writeable.
Moreover, you can store content at the root of USB drives and SD cards
too, but only if you'll only ever read—and never change—it with Mergeall.
Unless you're sure that a removable drive's copy won't be the subject of a future
Mergeall update, it should be nested in the drive's Termux folder.
Also keep in mind that Android's permission rules have varied much and often in the past, and seem likely to do so again in the future. While some media may grow more accessible and some less, it's impossible to foresee tradeoffs in a platform that seems to change the rules every other Tuesday (more on this in the following notebox).
More on the (in)accessibility of removable-drive content:
This section noted that internal storage is recommended for storing content on your
phone because it is more accessible to other apps, but this merits a callout here.
Due to the permission rules we met earlier, some apps that
do not request broader access, including Termux, are unable to update content on
removable media except in their own app-specific folders. This is why you must
oddly nest your updateable content on removable media, including SD cards
and USB drives, to manage it with Mergeall in this app.
Unfortunately, this can also make it impossible to update such content in some other apps. As we'll see in Appendix B, the alternative Pydroid 3 GUI app has the same access limits—on removable media, it can update only in its own app-specific folder. For such apps, this is a bit of a catch-22: you must nest content on removable media to update it, but this cuts it off from other apps' updates. More tangibly, Pydroid 3 cannot update any content nested in Termux's app-specific folder on removable media—and vice versa. The net effect is that your choice of updating app is binding, unless you move your content.
Luckily, this caveat's scope is limited: it impacts only updates on removable drives, and does not apply to all apps. You can still freely view nested content on these drives, and modify it in general-purpose apps like file explorers and text editors that request broader permissions than Termux and Pydroid 3. If it's important to also update content in apps with narrower permissions, though, internal storage is your best option. Especially with the next section's peril factored in, the strange access limitations of removable drives on Android makes them less well-suited for single-folder content storage. Watch for a rehash of this topic when we consider Pydroid 3 at the end of this guide; it becomes crucial for alternative apps.
Update: though uncertain at this writing, Android Q may impose the same access constraints on internal storage that KitKat imposed on removable media, and invalidate some of this section's coverage in the process. See Appendix C for more on the possible consequences for Mergeall—and every other program that saves shared data on the Android platform.
Finally, a bold-font caution: per normal Android operation (described by Android docs here and here), uninstalling Termux also silently deletes its app-specific folders on all your drives—including any content folders you've nested there. Unnested content copies, and unnested working folders in internal storage are immune to this, but those nested on your SD card and USB drives to support Mergeall updates may perish with Termux. Because this occurs without clear warning, it's substantially worse than the prior section's caveat on removable-media accessibility, and further bolsters the case for internal storage.
Specifically: on all devices tested, the
folders on both the removable SD card (at
and internal storage (at
/sdcard) were automatically deleted when
Termux was uninstalled, along with any and all content placed there by a user.
On one device, a file explorer also attempted to intervene and delete this
content too, but the system apparently beat it to the punch. There is
no reason to nest your content in an app-specific folder on internal storage—and
a root-level folder on internal storage avoids this issue altogether—but
there is no other option for updateable content on removable drives in Termux
(or other Python apps we'll meet in Appendix B).
This automatic deletion may sound scary, but it's designed to clear associated space when
you're done using an app, and really isn't much more dangerous than file-explorer deletions
if expected (and an
rm command in the Termux shell can be deadlier
still). Before uninstalling Termux, though, be careful to move your nested
content if you wish to retain it. In Android, uninstalls imply deletions.
And that's a wrap on the requirements front. In exchange for all this, Mergeall provides a free and proven incremental backup and changes propagation tool for the content you store on Android devices. Though not for everyone, this Mergeall approach requires neither device rooting nor card extraction—both of which can be perilous, if not impossible—and brings robust cloud-free storage management to your phone. If those benefits offset this section's requirements for your usage—or you just enjoy a technical challenge in general—read on for more on using this package's helper scripts to simplify Mergeall runs on Android.
Note from a possible future: If a standalone Mergeall app ever appears, it may obviate the need for both Termux command lines and content nesting in Termux's app-specific folders, but would still incur Android version and filesystem constraints. In particular, barring an exFAT fix, drives would still be best formatted as FAT32 for interoperability. Given the sloth with which bugs are fixed in commercial products like Samsung Android, using FAT32 today is likely not a short-term measure. If you're already using FAT32 drives, the only impact of an app upgrade would be an optional move of content to unnested drive roots.
Now that you've seen the what's required to use Mergeall on Android in general, this section shifts focus to the how. It provides detailed coverage of steps to follow to prepare your phone and drives for Mergeall use. Here's a quick rundown of these up-front tasks:
To be able to run Mergeall's Python scripts, you first need a command-line environment in which to run them. The Termux app provides a featured-rich Linux shell for Android that fits the bill; see its homepage for background. To get started, install the Termux app on your phone from the Google Play Store. Mergeall's command-line mode "just works" in Termux, because Android is really Linux under the hood.
Using Termux extra keys (or not):
Shortly after this guide was penned, Termux changed its extra-keys row
to be configurable, but in the process also removed keys crucial to
command-line editing (e.g., arrows).
To restore the former keys shown in this guide's Termux screenshots, visit
this page, and
scroll to its "Extra Keys Row" section. In brief: in Termux itself,
run a Unix
mkdir command to create the new folder
use a Unix editor like
nano to make a
termux.properties in the new folder and paste
in the line shown on that page; and restart Termux.
Option: some readers may prefer to disable the Termux row entirely and install an alternative on-screen keyboard that has all the important bits, like Hacker's Keyboard. Bonus: such a keyboard may also come in handy for tkinter GUIs run on Android (stay tuned for the details in Appendix B).
To allow Termux to update both internal storage and its app-specific folders
on removable drives, launch the Termux app you just installed on your phone,
and run the following command at its prompt (the
$ in command listings
here is Termux's default prompt; type the text that follows it, and press whatever
means "enter" on your keyboard to submit the command):
After that, allow access in the
popup that appears.
Note that you may need to rerun this command later with removable drives
connected, if you get permission errors. This command also creates the
/Android/data/com.termux nested folders on all connected
media automatically if they do not exist, but you can also make these manually
when you're preparing your drives on a PC (as we will a few steps ahead).
If you've forgotten why this permissions step is required (or came here from a search), see the earlier coverage. Also note that Termux commands that update your phone's internal storage from here on in this guide assume that this command has been run; without it, such commands will fail on permission errors.
Mergeall is coded in the Python programming language. As mentioned earlier, we need to use its source code on Android because its PC executables don't apply, and source code requires a Python interpreter on any platform. To install Python for use in Termux, simply type the following command at the Termux app's prompt on your phone, and answer "y" (yes) when prompted:
$ pkg install python
This gets the latest Python 3.X (version 3.7 at this writing), which works
best for Mergeall. If you want to verify the install, type just
python at the Termux prompt:
$ python Python 3.7.1 (default, Oct 21 2018, 18:20:26) [Clang 7.0.2 (https://android.googlesource.com/toolchain/clang 003100370607242d on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys, os, platform >>> sys.platform, os.name, platform.machine() ('linux', 'posix', 'aarch64')
As usual, a
Ctrl+d key combination at Python's
prompt gets you back to Termux's
Now that you've got Termux and Python ready to go, it's time to setup a working folder on your phone where you'll run scripts to sync content. This is a simple task, but you'll want to decide up front where you wish to work, to avoid having to repeat steps later.
First, the recommendation: an
/sdcard/work folder located in the
root of your internal-storage space is an ideal candidate for your working folder,
because it's fast, accessible to file explorers and most other apps, changeable
by programs run in Termux, and short to type. This folder has the following multiple
aliases on devices used in testing, but shorter is generally better on a smartphone:
You can work in any writeable folder on your phone, but others come with tradeoffs that make them less than ideal in this role. For example:
~(which is really
/data/data/com.termux) is writable, but fully app private: its lack of visibility in file explorers and other apps on non-rooted phones can make maintenance tasks like Bluetooth transfers harder.
/sdcard) might seem a natural too, but it's more to type, and will be automatically deleted if Termux is uninstalled (a broad issue discussed earlier).
For this guide, we'll assume and use the recommended working folder in internal
storage (and use its shorter
/sdcard name), because this is the
simplest and most-functional location. To create your folder, run the
following in Termux on your phone:
$ mkdir /sdcard/work
You can also create this folder in your favorite Android file explorer—one of the benefits of locating it in internal storage this way. Once you've got your folder set to go, you're ready to populate it with software, per the next step on our list.
Wherever you choose to work, unzip (i.e., extract) both this helper-scripts package and Mergeall's source code there for convenience. The helper scripts simply augment Mergeall, so you'll need both packages. You can use the standard Mergeall source-code package, because its core syncing code developed for Mac OS, Windows, and Linux works unchanged on Android—a testament to the benefits of portable software (though the story for Mergeall's GUI in Appendix B won't be nearly as seamless).
For readers newer to Unix systems, the following is one way to install this helper-scripts package on your phone by running commands in Termux itself. First, in Termux, go to the working folder you created in the prior step:
$ cd /sdcard/workNext, fetch (i.e., download) this helper-script package and unzip it to your new working folder; click the links here on your phone to perform the fetch part of this recipe, then type the commands in Termux:
Fetch the zipfile at learning-python.com/mergeall-android-scripts.zip $ mv ../Download/mergeall-android-scripts.zip . $ unzip -d . mergeall-android-scripts.zip
Finally, do similar to install Mergeall's source-code package in the same working folder:
Fetch the zipfile at learning-python.com/mergeall-products/Mergeall-source.zip $ mv ../Download/Mergeall-source.zip . $ unzip -d . Mergeall-source.zip
For an example of the resulting working folder, see this
This assumes that clicking the links above on your phone makes the zipfiles
show up in
/sdcard/Download, but that's normal Android public-folder behavior.
Two tips here. First, if you're new to Unix, be sure to use Termux's arrow keys to recall prior commands to edit and resubmit, and its tab completion to let Termux fill in most of these commands for you. The latter makes command lines much easier to type; for instance, a tab after "m" in the sessions above likely expands it to "mergeall-android-scripts.zip," one after "M" fills in the Mergeall full zipfile's name, and "Dow"+tab expands to "Download" similarly. Don't type what you don't have to type—especially on a phone.
Second, there are other ways to achieve most of the above that we'll skip here.
For instance, you can fetch files on a PC and beam them to your phone; can use
Android file explorers instead of command lines for most steps in this process
(the former often call an unzip an extract, and refer to
. as the
current path); and can even download the zipfiles directly in Termux using
curl after a
pkg install curl:
$ cd /sdcard/work $ curl -O https://learning-python.com/mergeall-products/Mergeall-source.zip $ unzip -d . Mergeall-source.zip
Termux has the
wget download command too.
Regardless of their sizes, Linux-based devices offer an embarrassment of options.
Depending on the size of your content, the next step may take the longest. To use your content on Android, its storage structure must satisfy the media's filesystem and location requirements we met earlier here and here. In brief:
Since this is probably the most convoluted of the steps in this list (and is also required of the GUI alternative in Appendix B), the following sections provide a more detailed breakdown by media type.
Use your PC to setup the removable drives you'll use on Android.
First, you'll need to meet the FAT32 requirement. If your drive is already using
FAT32, you can move on to the next paragraph. Otherwise, start by copying any existing
Android content on your drive to a temporary location so it can be restored later;
this includes existing
DCIM (camera) and
Android (app-data) folders,
if present. Then, format your drive as FAT32, and restore the prior existing content you
saved. If you're new to formatting, see the notebox ahead for tips.
Next, place your content-copy folder on your drive. As noted earlier, if you're sure you will never sync content to a removable drive on Android, this folder can be stored anywhere on the drive, including its root; copy or move it there using your usual tools. To allow Mergeall to update the drive's content, though, its content-copy folder must be placed in Termux's nested app-specific folder, as follows:
/Android/data/com.termuxfolder now if it does not exist, and copy your content folder there using whatever copy technique you normally use on your PC (e.g., file explorer or command line).
/Android/data/com.termuxfolder now if needed, and either move your content there if it's already located elsewhere on the drive, or copy it there anew.
For the last bullet above, you can move your content folder to the Termux app folder on the same drive with a single command on Unix-like PCs; here's an example on Mac OS (on Linux, use your media's paths; on Windows use drive letters instead of volume names, and move with DOS commands):
~$ mv /Volumes/EXT256-1/MY-STUFF /Volumes/EXT256-1/Android/data/com.termux/MY-STUFF
A caution here: you can move existing content in a file explorer too, but be sure a move in yours is a true move, and not a copy-and-delete. The speed difference can be staggering for large folders, and some Android file explorers may miss the mark on this.
Whether your content was copied or moved, it's now located in the Termux app folder—and four levels down from the drive's root—which can be inconvenient when coding paths and browsing for files. To make this easier to access on your PC, add a link (or shortcut) to it at your drive's root; here's how to do so on Mac OS with a symlink (that looks like this):
~$ ln -s /Volumes/EXT256-1/Android/data/com.termux/MY-STUFF /Volumes/EXT256-1/MY-STUFFThe link (or shortcut) is optional, but can save a lot of navigation time in file explorers, and typing in command lines. The root-level link probably won't work on Android, and you can't create one for removable drives on non-rooted phones because you don't have the required permissions on removable drives (this is the whole reason for nesting content in the first place); but it can save steps on PCs.
On Android, instead make a home-screen shortcut (that looks like
to your content folder; a longpress
in your file explorer is the usual first step. This makes more sense
for a resident SD card than a transient USB drive; but because your content is
now six levels deep when viewed on Android—nested in the
Android/data/com.termux folder of your drives'
/storage/xxxx-xxxx root—the shortcut can save a
whopping five taps per access, plus a file-explorer open.
If you're able (or required) to store your on-phone content in internal storage, you don't need to reformat as FAT32 or nest your content in the Termux app-specific folder. In fact, you shouldn't—internal storage is already formatted, and nested Termux app folders disappear on Termux uninstalls as noted earlier. Instead, simply create a folder for your content at the root of internal storage, and copy your content there. Both these steps must be done on your phone itself.
To begin, create your content folder in an Android file explorer, or use a Unix
mkdir command in Termux itself. For example, the following creates a
content folder named
MY-STUFF using a command in Termux; as usual,
your folder name may vary:
$ mkdir /sdcard/MY-STUFF
Then, copy your content to your phone. For most users, this will employ an intermediary USB drive: copy from PC to USB drive, then from USB drive to phone. A content copy on a USB drive you've already setup for Android in the prior section will suffice, and avoids another step.
A caution here too: on the phone, you can run the copy to internal storage in most Android file explorers, but some fail to copy over file modification times. At least one popular app, for instance, simply stamps every copied file with the current time, which is a showstopper for Mergeall—and no better than the pre-Oreo Android bug (in fact, this may be a lingering accommodation of it).
To make sure times make the move too, use Mergeall's
utility on your phone instead. There's more on this utility
in short, the copy on Termux looks like the following, assuming the settings we will
create in the next and final step are in place. That is,
you'll need to return here and run this after setting drive IDs in your bash profile file
(USB drive IDs are not available until that drive is prepped and attached,
and there's no way to show you this command without them):
$ cd $work $ py Mergeall-source/cpall.py $plug $data -skipcruft -vv -u > Admin-Mergeall/importlog.txt
Here too, once the content copy finishes, you may wish to create a home-screen
shortcut to its new location
on your phone. In this mode it's only two levels down in internal storage when
viewed on Android—at
/sdcard/MY-SYUFF—but the shortcut
still saves one tap per access, plus a file-explorer open.
A symlink doesn't make sense here, because it would be two levels down too (and
may be impossible in any event: a
ln -s at Termux's prompt didn't
/sdcard, despite having write permissions there).
A reminder: even if you use internal storage for on-phone content, you'll also need to setup a USB drive to propagate changes between your PC and phone (or save on-phone changes in general). That removable drive will still require FAT32 formatting for interoperability and content nesting for updates as described earlier (here and here), and can be used to initially populate your internal storage with content. Your on-phone content, though, is substantially easier to setup and use.
Tips on formatting your drives:
As noted, you'll generally need to reformat your
removable drives unless they already use FAT32, and this isn't a trivial task if you're
new to it. On Windows, the standard file Explorer won't format
larger drives as FAT32 (it forces exFAT instead), but a DOS
command and third-party options will; for pointers, try
this search and
On Mac OS, the Disc Utility app's Erase happily formats larger drives as FAT32 (if you select this and use uppercase drive names), and Linux users have many options. Android itself can format drives too (see your drive's Format in Settings ⇨ Device maintenance/care ⇨ Storage ⇨ Storage settings); unfortunately, it seems to silently choose FAT32 or exFAT for you based on drive size (a case of user-friendliness gone bad) so you'll have to use a PC for larger drives.
On all platforms, the usual warning about formatting applies: it erases your drive, so copy off anything valuable first. You may also want to add a
.nomedia file at the top of reformatted
removable drives that will be duplicates of the content kept on your phone
(e.g., on a go-between drive that will be used to propagate changes between your
phone and PC); where it works, this "hidden" file prevents Android's media scanner
from searching the redundant copies.
Last but not least, you'll now install and edit the bash profile file shipped in this package. This must happen last, because you may not know the Android names of your drives until they've been prepared and attached to your phone. Formatting, if run, changes the IDs Android uses to identify drives, and you'll need these IDs in this step.
Along with its scripts (and this guide), this helper-scripts package includes a
prototype bash profile file named
your--.bash_profile. This file has
simple settings that provide quick access to your Termux home and working folders,
as well as your content-copy directories on your phone and removable drives.
You can view this file's contents here;
it's mostly documentation (the lines that start with a
To use this file, delete the "your--" in its name; edit its path settings as both its
comments and the next section describe; and either save this file in your Termux home
folder or copy-paste its code into a
~/.bash_profile you already use.
Because your Termux home folder is app private, you will normally need to install the
profile file there using commands in Termux itself (not a file explorer). Here's the
required Unix incantation; this assumes you're
/sdcard/work and have already unzipped this package there:
$ cd ~ $ cp /sdcard/work/mergeall-android-scripts/your--.bash_profile . $ mv your--.bash_profile .bash_profile
For example usage of this file's settings at the Termux prompt, see this Android screen capture (and zoom in and zoom often). The bash profile file is also integral to using the helper scripts here: per the next and final setup section, it defines Android paths in a single location shared by all.
To make both commands and edits easier on smartphones, the helper scripts in this package use common content-copy path variables set in your bash profile file. This way, paths need be changed in that file only, and only when you first install this package or wish to use a new drive.
When you do need to change paths, the required edits are simple: in the
~/.bash_profile you created from the
prototype file, the settings to change are marked
as "EDIT ME." Replace these precoded paths' "xxxx-xxxx" IDs and literal
"MY-STUFF" with your own drive IDs and content-folder name, respectively.
Again, your drive IDs can be found in Android file explorers like
(make sure to match the case of drive ID letters exactly),
and folder name is whatever you wish it to be.
Here's one way to edit the profile file in Termux—because it's not
visible to other apps, you'll generally need to edit it in Termux itself too:
$ cd ~ $ vi .bash_profile $ source .bash_profile /sdcard/workThis runs
sourceto load your new settings after you're done, and uses the Unix
vitext editor. If
viis too cryptic for your tastes, the more user-friendly and WYSIWYG
nanotext editor also works in Termux after a
pkg install nano. You can also rename and edit the bash profile file on your PC and beam it over, but you'll still need to copy it into your home folder in Termux, because that folder is visible nowhere else.
Special cases: if you opt to use internal storage for your on-phone content copy
instead of SD card (per the requirements section earlier),
use the profile file's
/sdcard/MY-STUFF data-path setting instead.
You may need to also edit the scripts themselves to configure for your
Mergeall source-code package's install location if it's not in
(the current working directory), but this is not required if Mergeall's package
is unzipped to the same folder as this scripts package.
Because removable drives are identified by drive IDs in Android, the paths
used by helper scripts will change if a new removable drive is inserted. To reconfigure
for a new drive, simply update paths in your
~/.bash_profile with the editing
commands above; all Android helper scripts automatically
source (i.e., reload)
this file on startup, so they always use its current settings.
You may also need to rerun Termux's
termux-setup-storage when changing
to a new drive; if you get permission errors, this applies to you.
Helper-scripts coding notes:
The symlinks made at drive roots by the Mac OS move-and-link
script here might suffice for pathnames in other
scripts, but these links won't generally work on Android, and non-rooted phones don't have
permission to make such links in removable drives on Android itself.
~/storage/external-* links also record drive
paths, but they seem too ambiguous to use: they give IDs but not drive type, and are
not always current.
Instead, the helper scripts rely on the manual
settings in your bash profile file, both for generality, and because explicit is
usually better than implicit—especially when your content is at stake.
diffall.pybyte-for-byte comparison utility
Additional scripts run modification time (a.k.a. modtime) adjustment tools for FAT32
drives, and other Mergeall utilities are available as standard Unix command lines.
As a bonus, all the Android helper scripts here automatically save logfile output to
your working folder's
Admin-Mergeall subfolder, using filenames either
passed or generated; to see how this works, let's jump right into the available commands.
To sync or compare your content: select the Mergeall helper script
in this package whose name matches your goal; open the Termux app on your phone;
and run the selected script with a
bash scriptname.sh command
line at the Termux prompt. For example, the following launches a USB-to-phone
updates run, assuming this package has been unzipped in your working folder:
$ bash mergeall-android-scripts/mergeall-plug-to-data-update.sh
bash command just tells Termux to run all the precoded commands
in the chosen file, and scripts' filenames imply their action. For instance,
plug-to-data in this script's name syncs changes from USB drive to on-phone storage,
update means apply changes to the destination automatically;
data-to-plug syncs the other way, and
report means report
differences only without changing anything.
Worth reiterating: be sure to use shell tab completion to fill in the folder and script
names in these commands; in this case "bash m+tab/m+tab" is most of the command.
Optional: scripts located in your Termux home folder (
also be made more directly executable with a
chmod +x scriptname.sh
and run with just a
scriptname.sh; in Termux this works for
home-folder scripts only, not for working folders located
Mergeall's scripts—and hence this package's bash helper scripts that run them—both
process files and generate logfiles that detail their actions and results. On Android, script
logfile output is automatically saved in your working folder, to subfolder
$work/Android-Mergeall, in a file whose name is either:
bash scriptname.sh filename
Whether the logfile name is passed or generated, the logfiles subfolder is created automatically if needed.
You can also launch scripts with
& at the end of the bash command
bg after it starts) to run in the background,
so you can monitor output with a
tail -f logfile.txt. For instance,
the following session ensures that Termux is in your working directory by running the
gowork command defined in
~/.bash_profile; starts a USB-to-phone
update run, saving its output using a generated filename; and monitors script progress:
$ gowork $ bash mergeall-android-scripts/mergeall-plug-to-data-update.sh & $ tail -f Admin-Mergeall/mergeall-somedate-sometime.txt output scrolls here Ctrl+c to exit
Alternatively, the following reports file differences only, saves output to
an explicitly named file, and views output only after script completion
vi text editor (
nano also works after
an install as noted earlier, you can switch to another Termux session while
the merge runs, and you don't really need to
gowork again unless
you've gone elsewhere):
$ gowork $ bash mergeall-android-scripts/mergeall-plug-to-data-report.sh ma.txt $ vi Admin-Mergeall/ma.txt
Besides its main merge script, Mergeall includes the
deep-comparison utility, and the helper scripts include a script to launch it.
The following, for example, compares content copies byte for byte on Android,
monitoring output as it appears in a passed-in filename; there is no
from and to ordering for drives here, because this isn't a merge:
$ gowork $ bash mergeall-android-scripts/diffall-plug-and-data.sh diffs.txt & $ tail -f Admin-Mergeall/diffs.txt output scrolls here Ctrl+c to exit
As a guideline, you should run this comparison script occasionally to ensure that all files in your content copies are completely identical. Though Mergeall's time-and-size comparisons are fast and adequate for your day-to-day syncs, drives can and do go bad eventually. A byte-for-byte check proves the integrity of your content—albeit much more slowly (read the notebox ahead for details).
For a more graphic look at the Mergeall sync and diff helper scripts in action
on Android, check out the Mergeall-run screenshots
and the exemplary but heavily redacted
And for a comprehensive test that verifies the approaches laid out in this document,
study the files in this folder,
especially its test notes.
Though most phones would lose a straight-up footrace
with a PC, on Android, Mergeall itself is relatively fast because it compares just
file timestamps and sizes by design. The more exhaustive
utility, though, can run much longer for large content collections—which is
one reason for using incremental sync tools like Mergeall (though even-slower
full copies make it a slam dunk).
As an example, on-phone comparisons of a 150G content collection having 100K files and 10K folders on microSD and USB flashdrive work well, but take 2 hours in
diffall.py versus 2 minutes or less in Mergeall.
For long-running tasks like the former, be sure to acquire the Termux partial
wake lock in
to keep your program running even after your screen
Remember to release it when your task finishes to avoid battery drain.
And for true sports fans: the Mergeall comparison for the same 150G content folder on the same drives clocks in at a quicker 24 and 54 seconds on Mac OS El Capitan and Windows 7 machines, respectively—though the variables are many, and 2-minute on-phone merges still beat microSD-ectomies.
Update: results for internal storage are now in. On the same phone, and using the same USB flashdrive and 150G content collection, Mergeall and
diffall.py comparisons to internal storage run in just 19..43 seconds
and 29 minutes, respectively. This is 3 to 4 times faster
than microSD's results of 66..121 seconds and 2 hours on these tasks.
cpall.py script copies the full 150G from USB to
in just 21 minutes—slightly faster than a
and perhaps indicative of a USB port and/or drive that's a bigger bottleneck than
internal-storage write speed. Either way, internal storage takes the gold in the
on-phone content sprint.
Additional helper scripts in this package run Mergeall's
to adjust file modification times on FAT32 drives so they are in sync
with non-FAT drives after daylight-savings-time (DST) changes.
Because FAT32 is recommended for broadest interoperability on Android
today, you may need to run this utility twice a year,
especially if you propagate changes to or from a PC or your phone's
Without getting into all the details here, on DST rollovers, FAT32's local times may all be one hour off from the UTC-based times used on non-FAT drives. To diagnose this state, run Mergeall in report-only mode (its reams of messages will show every file differing by modtime), and inspect the times of the same file on both drives (they'll differ by one hour). Less than ideal, perhaps, but cross-platform content management is never dull.
Luckily, this state is also easily remedied. To make FAT32 times match other filesystems,
simply run the helper script as follows, with
data for on-phone
plug for USB drive; an
followed by a number-of-hours value; and an optional explicit logfile name:
$ gowork $ bash mergeall-android-scripts/fix-fat-dst-modtimes-data.sh -add 1 logfile.txt $ vi Admin-Mergeall/logfile.txt
In general, you'll want to run this script after DST rollovers on any FAT32 drives whose times must compare correctly to non-FAT32 drives. Although the possible combinations of filesystems and devices are many, most use cases fall into one of the following categories:
Adjust as needed for your usage, and see Mergeall's
itself for more details.
You can also run the bare fixer script directly, either on your PC (to adjust times on an
attached USB drive or a removed SD card), or on your phone (to adjust times on an attached USB
drive or a resident SD card); its helper
scripts like the one shown in the example above
simply make on-phone runs easier. For instance, a direct run may look like this (try
Mergeall-source for sourcepath in your phone's
\ on Windows):
$ python sourcepath/fix-fat-dst-modtimes.py folderpath -sub 1 > logfile
On caution here: you should generally restart your phone before syncing content to it after a DST rollover. Otherwise, Android may return file times saved earlier in a cache that are now invalid, and will not compare correctly to other files' times. This is arguably another Android bug; without a restart, its net result may be spurious differences or equivalences, either one of which can derail Mergeall syncs.
For example, after adjusting times on a FAT32 USB drive up by an hour at a recent DST rollover (March 2019 in the US), Android initially and erroneously reported all the drive's times as differing from those on UTC-based internal storage; a phone restart sufficed to make the drive's times agree with internal storage as expected. Restarts may or may not be required for on-phone content on removable drives: at the same DST rollover, FAT32 USB agreed with FAT32 SD and exFAT USB agreed with exFAT SD—without restarts. To be safe, though, restart to prevent Android's error-prone caching from causing problems.
Mergeall's source-code package also includes the
script, and its
test subfolder includes the ziptools package which
comes with zipfile create/extract scripts. If you elect to host your content on your phone's
internal storage instead of a removable card, for example, you might use either of
these to copy your content folder in full from a USB drive initially, or export
it to USB later.
Though not wrapped by helper scripts here, both of these tools can be run in your
$work folder with normal Unix command lines in Termux, using either
your bash-profile settings or custom paths. For instance, both of the following
command pairs import and export content copies on your phone ("py" is assumed to
be aliased to "python" in your bash profile file here, and routing the logfile is
up to you in this mode):
$ py Mergeall-source/cpall.py $plug $data -skipcruft -vv -u > Admin-Mergeall/importlog.txt & $ py Mergeall-source/cpall.py $data $plug -skipcruft -vv -u > Admin-Mergeall/exportlog.txt & $ py Mergeall-source/test/ziptools/zip-extract.py $plug/../import.zip $data $ py Mergeall-source/test/ziptools/zip-create.py $plug/../export.zip $data/* -skipcruft
As usual, tab completion helps with input here, and bash scripts, variables, or aliases
can shorten such commands.
For more on using
cpall.py, see the
for more on ziptools, see its homepage.
Termux itself comes with a Unix
unzip and a
pkg install zip adds a Unix
zip; both generally work
well, but are naturally subject to the Unix tools' constraints (e.g., file-size
limits may apply).
If you prefer a more interactive experience, Mergeall's console mode works on Android too. This mode asks you for input parameters, instead of expecting them in the command line. It doesn't have tab completion for paths, though, so it's a bit more typing or pasting, and may work better with a Bluetooth keyboard. See the screenshots of this mode's setup and results, and start it in Termux with a command like this:
$ py Mergeall-source/launch-mergeall-Console.py
Finally, you can also skip the helper scripts and bash profile altogether and run Mergeall's scripts directly, as described in its user guide. The following Unix command line, for instance, kicks off a Mergeall USB-to-SD updates run on Android manually:
$ python Mergeall-source/mergeall.py \ /storage/7284-2735/Android/data/com.termux/MY-STUFF \ /storage/25C9-1405/Android/data/com.termux/MY-STUFF \ -auto -backup -quiet -skipcruft \ > ../Admin-Mergeall/mergeall-$(date +%Y%m%d-%H%M%S).txt &
And if you can imagine typing this all on a smartphone's touchscreen, you'll probably appreciate why the helper scripts may be a better idea.
This section collects assorted usage pointers and reminders. Most are intended for users newer to Termux, Unix, or both.
emacsin Termux, or text-editor apps
nanois user friendly: run
pkg install nanoto get, and tap
Alt+xfor more space
pythoninstall is Python 3.7 today; install
python2for 2.X code (info)
pkg install Xinstalls Termux tools;
pip install Xinstalls Python tools in Termux
pip install Pillow, after running this command (but without its "-y")
In closing, if this Mergeall approach seems like a lot of work, it's because it is, but it's also a price inherent in maintaining content on Android smartphones with Termux command lines today. In exchange, this scheme provides a fully functional incremental-backup and changes-propagation tool that you can run directly on your Android device to synchronize with external drives—no device "rooting" or card juggling required.
That being said, some users of both Mergeall and Android may prefer to remove cards instead of using command lines, and this may be the only option for others. Still, any scheme that can free your content from the constraints of both closed devices and cloud storage is worth a look. Not only are storage devices much quicker than clouds, they can automatically neutralize the proprietary and even exploitive agendas that seem to underlie much of today's smartphone experience.
However you choose to proceed, taking control of your digital property is worth the effort.
This appendix introduces and summarizes the folders used in this guide. It's probably more useful to readers who haven't used command lines much in the past, but also serves as a glossary of sorts.
Because the locations at which you'll store your content on Android are central to the approach presented here, it's important to get a handle on these up front. Android identifies a storage location just like every other platform, with a pathname—a string of folder names that reflects folder nesting. Some of Android's naming conventions are unique, though, and some pathnames identify folders with specific roles. Here's a quick rundown on those that play a part in the Mergeall story:
This folder is really located in internal storage, but is too hidden
to be accessed as such. Being Linux-based, Android uses the
character to separate folders in a pathname like this; the one all the way to
the left means the device root (the top of the folder-nesting tree),
and the left-to-right order of
/-separated names gives their nesting.
This is where you normally "are" when you first open Termux (before
you run a
cd to go to a different folder).
It also goes by the names
which are just shell syntax that automatically equate to this folder's
full pathname above.
Because you can't see this folder outside the Termux app, its scope
is limited. Because of the way Termux works, though, this is where you'll
need to install and edit the bash profile file covered in this guide.
/sdcard /storage/emulated/0And yes,
/sdcardreally means internal storage, not your removable SD card, for historical—and even tortuous—reasons we'll omit here. Because internal storage is accessible to every app, it's ideal for content storage if your phone has enough space. It's also generally more convenient than your "home" folder for working in Termux, because you're not limited to Termux tools. If you store your content and working folders in internal storage, their pathnames would look like the following (your stuff's
MY-STUFFmay naturally vary):
Update: though uncertain at this writing, Android Q may impose the same update constraints on internal storage that KitKat imposed on removable media (described ahead), making it less flexible than implied here. See Appendix C for more details.
/storageroot on Android when they are attached, with a folder name derived from the drive's unique ID. For example, here's one for a USB drive with an ID of
/storage/25C9-1405Your drives' IDs will differ (they're supposed to be unique, after all), and you can find them in some Android file explorer apps after drives have been attached (and the system has a chance to mount it in the folder-nesting tree); see earlier for app pointers. A physical drive's Android ID will change if it's reformatted, but should generally remain constant otherwise.
Every app has one of these folders if it needs one; they also appear in
internal storage, and are meant for app-specific content that might be too
large or inconvenient to store in the
/data app-private folder.
As discussed in this guide, these app-specific folders are key to using
content on removable drives in Mergeall—whether from Termux command
lines, or the GUI of the next appendix.
App-specific folders like these are also automatically and implicitly deleted when the owning app is uninstalled. This is meant to clean up an app's resources, but makes these folders subpar for storing data. Because of that, you'll generally locate your working folder elsewhere. Your Mergeall content would ideally be elsewhere too, but updatable copies on limited-space phones may have no other option than removable storage, and this requires app-specific folders today.
Keep in mind that all of the above reflects Android's conventions. If you use the same removable drive across different platforms, the names of its folders will differ on each. The following, for example, is the pathname for the same folder on Android, Mac OS, Linux, and Windows; all reference the same folder, but you'll need to extrapolate from the Android flavor shown in this guide when using a drive elsewhere:
Android: /storage/25C9-1405/Android/data/com.termux/MY-STUFF Mac OS: /Volumes/EXT256-1/Android/data/com.termux/MY-STUFF Linux: /media/username/EXT256-1/Android/data/com.termux/MY-STUFF Windows: D:\Android\data\com.termux\MY-STUFF
Also keep in mind that the pathname forms above and used in this guide
may vary on some Android devices, and have varied over Android's history.
worked in the past too. The paths used here reflect Samsung Galaxy
devices using Android Pie, Oreo, and Nougat, so you'll have to translate if
those on your phone differ. Alas, Mergeall's budget is limited, and
stability is not among Android's core virtues.
For more details on Android pathnames, see books, the web, or other Android resources.
The main part of this guide recommends and covers running Mergeall by using command lines in the Termux app, because this approach is both straightforward and reliable, and is completely advertising and payment free. Termux provides a free, full-featured. and general-purpose Linux environment, which can run Mergeall's command-line mode with minimal setup tasks and robust performance. This is not, however, the only Python-on-Android game in town.
Most notably, the Pydroid 3 app has recently acquired some rudimentary though impressive support for using the Tk GUI library—and Python 3.X's tkinter interface to it—in Python programs. This makes it possible to use Mergeall's GUI on Android devices today, along with a host of other existing tkinter software. Less positively, this app's tkinter support also comes with substantial downsides, including both behavioral glitches and unfortunate advertising, that make this approach difficult to recommend in this guide.
That being said, the prospect of running PC GUIs on a smartphone is arguably seductive, and the possibilities that this app raises for Mergeall users merit both fair review and your own judgement. You may or may not prefer Pydroid 3 GUIs to Termux command lines in the end, but this appendix provides enough coverage of the former to help you make a good usage-mode choice.
Versions note: The findings in this appendix reflect Pydroid 3 versions 2.22 and 3.0, Python versions 3.6 and 3.7, an Android port of Tk version 8.6, and Android versions 7 though 9 (Nougat, Oreo, and Pie). All four of these systems play roles in issues noted along the way. While this how-to hopes to be revised for fixes in these systems, change is constant and rapid in the software world. If you're reading this in the future, be sure to check all three for current status. With any luck, the glitches list here will be completely invalid by the time you see it.
First and foremost, Mergeall's GUI does work in Pydroid 3 with only minor changes, and without the command lines required for Termux use. Under this alternative approach, your content syncs will be run in the Pydroid 3 app's IDE (its edit+run GUI) that looks like this, and the Mergeall GUI you will use to configure and run syncs on your phone will look as follows:
You can also view the screenshots above in slideshow mode here. Despite the tradeoffs we'll explore in this appendix, it's important to note that running tkinter programs this way is an enormous achievement, given the large disparities between PC and Android platforms.
Now that you've seen Mergeall's GUI at work, we need to define some terms that impact the way it renders in Pydroid 3. The first two sets of screenshots above were captured with this app's tkinter fullscreen and maximized viewing modes enabled, by setting the first "Tkinter" switch off and the second on in the app's Settings ⇨ System dialog. You can use neither, either, or both of these modes, but their utility can vary per GUI and usage:
Nevertheless, Mergeall seems to work best with both these modes enabled, because it has no persistent popup windows and doesn't need to catch program exits, and because screen space is a precious commodity on phones in general. Mergeall works without these modes too—and looks like the last set of screenshots above—but has less space, and may bear manual resizing which seems out of place on a phone.
As a more general observation, tkinter GUIs in Pydroid 3 can take a bit of getting used to, and some are substantially easier to operate with a Galaxy Note-type stylus or Bluetooth mouse. For instance:
On the other hand, tkinter GUIs are surprisingly usable on a phone with practice—and might even be compelling.
Is that an AndroWish under the hood?:
Programmers may be interested to know that the Pydroid 3
app's tkinter support might be based on the work of
project that uses the
graphics library to render Tk GUIs on Android devices, employs a
bridge between Android's Java code and Tk's C
code, and was itself based on an earlier project called SDLTk
(whose web whereabouts are unknown).
If so, this means that Pydroid 3's tkinter simply inherits both its utility
and flaws from AndroWish, though the Pydroid 3 commercial product also shares
responsibility for the software it bundles and sells.
Unfortunately, Pydroid 3's developer did not reply to requests for verification of AndroWish use. To be sure, Pydroid 3's tkinter bears multiple signs of AndroWish-ness. Among them: its two-finger zooms and three-finger zoomed pans work in AndroWish too; its unique rightclick bindings smack of AndroWish's
sdltk touchtranslate command;
its GUIs-cannot-spawn-GUIs constraint discussed ahead also exists in AndroWish;
its tendency to fire multiple events for one gesture parallels an AndroWish bug report;
its install tree includes SDL components used by AndroWish;
and its overall look-and-feel seems the same as AndroWish's.
Still, this isn't enough evidence for a positive identification.
What can be said is that Pydroid 3's tkinter is likely based on work of others that is not necessarily tied to Pydroid 3, and might be usable in other Python coding apps, in standalone apps of your own, and even on other platforms. While intriguing, such prospects are both beyond this guide's scope, and largely moot. The more important point here is that Pydroid 3 today brings Tk and tkinter on Android to the broad Python audience—something that the Tcl-biased AndroWish project seems unlikely to have ever pursued. Hence, we'll attribute Pydroid 3's tkinter to Pydroid 3 here because it's no less a part of this app than any other component it embeds, and note that running tkinter GUIs like Mergeall's on Android is a stellar achievement, regardless of its implementation.
The prior section paints a justifiably positive picture, but tkinter GUIs do not come without both procedural requirements and noticeable drawbacks in Pydroid 3. To bring Mergeall's GUI to life on your phone, you must first be willing to accept some constraints unique to Pydroid 3, and others common to Android in general. Specifically, you must:
Compared to the Termux approach, you do not need to install Python separately (it's part of Pydroid 3); install helper scripts (you'll use the GUI instead); or run a permissions setup command (Pydroid 3 requests permissions automatically).
However, you still need Mergeall's source-code package (you'll run its GUI's script); must patch two of its code files for Pydroid 3 alone (per the next section); and still have to work with Android version, filesystem, and permission constraints (per their earlier coverage).
While some of the steps above are self-explanatory, and we don't have space for an exhaustive dissection of the rest, the remainder of this appendix explores the most-salient points on the list above.
Among the requirements just listed, the third may merit explanation most.
To use Mergeall's GUI in Pydroid 3, you must first download
Mergeall's source-code package from
this page to any
writeable folder on your phone, and unzip (i.e., extract) it there.
/sdcard/Download is the usual target for
downloads, and will suffice for both a quick look and later syncs.
There's more on Mergeall's source-code package
here and here
if you've jumped into this appendix randomly.
After installing the Mergeall source-code package, you'll need to replace two of its files, because it's not possible to integrate Pydroid 3's changes into the source-code package today for reasons explained ahead. To do this, simply click the following links to fetch custom versions of two Mergeall source-code files, and move or copy them into the top level of your unzipped Mergeall source-code package folder on your phone. That is, replace the originals with the versions below:
If you click these links online, you'll receive reply pages that looks like this; download to your phone by clicking the "save" in these pages' "Raw text" lines. If you're working offline, try a rightclick/longpress save, or visit the folder these links reference.
The changes applied in these files are minor, and apply to Mergeall's GUI only. Hence, they won't break Mergeall's command-lines mode described earlier in this guide if it's run from the same modified source-code folder. Conversely, these changes will break Mergeall on PCs, so use the files above only on your phone's copy of the source-code package. All changes made are marked with "# ANDROID" comments; search for this string in the custom files if you want to see what differs on Android.
In short, these files' changes address some but not all limitations of tkinter GUIs in Pydroid 3.
For example, text labels were trimmed and control fonts shrunk for better fit;
some size settings were lowered for non-maximized mode; a workaround was installed
for a Pydroid 3
sys module glitch that prevented the GUI from spawning merges;
other workarounds were installed for Python
webbrowser issues that precluded
document views; a button was decolorized to avert color loss; and later releases added
dialog-text breaks, folder-choice prefills, and read-only text state, to avoid message
truncation, chooser dialogs, and keyboard popups, respectively.
Despite all the changes applied for Android, though, you must still avoid using some fonts in customizations because they trigger crashes prior to Pydroid 3 3.0 and force blind choices later, and should not rotate some devices while Mergeall is running because this can hang or kill Pydroid 3 altogether.
We'll take a closer look at these and other glitches in Pydroid 3's tkinter GUIs in a moment. First, though, let's run the GUI.
Once you've made the prior section's code changes, start Mergeall's GUI by opening
its source-code package's file
launch-mergeall-GUI.pyw in the
Pydroid 3 app's editor, and pressing the editor's big yellow run
Mergeall's GUI should appear, and work the same as it does on PCs and as described here—select FROM and TO folders, choose a REPORT or UPDATE, pick a logfile folder and other run options, and press the GUI's GO button to sync. Mergeall messages scroll in the GUI, and are saved to a logfile if you've requested one. As usual, you can open Mergeall's source-code file in either Pydroid 3 itself (see its folder icon), or most Android file explorers (tap the file and open with Pydroid 3).
As noted earlier, Mergeall tends to work best in fullscreen and maximized modes.
If desired, select these modes before running the GUI with the Settings ⇨ System
and press your phone's back button twice when you wish to exit Mergeall in
While you're in Pydroid 3's Settings, also consider using the Editor's monospace
font without line wrapping; Mergeall's code is otherwise difficult to view.
Viewing code is not required for the GUI, but you may wish to experiment with
configuration-file settings in
and the line between user and programmer is often gray at best.
This GUI approach works, but only for Mergeall syncs: other Mergeall
diffall.py comparisons and
cpall.py bulk copies
have no GUI, and are still best run with command lines in the Termux shell as covered
Pydroid 3 does have a Terminal option that is capable of running command lines too
(and can even run the main Mergeall script with a command typed
but it's much more limited than Termux for command-line work.
And while you can open and run such scripts in Pydroid 3's IDE, they won't work because
the IDE has no way to specify the command-line arguments they require.
As a special case, running Mergeall's GUI to sync to an empty folder has the
same effect as
cpall.py (and in fact uses the same code), but you won't
get as many status messages, and there's no GUI equivalent to
today. Command-line scripts need command lines (or an IDE that supports
Run GUIs faster with shortcuts:
Opening and running a source-code file in Pydroid 3's IDE normally requires multiple
steps—you must either open the app and navigate to your file, or navigate to
your file in a file explorer and open with the app. This, plus clicking the IDE's
run button, makes for substantially more steps than either a command in Termux or a
standalone app (and is one reason this guide downplays the GUI option).
That said, some Android file-explorer apps allow you to place a shortcut (a.k.a. link) to a source-code file on your home screen, like this and this. Once placed, starting a tkinter GUI program like Mergeall in Pydroid 3 is just two taps—click the shortcut, and click the IDE's run button. Pydroid 3 still has extra setup steps and glitches, but running a tkinter GUI can be nearly as easy as running an app. Usage notes: file explorers here and here are among those that make links, and be sure to set your file's default app or command to be Pydroid 3 in your file explorer (unless you like surprises more than monkeys).
Now that you know how to run a GUI sync on Android, you should also be aware up front that Pydroid 3's tkinter support may still not quite be ready for prime time. Running tkinter GUIs on Android is undoubtedly a major accomplishment, but the current implementation has some rough edges that Mergeall code changes cannot entirely smooth. While some don't impact Mergeall, and a few are probably of more interest to programmers than to end users, the full set of observed issues confronting Pydroid 3 tkinter GUIs merits a summary here:
Getting the fit right for smaller phones can be difficult. For example, because Pydroid 3's tkinter naively clips nearby widgets instead of truncating the text of labels that don't fit, Mergeall's original and descriptive PC-GUI labels were initially unusable on phones of all sizes, and had to be trimmed radically for use on screens as small as 5.5" (and may still be too wide on even smaller displays without new user configs). More-sophisticated tkinter GUIs may naturally require more redesign work, though shrinkage is limited by usability (Mergeall might someday run on your watch, but it probably shouldn't).
helvetica), but most other font family names trigger an immediate and silent hardcrash (including
system). Moreover, font sizes work as expected, but italic and bold font styles are simply ignored if used for
courier(though they work for other font families). To test this for yourself, edit and run this file in Pydroid 3.
Mergeall avoids crashes with font presets tailored for Pydroid 3 to use only supported fonts, but this constrains functionality, and is error prone in the context of end-user settings. You can still tailor its fonts in the configurations file, but customize with care (and see this file for names to use with platform-specific settings like fonts).
Update: Pydroid 3's 3.0 release in early April 2019
fixed its font crash by coercing any unknown font-family name to
but all of its font limitations described above are still present. That is,
unknown fonts no longer crash GUIs silently, but
substituted blindly, and no additional family names (or
styles) are now supported. It's a fix of sorts, but the app's tkinter clearly
requires more font work.
Due to testing constraints, the exact scope of this issue is unknown. It has been observed on Samsung J7, S8+, and Note9 devices running Android Nougat, Oreo, and Pie. On the more recent Samsung Note9, however, rotations in Pydroid 3 tkinter GUIs work fine in both Oreo and Pie after changing the display's screen-zoom setting from Small to either Medium or Large. This doesn't help on the older phones, though, and it's unclear if the older phones may benefit from applying system updates.
It may also be possible to work around this in Mergeall with a Java-based approach that locks orientation or ignores rotation, but this would reduce utility drastically (rotation can be an essential tool when space is limited), and the underlying bug should really be fixed globally for all programs. Ultimately, only Pydroid 3 tkinter GUIs exhibit this defect.
For now: if rotates don't work in tkinter GUIs on your device, either make sure your screen zoom is not Small and your device is up to date; wait for a fix in whatever software plays Brutus to Pydroid 3's rotation Caesar; or use Mergeall's GUI in either portrait or landscape orientation—only—and consider a rotation-locker app like one of these if you can't avoid flipping your phone.
webbrowsermodule for opening web pages and other content generally works in Pydroid 3, because the app presets the
BROWSERenvironment variable to an Android
amactivity-manager command-line template; when spawned, this template opens documents per the Android default-apps model. See the module's documentation for more on why this works, and this script for a demo of the technique. It's similar to
startcommands on Mac OS and Windows, respectively; on Android, chosen viewers run in parallel with their opener, and a back tap returns to the spawner immediately. Documents can also be opened with Python's
BROWSERsetting, but that apes what
webbrowser is weak enough on Android to classify as a
glitch (and initially warranted disabling widgets that rely on it).
For one thing, it requires local-file URLs to use a
on Android, unlike any other platform—and silently fails to open
if this is absent. For another, Android's default-apps model
is not as functional as file associations on other platforms, and reliance
on this limits
webbrowser's utility: as tested, text files
open with few and unconfigurable handler options, and local HTML files open
as raw HTML in a text editor instead of being rendered by a web browser.
Barring an unknown Android trick, opening HTML in a web browser requires
http:// server URL, which limits HTML documentation to online use only.
limitations impact the "Help" and "Popup logfile?" buttons, which display the
user-guide HTML and run-results text, respectively. These widgets were disabled
in earlier releases requiring users to open documents outside the GUI, but now
work as well as they can. To work around the local-HTML issue, Mergeall now opens its
by remote URL; this requires online access, but has the advantage
of always displaying the most recent version (HTML documents tend to change
as frequently as web browsers). To display
properly, Mergeall now uses
file:// prefixes to open per the default apps on
your devices; this may sometimes be too limited to be useful, but it's better than disabling
the feature altogether.
These workarounds were also adopted on Android by other programs
mentioned in this guide, all of which now open their help and
other documents from within their GUIs, just as they do on PCs.
For coding details, see Mergeall's
patched files, as well as those of Frigcal and PyEdit described
webbrowser's limitations, Android users might still be
better served by manually opening documents in some cases.
Caution: per the prior note, Pydroid 3 tkinter GUIs may slide off screen or hang
if phone orientation is changed during a spawned
rotate twice to restore the GUI where possible.
Update: this story was rewritten in light of new findings in mid-April 2019, but was modified yet again by new bugs discovered later in Pydroid 3's 3.0 release—a plot element large enough to warrant its own coverage in the next note.
webbrowsermodule is now fully broken on Android due to changes in release 3.0 of Pydroid 3. Document opens using this module work as described above in the prior 2.2 version of Pydroid 3, but they either fail with permission errors or are deliberately disabled in the latest versions of this app, and require additional workarounds applied to programs here on April 21. This complex breakage reflects two separate regressions in Pydroid 3:
DISPLAYenvironment variable in tkinter GUIs, which breaks
webbrowser—the module assumes an X11 context, and triggers a permission-error exception when calling tools in Python's
subprocessmodule to query
xdg-settings. See the new exception here. This happens during browser registration, and before the
BROWSERsetting of the next bullet can be applied.
BROWSERenvironment-variable setting to run an odd custom script that special-cases Jupyter
file://documents—but also senselessly refuses to open any other
file://URLs, exiting silently instead. This setting may help Jupyter documents which bypass
webbrowser, but renders the module unusable for all others. To see this biased hack for yourself, run
more `which mybrowser`in Pydroid 3 3.0's Terminal; it breaks other local-file URLs for the sake of a Jupyter kludge, and appears to do so intentionally.
Luckily, there are at least two coding workarounds for the damage to other programs:
DISPLAY to reenable
general and reset
BROWSER to its pre-3.0 activity-manager
command for file URLs; or avoid this module altogether by spawning the
activity-manager command with
os.system (again, see
webbrowser clients here were
updated to use the latter scheme because it leaves environment
settings intact for other Pydroid 3 use cases.
Perspective: although this workaround suffices for now, this episode puts the
app's viability as a host for application programs like Mergeall on shakier ground.
The tale of this bug has been rewritten multiple times here already,
webbrowser failures introduced by an
"upgrade" of Pydroid 3 that preferentially aids one program at the expense of all
others seem to tip the scales well towards unusability—especially given the lack of notification to paying customers.
At some point, prejudiced instability can be neither documented nor accommodated.
That point may remain uncrossed, but it's clearly in sight.
multiprocessingparallel-execution module does not work on Android, because required semaphore tools are not supported on this platform (dig deeper here and here). The impact of this will naturally vary per program. PyEdit's Grep command, for instance, depends on this module on PCs to run searches in parallel with the GUI; while it can be easily configured to use fully supported threads on Android instead, doing so makes it impossible for pure Python code to leverage multiple CPU cores—a speed factor whose significance on phones is unclear. Moreover, PyEdit benefited from preexisting code that leverages the API similarity of threads and
multiprocessing; other programs may require more extensive recoding.
sys.executableis left empty by Pydroid 3, which in turn makes the
subprocess.Popenprocess-spawn code in Mergeall's GUI fail. Mergeall works around this by hardcoding this setting to use the path to Pydroid 3's Python, but this should be fixed in Pydroid 3: this same bug crippled most of the larger programs tested, though some other programs' spawns were also hobbled by the next issue. The workaround's change also applies to the Pydroid 3 app only, which makes it impossible to integrate into base files (more on this ahead).
Update: Pydroid 3's 3.0 release in early April 2019 did not
sys.executable bug (this setting is still empty), but it did
manage to arbitrarily move its Python executable—and break Mergeall's workaround
to this bug in the process. See the new exception
Mergeall's workaround addressed this in its April 19
patch-code release, by using the result of a spawned
which python shell
command for hardcoding, instead of former or current paths to Pydroid 3 Pythons
The result is release and path agnostic, and will hopefully be immune to future
morph in an app that seems more prone to change irrelevant packaging structures
than fix reported bugs.
subprocessway after the prior note's patch. This is fatal to programs like PyMailGUI that cannot be used without their launchers and cannot be easily recoded to use top-level windows instead of freestanding processes (more details here).
While its exact cause remains a mystery, this restriction may be inherent in Pydroid 3 if its tkinter structure requires programs to be run by its IDE. Tellingly, tkinter programs also fail if run from Pydroid 3's own Terminal interface, and the IDE seems to scan specially for specific import statement forms that enable GUI support (see the next note). Moreover, AndroWish—the most likely underlying Tk implementation—appears to have similar constraints. If a workaround exists at all, it is hidden so well that it's effectively nonexistent.
fromstatement on a line all by itself, or the script's GUI won't work (the required scaffolding appears to be unloaded). This didn't require a change in Mergeall, but the PyToe program in PyGadgets did not open in Pydroid 3 until such a specific statement was added to its top-level script. Though less common, you may also need to add a tkinter import statement to programs that load and run tkinter code from files dynamically; an import in the loaded code won't be enough to enable tkinter support.
This surfaced when adding code to the main script of the
a 256,898-byte and 5,141-line prior version worked fine,
but a 258,031-byte and 5,160-line new version started failing for
no reason whatsoever—until wholly unrelated comment lines were removed
(a short launcher script's success was the first clue of size issues).
Given today's memory capacities, this size limit is absurd;
people do write 5k-line program files, and tools do generate them.
It also has no workaround; if your script starts failing inexplicably in
Pydroid 3 after adding new features, make sure its code file is under 256K bytes
(or 5K lines) in size—or write a two-line launcher that reads and
execs the file to subvert Pydroid 3's draconian constraint.
The PyEdit text editor, for example, has an explicit Quit button that works normally, but has no way to offer users a chance to save file changes on double-back exits. Worse, the Frigcal calendar has no explicit quit widget of its own, and hence no way to save calendars in maximized mode. Non-maximized mode does better with a window-border exit button that runs exit handlers properly, but doesn't use available space as well, and can still be the scene of a silent-but-deadly double-back exit. The lesson here: don't use a double-back in Pydroid 3 unless doing nothing is what you mean to do.
Luckily, there are at least three ways to work around this glitch if it impacts your device: use a physical Bluetooth keyboard; use Samsung's handwriting input option whose return key works correctly; or simply install an alternative on-screen keyboard whose return key works properly, such as Google's Gboard or Hacker's Keyboard. Both alternative keyboards support multiline-text input in programs like PyEdit and Frigcal, and the latter keyboard also helps with the next glitch on this list.
To work around this, use a Bluetooth keyboard, or install an alternative on-screen keyboard on your phone that can be opened on demand. Hacker's Keyboard, for instance, includes a permanent-notification option which allows you to pop up its on-screen keyboard at any time by clicking the app's entry in Android's notifications pulldown. The resulting keyboard is a complete solution in PyPhoto, and helps with newlines, menu shortcuts, and copy/paste in other programs too (see the notes above and below).
Frigcal program, for example,
readonly state in Tk
to mute annoying keyboard popups on most touch gestures that made the program unusable.
Mergeall's GUI wasn't impacted quite as badly, but its message-text
area still required
read-only state on Android (which means
disabled in Tk's
otherwise, slower swipes to scroll trigger an on-screen keyboard
popup—which overlays and obscures the very text you're trying to view.
Despite the work, keyboard popups may still appear more often than they should in
some Pydroid 3 tkinter GUIs; as usual on Android, your device's back button (and a
downswipe where supported) hides an on-screen keyboard that's obscuring important
content or GUI components.
charattribute and a
keysymthat identifies the press (as
'BackSpace', respectively). This differs from other platforms but can be worked around with binds to these specific keys (e.g., to
Worse, though: shifted keys, like
on most keyboards, send two useless keypress events—one for the shift
key itself, and another for the unshifted key pressed. A
Shift+* sequence, for example, sends a shift followed by the
8 key only; the
* never comes through.
Binding to specific keys doesn't help in these cases, and mapping unshifted
to shifted keys manually seems implausible across all keyboards and usage contexts.
Barring a workaround as yet unknown, this makes shifted keys unusable in some
tkinter GUIs run on Pydroid 3.
This glitch seems unrelated to the newlines issue noted earlier (which fails on just one keyboard), and on the upside appears limited to general keyboard handlers (tkinter text-input widgets process shifts correctly). But it impacts programs like PyCalc that allow numeric operator keys to be entered on keyboards. As is, PyCalc fully supports keyboards in its "cmd" popup input field, (whose use also seems to oddly correct the shifted-keys bug), but keyboard input for its main display is limited to non-shifted keys on Android alone.
Ctrl+cto copy and
Ctrl+vto paste in keyboards that support these. You'll find support for these keys in both Bluetooth keyboards, as well as alternative on-screen keyboards like Hacker's Keyboard (yes, a pattern is emerging). This works well, but may not be what a typical Android user would expect.
updatecalls. Only one impacts Mergeall and the other is narrow in scope, but both merit a heads-up for both users and programmers.
Most noticeably, on every Android device tested, Buttons lose
bg background-color setting when pressed: their color changes
to the recessed default, and stays that way until anything else on the
GUI is tapped. This can seem to make a Button
but is really just background colors being lost. Curiously, some buttons evade
color loss (e.g., those that open common dialogs may be immune, suggesting a
timing issue), but most colored buttons are impacted.
This colors glitch affects only the "Help" button in Mergeall, but may afflict other GUIs more broadly. The only known workaround requires changing code to either accept the Button's default color, or replace Buttons with Label widgets that use click bindings. Labels retain their background colors when clicked, and their cosmetic and functional difference is trivial (Labels don't give press feedback, but Buttons in Pydroid 3 sometimes don't either). As fix examples, Mergeall eventually chose default colors; the PyCalc calculator program adopted Labels to skirt the issue; and the Frigcal and PyToe programs have used Labels all along.
OptionMenuselection lists too large for the display
Altkey for menu shortcuts, in keyboards that support it
Probably the best spin on these is symptoms of still-maturing software.
There's more on the other programs noted in this list in the next section. The point here is that while some accommodation is to be expected when translating between PC and smartphone paradigms, issues like those above mean that running tkinter programs in Pydroid 3 is hardly seamless. Even allowing for fixes in its future releases, you'll clearly want to test-drive this app on tkinter programs you'd like to use to see how its support applies.
Despite the workarounds and glitches of the preceding section, the second to last of the Pydroid 3 requirements listed earlier is really its most regrettable and grievous downside: the free version of Pydroid 3 will flash distracting and distasteful fullscreen ads in your face every few runs, unless and until you pay a required $9.99 freemium fee. To be blunt, this comes off as ransom, and borders on extortion. It may be a norm in Android's culture, but it is rude and unacceptable in Python's, and makes it difficult to recommend Pydroid 3 to Mergeall users today.
Together with the prior section's glitches, this also makes it impossible to
integrate the changes required for Pydroid 3 tkinter GUIs into source-code
package base files. For example, the coding workarounds that accommodate
Pydroid 3's font limitations and
webbrowser breakage are specific
to Pydroid 3 alone, and the
sys.executable fix required
for spawns may not work in other apps.
Changing portable source-code packages
to support just a single app on Android is ethically dubious at
best—and just plain wrong when that app employs advertising.
Though less convenient, Pydroid 3 today is best supported
with the custom source-code files used here.
This guide's broader advice to Mergeall's users is to evaluate its GUI in Pydroid 3 to see if it warrants the fee in your context. It may, especially if you have other tkinter programs you wish to run on Android. As examples of this app's broader scope:
See this page for a separate how-to on running these programs on Android too, along with a summary of this appendix's Mergeall GUI coverage. Taken as a set, programs like these and others may justify the ransom's price, but this choice is yours to make.
Wherever you may land on this issue, there's no denying that Pydroid 3's advertising scheme casts programs like Mergeall in a supporting sales role that's both uncomfortable and unseemly. A free trial period would be much more palatable than obtrusive advertising, and will hopefully be a part of the future of the otherwise-remarkable Pydroid 3 app. Barring that, we'll have to eagerly await the Android debut of free and open-source alternatives for running the free and open-source tkinter.
It's a large world after all: Naturally, there may be other tkinter-on-Android options afoot or unknown at this guide's publication, and others may appear in the future. Please exercise the usual due diligence, and check back here for updates as change warrants and time allows. This guide wishes to be inclusive, but some commercial projects may be no more transparent than they are benevolent, and the quality of web-search results has grown inversely proportional to the avarice of companies providing them.
Finally, in addition to all the Pydroid 3 requirements described in the preceding sections, you must also keep in mind that this appendix's GUI approach shares all the other general constraints that this guide notes for the Termux command-lines approach. Specifically, to use Mergeall's GUI in Pydroid 3, you'll still need to:
In other words, Mergeall's GUI may run on Android, but it's still running on Android.
More on Pydroid 3 storage permissions:
Notice the last half of the general requirements listed in this section—it arises because
Pydroid 3 shares the exact same permission constraints that we saw earlier
for Termux. Namely, the programs that Pydroid 3 runs can read anywhere on internal storage
and removable media, and can write anywhere on internal storage, but writes on
removable media are limited to Pydroid 3's app-specific folder only, which is used
like this, and lives
at the following path on your
/storage/xxxx-xxxx drive (once you create it,
and swap the
xs for your drive's Android ID):
It's easy to verify this: see the permissions test script, and its results on both Pydroid 3 and Termux; the results do not differ between Android Nougat and Oreo and are unchanged by installing Pydroid 3's extra-permissions plugin. The takeaway from these results is that storage access rules are the same for both apps, and updates on removable media are limited in both to the app's own folder. The only real difference in the storage-permissions department is that Pydroid 3 replaces Termux's one-time script run with a one-time automatic request on first access to restricted folders.
As we've learned, this means that content on removable media must be nested in Pydroid 3's app-specific folder if it is to be updated by Mergeall in this app. It also means that items to be updated by both Termux and Pydroid 3 must be located in internal storage, because these apps can modify only their own app-specific folders on removable media. Importantly, your choice of Mergeall content folder on SD cards and USB drives wholly determines which access-limited app can change it. For instance, if Mergeall's GUI in Pydroid 3 processes removable-drive content in Termux folders—as in earlier screenshots—it is limited to comparisons only. And while our focus here is on Mergeall, this constraint applies to any GUI run in the app.
This is an unavoidable downside to using Android removable storage in these apps. The lesson here is that internal storage—usually at
always be used if possible, because it can be changed by both Termux and Pydroid 3
specifically, and by other apps generally. As an example of the latter set,
a Python Android app not covered in this guide because
it holds no advantages for Mergeall today, has the same storage-access limits per
Android's rules must be hard to beat.
Update: though uncertain at this writing, Android Q may impose the same access constraints on internal storage that KitKat imposed on removable media, and invalidate some of this section's coverage in the process. See Appendix C for more on the possible consequences for Mergeall—and every other program that saves shared data on the Android platform.
So there it is: the good, the bad, and the ugly of using tkinter GUIs like Mergeall's in the Pydroid 3 Android app. To be sure, the tkinter support embedded in this app is nothing short of stunning: it implements nearly all of Tk and tkinter on smartphones, and makes these devices substantially more fun to use, for both programmers and others. You'll have to decide for yourself whether its remaining defects—along with the advertising model of the app that brings it to you—justifies using this support on your phone.
Beyond Pydroid 3's tkinter support, there are additional GUI options for Python programs on Android that we don't have space to cover here. For example, while Termux does not offer tkinter support directly, it does provide some X Windows (a.k.a. X11) GUI support that may provide a path to using tkinter in its Linux environment. For a survey, see its links here and here, and the X11 search here. Even if viable, however, X11 support requires substantial extra setup work that may be too much to ask of users running Mergeall alone.
Other GUI options offer Android-friendlier alternatives, including Kivy, SL4A, and Qt's PyQt and PySide. Such tools allow programs to use Python for their core logic, but wrap it in GUIs which may better match the Android paradigm. Although their program recoding costs are not trivial, these tools become more attractive for tkinter GUIs that must be redesigned for smaller screens if their audiences are ever to include more than the most-stubborn tkinter fans.
At the end of the day, Pydroid 3's tkinter support is admirable and even amazing, but the approach it offers for Mergeall really just swaps command lines for a GUI run in an IDE that requires extra code changes and comes with noticeable seams. Whether you find it easier to sync your content by running a single command in Termux, or opening and running a source-code file in the Pydroid 3 IDE may very well depend on your background. For this guide's money, though, the latter is just enough extra steps to qualify as runner-up.
And now for the closing cliff-hanger. True to its thrashing-laden history, Android seems poised to radically alter storage-access permissions again and soon, with potentially profound impacts on both Mergeall and the Android platform in general. Although this change is still tentative, this appendix provides a preview of its implications for users of Android and Mergeall in one possible future.
This how-to has gone to great lengths to document the permission rules for internal and removable storage that existed at the time of publication, especially in sections here, here, and here. As covered, these rules apply to apps running on Androids KitKat through Pie, of which only Oreo and Pie are actually useable for Mergeall due to an earlier Android timestamps bug.
Regrettably, some of this guide's coverage of the rules for internal storage—the unremovable storage space in your phone that much Android documentation confusingly calls external—might be invalidated in the near future. It now appears that Android Q, released in beta form in March 2019, may impose the same crippling constraints on internal storage that Android KitKat earlier imposed on removable media. In brief, this so-called scoped storage proposal would restrict access to internal storage as follows:
The app sandboxed folder referenced in this list is what we've called the
app-specific folder (e.g., in
/sdcard/Android) in this
document, because its internal-storage version can be
accessed by other apps today. Apps already have an app-private folder
/data/data), but the new rules would limit apps' access
to their own app-specific folder in the currently general internal storage.
This model would be the death knell of same-device interoperability: because files in internal storage would no longer be freely usable across multiple apps, the scope of your on-phone content would be limited to a single role. This is fully unusable for modern computing tasks. It also manages to further trample the ethic of engineering freedom that has been promoted for decades by Python and the open-source world; a platform that imposes tools and rules on this scale would be closed by definition.
Some of the new constraints may be mitigated by wildly proprietary and convoluted interfaces, but these interfaces may be much slower than those of traditional POSIX files, and are unlikely to see widespread use in programs born on less restrictive systems and designed to run on multiple platforms. Either way, this change would impact nearly every program that updates shared content on Android devices (including Mergeall), and could very well be an endgame for this platform's broader potential.
For more background on this proposed change than this appendix can provide, read the terse and perpetually morphing announcement here; the pragmatic technical review by an Android file-explorer developer here; and the outrage of Android developers pleading for a reversal here, here, and here (the Mergeall team is guilty of the latter).
Caution: the language at some of those links is understandably explicit. If this change becomes permanent—and KitKat's example suggests it will—it threatens to make a host of existing programs and libraries based on the portable POSIX file standard unusable on Android; render Android's internal storage just as locked down and limited as that in iOS; grossly curtail what users can do with their content on their phones; and effectively kill the platform as an option for general-purpose computing.
Simply put, under the scoped-storage model, Android devices will be reduced to launchers of fenced-in apps, and lose all rights to the title of "PC alternative." Indeed, the better analogy for the proposed Android seems the cartridge-based Game Boy of eras past.
That being said, Mergeall itself will likely still work unchanged on Android if these restrictions are imposed, but content on internal storage may have to be structured in the same inconvenient and perilous way as that on removable media. Specifically, it might need to be nested in the updating app's app-specific folder in internal storage. This means the following sandboxed locations for the Termux and Pydroid 3 apps covered in this guide, respectively:
Unfortunately, content thus stored could generally be updated only by the sole app whose folder hosts its storage, and would be automatically deleted if that app is ever uninstalled. While these major constraints already apply to removable media today, extending them to internal storage may make the Mergeall program impractical for many or most users on Android.
There's a slim chance that the more permanent
shared-collection folder in internal storage would still be usable
for Mergeall content in the Termux and Pydroid 3 apps. That is,
you may be able to locate your content in internal storage just
one level deeper, at:
If viable, this might avoid the delete-on-uninstall peril of sandboxed folders,
and may support access from multiple apps. Also unfortunately, the beta release's
of proposed rules for the
Download folder suggests that files stored
there may be app specific too, though this is described thinly and ambiguously, and will
have to await empirical analysis on widely used devices.
In fact, the very definition of "access" to be restricted is unclear in the beta documents, and also awaits verification in the final release. For example, if the new internal-storage rules shadow those of removable media today, apps may still be able to read anywhere in internal storage, leaving a massive security hole completely unfilled by the new model. But shared-storage writes—essential for content creation of any significance—are almost certain to vanish, at least in their current direct form.
As a last resort, the beta release also suggests using Android's Storage Access Framework (SAF) to access files created by other apps (e.g., by passing fetched file descriptors to the native code layer that hosts Python on Android). But this proprietary Java-based library is unwieldly and slow; may not work for Mergeall's mass-updates use case; and seems a nonstarter for cross-platform Python code run by an app which sets permissions policies for scripts. Python uses the POSIX file interface on purpose: it makes programs portable to every modern computing platform of interest—except the proposed Android. The SAF's radical requirements would make this platform impractical to support.
In the end, Mergeall and similarly general tools may run on Android Q, but they're likely to be much more difficult and dubious to use, if not altogether pointless. Content collections that are accessible to just one app and may be subject to accidental removal would probably not be worth storing on your phone.
Naturally, it's impossible to predict if internal-storage restrictions will turn out to be anything more than beta-release drama, and you'll have to stay tuned for this story's conclusion. There is substantial developer pushback at present; apps like Termux and Pydroid 3 that run Python code may implement broader storage permissions and interfaces in response; phone manufacturers may implement internal storage in ways that cannot be foreseen; and an initially proposed compatibility mode promised to retain the current internal-storage permissions of apps used in this guide—at least until those apps targeted later releases, you uninstalled the apps, or you bought a new phone.
What is certain is that deliberately hobbling the Android platform is sure to limit both its users' options, and its devices' scope. Extending proprietary lockdown to internal storage would make portable tools like Mergeall nearly unusable on Android, and constrain the platform to artificially narrow—and even toyish—apps. Such apps might appeal to technically naive masses and generate proportionate advertising revenue in the process, but the closed platform hosting these apps will hold little interest for the skilled development community that produces nontrivial programs used elsewhere.
The net effect would deprive smartphone users of much useful software, and knowingly retard their devices' utility. This would be especially ironic at a time when foldable displays and ample storage hold the promise of expanded roles. Though more subtle and speculative, this change may also help drive smartphone users to cloud-based storage in the face of unworkable on-phone alternatives—boosting subscriptions, trackable content, and single-source control as darker byproducts. In this analysis, Android would remain a carefully crafted arena of exploitation, fueled by greed and justified by red-herring security arguments easily refuted by the platform's own success.
Let's hope that's not what Android's owner has in mind.
Update: Android blinked in late April 2019—or at least called a temporary cease-fire with its users. Per the blog post here, enforcement of the new internal-storage permission constraints described in this appendix has been delayed until Android R, the successor to Q expected to appear in 2020 and become common on consumer devices by 2021. While welcome news, this really just postpones the tragedy by one year for developers and end users alike. A lot can happen in that year, but people in both camps are left hoping that its manifest won't include a crippling change which reduces Android devices to media-consumption portals, designed to turn private lives into profit centers. The odds are sadly slim in a world where billions seem happy with the Faustian bargain.