Using Mergeall on Android with Termux (and Pydroid 3)

Version:	June 18, 2019 (revised Nov-2022)
Author:	© M. Lutz, learning-python.com
Quick links:	View and fetch this package
	View this guide's media
	View the related tkinter doc

This document describes how to use Mergeall to sync content on Android devices. It primarily focuses on running command lines in the Termux app, and more briefly explores a Mergeall GUI alternative on this platform. This coverage is provided freely, but with no warranties of any kind. Findings here reflect testing on Samsung Galaxy devices running Android 7, 8, and 9 (Nougat, Oreo, and Pie) and 10. Given the wide variety of products and rapid—really, supersonic—rate of change in the Android world, applicability to other devices and Androids cannot be guaranteed. Mergeall itself changes content by design: see its usage cautions before using the approaches outlined here.

Contents:

• Overview
• Mergeall Requirements on Android
  • Use Android 8 (Oreo) and Higher Only
  • Use FAT32 External Drives on Some Devices
  • Use Mergeall Command Lines in the Termux App
  • Grant Termux Extra Storage Permissions
  • Nest External-Drive Content in Termux App Folders
  • Retain Your Nested Content if Termux is Uninstalled
• Setting up Your Phone and Drives
  • 1. Install Termux
  • 2. Run termux-setup-storage
  • 3. Install Python
  • 4. Make Your Working Folder
  • 5. Install This Package and Mergeall
  • 6. Prepare Your Drives
  • 7. Edit Your Bash Profile
• Managing Your Content
  • Running Syncs and Diffs
  • The FAT Times Fixer Tool
  • Other Mergeall Tools and Techniques
• Random Tips
  • Termux Tips
  • Other Tips
• The Mergeall Termux Wrap-Up
• Appendix A: A Brief Primer on Android Pathnames
• Appendix B: The Pydroid 3 tkinter GUI Alternative
  • Mergeall's GUI in Action
  • Mergeall Pydroid 3 Requirements
  • Required Mergeall Code Changes
  • How to Run Syncs in Mergeall's GUI
  • Glitches in Pydroid 3 tkinter GUIs
  • The Advertising Scourge
  • General Requirements Still Apply
  • The Mergeall Pydroid 3 Wrap-Up
• Appendix C: Android 11—the Future is Closed?
  • The Permissions Game Changer
  • The Impact on Mergeall
  • The Impact on Android
  • Updates: The Rest of the Story

Overview

If you're able to meet its requirements below, Mergeall on Android provides incremental backups and changes propagation for content stored on your smartphone. With a few simple commands (or a fledgling GUI), you can sync content on your phone's internal storage or removable SD card both to and from an attached USB drive, and can do so right on your phone itself. The net effect turns your phone into a portable storage device for arbitrarily large content collections, without requiring card removal or device rooting, and without the cost, speed, and privacy negatives of cloud storage.

The package that hosts the guide you are reading is a supplement to the Mergeall system. It comes with precoded helper scripts that you can run directly on your phone from the Termux app's prompt. Though not required, these scripts are designed to simplify Mergeall's command-lines mode on Android, and minimize the risk of user errors. To make edits on smartphones easier, these scripts also use common settings in the included .bash_profile file, which you'll modify for your drives' IDs and folder names. Besides their role in scripts, these settings provide quick command-line access to your folders in general.

For Mergeall users who break out in hives at the thought of using command lines, this guide's Appendix B also includes coverage of using Mergeall's GUI in the Pydroid 3 app, and this package includes patch files needed to make the GUI work. While this approach comes with code changes, operational issues, and pay-or-else advertising that make it a secondary choice here, some users may prefer a GUI even if it entails extra drama. If you wish to explore this option, be sure to read the Termux coverage in the main part of this guide first, as most of it applies to Pydroid 3 GUIs too.

Mergeall Requirements on Android

The Mergeall system, originally developed for PCs, can be used unchanged on Android to synchronize on-phone content to and from a USB drive, without rooting your phone, and without requiring SD card removals—an arguably amazing feat in such a limited form factor. To make this work, though, you'll have to meet a few requirements up front. In short, Android has multiple issues and permission rules that require a custom approach; to use Mergeall on this platform, you:

• Must use Android 8 (Oreo) and higher—due to a former Android timestamps bug
• Might use FAT32 on removable drives—due to an Android exFAT interoperability bug
• Might run Mergeall from command lines in the Termux app—because its GUI is marginal
• Must grant Termux extra storage permissions—else storage options are limited
• Must nest removable-drive content in Termux app folders—else it cannot be updated
• Should retain any nested content if Termux is uninstalled—else it will be auto-deleted

The sections that follow go into these requirements in more detail. Later sections cover setup for these requirements, and usage of this package's helper scripts. The positive take here is that Mergeall on Android may be constrained, but it does work. While some users may prefer pulling their SD card and syncing on a PC, this is delicate work in the extreme, and isn't possible on all devices. With a USB drive, a little up-front work, and either simple command lines or a GUI, Mergeall brings a more PC-like experience to your on-phone content management.

Use Android 8 (Oreo) and Higher Only

The first requirement is an absolute: Mergeall works on Android devices, but only if they are running 2017's Android 8 (a.k.a. Oreo, and sometimes just O) or later. Earlier Androids have a showstopper bug which renders content-sync tools like Mergeall unusable. You can still use Mergeall to manage your phone's content prior to Oreo, but not on Android itself: remove your card and sync it on your PC.

The short story on the bug is that earlier Androids are unable to copy file modification times to either FAT32 and exFAT drives or internal storage on non-rooted phones: file content can be copied correctly, but copied files are always stamped with the time at which the copy was made, not that of the source. This is lethal to content-sync tools like Mergeall, which rely on file timestamps for change-detection speed. Without timestamp copies, on-phone content copies cannot be compared to other copies accurately, and quick content syncs are impossible.

This issue is well-known, long-lived, device-wide, and not specific to Mergeall, Python, or Termux. In fact, a shell cp -p copy command fails to copy times with an error before Oreo too—see the Nougat logfile. It stems from Android's former FUSE-based filesystem, which required root access to update file times but not content, and was eventually replaced with Samsung's SDCardFS. For more background, try the bug report here, the thread here, and searches like this and this. As an example of its wide swath, this bug also kills calendar-file backup copies in the Frigcal program, which makes a cameo appearance in Appendix B; prior to Oreo, many programs were impossible on smartphones.

Though likely unintentional, it's unfortunate that the world's most-used mobile operating system didn't support non-cloud content syncing until 2017 (and still doesn't for many prior-version users). The good news is that this bug is now fixed, making cloud-free storage on Android finally practical for non-trivial content—and just in time for the larger-storage devices beginning to appear.

Use FAT32 External Drives on Some Devices

Preface: This section documents an exFAT timestamps issue that exists in former Samsung Androids. The issue has been fixed as of 2020's Android 10 per the epilog, but will still be present on older unupdated devices. Hence, users of devices running Android 9 (Pie) and earlier are still advised to use FAT32 per this section, but users of newer or upgraded Samsung devices running Android 10 or higher may freely employ exFAT for external drives hosting Mergeall-managed content—and can safely skip both DST-rollover issues and this section.

The Issue

Due to a current Samsung Android bug regarding modification times recorded by the exFAT filesystem, removable drives used with Mergeall on Android should generally use FAT32 instead of exFAT for interoperability with other platforms. In short, the modification times of files created on exFAT drives are known to be incorrectly skewed as follows when using Android Nougat and Oreo (and later Pie) on multiple Samsung devices:

• Times created on Mac OS are skewed 16 hours ahead when used on Android
• Times created on Android may be skewed 16 hours behind when used on Mac OS and Windows

Both skews were observed in and are relative to the US Pacific time zone, which is normally 8 hours behind the UTC base. These are likely symptoms of the same exFAT bug in Samsung Android, and may eventually be fixed in its future versions, but they can wreak havoc with Mergeall file comparisons today.

It's important to note up front that, due to testing constraints, it's unknown whether this flaw lies only in Samsung software or is more widespread. exFAT is not part of Android's AOSP core-functionality stack (described here and here) which all phone makers augment with code from various sources (described here and here), and the issue is labeled "Android" in this document for brevity alone. Before making filesystem choices, you'll want to compare exFAT-drive file times on your own PC and Android devices to see if this defect impacts you. Even if limited to Samsung devices, though, this bug's scope is large.

It's also worth noting that you may be able to use exFAT in some contexts anyhow. It's now supported on most Androids; the first bullet above applies only to content created on Mac OS; this may be a Samsung-only issue; and exFAT-drive files may in some use cases squeak by as incorrect-but-good-enough for Mergeall comparisons because their mistakes are consistent. Moreover, users who will only ever backup on-phone content to a USB drive are exempt from exFAT PC interoperability concerns. Until this bug is fixed, though, it's dire enough to recommend FAT32 as the more reliable choice for your Mergeall-on-Android drives.

If you're willing to take that recommendation on faith, feel free to skip ahead to the next requirement now. For the curious and skeptical, though, the following traces the bug's forensics—beginning at the scene of the crime.

Act 1: Mac OS Content Is Skewed +16 Hours

The first symptom of this exFAT bug is easiest to diagnose: in testing on three recent Samsung devices (a J7, an S8+, and multiple Note9s), exFAT file timestamps created on some PC platforms are interpreted differently on Samsung Android than they are on any other platform tested. That is, the same file on the same exFAT drive reports a different modification time on Android alone.

In a comprehensive test conducted in early 2019 in the Pacific time zone:

• Files were created on the same exFAT-formatted USB flashdrive and microSD card on Mac OS El Capitan, Sierra, and High Sierra; Windows 7, 8, and 10; and Android Oreo and Nougat. Linux wasn't included, because its exFAT support is sketchy.
• All the created files reported modification times identically and correctly on all these platforms—except the two Androids.
• On the two Androids, only the files created on all three versions of Mac OS reported modification times incorrectly skewed 16 hours ahead of their correct values.
• Files created on Windows and Android reported times correctly (in this test, at least: stay tuned for more on Android times).

In other words, this test's results make the flaw appear to be a double bug: the Mac OS PC platform records exFAT timestamps in such a way that triggers an incorrect interpretation on the Android mobile platform. Files created on exFAT drives by Mac OS (only) report incorrect times on Android (only).

No, really. For a more graphic look at the problem, here is a folder of files created on Mac OS High Sierra, as it's seen on Mac OS El Capitan, Windows 10, and Android Oreo ("Today" on Mac OS means 1/15/2019, and 1/16 was still tomorrow when this test was run). As you can see, the file times are wrong by 16 hours on Android alone—files created around 8:20 AM report their times near 12:20 AM the next day.

Conversely, files created by this test on Android report the same and correct times on Mac OS, Windows, and Android. That makes this seem a one-way bug: it impacts just exFAT files created on Mac OS and viewed on Android, and not vice versa. You'd also never notice it if you stick to PCs (Windows times are correct on Mac OS and vice-versa), or limit your scope to just Windows and Android (Windows times are correct on Android and vice-versa). As the next section will explain, though, the latter deduction falls apart in the face of fresh evidence.

Act 2: File Saves May Be Skewed -16 Hours

But wait, it gets worse. The second symptom of Android exFAT anomaly is subtler and took more digging to unearth: in addition to the prior section's +16-hour Mac OS content skew, Samsung Android devices may also write file modification times incorrectly on exFAT drives. In later tests run on the same Samsung Android devices, files saved to internal storage and FAT32 drives recorded modification times correctly, but some saves to an exFAT USB drive erroneously recorded times that are interpreted as 16 hours behind the actual save times on every platform except Samsung Android itself.

No, really. The facts in this category are bizarre and even appear to be random. The incorrect times saved by Android are always incorrect on both Mac OS and Windows, but not all times saved by Android are incorrect (which is why this effect was uncovered later), and those that are wrong seem to depend on program and/or context (e.g., one program wrote new files' times correctly but existing files' times skewed—sometimes).

This makes this skew difficult to trigger, but easy to spot when it appears. As a brief example, a folder of files on an exFAT USB drive which began like this was updated and extended by assorted programs on Android around 11:30 AM on 2/27/2019 (again, in the Pacific time zone), yielding the following scenes on Android, Windows, and Mac OS; as you'll notice, some of the Android-written timestamps are wrong by -16 hours on the latter two.

That is, it's not just reads, and it's not just Mac OS: the whole conception of time in whatever software does exFAT on Samsung Android disagrees badly with others. This is really just the other side of a single bug's coin, and this guide doesn't have time or space to document it in more detail (and it's really not this document's job to do so in any event). But there's clearly something quite wrong with exFAT on Samsung Android today.

The Mergeall Damage

Gory details aside, this is another showstopper for timestamp-based programs like Mergeall: because exFAT files don't process timestamps correctly on Android, such files are generally unusable across any platform mix that includes Android. These files' incorrect times may work on Android itself—Mergeall doesn't care that times are correct, only that they differ when files are changed (i.e., consistently incorrect is good enough). However, the time skews can still cause problems when using exFAT drives in at least three noticeably devious ways:

• When Android copies files from an exFAT drive, it copies their incorrectly skewed times too. This suffices for later change detection: skewed times copied will compare correctly to skewed times on exFAT-drives. Unfortunately, this also makes the mistake permanent: if the copied files are ever copied unchanged from Android to PC—or the drive hosting such files on Android is ever used on a PC—the copied files will all be off by +16 hours, register as spurious differences, and be recopied unnecessarily. Though less tangible, the skewed times copied from exFAT may also pose other challenges if you ever need to manually reconcile copies in the future.
• When Android saves files, it records times correctly on both internal storage and FAT32 drives. Paradoxically, this can make the exFAT skew more error prone: if you ever save a file on your phone exactly 16 hours after its latest save recorded on an exFAT drive, the updated file will not register as a difference from the exFAT drive, and will not be propagated to your other devices. This seems unlikely to occur; the file would have to be saved on your phone only 16 hours (plus or minus 2 seconds) after the save recorded on exFAT, and this matters only if you will sync changes from your phone. Despite their rarity, though, lightning strikes do happen, and "unlikely" shouldn't be an acceptable term when it comes to your content.
• When Android saves files, it may record times incorrectly on exFAT drives. Such files may suffice for Mergeall comparisons on your phone itself: their incorrect -16-hour skew written to the drive will be offset by an incorrect +16-hour skew when read from the drive (or, equivalently, on-phone usage will ignore the flaws that skew file times on PCs), and consistently incorrect is again good enough. Still, "good enough" is also a term best avoided when your content is on the line; in this case, the incorrect times recorded by Android on such exFAT files will forever disagree with both your PC and off-phone reality in general, and again may hamper future reconciliations.

Just as insidious, there seems no way to compensate for the Android time skews automatically. For example, because exFAT drives may contain content created on multiple platforms, just some of your files, those saved on Mac OS, will report incorrect times on Android, and programs like Mergeall cannot tell which differences are valid and which reflect +16-hour Mac OS skew. The -16-hour write skew will be present on all PC platforms, and its nearly random nature seems even more impossible to work around.

Happily, FAT32 drives both interpret and record times correctly, and are thus immune to this bug. For instance, here's what the same file saved on Mac OS looks like on Mac OS and Windows; on Android, this same file's time is skewed on an exFAT drive, but correct on a FAT32 drive. This means FAT32 provides a path for using Mergeall on Android. Less happily, this precludes mixed FAT32/exFAT usage: because only exFAT drives skew times, their files won't compare correctly with the same files on FAT32 drives (as for others) and spurious differences and recopies will ensue.

The Verdict

In sum, this defect renders exFAT generally unusable on Samsung (and perhaps other) Android devices, and makes it impossible to recommend exFAT for Mergeall on this platform today.

Ultimately, this is a bug in Samsung or Android code that should be fixed. Although file times may have been broken on Android in the past, exFAT files today still report and record incorrect modification times on the Samsung Android platform alone, and neither file sources nor program contexts justify further deferring a repair. Barring a future Samsung or Android fix, though, your safest bet is to use FAT32 for all the removable drives you will sync with Mergeall on Android for both accuracy and interoperability.

This is regrettable, given exFAT's support for larger files and compatibility with other filesystems' times. Per its well-known limitations, FAT32 doesn't support files larger than 4G, and you may need to manually adjust your FAT32 drives' modification times twice a year to keep them in sync with non-FAT drives after daylight-savings-time (DST) rollovers. The internal storage in both your PC and phone, for example, is usually a non-FAT drive whose UTC times clash with FAT's local times.

See the FAT times fixer tool covered ahead for a simple way to make the adjustment; if you use FAT32, you'll run this tool at DST rollovers on your intermediate USB drive (to match your PC and phone), and possibly on a removable SD card (to match your intermediate drive). You might also run this tool on Samsung Android to adjust the times it skews on exFAT drives, but this will then skew them on all other devices (yes, !), and requires extra steps too; FAT-32 seems a better fix.

The upside is that, thanks to Microsoft patent constraints, FAT32 is still much more widely usable today than exFAT. More importantly, its timestamps are implemented more consistently and interoperably; for Mergeall on Android, this is a killer asset.

Use Mergeall Command Lines in the Termux App

On Android, Mergeall's Tk-based GUI works marginally with caveats, but its command-line mode works fully and qualifier free. To use this mode, you simply need to use Mergeall's source code, and type command lines to launch it. Although there are multiple ways to run Mergeall's Python source code on Android, the Termux app provides a full-featured Linux experience that seems best suited to the Mergeall task. Neither Termux nor command lines are required for Mergeall on Android, but they're suggested for most users.

In more detail, Mergeall's apps and executables for PCs (Mac OS, Windows, and Linux) do not apply to Android, and its reliance on the Tk library (and Python 3.X's tkinter interface to it) means that using its GUI's source code on this platform entails major tradeoffs. Although there is some new support for using Tk in Python programs run on Android today, it comes with rude advertising, substantial glitches, and extra steps that make it difficult to recommend for Mergeall syncs. See the notebox ahead for more on this story; it's evolving, but at this writing Tk may still be a nonstarter for many Mergeall users on Android.

Luckily, Mergeall is built with a decoupled architecture that allows its main code to be run separately from its GUI. To use Mergeall's command-line mode on your phone, you'll need to:

• Install the Termux app for running Mergeall command lines
• Install Python for running Mergeall's source code within Termux
• Install Mergeall's source-code package
• Run Mergeall's source code with command lines in Termux to sync your content

We'll walk through all these steps in more detail later in this guide. This is obviously more work than installing and opening a standalone app, but it's the easiest workable option for using Mergeall on Android today. On the bright side, this isn't nearly as difficult as it may sound. This package's helper scripts simplify the required command lines; the necessary installs are really just a few minutes of easy up-front work; and Termux provides a complete and general-purpose Linux command-line environment where Mergeall's source code is fully at home. As we'll learn later, a content sync is as simple as entering this at the Termux prompt:

bash mergeall-android-scripts/mergeall-plug-to-data-update.sh

This is roughly just a dozen taps, with the tab completion stressed repeatedly in this guide (and can be even faster if saved to a file). Although Mergeall's GUI comes with major downsides on Android today, its helper-script commands should still be easy enough to appease even the most chronically command-line phobic.

Grant Termux Extra Storage Permissions

In addition to installing Termux, you must run its utility script termux-setup-storage to create Termux's writeable app-specific folders, and allow it to update files in internal storage and access removable drives more broadly. See this page for more on the Termux setup command. Without it, Termux—and the Python scripts like Mergeall that it runs—cannot read or update most of the folders on your phone's internal storage or removable drives. This makes content-processing programs like Mergeall unusable in Termux until you run this command.

The Before Permissions Picture

To illustrate the Termux permissions story, let's run some Python code within the Termux app. To do this yourself you'd need to install Termux and Python per ahead, but there's no reason to do so yet; this is just a demo. This also assumes some familiarity with pathnames (jump ahead to this section's notebox if you need more background); assumes basic Python knowledge (but you'll probably get the gist of this simple code either way); and uses Termux 0.65 (if your mileage varies in the future, it's probably due to a new Termux, a new Android, or both).

Before the Termux setup command is run, your storage options are severely limited: on non-rooted phones, you can read and update files in the fully private Termux app folder /data/data/com.termux (the one holding your ~ home), but not much more. In Termux's Python, with internal storage at /sdcard, a removable drive named /storage/8BB9-1202, and tracebacks trimmed for space here:

>>> open('/data/data/com.termux/xxx.txt', 'w').write('1234')
4
>>> open('/sdcard/xxx.txt', 'w').write('1234')
PermissionError: [Errno 13] Permission denied: '/sdcard/xxx.txt'
>>>
>>> open('/storage/8BB9-1202/xxx.txt', 'w').write('1234')
PermissionError: [Errno 13] Permission denied: '/storage/8BB9-1202/xxx.txt'
>>>
>>> open('/storage/8BB9-1202/Android/data/com.termux/xxx.txt', 'w').write('1234')
FileNotFoundError: [Errno 2] No such file or directory: '/storage/8BB9-1202/Android/data/com.termux/xxx.txt'

The Termux app-private folder at /data is the standout exception. It supports changes out of the box, but is largely useless for Mergeall, because this folder cannot be accessed outside the Termux app on non-rooted devices. This includes file explorers, photo viewers, and text editor apps, and disqualifies this folder for most Mergeall roles. Because you really don't want to store your content in a location that can be accessed by just one app (and a command-line-only one at that), Termux's scope is very narrow when first installed. (But see the update ahead for exceptions.)

Technically, at this point Termux can also read and update its nested Android/data/com.termux app-specific folders like the one in the last command above, provided you make them manually on either removable drives or internal storage first. But you would have to do so in another app or on a PC: because Termux cannot access the containing folders yet, it cannot build the required folder paths under them. To do more, we have to move on.

The After Permissions Picture

After running the Termux setup command, your storage prospects improve radically. You—and hence Python scripts like Mergeall—have update access to all of internal storage, plus Termux's nested app-specific folders on removable drives:

>>> open('/data/data/com.termux/xxx.txt', 'w').write('1234')
4
>>> open('/sdcard/xxx.txt', 'w').write('1234')
4
>>> open('/storage/8BB9-1202/xxx.txt', 'w').write('1234')
PermissionError: [Errno 13] Permission denied: '/storage/8BB9-1202/xxx.txt'
>>>
>>> open('/storage/8BB9-1202/Android/data/com.termux/xxx.txt', 'w').write('1234')
4

Read access for these folders is essentially the same as the write access shown, except that the failing folder here can be read without error too. That is, after augmenting its permissions with termux-setup-storage, Termux, and the programs like Mergeall that it runs:

• Can read and write files anywhere in the app-private folder (/data), though it's not very useful
• Can read and write files anywhere in internal storage (/sdcard), though its space may be limited
• Can read files anywhere on removable media (/storage/xxxx-xxxx), like SD cards and USB drives
• Can write files on removable media only in its own app-specific folder

As noted, Termux's setup command creates its app-specific folders, like the one used in the last command run above, but you can create these folders manually too (and we will in the setup part of this guide). Running the setup command creates these nested app folders if they don't already exist; this ensures that Termux, and the scripts it runs, have a place to make changes on removable drives.

More Permissions Examples

For a more comprehensive look at accessible paths which we'll skip in this guide, see:

• This Python test script, and its Termux results
• This extended Python updates session
• This system session that proves USB drives follow the same rules as SD cards
• This before-and-after termux-setup-storage Termux session

Nest External-Drive Content in Termux App Folders

After you run the Termux permissions command of the prior section, Mergeall is able to both read and change content anywhere in internal storage, and can read content anywhere on an external (i.e., removable) drive. To allow Mergeall to make changes on external drives, though, your content must be nested in the drives' Termux app-specific folders. In Mergeall-speak, this nesting is not necessary if an external drive will only ever be the from copy in synchronizations, but it is required if the drive will ever play the role of to.

In more-concrete terms, the Mergeall content folders that you will only read on your phone can be located anywhere, but those you will change (i.e., update) must reside at either of the following Android locations:

• Anywhere in internal storage, known as /sdcard (and sometimes /storage/emulated/0)
• Nested in the Android/data/com.termux folder of your external drives' /storage/xxxx-xxxx roots

You can use the first of these for on-phone content copies if you have enough space, but must use the second for content on SD cards and USB drives that you wish to update. Since this is an arguably bizarre constraint, let's take a closer look at its rules.

Content in Internal Storage

First, the simple case: if you're lucky enough to have a phone with enough free space to store your content, internal storage is the easiest approach for on-phone storage, and recommended over SD cards. Internal storage:

• Is usually much faster—5X to 10X faster at writing on one device
• Yields paths that are quicker to navigate and type in commands
• Does not require nesting in the Termux (or other) app-specific folder
• Is generally more accessible to other apps per the notebox ahead
• Is immune to the automatic-deletion peril described in the next section

There's more on internal storage's speed for Mergeall-specific tasks ahead. Internal storage is also growing larger in recent phones, and is the only option for on-phone storage if your device doesn't have a removable card. To store your Mergeall content in internal storage, simply copy it to a folder like the following on your phone; /sdcard really means internal storage, and the MY-STUFF folder name here can be any name you wish:

/sdcard/MY-STUFF

This seems simple—and even PC-like—but internal storage isn't enough by itself. For one thing, even if your on-phone content is in internal storage, you'll still need to sync to and/or from an external USB drive in most use cases. For another, if your phone's available internal storage space isn't large enough for your content (and many today are not), you'll need to store your on-phone content on your device's SD card instead. Since both USB drives and SD cards require nested storage to support updates, let's turn to this model next.

Content on Removable Drives

Removable storage is generally slower, but it's required if your phone's internal space is too small, and is still the ultimate in content control. To store your changeable Mergeall content on external (i.e., removable) storage—including USB drives and your phone's removable SD card—simply copy it to a folder nested in the drive's Termux app-specific folder like this; again, MY-STUFF means your stuff's folder:

/storage/25C9-1405/Android/data/com.termux/MY-STUFF

Android uses IDs like this path's 25C9-1405 to uniquely identify external drives. Your drives' IDs will naturally vary too, and are displayed in popular Android file explorers (including this). More unusual here: on both SD cards and USB drives, content folders like this path's MY-STUFF must be nested six-levels deep in the drives' Termux app-specific folder as shown, if you wish to update them with Mergeall on your phone. This differs from internal storage, where anything under the /sdcard root works.

Termux's explanation for why this is so is told on pages here and here. This restriction on updating removable drives came online in Android KitKat—to the dismay of both users and developers—and has seen multiple later mutations, including app-initiated run-time requests for storage permissions. Termux, and all other Python apps tested, however, remain limited to their own app-specific folders for updates to SD cards and USB drives.

This may seem an odd requirement given that internal storage is wide open to changes, but it cannot be lifted in the context of Termux command lines today, and isn't as grievous in practice as it may sound. For instance, you can still access your content normally in a file explorer (as captured in this SD-card screen shot), and a home-screen shortcut (like this one) provides quick access to your nested content folder; see your file explorer to set one up.

To Nest or Not to Nest

Keep in mind that no nesting is required for content stored in your phone's internal storage (i.e., in /sdcard), because it's freely writeable. Moreover, you can store content at the root of USB drives and SD cards too, but only if you'll only ever read—and never change—it with Mergeall. Unless you're sure that a removable drive's copy won't be the subject of a future Mergeall update, it should be nested in the drive's Termux folder.

Also keep in mind that Android's permission rules have varied much and often in the past, and seem likely to do so again in the future. While some media may grow more accessible and some less, it's impossible to foresee tradeoffs in a platform that seems to change the rules every other Tuesday (more on this in the following notebox).

Retain Your Nested Content if Termux is Uninstalled

Finally, a bold-font caution: per normal Android operation (described by Android docs here and here), uninstalling Termux also silently deletes its app-specific folders on all your drives—including any content folders you've nested there. Unnested content copies, and unnested working folders in internal storage are immune to this, but those nested on your SD card and USB drives to support Mergeall updates may perish with Termux. Because this occurs without clear warning, it's substantially worse than the prior section's caveat on removable-media accessibility, and further bolsters the case for internal storage.

Specifically: on all devices tested, the Android/data/com.termux folders on both the removable SD card (at /storage/xxxx-xxxx) and internal storage (at /sdcard) were automatically deleted when Termux was uninstalled, along with any and all content placed there by a user. On one device, a file explorer also attempted to intervene and delete this content too, but the system apparently beat it to the punch. There is no reason to nest your content in an app-specific folder on internal storage—and a root-level folder on internal storage avoids this issue altogether—but there is no other option for updateable content on removable drives in Termux (or other Python apps we'll meet in Appendix B).

This automatic deletion may sound scary, but it's designed to clear associated space when you're done using an app, and really isn't much more dangerous than file-explorer deletions if expected (and an rm command in the Termux shell can be deadlier still). Before uninstalling Termux, though, be careful to move your nested content if you wish to retain it. In Android, uninstalls imply deletions.

And that's a wrap on the requirements front. In exchange for all this, Mergeall provides a free and proven incremental backup and changes propagation tool for the content you store on Android devices. Though not for everyone, this Mergeall approach requires neither device rooting nor card extraction—both of which can be perilous, if not impossible—and brings robust cloud-free storage management to your phone. If those benefits offset this section's requirements for your usage—or you just enjoy a technical challenge in general—read on for more on using this package's helper scripts to simplify Mergeall runs on Android.

Setting up Your Phone and Drives

Now that you've seen the what's required to use Mergeall on Android in general, this section shifts focus to the how. It provides detailed coverage of steps to follow to prepare your phone and drives for Mergeall use. Here's a quick rundown of these up-front tasks:

1. Install Termux—to run Mergeall commands
2. Run termux-setup-storage—to access content
3. Install Python—to run Mergeall's source code
4. Make your working folder—to run code and view results
5. Install this package and Mergeall—to sync your content
6. Prepare your drives—to add content and meet specs
7. Edit your bash profile—to set paths used by scripts

1. Install Termux

To be able to run Mergeall's Python scripts, you first need a command-line environment in which to run them. The Termux app provides a featured-rich Linux shell for Android that fits the bill; see its homepage for background. To get started, install the Termux app on your phone from the Google Play Store. Mergeall's command-line mode "just works" in Termux, because Android is really Linux under the hood.

2. Run termux-setup-storage

To allow Termux to update both internal storage and its app-specific folders on removable drives, launch the Termux app you just installed on your phone, and run the following command at its prompt (the $ in command listings here is Termux's default prompt; type the text that follows it, and press whatever means "enter" on your keyboard to submit the command):

$ termux-setup-storage

After that, allow access in the popup that appears. Note that you may need to rerun this command later with removable drives connected, if you get permission errors. This command also creates the required /Android/data/com.termux nested folders on all connected media automatically if they do not exist, but you can also make these manually when you're preparing your drives on a PC (as we will a few steps ahead).

If you've forgotten why this permissions step is required (or came here from a search), see the earlier coverage. Also note that Termux commands that update your phone's internal storage from here on in this guide assume that this command has been run; without it, such commands will fail on permission errors.

3. Install Python

Mergeall is coded in the Python programming language. As mentioned earlier, we need to use its source code on Android because its PC executables don't apply, and source code requires a Python interpreter on any platform. To install Python for use in Termux, simply type the following command at the Termux app's prompt on your phone, and answer "y" (yes) when prompted:

$ pkg install python

This gets the latest Python 3.X (version 3.7 at this writing), which works best for Mergeall. If you want to verify the install, type just python at the Termux prompt:

$ python
Python 3.7.1 (default, Oct 21 2018, 18:20:26)
[Clang 7.0.2 (https://android.googlesource.com/toolchain/clang 003100370607242d on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, os, platform
>>> sys.platform, os.name, platform.machine()
('linux', 'posix', 'aarch64')

As usual, a Ctrl+d key combination at Python's >>> prompt gets you back to Termux's $ prompt.

4. Make Your Working Folder

Now that you've got Termux and Python ready to go, it's time to setup a working folder on your phone where you'll run scripts to sync content. This is a simple task, but you'll want to decide up front where you wish to work, to avoid having to repeat steps later.

First, the recommendation: an /sdcard/work folder located in the root of your internal-storage space is an ideal candidate for your working folder, because it's fast, accessible to file explorers and most other apps, changeable by programs run in Termux, and short to type. This folder has the following multiple aliases on devices used in testing, but shorter is generally better on a smartphone:

/sdcard/work
/storage/emulated/0/work

You can work in any writeable folder on your phone, but others come with tradeoffs that make them less than ideal in this role. For example:

• Your Termux home folder known as ~ (which is really files/home in /data/data/com.termux) is writable, but fully app private: its lack of visibility in file explorers and other apps on non-rooted phones can make maintenance tasks like Bluetooth transfers harder.
• The Termux app's nested folder in internal storage (Android/data/com.termux in /sdcard) might seem a natural too, but it's more to type, and will be automatically deleted if Termux is uninstalled (a broad issue discussed earlier).

For this guide, we'll assume and use the recommended working folder in internal storage (and use its shorter /sdcard name), because this is the simplest and most-functional location. To create your folder, run the following in Termux on your phone:

$ mkdir /sdcard/work

You can also create this folder in your favorite Android file explorer—one of the benefits of locating it in internal storage this way. Once you've got your folder set to go, you're ready to populate it with software, per the next step on our list.

5. Install This Package and Mergeall

Wherever you choose to work, unzip (i.e., extract) both this helper-scripts package and Mergeall's source code there for convenience. The helper scripts simply augment Mergeall, so you'll need both packages. You can use the standard Mergeall source-code package, because its core syncing code developed for Mac OS, Windows, and Linux works unchanged on Android—a testament to the benefits of portable software (though the story for Mergeall's GUI in Appendix B won't be nearly as seamless).

For readers newer to Unix systems, the following is one way to install this helper-scripts package on your phone by running commands in Termux itself. First, in Termux, go to the working folder you created in the prior step:

$ cd /sdcard/work

Fetch the zipfile at learning-python.com/mergeall-android-scripts.zip

$ mv ../Download/mergeall-android-scripts.zip . 
$ unzip -d . mergeall-android-scripts.zip

Finally, do similar to install Mergeall's source-code package in the same working folder:

Fetch the zipfile at learning-python.com/mergeall-products/Mergeall-source.zip

$ mv ../Download/Mergeall-source.zip . 
$ unzip -d . Mergeall-source.zip

For an example of the resulting working folder, see this screen capture. This assumes that clicking the links above on your phone makes the zipfiles show up in /sdcard/Download, but that's normal Android public-folder behavior.

Two tips here. First, if you're new to Unix, be sure to use Termux's arrow keys to recall prior commands to edit and resubmit, and its tab completion to let Termux fill in most of these commands for you. The latter makes command lines much easier to type; for instance, a tab after "m" in the sessions above likely expands it to "mergeall-android-scripts.zip," one after "M" fills in the Mergeall full zipfile's name, and "Dow"+tab expands to "Download" similarly. Don't type what you don't have to type—especially on a phone.

Second, there are other ways to achieve most of the above that we'll skip here. For instance, you can fetch files on a PC and beam them to your phone; can use Android file explorers instead of command lines for most steps in this process (the former often call an unzip an extract, and refer to . as the current path); and can even download the zipfiles directly in Termux using curl after a pkg install curl:

$ cd /sdcard/work
$ curl -O https://learning-python.com/mergeall-products/Mergeall-source.zip
$ unzip -d . Mergeall-source.zip

Termux has the wget download command too. Regardless of their sizes, Linux-based devices offer an embarrassment of options.

6. Prepare Your Drives

Preface: As noted earlier, the requirement to format removable drives as FAT32 has been lifted as of Samsung Android 10, but still applies to drives used on older versions of that platform. Given that Android updates are not all mandatory, this yields unavoidable bifurcation; this section's parenthetical updates address the schism.

Depending on the size of your content, the next step may take the longest. To use your content on Android, its storage structure must satisfy the media's filesystem and location requirements we met earlier here and here. In brief:

• All removable drives to be used with Mergeall on Android should generally be formatted with the FAT32 filesystem for best interoperability (till Samsung Android 10), and must nest content in their Termux app-specific folders if you plan to update them with Mergeall. These requirements apply to both removable SD cards used for on-phone storage, and USB drives used as intermediaries between phones and PCs. Setup and content copies for both can be done on your PC—an advantage of removable drives.
• If you opt to store on-phone content on internal storage instead of SD card, it does not require FAT32 formatting or content nesting, but content copies must be done on your phone itself, and your USB go-between drive must still satisfy the prior bullet's rules.

Since this is probably the most convoluted of the steps in this list (and is also required of the GUI alternative in Appendix B), the following sections provide a more detailed breakdown by media type.

To Setup Removable SD Cards and USB Drives

Use your PC to setup the removable drives you'll use on Android. First, you'll need to meet the FAT32 requirement (till Samsung Android 10). If your drive is already using FAT32, you can move on to the next paragraph. Otherwise, start by copying any existing Android content on your drive to a temporary location so it can be restored later; this includes existing DCIM (camera) and Android (app-data) folders, if present. Then, format your drive as FAT32, and restore the prior existing content you saved. If you're new to formatting, see the notebox ahead for tips.

Next, place your content-copy folder on your drive. As noted earlier, if you're sure you will never sync content to a removable drive on Android, this folder can be stored anywhere on the drive, including its root; copy or move it there using your usual tools. To allow Mergeall to update the drive's content, though, its content-copy folder must be placed in Termux's nested app-specific folder, as follows:

• If you did format your drive, create its /Android/data/com.termux folder now if it does not exist, and copy your content folder there using whatever copy technique you normally use on your PC (e.g., file explorer or command line).
• If you did not format your drive, create its /Android/data/com.termux folder now if needed, and either move your content there if it's already located elsewhere on the drive, or copy it there anew.

For the last bullet above, you can move your content folder to the Termux app folder on the same drive with a single command on Unix-like PCs; here's an example on Mac OS (on Linux, use your media's paths; on Windows use drive letters instead of volume names, and move with DOS commands):

~$ mv /Volumes/EXT256-1/MY-STUFF /Volumes/EXT256-1/Android/data/com.termux/MY-STUFF

A caution here: you can move existing content in a file explorer too, but be sure a move in yours is a true move, and not a copy-and-delete. The speed difference can be staggering for large folders, and some Android file explorers may miss the mark on this.

Whether your content was copied or moved, it's now located in the Termux app folder—and four levels down from the drive's root—which can be inconvenient when coding paths and browsing for files. To make this easier to access on your PC, add a link (or shortcut) to it at your drive's root; here's how to do so on Mac OS with a symlink (that looks like this):

~$ ln -s /Volumes/EXT256-1/Android/data/com.termux/MY-STUFF /Volumes/EXT256-1/MY-STUFF

On Android, instead make a home-screen shortcut (that looks like this) to your content folder; a longpress in your file explorer is the usual first step. This makes more sense for a resident SD card than a transient USB drive; but because your content is now six levels deep when viewed on Android—nested in the Android/data/com.termux folder of your drives' /storage/xxxx-xxxx root—the shortcut can save a whopping five taps per access, plus a file-explorer open.

To Setup Internal Storage on Your Phone

If you're able (or required) to store your on-phone content in internal storage, you don't need to reformat as FAT32 (even before Samsung Android 10), or nest your content in the Termux app-specific folder. In fact, you shouldn't—internal storage is already formatted, and nested Termux app folders disappear on Termux uninstalls as noted earlier. Instead, simply create a folder for your content at the root of internal storage, and copy your content there. Both these steps must be done on your phone itself.

To begin, create your content folder in an Android file explorer, or use a Unix mkdir command in Termux itself. For example, the following creates a content folder named MY-STUFF using a command in Termux; as usual, your folder name may vary:

$ mkdir /sdcard/MY-STUFF

Then, copy your content to your phone. For most users, this will employ an intermediary USB drive: copy from PC to USB drive, then from USB drive to phone. A content copy on a USB drive you've already setup for Android in the prior section will suffice, and avoids another step.

A caution here too: on the phone, you can run the copy to internal storage in most Android file explorers, but some fail to copy over file modification times. At least one popular app, for instance, simply stamps every copied file with the current time, which is a showstopper for Mergeall—and no better than the pre-Oreo Android bug (in fact, this may be a lingering accommodation of it).

To make sure times make the move too, use Mergeall's cpall.py bulk-copy utility on your phone instead. There's more on this utility ahead; in short, the copy on Termux looks like the following, assuming the settings we will create in the next and final step are in place. That is, you'll need to return here and run this after setting drive IDs in your bash profile file (USB drive IDs are not available until that drive is prepped and attached, and there's no way to show you this command without them):

$ cd $work
$ py Mergeall-source/cpall.py $plug $data -skipcruft -vv -u > Admin-Mergeall/importlog.txt

Here too, once the content copy finishes, you may wish to create a home-screen shortcut to its new location on your phone. In this mode it's only two levels down in internal storage when viewed on Android—at /sdcard/MY-SYUFF—but the shortcut still saves one tap per access, plus a file-explorer open. A symlink doesn't make sense here, because it would be two levels down too (and may be impossible in any event: a ln -s at Termux's prompt didn't work for /sdcard, despite having write permissions there).

A reminder: even if you use internal storage for on-phone content, you'll also need to setup a USB drive to propagate changes between your PC and phone (or save on-phone changes in general). That removable drive will still require FAT32 formatting for interoperability (till Samsung Android 10), and content nesting for updates as described earlier (here and here), and can be used to initially populate your internal storage with content. Your on-phone content, though, is substantially easier to setup and use.

7. Edit Your Bash Profile

Last but not least, you'll now install and edit the bash profile file shipped in this package. This must happen last, because you may not know the Android names of your drives until they've been prepared and attached to your phone. Formatting, if run, changes the IDs Android uses to identify drives, and you'll need these IDs in this step.

Installing the Profile File

Along with its scripts (and this guide), this helper-scripts package includes a prototype bash profile file named your--.bash_profile. This file has simple settings that provide quick access to your Termux home and working folders, as well as your content-copy directories on your phone and removable drives. You can view this file's contents here; it's mostly documentation (the lines that start with a #).

To use this file, delete the "your--" in its name; edit its path settings as both its comments and the next section describe; and either save this file in your Termux home folder or copy-paste its code into a ~/.bash_profile you already use. Because your Termux home folder is app private, you will normally need to install the profile file there using commands in Termux itself (not a file explorer). Here's the required Unix incantation; this assumes you're working in /sdcard/work and have already unzipped this package there:

$ cd ~
$ cp /sdcard/work/mergeall-android-scripts/your--.bash_profile .
$ mv your--.bash_profile .bash_profile 

For example usage of this file's settings at the Termux prompt, see this Android screen capture (and zoom in and zoom often). The bash profile file is also integral to using the helper scripts here: per the next and final setup section, it defines Android paths in a single location shared by all.

Editing Your Path Settings

To make both commands and edits easier on smartphones, the helper scripts in this package use common content-copy path variables set in your bash profile file. This way, paths need be changed in that file only, and only when you first install this package or wish to use a new drive.

When you do need to change paths, the required edits are simple: in the ~/.bash_profile you created from the prototype file, the settings to change are marked as "EDIT ME." Replace these precoded paths' "xxxx-xxxx" IDs and literal "MY-STUFF" with your own drive IDs and content-folder name, respectively. Again, your drive IDs can be found in Android file explorers like this (make sure to match the case of drive ID letters exactly), and folder name is whatever you wish it to be. Here's one way to edit the profile file in Termux—because it's not visible to other apps, you'll generally need to edit it in Termux itself too:

$ cd ~
$ vi .bash_profile
$ source .bash_profile
/sdcard/work

Special cases: if you opt to use internal storage for your on-phone content copy instead of SD card (per the requirements section earlier), use the profile file's /sdcard/MY-STUFF data-path setting instead. You may need to also edit the scripts themselves to configure for your Mergeall source-code package's install location if it's not in . (the current working directory), but this is not required if Mergeall's package is unzipped to the same folder as this scripts package.

Because removable drives are identified by drive IDs in Android, the paths used by helper scripts will change if a new removable drive is inserted. To reconfigure for a new drive, simply update paths in your ~/.bash_profile with the editing commands above; all Android helper scripts automatically source (i.e., reload) this file on startup, so they always use its current settings. You may also need to rerun Termux's termux-setup-storage when changing to a new drive; if you get permission errors, this applies to you.

Managing Your Content

Running Syncs and Diffs

• The main Mergeall content backup/propagation script, mergeall.py
• Mergeall's diffall.py byte-for-byte comparison utility
• Unix move+link commands to setup your content copies for Mergeall use on Android

Additional scripts run modification time (a.k.a. modtime) adjustment tools for FAT32 drives, and other Mergeall utilities are available as standard Unix command lines. As a bonus, all the Android helper scripts here automatically save logfile output to your working folder's Admin-Mergeall subfolder, using filenames either passed or generated; to see how this works, let's jump right into the available commands.

Syncing Commands

To sync or compare your content: select the Mergeall helper script in this package whose name matches your goal; open the Termux app on your phone; and run the selected script with a bash scriptname.sh command line at the Termux prompt. For example, the following launches a USB-to-phone updates run, assuming this package has been unzipped in your working folder:

$ bash mergeall-android-scripts/mergeall-plug-to-data-update.sh

The bash command just tells Termux to run all the precoded commands in the chosen file, and scripts' filenames imply their action. For instance, plug-to-data in this script's name syncs changes from USB drive to on-phone storage, and update means apply changes to the destination automatically; data-to-plug syncs the other way, and report means report differences only without changing anything.

Worth reiterating: be sure to use shell tab completion to fill in the folder and script names in these commands; in this case "bash m+tab/m+tab" is most of the command. Optional: scripts located in your Termux home folder (~) can also be made more directly executable with a chmod +x scriptname.sh and run with just a scriptname.sh; in Termux this works for home-folder scripts only, not for working folders located elsewhere.

Command Variations

Mergeall's scripts—and hence this package's bash helper scripts that run them—both process files and generate logfiles that detail their actions and results. On Android, script logfile output is automatically saved in your working folder, to subfolder $work/Android-Mergeall, in a file whose name is either:

• Passed as an argument on the script's command line—in filename, for bash scriptname.sh filename
• Generated to be unique with a date-time stamp—in mergeall-date-time.txt, when filename is omitted

Whether the logfile name is passed or generated, the logfiles subfolder is created automatically if needed.

You can also launch scripts with & at the end of the bash command (or use Ctrl+z and bg after it starts) to run in the background, so you can monitor output with a tail -f logfile.txt. For instance, the following session ensures that Termux is in your working directory by running the gowork command defined in ~/.bash_profile; starts a USB-to-phone update run, saving its output using a generated filename; and monitors script progress:

$ gowork
$ bash mergeall-android-scripts/mergeall-plug-to-data-update.sh &
$ tail -f Admin-Mergeall/mergeall-somedate-sometime.txt
output scrolls here
Ctrl+c to exit

Alternatively, the following reports file differences only, saves output to an explicitly named file, and views output only after script completion with the vi text editor (nano also works after an install as noted earlier, you can switch to another Termux session while the merge runs, and you don't really need to gowork again unless you've gone elsewhere):

$ gowork
$ bash mergeall-android-scripts/mergeall-plug-to-data-report.sh ma.txt
$ vi Admin-Mergeall/ma.txt

The diffall.py Utility

Besides its main merge script, Mergeall includes the diffall.py deep-comparison utility, and the helper scripts include a script to launch it. The following, for example, compares content copies byte for byte on Android, monitoring output as it appears in a passed-in filename; there is no from and to ordering for drives here, because this isn't a merge:

$ gowork
$ bash mergeall-android-scripts/diffall-plug-and-data.sh diffs.txt &
$ tail -f Admin-Mergeall/diffs.txt
output scrolls here
Ctrl+c to exit

As a guideline, you should run this comparison script occasionally to ensure that all files in your content copies are completely identical. Though Mergeall's time-and-size comparisons are fast and adequate for your day-to-day syncs, drives can and do go bad eventually. A byte-for-byte check proves the integrity of your content—albeit much more slowly (read the notebox ahead for details).

Syncs and Diffs in Action

For a more graphic look at the Mergeall sync and diff helper scripts in action on Android, check out the Mergeall-run screenshots with and without tails, and the exemplary but heavily redacted Mergeall and diffall.py logfiles. And for a comprehensive test that verifies the approaches laid out in this document, study the files in this folder, especially its test notes.

The FAT Times Fixer Tool

Additional helper scripts in this package run Mergeall's fix-fat-dst-modtimes-data.py to adjust file modification times on FAT32 drives so they are in sync with non-FAT drives after daylight-savings-time (DST) changes. Because FAT32 is recommended for broadest interoperability on Android today (until Samsung Android 10), you may need to run this utility twice a year, especially if you propagate changes to or from a PC or your phone's internal storage.

Without getting into all the details here, on DST rollovers, FAT32's local times may all be one hour off from the UTC-based times used on non-FAT drives. To diagnose this state, run Mergeall in report-only mode (its reams of messages will show every file differing by modtime), and inspect the times of the same file on both drives (they'll differ by one hour). Less than ideal, perhaps, but cross-platform content management is never dull.

Luckily, this state is also easily remedied. To make FAT32 times match other filesystems, simply run the helper script as follows, with data for on-phone storage and plug for USB drive; an -add or -sub followed by a number-of-hours value; and an optional explicit logfile name:

$ gowork
$ bash mergeall-android-scripts/fix-fat-dst-modtimes-data.sh -add 1 logfile.txt
$ vi Admin-Mergeall/logfile.txt

In general, you'll want to run this script after DST rollovers on any FAT32 drives whose times must compare correctly to non-FAT32 drives. Although the possible combinations of filesystems and devices are many, most use cases fall into one of the following categories:

• If your on-phone content is in internal storage, run the fixer script on your FAT32 USB drives only. On devices tested, internal storage uses UTC times just like PC internal drives, so updating an intermediate FAT32 USB drive will bring it in sync with both your PC and your phone.
• If your on-phone content is on a removable SD card formatted as FAT32, run the fixer script on both your FAT32 USB drives (to match your PC's UTC times), and your removable SD card (to match your intermediate drive's fixed times).
• If you use Mergeall for incremental backups only and never propagate content to or from a PC, a FAT32 USB drive must still be adjusted to match UTC-based internal storage or an exFAT-formatted removable SD card, but need not be adjusted to match a FAT32 SD card.
• If you format your removable drives as UTC-based exFAT (despite the earlier advice), they need not be adjusted to match UTC-based internal storage or exFAT-formatted SD cards, but an adjustment is still required to make an exFAT USB drive match a FAT32 SD card.

Adjust as needed for your usage, and see Mergeall's user guide and the fixer script itself for more details. You can also run the bare fixer script directly, either on your PC (to adjust times on an attached USB drive or a removed SD card), or on your phone (to adjust times on an attached USB drive or a resident SD card); its helper scripts like the one shown in the example above simply make on-phone runs easier. For instance, a direct run may look like this (try Mergeall-source for sourcepath in your phone's $work folder, and use \ on Windows):

$ python sourcepath/fix-fat-dst-modtimes.py folderpath -sub 1 > logfile

On caution here: you should generally restart your phone before syncing content to it after a DST rollover. Otherwise, Android may return file times saved earlier in a cache that are now invalid, and will not compare correctly to other files' times. This is arguably another Android bug; without a restart, its net result may be spurious differences or equivalences, either one of which can derail Mergeall syncs.

For example, after adjusting times on a FAT32 USB drive up by an hour at a recent DST rollover (March 2019 in the US), Android initially and erroneously reported all the drive's times as differing from those on UTC-based internal storage; a phone restart sufficed to make the drive's times agree with internal storage as expected. Restarts may or may not be required for on-phone content on removable drives: at the same DST rollover, FAT32 USB agreed with FAT32 SD and exFAT USB agreed with exFAT SD—without restarts. To be safe, though, restart to prevent Android's error-prone caching from causing problems.

Other Mergeall Tools and Techniques

Mergeall's source-code package also includes the cpall.py folder-copy script, and its test subfolder includes the ziptools package which comes with zipfile create/extract scripts. If you elect to host your content on your phone's internal storage instead of a removable card, for example, you might use either of these to copy your content folder in full from a USB drive initially, or export it to USB later.

Though not wrapped by helper scripts here, both of these tools can be run in your $work folder with normal Unix command lines in Termux, using either your bash-profile settings or custom paths. For instance, both of the following command pairs import and export content copies on your phone ("py" is assumed to be aliased to "python" in your bash profile file here, and routing the logfile is up to you in this mode):

$ py Mergeall-source/cpall.py $plug $data -skipcruft -vv -u > Admin-Mergeall/importlog.txt &
$ py Mergeall-source/cpall.py $data $plug -skipcruft -vv -u > Admin-Mergeall/exportlog.txt &

$ py Mergeall-source/test/ziptools/zip-extract.py $plug/../import.zip $data
$ py Mergeall-source/test/ziptools/zip-create.py $plug/../export.zip $data/* -skipcruft

As usual, tab completion helps with input here, and bash scripts, variables, or aliases can shorten such commands. For more on using cpall.py, see the script; for more on ziptools, see its homepage. Termux itself comes with a Unix unzip and a pkg install zip adds a Unix zip; both generally work well, but are naturally subject to the Unix tools' constraints (e.g., file-size limits may apply).

If you prefer a more interactive experience, Mergeall's console mode works on Android too. This mode asks you for input parameters, instead of expecting them in the command line. It doesn't have tab completion for paths, though, so it's a bit more typing or pasting, and may work better with a Bluetooth keyboard. See the screenshots of this mode's setup and results, and start it in Termux with a command like this:

$ py Mergeall-source/launch-mergeall-Console.py

Finally, you can also skip the helper scripts and bash profile altogether and run Mergeall's scripts directly, as described in its user guide. The following Unix command line, for instance, kicks off a Mergeall USB-to-SD updates run on Android manually:

$ python Mergeall-source/mergeall.py \
    /storage/7284-2735/Android/data/com.termux/MY-STUFF \
    /storage/25C9-1405/Android/data/com.termux/MY-STUFF \
    -auto -backup -quiet -skipcruft \
        > ../Admin-Mergeall/mergeall-$(date +%Y%m%d-%H%M%S).txt &

And if you can imagine typing this all on a smartphone's touchscreen, you'll probably appreciate why the helper scripts may be a better idea.

Random Tips

This section collects assorted usage pointers and reminders. Most are intended for users newer to Termux, Unix, or both.

Termux Tips

• Swipe from the left and longpress Keyboard to display the control-key bar
• Use New Session to work in multiple terminals (e.g., edit in one, run in another)
• To edit text files on your phone, use nano, vi, or emacs in Termux, or text-editor apps
• nano is user friendly: run pkg install nano to get, and tap Alt+x for more space
• Longpress cut-and-paste works everywhere (e.g., to copy text to editors or commands)
• The Termux Linux system is full-featured but "prefixed": its files live in ~/files (not in /)
• Termux command lines and filenames are case sensitive: use uppers and lowers well
• Termux's default python install is Python 3.7 today; install python2 for 2.X code (info)
• pkg install X installs Termux tools; pip install X installs Python tools in Termux
• To install Pillow: pip install Pillow, after running this command (omit its "-y" to approve changes)
• For more on using Termux, see its usage pages here, here, and here

Other Tips

• Some, but not all, Android file explorer apps give removable-drive IDs; this does
• Home-screen shortcuts give quick access to content and working folders; see file explorers
• Command-line tab completion and arrow recall are essential on phones; use them
• Internal storage was 5X and 10X faster on tested writing tasks than a microSD card; use it
• Internal storage was also 3X to 4X faster on tasks related to Mergeall specifically; use it
• Network drives with usable paths might provide other Mergeall options (not explored here)
• Syncs between internal storage and removable SD cards work too (but use cases are fewer)
• A Bluetooth keyboard and mouse/touchpad makes smartphone command-lines more humane
• A Bluetooth mouse is nearly required for moves, resizes, and some controls in tkinter GUIs
• For thoughts on whether you should host content on mobile devices, see this earlier post

The Mergeall Termux Wrap-Up

In closing, if this Mergeall approach seems like a lot of work, it's because it is, but it's also a price inherent in maintaining content on Android smartphones with Termux command lines today. In exchange, this scheme provides a fully functional incremental-backup and changes-propagation tool that you can run directly on your Android device to synchronize with external drives—no device "rooting" or card juggling required.

That being said, some users of both Mergeall and Android may prefer to remove cards instead of using command lines, and this may be the only option for others. Still, any scheme that can free your content from the constraints of both closed devices and cloud storage is worth a look. Not only are storage devices much quicker than clouds, they can automatically neutralize the proprietary and even exploitive agendas that seem to underlie much of today's smartphone experience.

However you choose to proceed, taking control of your digital property is worth the effort.

Appendix A: A Brief Primer on Android Pathnames

This appendix introduces and summarizes the folders used in this guide. It's probably more useful to readers who haven't used command lines much in the past, but also serves as a glossary of sorts. This appendix was also updated in 2021 for some findings in Android 11 covered in full off page; this spoils part of Appendix C's surprise, but makes this primer more useful.

Because the locations at which you'll store your content on Android are central to the content-management approaches presented in this doc, it's important to get a handle on these up front. Android identifies a storage location just like every other platform, with a pathname—a string of folder names that reflects folder nesting. Some of Android's naming conventions are unique, though, and some pathnames identify folders with specific roles. Here's a quick rundown on those that play a part in the Mergeall story:

App-private folder

The Termux app used by this guide's approach installs and runs its programs in its Android app-private folder. This folder is fully private to the Termux app: it cannot be accessed by any other apps on non-rooted phones, including file explorers, and is automatically deleted when Termux is uninstalled. This makes it less than ideal for general-purpose storage. The Termux app-private folder is located at this pathname on your phone:

/data/data/com.termux

This folder is really located in internal storage, but is too hidden to be accessed as such. Being Linux-based, Android uses the / character to separate folders in a pathname like this; the one all the way to the left means the device root (the top of the folder-nesting tree), and the left-to-right order of /-separated names gives their nesting. The com.termux part at the end is how Termux is known to Android; other apps have similar dotted-path IDs—and similar app-private folders.

Update: as noted earlier, Termux eventually added support for the Storage Access Framework (SAF), which allows specially equipped file-explorer apps to access the Termux app-private home folder from outside the Termux app. This isn't normal access and subverts Android permissions, but it does allow broader use of this otherwise private folder. For more details, see the later coverage in Android Deltas Sync.

User-home folder

Your Termux "home" folder lives in the Termux app-private folder too, at the following pathname:

/data/data/com.termux/files/home 

This is where you normally "are" when you first open Termux (before you run a cd to go to a different folder). It also goes by the names ~ and $HOME, which are just shell syntax that automatically equate to this folder's full pathname above. Because you can't see this folder outside the Termux app, its scope and utility are limited (sans tools covered in the update above). Because of the way Termux works, though, this is where you'll need to install and edit the bash profile file covered in this guide.

Internal-storage folder

Android allows all apps and their users to access and share available space on the phone's internal storage freely (subject to permission requirements for apps that we'll skip here for space). In fact, this space is often called just shared storage in recognition of its cross-app role. Confusingly, Android docs also sometimes call this storage "external" to distinguish it from storage associated with a specific app, but this doc prefers to use that term for physically removable drives.

Thanks to Android history—and the obfuscation magic of Linux links—there are at least two pathnames that can be used to refer to internal shared storage on most devices (but use the shorter unless you happen to enjoy pointless tapping):

/sdcard
/storage/emulated/0

Some devices provide alternate paths to this space that are less common. And yes, /sdcard really means internal storage, not your removable SD card, for legacy—and even tortuous—reasons we'll omit here. Because internal storage is accessible to every app, and is never automatically removed (short of a factory reset), it's ideal for content storage if your phone has enough space. It's also generally more convenient than your home folder for working in Termux, because you're not limited to Termux tools.

If you opt to store both your content and working folders in internal shared storage for best utility and permanence, their pathnames might look like the following (though /sdcard may have to be something else on some phones, and your stuff's MY-STUFF may naturally vary):

/sdcard/MY-STUFF
/sdcard/work

Update: though uncertain at this writing, Android 11 may impose the same update constraints on internal storage that KitKat imposed on removable media (described earlier), making it less flexible than implied here. See Appendix C for more details.

Update: per the most recent news, most of internal shared storage will still be freely accessible to at least some apps in Android 11, but will also run radically slower than internal app-private and app-specific storage, and app-specific folders within internal storage will be locked down to the owning app (more on these folders ahead).

Removable-drive folders

Removable media—including both SD cards and USB drives—show up under the /storage root on Android when they are attached, with a folder name derived from the drive's unique ID and composed of hex digits. For example, here's one for a USB drive with an ID of 25C9-1405:

/storage/25C9-1405

Your drives' IDs will differ (they're supposed to be unique, after all), and you can find them in some Android file explorer apps after drives have been attached (and the system has a chance to mount it in the folder-nesting tree); see earlier for app pointers. A physical drive's Android ID will change if it's reformatted, but should generally remain constant otherwise. Either way, the drive's path allows programs to perform device-agnostic processing of content over USB.

At least that's the removable-drive story on Samsung Android devices, and it's prone to vary across vendors (it depends on how drives are mounted in the underlying Linux system and vendor extensions). USB drives are also generally available at their ID-based folders in /mnt/media_rw too, for instance, but using that path may require special permissions or phone rooting that are beyond this appendix's scope.

Update: per the most recent news, Android 11 imposes new constraints on USB-drive access, making it less usable than implied here. Though prone to change, general USB paths are revoked, requiring program rewrites to use proprietary Java-based APIs instead of traditional POSIX techniques. See the doc for possible work-arounds.

Removable-drive app-specific folders

Like internal storage, removable drives are accessible to all apps (subject to permission rules we'll skip here too). As covered in this guide, though, Termux does not support updates to a removable drive except in the drive's Termux app-specific folder. This folder isn't app private, but it follows the same pathname format: if you store your content folder on a removable drive, its pathname would look like the following (but use your own drive ID and folder name, instead of 25C9-1405 and MY-STUFF):

/storage/25C9-1405/Android/data/com.termux/MY-STUFF

Every app has one of these app-specific folders if it needs one; they are meant for app-related content that might be too large or inconvenient to store in the /data app-private folder. As discussed in this guide, these app-specific folders are key to using content on removable drives in Mergeall—whether from Termux command lines, or the GUI of the next appendix. Storing content here enables Mergeall updates on removable drives, though this folder's limited scope rules it out for cross-app updates.

App-specific folders like these are also automatically and implicitly deleted when the owning app is uninstalled. This is meant to clean up an app's resources, but makes these folders subpar for storing data. Because of that, you'll generally locate your working folder elsewhere. Your Mergeall content would ideally be elsewhere too, but updatable copies on limited-space phones may have no other option than removable storage, and this requires app-specific folders today.

Internal-storage app-specific folders

For completeness, internal storage hosts app-specific folders too, which follow the same pathname pattern as removable drives, and might look like the following if your content is nested in Termux-specific storage:

/sdcard/Android/data/com.termux/MY-STUFF

This guide downplays this folder, because it's less functional: just like their removable cousins, app-specific folders in internal storage are automatically deleted when the owning app is uninstalled. Internal app-specific folders are also more accessible than app-private space (e.g., file explorers work), but not any more so than the rest of internal storage. Hence, you're normally better off using less-transient locations in internal storage for your content; anywhere else in /sdcard is both safer and just as usable.

When this guide was written, app-specific folders in internal storage were largely lumped together with the rest of internal space, because they did not restrict updates to the owning app—unlike their removable kindred. If this changes (and the following updates suggest it may), it will make app-specific storage folders even less desirable than internal shared storage at large.

Update: though uncertain at this writing, Android 11 may impose new rules on internal app-specific storage, making it less usable than implied here. See Appendix C for more details.

Update: per later news, Android 11 narrows the accessibility of internal app-specific folders so much that they are generally unusable for content managed by more than one app. Sans special new app permissions, files to be processed in both Termux and Pydroid 3 must be kept in shared storage (i.e., /sdcard/other) instead.

Update: Android file-explorer apps ultimately found a loophole which allowed them to provide access to app-specific folders in Androids 11 and 12. Android 13 aims to close this loophole, but the outcome of this is unclear at this writing. See the later coverage in Android Deltas Sync.

Downloaded-content folder

Finally, something simple: documents and packages you download in your web browser normally wind up at the following location in internal storage (or a similarly descriptive locale on other phones):

/sdcard/Download

This is where Mergeall's source-code and android-scripts zipfiles show up if you fetch them from their web pages named in this doc. Because internal storage is completely general and accessible, you can store and use their unzipped content in this folder if you wish (but watch for possible mods to this too in an Android 11 future).

A Storage-Paths Cheat Sheet

As a concise summary of the rules we've just met, Android groups internal and removable storage into accessibility and permanence categories as follows—all subject to the Android device and version variability which defies authoritative docs:

• Internal app-private storage
  • Lives at /data/data/appname
  • Can be accessed normally only by the app
  • Is removed when the app is uninstalled
• Internal app-specific storage
  • Lives at /sdcard/Android/data/appname
  • Has access that varies by Android version
  • Is removed when the app is uninstalled
• Internal shared storage
  • Lives at /sdcard/other
  • Is generally accessible to all apps
  • Is never removed on app uninstalls
• Removable app-specific storage
  • Lives at /storage/xxxx-xxxx/Android/data/appname
  • Has access that varies by Android version
  • Is removed when the app is uninstalled
• Removable shared storage
  • Lives at /storage/xxxx-xxxx/other
  • Has access that varies by Android version
  • Is never removed with app installs

Keep in mind that all of these conventions pertain to Android only. If you use the same removable drive across different platforms, the names of its folders will differ on each. The following, for example, is the pathname for the same folder on Android, Mac OS, Linux, and Windows; all reference the same folder, but you'll need to extrapolate from the Android flavor shown in this guide when using a drive elsewhere:

Android: /storage/25C9-1405/Android/data/com.termux/MY-STUFF
Mac OS:  /Volumes/EXT256-1/Android/data/com.termux/MY-STUFF
Linux:   /media/username/EXT256-1/Android/data/com.termux/MY-STUFF
Windows: D:\Android\data\com.termux\MY-STUFF

And again, also keep in mind that the pathname forms above and used in this guide may vary on some Android devices, and have varied over Android's history. For instance, /storage/extSdCard and /storage/sdcard0 worked in the past too. The paths used here mostly reflect Samsung Galaxy devices using Androids 7 (Nougat) through 10, so you'll have to translate if those on your phone differ. Alas, Mergeall's budget is limited, and stability is not among Android's core virtues.

For more details on Android pathnames, see books, the web, or other Android resources.

Appendix B: The Pydroid 3 tkinter GUI Alternative

The main part of this guide recommends and covers running Mergeall by using command lines in the Termux app, because this approach is both straightforward and reliable, and is completely advertising and payment free. Termux provides a full-featured and general-purpose Linux environment, which can run Mergeall's command-line mode with minimal setup tasks and robust performance. This is not, however, the only Python-on-Android game in town.

Most notably, the Pydroid 3 app has recently acquired some rudimentary though impressive support for using the Tk GUI library—and Python 3.X's tkinter interface to it—in Python programs. This makes it possible to use Mergeall's GUI on Android devices today, along with a host of other existing tkinter software. Less positively, this app's tkinter support also comes with substantial downsides, including both behavioral glitches and unfortunate advertising, that make this approach difficult to recommend in this guide.

That being said, the prospect of running PC GUIs on a smartphone is arguably seductive, and the possibilities that this app raises for Mergeall users merit both fair review and your own judgement. You may or may not prefer Pydroid 3 GUIs to Termux command lines in the end, but this appendix provides enough coverage of the former to help you make a good usage-mode choice.

Mergeall's GUI in Action

First and foremost, Mergeall's GUI does work in Pydroid 3 with only minor changes, and without the command lines required for Termux use. Under this alternative approach, your content syncs will be run in the Pydroid 3 app's IDE (its edit+run GUI) that looks like this, and the Mergeall GUI you will use to configure and run syncs on your phone will look as follows:

Portrait orientation

Start, browse, report, finish, update, finish

Landscape orientation

Start, browse, report, finish, update, finish

Other viewing modes

Non-maximized, Non-maximized, and Non-fullscreen

You can also view the screenshots above in slideshow mode here. Despite the tradeoffs we'll explore in this appendix, it's important to note that running tkinter programs this way is an enormous achievement, given the large disparities between PC and Android platforms.

Pydroid 3 Viewing Modes

Now that you've seen Mergeall's GUI at work, we need to define some terms that impact the way it renders in Pydroid 3. The first two sets of screenshots above were captured with this app's tkinter fullscreen and maximized viewing modes enabled, by setting the first "Tkinter" switch off and the second on in the app's Settings ⇨ System dialog. You can use neither, either, or both of these modes, but their utility can vary per GUI and usage:

• Fullscreen mode saves space by hiding the Android top-of-screen status bar. But it may cause conflicts with Android's notifications pulldowns whenever you tap near the top of the screen (especially when trying to move windows with touch).
• Maximized mode uses space well by expanding the first-launched window to fill the entire display without a window border, and automatically resizes the window to fit the new aspect ratio on each rotation. But it doesn't work for persistent popup windows (they're overlaid and hidden), and is dangerous or unusable for programs that need to process or verify exits (the double-back tap alternative to a window-border's "X" exit button does not run program exit handlers).

Despite their downsides, Mergeall seems to work best with both these modes enabled, because it has no persistent popup windows and doesn't need to catch program exits, and because screen space is a precious commodity on phones in general. Mergeall works without these modes too—and looks like the last set of screenshots above—but has less space, and may bear manual resizing which seems out of place on a phone.

General Usage Notes

As a more general observation, tkinter GUIs in Pydroid 3 can take a bit of getting used to, and some are substantially easier to operate with a Galaxy Note-type stylus or Bluetooth mouse. For instance:

• There is no window manager, apart from the IDE's canvas: multiwindow programs and window resizes are supported, but there is no minimize, and the IDE can run only one program at a time.
• As shipped, rightclicks are drive-by swipes instead of press-and-holds or mouse rightclicks. In programs that support it, either swipe with touch or stylus, swipe with a mouse cursor while holding its left button, or use a control-key + mouse-click combo with a Bluetooth keyboard.
• Some smaller controls are difficult to operate with touch alone, and may benefit much from a stylus, mouse, or two-finger spread/pinch zooms which expand/shrink the view generally and three-finger pans which scroll zoomed views.
• Window resizes and moves for multiwindow GUIs run in non-maximized mode are frustrating to perform with touch, and nearly impossible without a stylus or mouse (or the patience of a saint).
• Some larger GUIs may warrant, if not require, landscape orientation for better fit, maximized viewing mode to avoid display overflow and slides, and navigation bar hiding and/or fullscreen mode for extra space.
• Some larger GUIs may also benefit from lower Pydroid 3 DPI settings that shrink the GUI in full, thereby allowing for more on-screen content. For more details on this option and GUI-to-display fit in general see the related doc's sidebar.
• Though subjective, most tkinter GUIs reflect their PC heritage, and some of their cosmetics may seem foreign to users accustomed to apps designed specifically for smaller Android devices.

On the other hand, tkinter GUIs are surprisingly usable on a phone with practice—and might even be compelling.

Mergeall Pydroid 3 Requirements

The prior section paints a justifiably positive picture, but tkinter GUIs do not come without both procedural requirements and noticeable drawbacks in Pydroid 3. To bring Mergeall's GUI to life on your phone, you must first be willing to accept some constraints unique to Pydroid 3, and others common to Android in general. Specifically, you must:

• Install the Pydroid 3 app on your phone: see its Play Store page
• Fetch and unzip Mergeall's source-code package on your phone from here
• Replace two of Mergeall's source-code files with versions that work around Pydroid 3 issues
• Use Mergeall by opening and running its GUI's source-code file in Pydroid 3's IDE on every sync
• Accept a few remaining Pydroid 3 idiosyncrasies and flaws that cannot be addressed in code
• Pay Pydroid 3's $9.99 freemium fee, or tolerate its rude pay-or-else fullscreen ads
• Satisfy all the general requirements described earlier for the Termux approach

Compared to the Termux approach, you do not need to install Python separately (it's part of Pydroid 3); install helper scripts (you'll use the GUI instead); or run a permissions setup command (Pydroid 3 requests permissions automatically).

However, you still need Mergeall's source-code package (you'll run its GUI's script); must patch two of its code files for Pydroid 3 alone (per the next section); and still have to work with Android version, filesystem, and permission constraints (per their earlier coverage).

While some of the steps above are self-explanatory, and we don't have space for an exhaustive dissection of the rest, the remainder of this appendix explores the most-salient points on the list above.

Required Mergeall Code Changes

Among the requirements just listed, the third may merit explanation most. To use Mergeall's GUI in Pydroid 3, you must first download Mergeall's source-code package from this page to any writeable folder on your phone, and unzip (i.e., extract) it there. Internal storage's /sdcard/Download is the usual target for downloads, and will suffice for both a quick look and later syncs. There's more on Mergeall's source-code package here and here if you've jumped into this appendix randomly.

After installing the Mergeall source-code package, you'll need to replace two of its files, because it's not possible to integrate Pydroid 3's changes into the source-code package today for reasons explained ahead. To do this, simply click the following links to fetch custom versions of two Mergeall source-code files, and move or copy them into the top level of your unzipped Mergeall source-code package folder on your phone. That is, replace the unzipped originals with the versions below:

• launch-mergeall-GUI.pyw
• mergeall_configs.py

If you click these links online, you'll receive reply pages that looks like this; download to your phone by clicking the "save" in these pages' "Raw text" lines. If you're working offline, try a rightclick/longpress save, or visit the folder these links reference. Tip: some browsers may append a bogus ".txt" to the end of a saved ".pyw" file's name; delete the ".txt" manually by renaming if needed.

What Was and Wasn't Changed

The changes applied in these files are minor, and apply to Mergeall's GUI only. Hence, they won't break Mergeall's command-lines mode described earlier in this guide if it's run from the same modified source-code folder. Conversely, these changes will break Mergeall on PCs, so use the files above only on your phone's copy of the source-code package. All changes made are marked with "# ANDROID" comments; search for this string in the custom files if you want to see what differs on Android.

In short, these files' changes address some but not all limitations of tkinter GUIs in Pydroid 3. For example, text labels were trimmed and control fonts shrunk for better fit; some size settings were lowered for non-maximized mode; a workaround was installed for a Pydroid 3 sys module glitch that prevented the GUI from spawning merges; other workarounds were installed for Python webbrowser issues that precluded document views; a button was decolorized to avert color loss; and later releases added dialog-text breaks, folder-choice prefills, and read-only text state, to avoid message truncation, chooser dialogs, and keyboard popups, respectively.

Despite all the changes applied for Android, though, you must still avoid using some fonts in customizations because they trigger crashes prior to Pydroid 3 3.0 and force blind choices later, and should not rotate some devices while Mergeall is running because this can hang or kill Pydroid 3 altogether (until its version 4.0).

We'll take a closer look at these and other glitches in Pydroid 3's tkinter GUIs in a moment. First, though, let's run the GUI.

How to Run Syncs in Mergeall's GUI

Once you've made the prior section's code changes, start Mergeall's GUI by opening its source-code package's file launch-mergeall-GUI.pyw in the Pydroid 3 app's editor, and pressing the editor's big yellow run button.

Mergeall's GUI should appear, and work the same as it does on PCs and as described here—select FROM and TO folders, choose a REPORT or UPDATE, pick a logfile folder and other run options, and press the GUI's GO button to sync. Mergeall messages scroll in the GUI, and are saved to a logfile if you've requested one. As usual, you can open Mergeall's source-code file in either Pydroid 3 itself (see its folder icon), or most Android file explorers (tap the file and open with Pydroid 3).

Configuring Pydroid 3

As noted earlier, Mergeall tends to work best in fullscreen and maximized modes. If desired, select these modes before running the GUI with the Settings ⇨ System dialog, and press your phone's back button twice when you wish to exit Mergeall in maximized mode. While you're in Pydroid 3's Settings, also consider using the Editor's monospace font without line wrapping; Mergeall's code is otherwise difficult to view. Viewing code is not required for the GUI, but you may wish to experiment with configuration-file settings in mergeall_configs.py, and the line between user and programmer is often gray at best.

Update: two related tips here. First, unlike Termux's run-on-demand tool, Pydroid 3's extra-permissions request popup has historically been triggered by a first permission error; if Python scripts generate permission errors in Pydroid 3, it may help to create a new file in the app's IDE and save it to a folder outside Pydroid 3's own, to force the permissions request. Also, there's a new sidebar in the companion tkinter doc with info on setting Pydroid 3's DPI settings; this can improve GUI-to-display fit.

Other Mergeall Tools

This GUI approach works, but only for Mergeall syncs: other Mergeall tools like diffall.py comparisons and cpall.py bulk copies have no GUI, and are still best run with command lines in the Termux shell as covered earlier. Pydroid 3 does have a Terminal option that is capable of running command lines too (and can even run the main Mergeall script with a command typed manually), but it's much more limited than Termux for command-line work. And while you can open and run such scripts in Pydroid 3's IDE, they won't work because the IDE has no way to specify the command-line arguments they require.

As a special case, running Mergeall's GUI to sync to an empty folder has the same effect as cpall.py (and in fact uses the same code), but you won't get as many status messages, and there's no GUI equivalent to diffall.py today. Command-line scripts need command lines (or an IDE that supports them).

Glitches in Pydroid 3 tkinter GUIs

Now that you know how to run a GUI sync on Android, you should also be aware up front that some of Pydroid 3's tkinter support may still not quite be ready for prime time. Running tkinter GUIs on Android is undoubtedly a major accomplishment, but the current implementation still has rough edges that Mergeall code changes cannot entirely smooth. In the interest of full disclosure, this section covers the most prominent technical glitches known to confront Pydroid 3 tkinter GUIs.

Three brief scope notes about the items described here: some don't impact Mergeall, but do impact other programs; some are probably of more interest to programmers than to end users, though the distinction for source-code programs is gray; and some have been repaired as of this doc's latest update, and are marked "(Fixed)" in this list. Also note: if JavaScript is enabled in your browser, a "Show Index" button below toggles a succinct index list summarizing this section's content:

Show Index

GUIs may require redesign for fit

Some PC GUIs may be too large to fit well in the smartphone form factor, and require changes to avoid widget truncation in maximized mode, or horizontal slides in non-maximized mode. In Mergeall, this meant trimming text labels and shrinking control fonts for better fit on smaller screens, and lowering some preset size configurations for non-maximized mode (though they're irrelevant at maximized size).

Getting the fit right for smaller phones can be difficult. For example, because Pydroid 3's tkinter naively clips nearby widgets instead of truncating the text of labels that don't fit, Mergeall's original and descriptive PC-GUI labels were initially unusable on phones of all sizes, and had to be trimmed radically for use on screens as small as 5.5" (and may still be too wide on even smaller displays without new user configs).

More-sophisticated tkinter GUIs may naturally require more redesign work. This is especially true for those that open multiple active windows at once—a desktop-metaphor paradigm at odds with both Pydroid 3's rendering options and Android's one-window-at-a-time app model. While new tkinter GUIs can be designed with such models in mind, many existing GUIs' shrinkability will be firmly limited by usability; Mergeall might someday run on your watch, but it probably shouldn't.

Update: it's worth adding that Android GUIs can also be impacted by global user settings for font size, screen zoom, and display resolution; GUIs have no control over these, and might not accommodate pathological cases. Pydroid 3 tools such as maximized mode and low DPI settings can often improve rendering, but users share some responsibility on this front. For tips on better fitting GUIs to displays, see the related doc's sidebar.

(Partly fixed) Font support is limited and crashy

Font support in Pydroid 3's tkinter is weak enough to ensnare many PC GUIs, including Mergeall's. In general, font-family names courier, times, and helvetica work (and monaco is courier, and arial is helvetica), but most other font family names trigger an immediate and silent hardcrash (including system). Moreover, font sizes work as expected, but italic and bold font styles are simply ignored if used for courier (though they work for other font families). To test this for yourself, edit and run this file in Pydroid 3.

Mergeall avoids crashes with font presets tailored for Pydroid 3 to use only supported fonts, but this constrains functionality, and is error prone in the context of end-user settings. You can still tailor its fonts in the configurations file, but customize with care (and see this file for names to use with platform-specific settings like fonts).

Update: Pydroid 3's 3.0 release in early April 2019 fixed its font crash by coercing any unknown font-family name to helvetica, but its other font limitations described above are still present. That is, unknown fonts no longer crash GUIs silently, but helvetica is substituted blindly, and no additional family names (or courier styles) are now supported. It's a fix of sorts, but the app's tkinter clearly requires more font work.

(Fixed) Phone rotations may kill or hang GUIs

Even more glaringly, on some devices, rotating the screen between portrait and landscape orientation while any tkinter program is running either kills the GUI immediately and silently, or fully hangs the Pydroid 3 app so badly that it must be forcibly killed and restarted. Whether you get a kill or hang depends on which GUI you run, but both effects are clearly less than ideal.

Due to testing constraints, the exact scope of this issue is unknown. It has been observed on Samsung J7, S8+, and Note9 devices running Android Nougat, Oreo, and Pie. On the more recent Samsung Note9, however, rotations in Pydroid 3 tkinter GUIs work fine in both Oreo and Pie after changing the display's screen-zoom setting from Small to either Medium or Large. This doesn't help on the older phones, though, and it's unclear if the older phones may benefit from applying system updates.

It may also be possible to work around this in Mergeall with a Java-based approach that locks orientation or ignores rotation, but this would reduce utility drastically (rotation can be an essential tool when space is limited), and the underlying bug should really be fixed globally for all programs. Ultimately, only Pydroid 3 tkinter GUIs exhibit this defect.

For now: if rotates don't work in tkinter GUIs on your device, either make sure your screen zoom is not Small and your device is up to date; wait for a fix in whatever software plays Brutus to Pydroid 3's rotation Caesar; or use Mergeall's GUI in either portrait or landscape orientation—only—and consider a rotation-locker app like one of these if you can't avoid flipping your phone.

Update: the screen-rotation issue described by this section has been fixed as of Pydroid 3's 4.X release in 2020. Specifically, screen rotations have been verified to work well and without requiring screen-zoom settings when using Pydroid 3 version 4.01 on Samsung Android versions 7 (Nougat) through 10. Please upgrade your Pydroid 3 app as needed to avoid this issue.

(Fixed) Phone rotations during activities misplace or hang GUIs

In the same department as the prior note, changing phone orientation during a spawned activity may cause a running Pydroid 3 tkinter GUI to slide off screen, or hang similarly. Slides happen even on phones that support GUI rotation in general, and have been observed after rotating both Shares and document handlers opened by the GUI (e.g., web-browser help displays). Rotations are clearly a Pydroid 3 tkinter problem area; as a workaround, rotating twice generally restores a slid GUI—but only on devices that support GUI rotation.

Update: the screen-rotation issue described by this section has been fixed as of Pydroid 3's 4.X release in 2020; see the preceding item's update for more details, and update your Pydroid 3 to avoid this issue.

Python's webbrowser works marginally in Pydroid 3's 2.2 release

Python's webbrowser module for opening web pages and other content generally works in Pydroid 3, because the app presets the BROWSER environment variable to an Android am activity-manager command-line template; when spawned, this template opens documents per the Android default-apps model. See the module's documentation for more on why this works, and this script for a demo of the technique. It's similar to open and start commands on Mac OS and Windows, respectively; on Android, chosen viewers run in parallel with their opener, and a back tap returns to the spawner immediately. Documents can also be opened with Python's os.system and the BROWSER setting, but that apes what webbrowser does automatically.

Even so, webbrowser is weak enough on Android to classify as a glitch (and initially warranted disabling widgets that rely on it). For one thing, it requires local-file URLs to use a file:// prefix on Android, unlike any other platform—and silently fails to open if this is absent. For another, Android's default-apps model is not as functional as file associations on other platforms, and reliance on this limits webbrowser's utility: as tested, text files open with few and unconfigurable handler options, and local HTML files open as raw HTML in a text editor instead of being rendered by a web browser. Barring an unknown Android trick, opening HTML in a web browser requires an http:// server URL, which limits HTML documentation to online use only.

In Mergeall's GUI, these limitations impact the "Help" and "Popup logfile?" buttons, which display the user-guide HTML and run-results text, respectively. These widgets were disabled in earlier releases requiring users to open documents outside the GUI, but now work as well as they can. To work around the local-HTML issue, Mergeall now opens its user guide by remote URL; this requires online access, but has the advantage of always displaying the most recent version (HTML documents tend to change as frequently as web browsers). To display logfiles properly, Mergeall now uses file:// prefixes to open per the default apps on your devices; this may sometimes be too limited to be useful, but it's better than disabling the feature altogether.

These workarounds were also adopted on Android by other programs mentioned in this guide, all of which now open their help and other documents from within their GUIs, just as they do on PCs. For coding details, see Mergeall's patched files, as well as those of Frigcal and PyEdit described here and here. Caveat: given webbrowser's limitations, Android users might still be better served by manually opening documents in some cases. Caution: per the prior note, Pydroid 3 tkinter GUIs may slide off screen or hang if phone orientation is changed during a spawned activity; rotate twice to restore the GUI where possible.

Update: this story was rewritten in light of new findings in mid-April 2019, but was modified yet again by new bugs discovered later in Pydroid 3's 3.0 release—a plot element large enough to warrant its own coverage in the next note.

Python's webbrowser is fully broken in Pydroid 3's 3.0 release

Stop the presses—as of early April 2019, Python's webbrowser module is now fully broken on Android due to changes in release 3.0 of Pydroid 3. Document opens using this module work as described above in the prior 2.2 version of Pydroid 3, but they either fail with permission errors or are deliberately disabled in the latest versions of this app, and require additional workarounds applied to programs here on April 21. This complex breakage reflects two separate regressions in Pydroid 3:

• The app's 3.0 release now sets the DISPLAY environment variable in tkinter GUIs, which breaks webbrowser—the module assumes an X11 context, and triggers a permission-error exception when calling tools in Python's subprocess module to query xdg-settings. See the new exception here. This happens during browser registration, and before the BROWSER setting of the next bullet can be applied.
• The app's 3.0 release has changed its BROWSER environment-variable setting to run an odd custom script that special-cases Jupyter file:// documents—but also senselessly refuses to open any other file:// URLs, exiting silently instead. This setting may help Jupyter documents which bypass webbrowser, but renders the module unusable for all others. To see this biased hack for yourself, run echo $BROWSER and more `which mybrowser` in Pydroid 3 3.0's Terminal; it breaks other local-file URLs for the sake of a Jupyter kludge, and appears to do so intentionally.

Luckily, there are at least two coding workarounds for the damage to other programs: manually unset DISPLAY to reenable webbrowser in general and reset BROWSER to its pre-3.0 activity-manager command for file URLs; or avoid this module altogether by spawning the activity-manager command with os.system (again, see this demo). All former webbrowser clients here were updated to use the latter scheme because it leaves environment settings intact for other Pydroid 3 use cases.

Perspective: although this workaround suffices for now, this episode puts the app's viability as a host for application programs like Mergeall on shakier ground. The tale of this bug has been rewritten multiple times here already, and new webbrowser failures introduced by an "upgrade" of Pydroid 3 that preferentially aids one program at the expense of all others seem to tip the scales towards unusability—especially given the lack of notification to paying customers. At some point, prejudiced instability can be neither documented nor accommodated. That point may remain uncrossed, but seems clearly in sight in 3.0.

Update: as of Pydroid 3's 4.0 release, the DISPLAY environment variable is no longer set, but the 3.0 Jupyter-biased kludge remains in place, leaving Python's webbrowser module still broken for all other use cases through Android 10. Fix, please!

Python's multiprocessing doesn't work on Android (but threads do)

While not a Pydroid 3 flaw and irrelevant in Mergeall, Python's multiprocessing parallel-execution module does not work on Android, because required semaphore tools are not supported on this platform (dig deeper here and here). The impact of this will naturally vary per program. PyEdit's Grep command, for instance, depends on this module on PCs to run searches in parallel with the GUI; while it can be easily configured to use fully supported threads on Android instead, doing so makes it impossible for pure Python code to leverage multiple CPU cores—a speed factor whose significance on phones is unclear. Moreover, PyEdit benefited from preexisting code that leverages the API similarity of threads and multiprocessing; other programs may require more extensive recoding.

(Fixed) Pydroid 3 breaks spawns by leaving sys.executable empty

Mergeall's GUI runs the main work of merges as a separate process by design (this lets you run merges outside the GUI too). Oddly, Python's sys.executable is left empty by Pydroid 3, which in turn makes the subprocess.Popen process-spawn code in Mergeall's GUI fail. Mergeall works around this by hardcoding this setting to use the path to Pydroid 3's Python, but this should be fixed in Pydroid 3: this same bug crippled most of the larger programs tested, though some other programs' spawns were also hobbled by the next issue. The workaround's change also applies to the Pydroid 3 app only, which makes it impossible to integrate into base files (more on this ahead).

Update: Pydroid 3's 3.0 release in early April 2019 did not fix its sys.executable bug (this setting is still empty), but it did move its Python executable to a different path—and break Mergeall's initial workaround to this bug in the process. See the new exception here. Mergeall's updated workaround addresses this in its April 19 patch-code release, by using the result of a spawned which python shell command, instead of former or current paths to Pydroid 3 Pythons (see the demo code). The result is release and path agnostic, and will hopefully be immune to future morph in an app which today seems more likely to sprout packaging-structure changes than fix reported bugs.

Update: the Pydroid 3 bug described in this note has been fixed. Pydroid 3 doesn't post a formal changes log, but sometime around its 4.0 release in 2020, this app was fixed to properly set sys.executable to the running Python interpreter's path in tkinter GUIs. This attribute was left empty in earlier releases, necessitating the initial hardcoding and later which work-around of the preceding update. The which work-around is harmless, but is no longer required—except on devices running older Pydroid 3 versions without the fix. For an example of how to apply the work-around only when needed, see the updated demo code.

Two postmortem notes about this glitch. First of all, it was later discovered that Pydroid 3 left sys.executable empty in 2.2 and 3.0 only for programs that included a statement which imported tkinter, and were thus identified as GUIs. This was true even if the import statement was preceded with a # to stub it out as a comment. Programs without an import, or a comment resembling one, had a correct Python path. A naive scan for imports is enough to invoke much magic in Pydroid 3—both dark and otherwise.

Secondly, though it's impossible to be certain without access to Pydroid 3's build system, the prior update's Python move in Pydroid 3 3.0 may reflect either Android NDK changes, or the Android app installer's selection of an ABI-specific variant from a fat APK (i.e., different versions of the compiled code may imply different paths on different devices). Regardless of relocation causes, though, the Python interpreter's path should have been—and now is—reliably stored in sys.executable; else, a variable path cannot be accommodated without work-around heroics like those deployed here. More Android ABI details here and here.

GUIs cannot spawn GUIs

Though it doesn't impact Mergeall, tkinter GUI programs can spawn other tkinter GUI programs in Pydroid 3, but the spawned programs' windows never appear for reasons unknown. Both Frigcal's and PyGadgets' launchers, for instance, start programs that run (e.g., they can write to files) but remain invisible, even though Mergeall's GUI successfully spawns a non-GUI process the same subprocess way after the prior note's patch. This is fatal to programs like PyMailGUI that cannot be used without their launchers and cannot be easily recoded to use top-level windows instead of freestanding processes (more details here).

While its exact cause remains a mystery, this restriction may be inherent in Pydroid 3 if its tkinter structure requires programs to be run by its IDE. Tellingly, tkinter programs also fail if run from Pydroid 3's own Terminal interface, and the IDE seems to scan specially for specific import statement forms that enable GUI support (see the next note). Moreover, AndroWish—the most likely underlying Tk implementation—appears to have similar constraints. If a workaround exists at all, it is hidden so well that it's effectively nonexistent.

Pydroid 3 is picky about imports

Pydroid 3 must see a script's tkinter import or from statement on a line all by itself, or the script's GUI won't work (the required scaffolding appears to be unloaded). This didn't require a change in Mergeall, but the PyToe program in PyGadgets did not open in Pydroid 3 until such a specific statement was added to its top-level script. Though less common, you may also need to add a tkinter import statement to programs that load and run tkinter code from files dynamically; an import in the loaded code won't be enough to enable tkinter support.

Pydroid 3 crashes badly on larger source files

And now for something completely unnecessary: Pydroid 3's editor has a hard—and arguably naive—limit on the sizes of files it can open, and won't even warn you when the limit is breached. Instead, if you open a flawless script whose source file is larger than roughly 256K bytes (or 5K lines), its runs will fail with Android error-message popups stating that Pydroid 3 has stopped; exiting the script's GUI may exit Pydroid 3 altogether; and the Pydroid 3 IDE itself may start failing randomly but regularly with the same messages. It's unclear if bytes or lines kills the IDE, but large files seem its Achilles' heel.

This surfaced when adding code to the main script of the PyEdit program: a 256,898-byte and 5,141-line prior version worked fine, but a 258,031-byte and 5,160-line new version started failing for no reason whatsoever—until wholly unrelated comment lines were removed (a short launcher script's success was the first clue of size issues). Given today's memory capacities, this size limit is absurd; people do write 5k-line program files, and tools do generate them. It also has no workaround; if your script starts failing inexplicably in Pydroid 3 after adding new features, make sure its code file is under 256K bytes (or 5K lines) in size—or write a two-line launcher that reads and execs the file to subvert Pydroid 3's draconian constraint.

Double-back exits don't run exit handlers

Pydroid 3 has a double-back-click option which exits a GUI, and is often required in maximized viewing mode due to the absence of a window-border exit button (the top-right corner's "X" in windows like this). Strangely, double-back exits close GUIs immediately without running their exit handlers. While such silent exits have little consequence to users in Mergeall, they can be catastrophic in other programs that must finalize work on closes, and are at best a use-at-your-own-peril feature.

The PyEdit text editor, for example, has an explicit Quit button that works normally, but has no way to offer users a chance to save file changes on double-back exits. Worse, the Frigcal calendar has no explicit quit widget of its own, and hence no way to save calendars in maximized mode. Non-maximized mode does better with a window-border exit button that runs exit handlers properly, but doesn't use available space as well, and can still be the scene of a silent-but-deadly double-back exit. The lesson here: don't use a double-back in Pydroid 3 unless doing nothing is what you mean to do.

Update: Pydroid 3's 4.0 release in 2020 seems to have changed double-back exits to be single-back exits: just one system-back tap now suffices to close a GUI on Samsung devices tested. This seems inadvertent (it's still dubbed "double" in option menus), and may be a new buglet; either way, exit handlers are still skipped on the exit—perilously.

Newlines may need help

Though not a factor in Mergeall, pressing the return (a.k.a. enter) key on some Samsung on-screen keyboards does not move the cursor to a new line in text widgets of tkinter GUIs in Pydroid 3. For reasons unknown, the return key's newline character never makes the trip between keyboard and widget, which renders general text-input components virtually unusable. This bug may be limited to specific versions of the Samsung keyboard—it's been observed only when using version 3.2 on Android Oreo, and both older and newer versions' return keys work correctly. But where returns fail, they appear to fail in Pydroid 3 tkinter GUIs exclusively.

Luckily, there are at least three ways to work around this glitch if it impacts your device: use a physical Bluetooth keyboard; use Samsung's handwriting input option whose return key works correctly; or simply install an alternative on-screen keyboard whose return key works properly, such as Google's Gboard or Hacker's Keyboard. Both alternative keyboards support multiline-text input in programs like PyEdit and Frigcal, and the latter keyboard also helps with the next glitch on this list.

Keypresses may need help

Though this also doesn't impact Mergeall, and isn't a bug in any of the systems involved, PC GUIs designed to expect a keyboard may require redesign or logistical workarounds when used in Pydroid 3, because the on-screen keyboard may not be readily available. The PyPhoto program, for example, requires keypresses to process photos, and its main display has no text input field to tap—the only way to force most preinstalled on-screen keyboards to appear.

To work around this, use a Bluetooth keyboard, or install an alternative on-screen keyboard on your phone that can be opened on demand. Hacker's Keyboard, for instance, includes a permanent-notification option which allows you to pop up its on-screen keyboard at any time by clicking the app's entry in Android's notifications pulldown. The resulting keyboard is a complete solution in PyPhoto, and helps with newlines, menu shortcuts, and copy/paste in other programs too (see the notes above and below).

Keypresses may need silencing

Of course, too much of a good thing can also be a bad thing: Android's automatic on-screen keyboard popup may be an unwelcome feature in some Pydroid 3 tkinter GUIs. Interacting with any text field mindlessly triggers the popup—unless it's silenced by code changes which mark text fields as uneditable, with Tk's readonly or disabled states.

The Frigcal program, for example, required readonly state in Tk entry widgets to mute annoying keyboard popups on most touch gestures that made the program unusable. Mergeall's GUI wasn't impacted quite as badly, but its message-text area still required read-only state on Android (which means disabled in Tk's text); otherwise, slower swipes to scroll trigger an on-screen keyboard popup—which overlays and obscures the very text you're trying to view. Despite the work, keyboard popups may still appear more often than they should in some Pydroid 3 tkinter GUIs; as usual on Android, your device's back button (and a downswipe where supported) hides an on-screen keyboard that's obscuring important content or GUI components.

Keypresses botch shifts in some programs

A technical keyboard issue not relevant to Mergeall but a potential pitfall to others: Pydroid 3's tkinter also fails to send expected details to general keypress handlers for all physical and on-screen keyboards tested. For instance, the return and backspace keys send tkinter event objects with an empty char attribute and a keysym that identifies the press (as 'Return' and 'BackSpace', respectively). This differs from other platforms but can be worked around with binds to these specific keys (e.g., to <Return>).

Worse, though: shifted keys, like + and * on most keyboards, send two useless keypress events—one for the shift key itself, and another for the unshifted key pressed. A Shift+* sequence, for example, sends a shift followed by the unshifted 8 key only; the * never comes through. Binding to specific keys doesn't help in these cases, and mapping unshifted to shifted keys manually seems implausible across all keyboards and usage contexts. Barring a workaround as yet unknown, this makes shifted keys unusable in some tkinter GUIs run on Pydroid 3.

This glitch seems unrelated to the newlines issue noted earlier (which fails on just one keyboard), and on the upside appears limited to general keyboard handlers (tkinter text-input widgets process shifts correctly). But it impacts programs like PyCalc that allow numeric operator keys to be entered on keyboards. As is, PyCalc fully supports keyboards in its "cmd" popup input field (whose use also seems to oddly correct the shifted-keys bug), but keyboard input for its main display is limited to non-shifted keys on Android alone.

Copy/paste may need help

Android-style copy and paste via longpresses is not available in tkinter GUIs run in Pydroid 3. Instead, to copy and paste text between any text-input widgets: first, highlight text with touch, stylus, or mouse; then either use program-specific widgets (e.g., menu options) where available to copy and paste, or use the PC-like key combinations Ctrl+c to copy and Ctrl+v to paste in keyboards that support these. You'll find support for these keys in both Bluetooth keyboards, as well as alternative on-screen keyboards like Hacker's Keyboard (yes, a pattern is emerging). This works well, but may not be what a typical Android user would expect.

Buttons glitch #1: losing background color

The fundamental Button widget in Pydroid 3's tkinter has a pair of issues that seem to reflect state problems and do not respond to tkinter update calls. Only one impacts Mergeall and the other is narrow in scope, but both merit a heads-up for both users and programmers.

Most noticeably, on every Android device tested, Buttons lose their bg background-color setting when pressed: their color changes to the recessed default, and stays that way until anything else on the GUI is tapped. This can seem to make a Button appear pressed, but is really just background colors being lost. Curiously, some buttons evade color loss (e.g., those that open common dialogs may be immune, suggesting a timing issue), but most colored buttons are impacted.

This colors glitch affects only the "Help" button in Mergeall, but may afflict other GUIs more broadly. The only known workaround requires changing code to either accept the Button's default color, or replace Buttons with Label widgets that use click bindings. Labels retain their background colors when clicked, and their cosmetic and functional difference is trivial (Labels don't give press feedback, but Buttons in Pydroid 3 sometimes don't either). As fix examples, Mergeall eventually chose default colors; the PyCalc calculator program adopted Labels to skirt the issue; and the Frigcal and PyToe programs have used Labels all along (since their ports to Mac OS, which has similar button-color limitations).

Buttons glitch #2: getting stuck

More rarely, Buttons may actually become stuck in a pressed state, in which they remain visibly recessed, and may respond to a future press by triggering two press events at once. Luckily, this glitch appears to be very limited in scope; it has been observed on only one device running Android Nougat (a Samsung S8+) and only occasionally on this device, and has not been seen on any other devices tested—including Samsung devices running Oreo and Pie, and an older Samsung J7 also running Nougat. The flaw trigger on the S8+ is a complete unknown (like much in Android forensics), and there is no known workaround. If your tkinter buttons are truly stuck "on," tap elsewhere in the GUI to "unpress" them.

Android issues cast a wide net

Although not specific to Pydroid 3, the Android issues we've seen besetting Mergeall afflict other tkinter GUIs too. In the Frigcal calendar GUI, for instance, calendar updates are limited to internal storage or app-specific folders on removable drives, and work only on Android Oreo and later due to the same timestamp bug that constrains Mergeall. Violating these rules produces the respective crash scenes here and here. In Frigcal, the timestamp bug kills calendar backups and makes it view-only before Oreo. In other GUIs, Android's bugs and permission rules may limit functionality similarly.

And so on

This appendix cannot provide a comprehensive list of Pydroid 3 tkinter defects—and it's impossible to report them to lists elsewhere, or even reference such lists, when the underlying implementation is unknown—but testing performed so far has uncovered additional pitfalls which merit brief cautions here. In the other-issues bucket, Pydroid 3's tkinter also:

• Munges Unicode symbols in non-maximized window titles, and sometimes on buttons
• Whether feature or anomaly, draws a weird rectangle around the cursor in some contexts
• Truncates instead of scrolling OptionMenu selection lists too large for the display
• Badly truncates text in common dialogs, instead of wrapping it like every other platform
• Posts fullscreen windows at display top where they are apt to argue with Android's status-bar pulldown
• Does not recognize the Alt key for menu shortcuts, in keyboards that support it
• Uses Linux file- and folder-chooser dialogs that can be tedious to use on phones, and may be best avoided
• Fails to dispatch stylus-hover events, though mouse hovers work fine (and fingers don't hover)
• As noted earlier, uses rightclick triggers that seem unusual, and is limited to running one GUI at a time
• As also noted, has a maximized mode more suited for single-window apps than multi-window programs
• May truncate windows—and even flash spastically—if the navigation bar is hidden while running a GUI
• Has been known to fire multiple events for a single gesture
• Requires an occasional restart to restore competence
• And crashes without clear cause more commonly than it should

Probably the best spin on these is symptoms of still-maturing software.

There's more on the other programs noted in the preceding list in the next section. The point here is that while some accommodation is to be expected when translating between PC and smartphone paradigms, issues like those above mean that running tkinter programs in Pydroid 3 is hardly seamless. Even allowing for fixes in its future releases, you'll clearly want to test-drive this app on tkinter programs you'd like to use to see how its support applies.

The Advertising Scourge

Despite the workarounds and glitches of the preceding section, the second to last of the Pydroid 3 requirements listed earlier is really its most regrettable and grievous downside: the free version of Pydroid 3 will flash distracting and distasteful fullscreen ads in your face every few runs, unless and until you pay a required $9.99 freemium fee. To be blunt, this comes off as ransom, and borders on extortion. It may be a norm in Android's culture, but it is rude and unacceptable in Python's, and makes it difficult to recommend Pydroid 3 to Mergeall users today.

Together with the prior section's glitches, this also makes it impossible to integrate the changes required for Pydroid 3 tkinter GUIs into source-code package base files. For example, the coding workarounds that accommodate Pydroid 3's font limitations and webbrowser breakage are specific to Pydroid 3 alone, and the sys.executable fix required for spawns may not work in other apps. Changing portable source-code packages to support just a single app on Android is ethically dubious at best—and just plain wrong when that app employs advertising. Though less convenient, Pydroid 3 today is best supported with the custom source-code files used here.

This guide's broader advice to Mergeall's users is to evaluate its GUI in Pydroid 3 to see if it warrants the fee in your context. It may, especially if you have other tkinter programs you wish to run on Android. As examples of this app's broader scope:

• The smaller tkinter tutorial examples in the book Programming Python, 4th Edition work unchanged in Pydroid 3, like this, this, this, this, and this, though some require path settings for imports that can be forced with Python sys.path appends.
• Other major tkinter applications available at Mergeall's hosting site—including Frigcal, PyEdit, and the PyGadgets utilities PyCalc, PyClock, PyPhoto, and PyToe—also run as shown in Pydroid 3 with minimal changes similar to Mergeall's, but benefit more from tkinter support than Mergeall because they lack command-line modes.

See this page for a separate how-to on running these programs on Android too, along with a summary of this appendix's Mergeall GUI coverage. Taken as a set, programs like these and others may justify the ransom's price, but this choice is yours to make.

Wherever you may land on this issue, there's no denying that Pydroid 3's advertising scheme casts programs like Mergeall in a supporting sales role that's both uncomfortable and unseemly. A free trial period would be much more palatable than obtrusive advertising, and will hopefully be a part of the future of the otherwise-remarkable Pydroid 3 app. Barring that, we'll have to eagerly await the Android debut of free and open-source alternatives for running the free and open-source tkinter.

General Requirements Still Apply

Finally, in addition to all the Pydroid 3 requirements described in the preceding sections, you must also keep in mind that this appendix's GUI approach shares all the other general constraints that this guide notes for the Termux command-lines approach. Specifically, to use Mergeall's GUI in Pydroid 3, you'll still need to:

• Use Oreo 8 and higher only, due to a prior Android timestamp bug
• Use FAT32 eternal drives on some devices, due to a former Samsung exFAT bug
• Use Mergeall's source-code package, because that's what Pydroid 3 runs too
• Grant Pydroid 3 extra storage permissions, requested automatically in this app
• Nest updateable content on removable drives in Pydroid 3's app-specific folder
• Beware of losing content nested per the prior bullet if you uninstall Pydroid 3

In other words, Mergeall's GUI may run on Android, but it's still running on Android.

The Mergeall Pydroid 3 Wrap-Up

So there it is: the good, the bad, and the ugly of using tkinter GUIs like Mergeall's in the Pydroid 3 Android app. To be sure, the tkinter support embedded in this app is nothing short of stunning: it implements nearly all of Tk and tkinter on smartphones, and makes these devices substantially more fun to use, for both programmers and others. You'll have to decide for yourself whether its remaining defects—along with the advertising model of the app that brings it to you—justifies using this support on your phone.

Beyond Pydroid 3's tkinter support, there are additional GUI options for Python programs on Android that we don't have space to cover here. For example, while Termux does not offer tkinter support directly, it does provide some X Windows (a.k.a. X11) GUI support that may provide a path to using tkinter in its Linux environment. For a survey, see its links here and here, and the X11 search here. Even if viable, however, X11 support requires substantial extra setup work that may be too much to ask of users running Mergeall alone.

Other GUI options offer Android-friendlier alternatives, including Kivy, SL4A, and Qt's PyQt and PySide. Such tools allow programs to use Python for their core logic, but wrap it in GUIs which may better match the Android paradigm. Although their program recoding costs are not trivial, these tools become more attractive for tkinter GUIs that must be redesigned for smaller screens if their audiences are ever to include more than the most-stubborn tkinter fans.

At the end of the day, Pydroid 3's tkinter support is admirable and even amazing, but the approach it offers for Mergeall really just swaps command lines for a GUI run in an IDE that requires extra code changes and comes with noticeable seams. Whether you find it easier to sync your content by running a single command in Termux, or opening and running a source-code file in the Pydroid 3 IDE may very well depend on your background. For this guide's money, though, the latter is just enough extra steps to qualify as runner-up.

Appendix C: Android 11—the Future is Closed?

Preface: This appendix was written when the changes it describes were planned for Android Q. After publication, these changes were pushed out to Android R, and Androids Q and R were rebranded 10 and 11—both respectively and confusingly. Android's names have been updated here, but the rest of this story was left intact; its potentially paralyzing changes still loom large for apps and users in Android 11.

And now for the closing cliff-hanger. True to its thrashing-laden history, Android seems poised to radically alter storage-access permissions again and soon, with potentially profound impacts on both Mergeall and the Android platform in general. Although this change is still tentative, this appendix provides a preview of its implications for users of Android and Mergeall in one possible future.

The Permissions Game Changer

This how-to has gone to great lengths to document the permission rules for internal and removable storage that existed at the time of publication, especially in sections here, here, and here. As covered, these rules apply to apps running on Androids KitKat through Pie, of which only Oreo and Pie (and later, 10) are actually useable for Mergeall due to an earlier Android timestamps bug.

Regrettably, some of this guide's coverage of the rules for internal storage—the unremovable storage space in your phone that much Android documentation misleadingly calls external—might be invalidated in the near future. It now appears that Android 10, released in beta form in March 2019 (when it was known as Q), may impose the same crippling constraints on internal storage that Android KitKat earlier imposed on removable media. In brief, this so-called scoped storage proposal would restrict access to internal storage as follows:

• An app would have free access only to files it created in either its own sandboxed folder or a common media folder
• All the files in an app's sandboxed folder would be automatically and silently deleted when the app is uninstalled
• Files in common media folders may survive app uninstalls, but could be freely accessed only by their creating app
• The portable and standard POSIX file API would not be directly usable for content outside an app's own sandboxed folder

The app sandboxed folder referenced in this list is what we've called the app-specific folder (e.g., in /sdcard/Android) in this document, because its internal-storage version can be accessed by other apps today. Apps already have an app-private folder (e.g., in /data/data), but the new rules would limit apps' access to their own app-specific folder in the currently general internal storage.

This model would be the death knell of same-device interoperability: because files in internal storage would no longer be freely usable across multiple apps, the scope of your on-phone content would be limited to a single role. This is fully unusable for modern computing tasks. By sandboxing apps, the proposed Android severs the data workflow at the heart of content development. It also manages to further trample the ethic of engineering freedom promoted for decades by the open-source world; a platform that imposes tools and rules on this scale would be closed by definition.

Some of the new constraints may be mitigated by wildly proprietary and convoluted interfaces, but these interfaces may be much slower than those of traditional POSIX files, and are unlikely to see widespread use in programs born on less restrictive systems and designed to run on multiple platforms. Either way, this change would impact nearly every program that updates shared content on Android devices (including Mergeall), and could very well be an endgame for this platform's broader potential.

For more background on this proposed change than this appendix can provide, read the terse and perpetually morphing announcement here; the pragmatic technical review by an Android file-explorer developer here; and the outrage of Android developers pleading for a reversal here, here, and here (the Mergeall team is guilty of the latter).

Caution: the language at some of those links is understandably explicit. If this change becomes permanent—and KitKat's example suggests it will—it threatens to make a host of existing programs and libraries based on the portable POSIX file standard unusable on Android; render Android's internal storage just as locked down and limited as that in iOS; grossly curtail what users can do with their content on their phones; and effectively kill the platform as an option for general-purpose computing.

Simply put, under the scoped-storage model, Android devices may be reduced to launchers of fenced-in apps, and lose all rights to the title of "PC alternative." Indeed, the better analogy for the proposed Android seems the cartridge-based Game Boy of eras past.

The Impact on Mergeall

That being said, Mergeall itself will likely still work unchanged on Android if these restrictions are imposed, but content on internal storage may have to be structured in the same inconvenient and perilous way as that on removable media. Specifically, it might need to be nested in the updating app's app-specific folder in internal storage. This means the following sandboxed locations for the Termux and Pydroid 3 apps covered in this guide, respectively:

/sdcard/Android/data/com.termux/MY-STUFF
/sdcard/Android/data/ru.iiec.pydroid3/MY-STUFF

Unfortunately, content thus stored could generally be updated only by the sole app whose folder hosts its storage, and would be automatically deleted if that app is ever uninstalled. While these major constraints already apply to removable media today, extending them to internal storage may make the Mergeall program impractical for many or most users on Android.

There's a slim chance that the more permanent Download shared-collection folder in internal storage would still be usable for Mergeall content in the Termux and Pydroid 3 apps. That is, you may be able to locate your content in internal storage just one level deeper, at:

/sdcard/Download/MY-STUFF

If viable, this might avoid the delete-on-uninstall peril of sandboxed folders, and may support access from multiple apps. Also unfortunately, the beta release's documentation of proposed rules for the Download folder suggests that files stored there may be app specific too, though this is described thinly and ambiguously, and will have to await empirical analysis on widely used devices.

In fact, the very definition of "access" to be restricted is unclear in the beta documents, and also awaits verification in the final release. For example, if the new internal-storage rules shadow those of removable media today, apps may still be able to read anywhere in internal storage, leaving a massive security hole completely unfilled by the new model. But shared-storage writes—essential for content creation of any significance—are almost certain to vanish, at least in their current direct form.

As a last resort, the beta also suggests using Android's Storage Access Framework (SAF) to access files created by other apps (e.g., by passing file descriptors down to the native code layer that hosts Python on Android). But this proprietary library is Java based, unwieldly, and slow; may not work for Mergeall's mass-updates use case; and seems a nonstarter for cross-platform Python code run by an app which sets permissions. Python programs use the POSIX file interface on purpose: it makes them portable to every modern computing platform of interest—except the proposed Android. For much code, the extreme requirements of the SAF would make this platform unsupportable.

In the end, Mergeall and similarly general tools may run on Android 10, but they're likely to be much more difficult and dubious to use, if not altogether pointless. Content collections that are accessible to just one app and may be subject to accidental removal would probably not be worth storing on your phone.

The Impact on Android

Naturally, it's impossible to predict if internal-storage restrictions will turn out to be anything more than beta-release drama, and you'll have to stay tuned for this story's conclusion. There is substantial developer pushback at present; apps like Termux and Pydroid 3 that run Python code may implement broader storage permissions and interfaces in response; phone manufacturers may implement internal storage in ways that cannot be foreseen; and an initially proposed compatibility mode promised to retain the current internal-storage permissions of apps used in this guide—at least until those apps targeted later releases, you uninstalled the apps, or you bought a new phone.

What is certain is that deliberately hobbling the Android platform is sure to limit both its users' options, and its devices' scope. Extending proprietary lockdown to internal storage would make portable tools like Mergeall nearly unusable on Android, and constrain the platform to artificially narrow—and even toyish—apps. Such apps might appeal to technically naive masses and generate proportionate advertising revenue in the process, but the closed platform hosting these apps will hold little interest for the skilled development community that produces nontrivial programs used elsewhere.

The net effect would deprive smartphone users of much useful software, and knowingly retard their devices' utility. This would be especially ironic at a time when foldable displays and ample storage hold the promise of expanded roles. Though more subtle and speculative, this change may also help drive smartphone users to cloud-based storage in the face of unworkable on-phone alternatives—boosting subscriptions, trackable content, and single-source control as darker byproducts. In this analysis, Android would remain a carefully crafted arena of exploitation, fueled by greed and justified by red-herring security arguments easily refuted by the platform's own success.

Let's hope that's not what Android's owner has in mind.

Updates: The Rest of the Story

Because Android's internal-storage permissions were still an early and evolving epic when this guide was written, this section has been added to provide updates that arose after publication. Some of its content may be developer oriented, but its twists seem worthy of dramatists of all kinds.

Top

Code

Media

Tkinter

Mergeall

News

Blog

Apps

Input

©M.Lutz