The Android 11 Updates Page

Published Jan-2021, updated Nov-2021

This page hosts updates concerning Android 11's implications for POSIX-based software, especially Python programs and Python/tkinter GUIs run as source-code in the Pydroid 3 and Termux apps. It began as an inline note at the end of this appendix—whose background context is effectively prerequisite reading here—but grew into content worthy of its own stand-alone page somewhere around update #3. Moving this outside the Mergeall-on-Android package also better accommodates future revisions, which seem about as certain in the Android world as death and taxes in the real one.

Main Contents

• Prequel: Crisis Postponed till 11 (Apr-2019)
• A New Hope: All Files Access (Oct-2020)
• The Empire Strikes Back: 11's Damage (Jan-2021)
• Retreat of the Jedi: USB is DOA (Feb-2021)
• The Force Awakens: Fold3 vs Deltas (Oct-2021)

While this page is really just an extension of the Mergeall main doc and its tkinter cousin, you can view its own media as a set in this gallery. Also note that this page mentions Android 12 briefly here and here, but mostly focuses on Android 11; see the later update for additional issues posed for Python programs by Android 12 and later.

Android blinked in late April 2019—or at least called a temporary cease-fire with its users. Per the blog post here, enforcement of the new internal-storage permission constraints described in this appendix has been delayed until Android 11, the successor to 10 expected to appear in 2020 and become common on consumer devices by 2021 (and originally named R).

While welcome news, this really just postpones the tragedy by one year for developers and end users alike. A lot can happen in that year, but people in both camps are left hoping that its manifest won't include a crippling change which reduces Android devices to media-consumption portals, designed to turn private lives into profit centers. The odds are sadly slim in a world where billions seem happy with the Faustian bargain.

Per the latest Android 11 specs and docs in fall 2020, it now appears that programs which require broader read+write access to shared internal storage that survives app uninstalls may be able to acquire it with the new MANAGE_EXTERNAL_STORAGE manifest permission, plus per-app system settings made at users' discretion. Per current sources, some Java-based code may also be required to invoke or manage this permission at runtime, but the code could be self-contained, and seems possible in Python (see Pyjnius and others).

This new option, known as All files access, might be perfect for file managers, virus scanners, and incremental backup/sync programs like Mergeall that allow their users to process arbitrary folders on earlier Androids. It appears to grant access nearly as permissive as before, though other apps' Android/data folders may be off-limits. The new option also remains to be evaluated by the Mergeall team, but you can read about it for now at these pages (while they last; Android developer pages trend towards transience): Android 11 storage; the All files access exception; an FAQ on the subject; and Google Play edicts here and here.

Per the last of those links, using the new permission in apps uploaded to the Play store requires special approval—a point of control which hopefully won't be misused (see the recent Fortnite fiasco), but can be subverted in some contexts by Android app sideloading. Per the other links, the new permission may also make the Storage Access Framework unnecessary, but it's unknown if it will be available in Python-hosting apps like Termux and Pydroid 3, or require creation of fully standalone apps. Either way, apps using All files access cannot be uploaded to Google Play until early 2021, so you'll have to stay tuned for the conclusion to this saga (and appendix).

Thanks to an unexpected auto-update from Samsung, the Mergeall testing team has now had a chance to evaluate the apps and programs described in the main doc under Android 11. This update reflects initial testing with that platform on an unrooted Samsung Galaxy Note20 Ultra, as well as later results on a Pixel (per the update box below). While its findings may or may not apply to other devices, and are prone to change over time, users and developers alike are stakeholders in the Android 11 tale told here.

Contents

To tell this story efficiently and fairly, this section presents its results as a sort of incremental developer journal, which is amended as new facts emerge. Some of its coverage will make more sense if you have a basic understanding of Android storage categories and paths; for a quick intro, try this primer. Taking that background as a given, here is the wreckage uncovered while testing Android 11, and covered in this update:

1. Pydroid 3: Shifted Keys Doubled and Keyboards MIA (not just in 11)
2. Pydroid 3: Help and Other Activity-Manager Popups No Longer Work
3. Termux: Permissions Spontaneously Lost, Requiring a Reinstall
4. Global: USB Drives Can No Longer Be Accessed by Pathname or POSIX
5. Global: Apps That Process Files in Shared Storage Are Radically Slower
6. And So On: Scoped Storage’s Other Victims—and Other Shoe

Overview

The good news is that apps which still target earlier Androids get a temporary opt-out of scoped storage in Android 11, and seem, at least at first glance, to mostly work as they did before. In fact, permissions for such apps—now branded "legacy"—are in some ways more liberal than before. Here's the Android 11 permission stories captured in Termux and Pydroid 3 after their permission-granting steps. Compare these to behavior on Android 10 and earlier presented in the main doc; we'll revisit these results ahead (as well as new limits in file explorers), but these apps now can access microSD-card files more freely, and the root of internal shared storage at /sdcard and its equivalents is still fully usable for content storage and management.

The bad news is that the reprieve is nowhere near complete: as expected, Android 11 does break the Python programs mentioned in this document, in the multiple ways covered below. Most grievously, general access to USB drives has been revoked, even for "legacy" apps; and shared storage has been slowed nearly into oblivion, taking many long-running file-processing tasks with it. On top of this, Android 11 introduces new glitches to tkinter GUIs, trims the utility of tools like file explorers and web browsers, and bizarrely bungles permissions in some apps. In worst cases, the Android 11 breakages have the potential to render Python programs wholly unusable on this platform, in both source-code and standalone-app forms.

Please note up front that findings here mostly reflect an Android 11 upgrade rolled out to Samsung devices in late December 2020 and briefly tested over a few weeks. This upgrade has additional unrelated problems given scant or no coverage here, that seem to call its quality into question in general (e.g., twilight-zone states in file-explorer apps). To be sure, 11 still feels like a beta, and some of its ruinations noted below may be unintentional bugs which may be addressed soon in updates from apps, Samsung, or Android. Breakages which reflect deliberate functionality downgrades, however, may make the Android platform permanently out of scope for POSIX-based tools like Python.

Updates

Also keep in mind that many of this update's threads have evolved over time. While some issues' descriptions are being changed in place to reflect new findings, others are being amended in their blue-shaded end-of-section noteboxes as new info dribbles in. Be sure to see the updates in these boxes—including the first that follows—for full disclosure.

First up on our damages tour is a new GUI glitch that applies to Pydroid 3 alone, and will hopefully be repaired in an app update soon. At present, though, tkinter text-input widgets are unusable in this app, because shifted input keys are doubled, and on-screen keyboards don't survive an app switch. Unlike all other breakages on this list, this one wasn't caused by Android 11 (it wound up being app bugs), but it hampers tkinter GUIs run on 11 nonetheless.

Details

In tkinter GUIs run by this app's current version on Android 11, all text-input widgets erroneously generate double characters when shifted. For example, shift+a generates AA in these widgets, and shift+@ (or just 2 on some keyboards) generates an @2 combo. The stuttered effect is captured in this GUI's screenshot. This key doubling happens on both Bluetooth and on-screen keyboards, and for all on-screen keyboards tested (Hacker's Keyboard, Samsung's and Microsoft's keyboards, and Google's Gboard), and it occurs before any extra keyboards are installed.

In addition, screen taps no longer restore on-screen keyboards, if the Pydroid 3 app is running a tkinter GUI and is temporarily left for another app or context. Switching to and from Recents, for example, leaves the app and its GUI without a keyboard. Your only input recourse is to restart the GUI, or summon Hacker's Keyboard via its permanent notification.

These glitches were observed on both a Samsung Galaxy Note20 Ultra and a Google Pixel 4a, in both cases running Pydroid 3 version 4.01_arm64; it's likely that they impact other devices too. The glitches do not occur on other Samsung devices running earlier versions of both Android and Pydroid 3, and versions 2 and 3 of Pydroid 3 are both free of this bug (though they have repaired others over time).

Prognosis

There are no known user or program work-arounds for these regressions. These are bugs somewhere in the Pydroid/SDLTk/SDL2 stack which implements tkinter in the Pydroid 3 app. They've been reported to Pydroid 3, and seem likely to be fixed soon. At present, though, they make most tkinter programs largely unusable for many or all devices running Android 11. Key doubling can be addressed by pressing Backspace after each shifted key, but it's preclusively painful; and lost keyboards can be avoided by not leaving the app, but this isn't ideal on a device built for multitasking. If you care about running tkinter GUIs on Android in Pydroid 3, stick with earlier releases until fixes appear.

Updates

This glitch's source and scope were initially unclear, but later testing proved that it wasn't caused by Android 11 or limited to Samsung devices. The description above now reflects this, but the box below details these later findings.

Next up is a system-level glitch genuinely new in Android 11, and observed again only in Pydroid 3. In this app alone, activity-manager command lines used to spawn a web-page or text-file view intent no longer work on Android 11; the user's request appears to silently do nothing, as the command fails on permission errors. This will also hopefully be a quick fix for this app, but it renders media views in most tkinter GUIs broken today.

Details

In many of the tkinter GUIs covered in this doc and run by Pydroid 3 on Android, viewer popups for HTML Help files and other text content are generated by spawning Android am start activity-manager command lines with Python's os.system(). These commands launch a view intent that opens files by URL or path (demoed here); negate a Pydroid 3 hack which disables Python's webbrowser module for most files (as described here, the kludge special-cases file:// docs for Jupyter and breaks all others); and work well on Androids Oreo through 10.

As of the Android 11 upgrade, unfortunately, these activity-manager command lines now fail on permission errors on both Galaxy and Pixel devices. The failure is nearly mute, except for a clue left in the adb logcat output that notes "com.android.shell does not belong to uid=xxxx." To most users, though, the net effect is that GUI Help buttons, and other viewer popups, now silently do nothing.

Importantly, this occurs in Pydroid 3 only. On Android 11, the activity-manager command works in the Termux app—at both its terminal and in Python code it runs—but fails in both contexts in Pydroid 3. Here's the scene in Termux and Pydroid 3. Moreover, the same activity-manager command works in adb shell (see the log), and other shell commands (e.g., ls) run correctly in Pydroid 3 from both its terminal and Python code via os.system(), as before; only activity-manager commands fail, and only in Pydroid 3 on Android 11.

Prognosis

There are no known user work-arounds for this regression. It's unclear if it reflects Pydroid 3 permission choices, or a deliberate narrowing of permissions in Android 11. The fact that activity-manager commands work well in Termux, though, strongly suggests the former, which seems more hopeful of an early fix. If this instead stems from a new Android 11 constraint, it might still be addressed by changing programs to use the webbrowser module, but only if Pydroid 3's biased and limiting hack is ever backed out.

Lest this page be accused of picking on Pydroid 3, let's move on to an issue seen only in Termux (so far). Though triggers are elusive, Android 11 has been seen to revoke file-access permissions from the Termux app, leaving it partly disabled. This happens only in 11, and feels like an unintentional glitch that's likely to be repaired soon; but it calls the robustness of Android 11's scoped-storage changes into question.

Details

As described both in the main doc and here, the Termux app requires its termux-setup-storage command to be run to trigger its permission-granting dialog. However, even after this is done—and for reasons disturbingly unknown—Termux may abruptly lose access to every internal-storage folder except its own app-specific/private folders. For example, an ls /sdcard works fine in the app initially, but then stops working suddenly. Just as strangely, reads of non-app folders are disabled after the switch, but writes are still allowed. This makes no sense, though it seems to have been confirmed in a related Termux dev thread about Android 11 storage permissions.

To date, this issue has been observed only in Termux, whose session logs before and after the access demotion capture its impact in full detail. In essence, this app is randomly downgraded to the sort of permissions it might hold without termux-setup-storage, and by default if it targeted Android 11 and its scoped-storage model. The latter seems unlikely for this app's tested version (0.101), given its September 2020 date.

Prognosis

This permission downgrade was seen twice in as many days with no apparent trigger on the Note20—and made a surprise reappearance during later testing on the Pixel 4a. This seems at least anecdotal evidence that the scoped-storage opt-out for legacy apps in Android 11 is still a bit flaky. Probably, the Termux app will require a new release to fully support Android 11's permissions model, though this may sacrifice more in the bargain. As this downgrade has not been seen in Pydroid 3, running Mergeall's GUI in that app suffices as a work-around for that program and others today.

And now a potential endgame for a host of programs. As of Android 11, USB drives can no longer be accessed with their former /storage/xxxx-xxxx drive-ID mount paths, and the POSIX file API. Both worked well for years on unrooted Galaxy devices (specifically, Notes 9 through 20, running Androids Oreo through 10), but are unusable on Android 11. This is true for both Galaxy and Pixel devices, and seems likely elsewhere. If permanent, this makes it effectively impossible for portable POSIX-based programs like Mergeall to sync on-phone content to and from USB external drives, and leaves users who have non-trivial content collections with few good options on these devices, as we'll see here. For such programs and users, phones become "just phones"—which may be the bitter but true point of 11.

Details

This regression is best described by example. Given an attached microSD card and USB flashdrive with IDs 6C2A-1618 and 8145-191C, respectively, here is the Android 11 story in wireless adb; at terminals in Pydroid 3 and Termux, and again in Python code run in Pydroid 3 and Termux. As you can see, internal shared storage and the embedded microSD card are still addressable at /sdcard and /storage/6C2A-1618, respectively; the USB drive which should be at its former /storage/8145-191C is not, and seems the only loser in the brave new Android 11 world.

In contrast, most file explorer apps do provide access to USB-drive files under Android 11; here's the phone and one of its USB folders in Total Commander. Tellingly, though, file explorers now open these files with a jumble of indirect or specialized techniques that seems suggestive of toolset incoherence:

• By content://com.android.externalstorage.* or similar URIs, which invoke content providers to access files indirectly
• By temporary copies in the /storage/emulated/0/.localcache/usb cache folder, which hosts only files opened in this manner
• By /mnt/media_rw/xxxx-xxxx drive-ID mount paths, which are inaccessible everywhere else, including adb

This regression applies to source code run in both Termux and Pydroid 3; standalone Python Android apps built with tools like python-for-android and BeeWare; and any other program on Android that looks for USB content at its former mounted path. While it's relevant only to programs that make use of USB drives (and many do not), this is nearly a show-stopper for POSIX-based backup/sync programs like Mergeall; device-neutral pathnames are crucial to these programs' code, and the speed and convenience of USB drives themselves are essential to these programs' users.

Work-Arounds—or Not

Unlike other Android 11 issues in this update, there are a handful of candidate options for addressing this regression—including MTP, network drives, phone rooting, and microSD-card removal—but none qualify as universal solution, and all fall well short of external USB drives in terms of velocity, accuracy, and simplicity. There just is no substitute for USB drives today when it comes to content usability.

Because such broad claims demand evidence, the Mergeall team explored the handful of common alternatives to direct USB-drive access listed below. All were tested on an Android-11 Note20 Ultra with compare-only Mergeall runs against on-phone and on-PC copies of the same 124K-file, 10K-subfolder, and 178G content collection. This is a large but realistic use case, that's easy to manage with on-phone, on-demand syncs, thanks to Mergeall's incremental processing and fast USB drives. Like USB drives, all the alternatives below allow this content collection's copies to be addressed by pathname—the underlying requirement for both selection by GUI choosers or command-line arguments, and device-agnostic processing by portable POSIX-based programs.

Android's former USB-drive access met the pathname requirement perfectly; allowed Mergeall to be run directly on phones in Python apps; completed the same or similarly sized comparisons on phone in roughly 30 seconds on an Android-10 Note20 Ultra with an external SSD (and about a minute on an Android-Pie Note10+ with a slower flashdrive); and scored high marks on precision, reliability, and ease of use. Assuming this option has been forever locked down by Android 11, let's see how the challengers stack up:

Phone rooting

On phones that support rooting, it may still be possible to directly access a mounted USB drive, at its Android /mnt/media_rw/xxxx-xxxx pathname. If so, this would support on-phone syncs, and retain both the speed and convenience of the former /storage/xxxx-xxxx paths. More broadly, rooting grants super powers to access functionality taken away by vendors who seem to think that you are a security risk to your own phone. That's a viscerally Good Thing.

Regrettably, phone rooting is a complex task, may void warranties, and is out of scope for many or most users. It's also nearly impossible on many phones—including the Samsung US models in the Mergeall team's testing stable. (The later Pixel-rooting update at the end of this section provides more details and a counterargument, but rooting's scope is still drastically limited.) Let's see what else we can try.

Removable SD cards

On phones that support an embedded microSD card, pulling the card out is always a last-resort way to manage your on-phone content. In this mode, you remove the card; bring it up to date by running a sync to it on your PC; reinsert the card in your phone; and, optionally, run a sync on your phone from the card to faster internal storage. This mode can be used on any PC platform, and its on-phone syncs still work well on Android 11 using Mergeall's GUI or command-line interfaces. Moreover, the card is accessible by its usual pathname everywhere, and the syncs on both PC and phone are radically faster than all other options ahead on this list.

Unfortunately, microSD card ports are not available on all Android phones; juggling cards in and out of possibly encased phones this way is both inconvenient and fragile in the extreme; microSD is much slower than most USB 3.X drives available today; and it's possible that microSD cards may soon perish from this smartphone land altogether, rendering this option a nonstarter (at this writing, Samsung has just ditched them on its new S21). Really, using on-phone syncs to avoid card fumbling was a primary motivation for porting Mergeall to Android in the first place; let's see if we can do any better.

MTP over USB

MTP is what normally happens when you plug your phone into a PC by USB cable. It's the native interface provided by Android to access your phone's content from a PC, and was adopted by Android over its earlier USB mass storage, purportedly because it sidestepped filesystem and partition constraints, allowed access controls, and minimized user operations (though it also greatly diminished user options—prescient, perhaps, of Androids to come). MTP works reasonably well for the occasional copy/paste of a few files, and can also be used for broader syncs by accessing the attached phone's mount-point path on PCs.

When used for syncs, your phone is attached to your PC by a USB cable, and acts like an external drive; to sync to or from the phone, Mergeall (or similar) is run on the PC instead of the phone itself, using the phone's MTP pathname. Linux supports this out of the box at path /run/user/1000/gvfs/mtp:*, and mounted MTP drives can be accessed as /Volumes/Phone and drive letters on macOS (f.k.a. Mac OS) and Windows, respectively, with third-party extensions (Windows options were never tested, for reasons about to be disclosed).

Unfortunately, MTP syncs are painfully slow: the Mergeall compare-only run that formerly finished in some 30 seconds on phone by USB drive takes a whopping and fully impractical 53 minutes by MTP on an Ubuntu Linux machine, and about 43 minutes on two macOS boxes. MTP as tested also omitted all .* "hidden" files in content lists on Linux (they'll be removed by syncs from phone to PC, even if important); and one macOS driver returned MTP-origin timestamps a comical 15 years in the future, and did not support phone writes (the former kills syncs in general, and the latter precludes to-phone updates). The macOS driver fixed its timestamps in a later release, and Linux's built-in support always handled both timestamps and writes correctly, but MTP's speed problems makes all this moot.

Plainly, MTP is too slow for prime time. While some users with very small data sets and very large patience might be able to sync by MTP (or copy in full), this mode is clearly a mixed bag, and completely impractical for users who need to sync significant content collections on demand. Given all the hype behind cloud storage, it seems reasonable to assume that wireless options would be more honed and viable than MTP; let's take a look.

Samba servers

Samba is a free implementation of the SMB file-server protocol (which has a substantial Microsoft heritage). Like MTP, you can use Samba from PC clients for both tactical drag-and-drop copies, as well as general syncs based on mount-point paths, though the performance needs of the latter tend to be much more intense. When used for syncs, a Samba server app is run on your phone, your PC connects to it wirelessly as a client (which in this doc means by Wi-Fi on the same local network unless stated otherwise), and syncs are run on the PC using the Samba server's mounted path.

This scheme was unusable out of the gate. On both Linux and macOS, the Samba server proved impractically slow—and similar to MTP in speed. A compare on Linux using a cifs mount at /mnt/landrive, for example, took an unrealistic 36 minutes, and the same test on macOS by Finder connect at /Volumes/Share was so slow that the test was abandoned unfinished at nearly one hour in. These all compare with USB's 30-seconds showing.

The Samba story on Windows 10 was even worse: connection was nearly impossible, due to the combination of that platform's hard-coded ports (it supports only 445 or 13X), and Android's port-number restrictions (it makes ports < 1027 off-limits sans rooting). Third-party port forwarders and SMB mounters may help on Windows, but all those explored seemed subpar in terms of complexity, quality, and status; in fact, the Windows mounter recommended by one Android Samba-server app has ceased to be.

Obviously, Samba's speed and usability constraints disqualify it as a replacement for USB access on Android devices today when it comes to on-demand syncs (except for content collections so small that a simple drag-and-drop would suffice—and probably be simpler by MTP). Can other network-drive options do any better?

WebDAV servers

WebDAV is an open file-server protocol that extends HTTP, and boasts support in Apache. Like Samba, using this for syncs involves running a WebDAV server app on your phone, connecting to the phone wirelessly by PC, and running syncs on the PC using the server's mount path. This option seemed promising at first: the purchased server tested was much faster than both MTP and Samba, turning in a respectable 5-6 minute compare on both macOS (via Finder connect at /Volumes/192.168.1.2) and Linux (via davfs2 mount at /mnt/webdav; Nautilus was slower). Here's a shot of the server and a log from its Mergeall client.

Unfortunately, this scheme still ran much slower than USB's 30-second compares (10-12X, to be precise). Much worse, it yielded data corruption and transfer errors where supported at all. macOS's built-in client support produced glaring data corruption while viewing photos on the phone, choked on munged filenames per the footnote ahead, and was ruled out quickly. Linux's support fared better and transferred data properly when it worked; but it was also complex to configure, and in the end proved too unreliable to use: it sported frequent disconnects from the server during syncs (causing file errors and skips in Mergeall), as well as sporadic segfaults and system errors (of the sort Ubuntu wants to hear about).

Also like Samba, WebDAV is not supported by Windows 10 on non-default port numbers required by Android, and the third-party tool tested to work around this was both slow and inaccurate, in one test completely skipping some of the subfolders in trees being synced. While some Linux failures might be attributed to the server app used, and restarting Wi-Fi on the client PC or server phone sometimes helped temporarily, another WebDAV app on the Play store failed to start altogether after install, and the disconnects pose major usability hurdles.

Clearly, WebDAV's performance and reliability issues mean that it can't stand in for a USB drive on Android today either. While some options were untested and your mileage may naturally vary with any of them, this page's space and patience are now both nearly exhausted, Let's take a quick look at one last option.

FTP servers

When it comes to file transfers, FTP is the classic. In the name of completeness, an on-phone FTP-server app was also tested as an alternative to direct USB connection. When mounted as a PC pathname on macOS, this scheme ran the benchmark comparison in a promising 17 minutes by local Wi-Fi (its port loopback over USB cable was unusable). This is 34X slower than USB's 30 seconds, but might pass in some smaller use cases.

Sadly, though, the FTP server also botched modtimes badly: it returned them as 12:00 AM for every file older than 6 months, and arbitrarily bungled their seconds component for all others. The net effect is lethal to timestamp-based syncs, because every file registers as a difference. In addition, macOS mounted the server as read-only, which precludes to-phone propagation; and this scheme shared the macOS filenames and symlinks munges of the upcoming footnote, which can trigger file loss and errors. By contrast, USB compares correctly, and radically faster.

For FTP's complete results, see this logfile. While they may vary for other apps and platforms untested, finding workable USB alternatives now seems akin to hunting unicorns. Networking is innately slower than USB, and even if you stumble onto a scheme that meets your goals, it will likely come with downsides and complexities that push it well out of scope for all but the most hard-core phone users. For this page, the hunt has reached its end.

Per the brief look, though, FTP, like other server schemes and MTP, might suffice for tactical drag-and-drop copies, but isn't up to the task of larger-archive syncs. The latter demands the speed, accuracy, and simplicity of the direct USB access that's regrettably been yanked away in Android 11—a fate ironically doled out to FTP itself at the same hands.

Footnote: macOS FAT munges vs servers

Testing WebDAV servers also brought to light a subtle potential for filename mismatches when illegal characters are propagated from macOS on FAT32 and exFAT drives, and served elsewhere. This gotcha is covered off page here because it's not limited to Android 11. Briefly, spurious diffs can be caused by the Mac's mapping of illegals to/from Unicode privates (e.g., | ⇔ \uf027), which may not be unmapped by servers running on other platforms. This further clouds the viability of WebDAV for macOS users. Although some Samba servers reverse the munge, its real fix lies in avoiding covert magic that works only in rarified Apple-only contexts. A fat chance indeed (and yes, the pun is a stretch).

Update: this munge was later seen to also affect both MTP and FTP servers used from macOS (filenames mapped by macOS won't match originals when served by these protocols to macOS); and a related macOS munge similarly triggers diffs for Unix symlinks written to Windows-filesystem drives by macOS (symlinks are written as simple files, which won't match the original items' types on macOS when served from elsewhere). In both cases, macOS's silent changes can lead to file loss or errors on later non-USB syncs. See the off-page note's updates for more details.

So there you have it: the desperate and failed quest for alternatives to the direct USB-drive access that Android 11 seems to have rudely ripped away from its users—all for the sake of a mandatory security policy that could have easily been left to the discretion of a user toggle. In sum, MTP and wireless network syncs are largely just as slow and error prone as expected; phone rooting isn't desirable for most users and impossible for many; and microSD cards are a slow and clumsy substitute that isn't available on many phones.

In other words, there is no work-around to be found here. While we can still hope that USB drives will make a comeback, the best bet today for users with non-trivial content collections seems grabbing a phone that supports a removable microSD card. While they last, at least; alas, some Android smartphones seem on the fast track to becoming content islands soon (yes, really).

Prognosis

This regression's source and scope are unknown, but the demise of USB paths has been reported to both Samsung and Android. If it's an unintended bug in either code base, it may be fixed soon. If it's a "by design" narrowing of functionality (and the file-opening behavior of file explorers suggests it may be), it makes Mergeall, and similar POSIX-based tools that rely on USB drives for data transfer, mostly unusable on Android 11.

Such tools have long relied on the fact that USB drives are accessed with the same sorts of pathnames as everything else. Without this device agnosticism, programs would need to allow for USB drives as a special case globally, and expand both file choosers and IO-related code to support them uniquely. This would be a massive rewrite effort required only for Android; would have to use Java-based and Java-biased APIs; and is unlikely to happen in portable software like Mergeall.

For device owners, the demise of addressable USB paths would mean having to either resort to painful microSD-ectomies, use impractically slow syncs over MTP or wireless protocols, or subscribe to cloud-based options eagerly promoted by device vendors but no better than the other alternatives we've explored here. As lamented previously, such narrowed utility would probably qualify as a death knell for many of the broader aspirations of the Android platform.

Updates

Or so ends the initial take. The rest of the story behind the demise of USB access unfolded over time with vendor feedback and new devices, so please see the updates box that follows (and the conclusion it references) for later discoveries and notes.

Finally, yet another breakage with potentially broad and dire consequences. Mergeall compare-only runs have now been clocked at 10 times slower on Android 11 than they were on Android 10 and earlier, and can be a colossal 100X slower on 11 in some contexts. This impacts usage in both Termux and Pydroid 3, and might make POSIX-based data-processing programs impractical on Android alone—whether they manage to work around 11's other constraints or not. And it's not just Mergeall: scans for sizes in all file explorers tested are also much slower on Android 11, and unusably so for large folders. The slowdowns ultimately reflect a deliberate performance downgrade for shared storage (at /sdcard and its ilk), which seems related to a filesystem change, and may trigger other throttling. But the full story requires a short empirical journey to see how deep the damage goes.

Initial Findings: File Scans Crawl

To measure this regression, Mergeall was run on-phone in compare-only mode, with both FROM and TO set to the path of the same large folder in internal storage shared by all apps. Compare-only runs like this primarily exercise per-folder directory listings and per-file stat calls, but do not update files. The test folder held 124K files and 10K subfolders in 178G space (this is the same use case as in the preceding section but without a USB drive), and was identical on the test devices: an Android-11 Note20 Ultra, and an Android-Pie Note10+. As noted earlier, this is a large data set, but it's easily manageable on any device with Mergeall's incremental processing.

The results: when run via Mergeall's GUI in Pydroid 3, this test took an average 104 seconds and 168 worst-case seconds on the Android-11 Note20, but just 15 average seconds and 11 best-case seconds on the Android-Pie Note10. Times for Mergeall command-line runs in Termux for the same folders were just as slow under Android 11—and frequently slower, often checking in at just over 200 seconds. For completeness, a two-year-old Oreo Note9 ran the test in 16 seconds, though the data set differed slightly.

In total, Android 11 posted a 7X average slowdown from earlier Androids in Pydroid 3 on these tests, and often 10X or worse. While the effect is nowhere near the prior note's massive 53 minutes logged for MTP (and cannot account for the slowness of other server options), it still qualifies as extreme. Moreover, the Note20 was already running comparable Mergealls 35-45% faster than the Note10 before the Note20 was forcibly upgraded to Android 11; hence, the true Android 11 sloth is larger than comparisons to the Note10 today suggest, as its slowdown also negates the former speedup.

Nor is Mergeall the only victim: all file explorer apps tested are impacted by this regression too. They appear to also be on the order of 10x slower when scanning folders to compute sizes on Samsung's Android 11. Their API choices are naturally unknown (they may or may not use POSIX calls), and their slowdowns haven't been formally quantified (i.e., app sloth exceeds tester patience). The slowdown is so obviously bad, though, that a Properties scan virtually hangs; this surely must have been noticed before this release pushed its way onto so many customers' phones.

But it gets worse. Because the initial results we just met were so bad, a simple Python test script was coded to isolate and exercise the apparently slower IO operations used by Mergeall compare-only runs; see the test script here. For identical folder trees, testing showed this script to be a horrific 40X slower on Android 11 than Android Pie, which is substantially worse than 11's 7-10X slowdown for the full Mergeall program. In both Termux and Pydroid 3, Android 11 on the Note20 typically took 80 to 100 seconds to complete the script's tree walk, but Pie on the Note10 finished it in just 2 to 2.5.

And it gets weirder. Per later testing (really, fortuitous accident), it turns out that Android 11 runs the test script dramatically faster—sometimes finishing in just 10 seconds—if any other app is interacted with while the app running the script continues in the background. This happens in both Termux and Pydroid 3, and a simple scroll in Settings suffices to speed the running script tenfold. A similar speed boost is seen in Mergeall's Pydroid 3 GUI if it's swiped aside to use other apps while it's running.

And even weirder. On further inspection, it also turns out that a same or similar speedup occurs in the Termux and Pydroid 3 apps' terminals, as well as tkinter GUIs run in Pydroid 3, if you simply keep tapping an app's screen while it's busy with a long-running task—like the test script or a full Mergeall run. Presumably, the screen activity forces task-scheduling logic to count the app as still active and avoids lowering the app's priority, or the CPU's speed. Yes, really:

• Immediately after a 125-second run, the test script noted earlier finished in just 10 seconds with constant screen tapping
• A compare-only Mergeall run by its tkinter GUI in Pydroid 3 took 90 seconds when left alone, and just 20 when repeatedly tapped
• And a fully non-Python file-explorer app's scan to calculate folder space took 120 seconds without taps, but just 20 with them

In truth, taps aren't strictly required—drawing imaginary circles on the app's display while it is running tasks will suffice. And no, it's not April Fool's day yet (though these apps seem perilously close to qualifying as comedy props).

Before you ask: yes, the battery was amply charged when all these observations were made (low-battery mode wasn't in play). And yes, there's proof: here's a Termux screen capture showing a 10X boost; its first two runs were not tapped, but the second two were. Untapped performance sometimes varies, but tapped runs are always much faster, and this phenomenon is system-wide per the last bullet above.

There is no explanation for this behavior yet, despite toggling every related setting on the device (spoiler: see the update ahead). But it strongly suggests that a new task throttling based on screen activity in Android 11 (or Samsung's flavor of it) is a major factor in slowdowns, and a likely accomplice of IO sloth. This throttling may be done in the name of CPU sharing, thermal management, or an overly ambitious battery optimization, and might be triggered by other factors or slowdowns (e.g., IO churn might aggravate it, or vice-versa). Some such app throttling may even be normal on phones, but it has assumed deadly proportions in 11, and makes many long-running tasks nearly unusable.

That being said, POSIX-based IO also appears slower in 11: tapped apps still don't match Pie's speed, and Android has already given notice about a deliberate 2X slowdown (more in the updates ahead). Moreover, the screen-activity throttling is either new or escalated in Android 11; is fairly labeled a bug which starves apps' long-running tasks of performance; and qualifies as a major regression and a likely dealbreaker for numerous programs on this platform.

Unfortunately, the Mergeall team has no tools to pursue this further. Performance analysis is murky at best on Android (especially now that per-app CPU usage seems off-limits to mere mortals), so this will have to await more-privileged parties' attention. Please watch the Android Issue Tracker bug report for updates. Technically speaking, neurotic tapping may be a kind of a work-around for this issue, but a fix in Android 11 (or Samsung's flavor of it) seems a much more rational resolution.

Prognosis

To recap: it appears that there are two major performance regressions in Android 11 that impact Mergeall—both a specific file-processing handicap, and a perhaps broader screen-activity throttling—and no reasonable user or program work-around is known for either. It's unclear where causes lie, but the issues have been reported to both Samsung and Android, and may also be fixed soon: Samsung has already promised a fix for some known Android 11 speed issues later in January, though it's unknown if the fix will address the performance issues covered here.

If the slowdowns are instead intentional consequences of Android 11's new mandates, they will severely paralyze (if not fully preclude) a host of POSIX-based and other software on Android 11, including all programs coded in Python that access files or perform non-trivial tasks normally, whether they are run as source-code or apps. In this scenario, program performance may have been deliberately sacrificed for the sake of Android 11's scoped-storage agenda. Regrettably, Android itself already seems to have suggested as much.

Time will tell if a half century of Unix practice is among the casualties of Android's newly found security obsession. All indicators today suggest it is, though only as long as developers enable such choices by rewarding this increasingly closed platform with grossly proprietary code. Dramatic to be sure, but this alarm badly needs sounding. The road to corporate tyranny is almost always paved with both customer complacency and developer complicity. Until we're told otherwise, though, we can still choose not to lay the bricks.

Updates

Like USB's demise, the rest of this speed-regression story filtered in over time, as vendors replied and new devices were tested. Be sure to see the updates box that follows for full closure on this thread.

And beyond all this, of course, the full impact of Android 11's scoped-storage permission changes remains to be seen, if and when the "legacy" Python-programming apps leveraged in this guide target Android 11. Some apps are already impacted by the new chain-link fences:

Web browsers

Samsung's own web-browser app can no longer open file:// local-file URLs—which reduces users' options for viewing documents and website pages off-line. Firefox (version 84) was the only browser tested that still fully supported such off-line viewing with Android 11 on a Samsung Galaxy device—though not on a Pixel 4a, oddly, where Opera also fails to display on 11 what it can on 10. The web-server issues later in this list compound these lockdowns' impact.

File explorers

On unrooted phones, none of the seven file-explorer apps tested provide access to the app-specific folders of other apps at Android/data—which is where files ironically must be located for usably quick access. Here's the Pixel 4a running Google's own Files. It is unknown if such apps have yet adopted or been granted Android 11's All files access permission, and this is prone to change (see the updates box below for the latest details). But these apps are hog-tied today, leaving a Catch-22 for scoped storage which drastically limits content utility: how do you store and use files in faster folders that file explorers cannot access?

Network servers

As covered earlier, all of the file-server apps tested for wireless syncs were hampered by port-number restrictions and connection issues on Android 11. Web-server apps were untested on 11, and some are marginally usable for local website views on 10 (like this and this), but others are already suffering slow demise on this platform. It's unclear which version of Android began the closures, but its network support is clearly shrinking.

While the Python apps noted in this coverage have yet to emerge from their opt-out havens, their current stay of execution already comes with the visible barbs noted previously on this list: USB access is MIA, storage performance has been dialed back to zero, and glitches abound.

Again, please take this update in its context: it reflects just initial Android 11 testing on two devices; some its findings will likely be clarified or reversed over time; and this page may be updated as fixes and new information come in (watch the top-of-page date for revisions). For the present, though, Python and tkinter programs have clearly been harmed substantially by Android 11. Although a Return-of-the-Jedi update here is still slimly possible, it grows less likely with each new "aw crap" issued by the Mergeall testing team.

In closing this update, it's worth noting for fairness that other platforms also suffer frequent breakages at the hands of today's mosh-pit development model. Witness this site's coverage of recent regressions in Linux here and here; macOS here and here; and Windows here and here. While the ultimate goal behind Android 11's many regressions is a matter of speculation, its morph more clearly jives with the rude and reckless motifs now sadly common across the software field. Android has an especially clear pattern of both mandating languages and tools, and regularly breaking the programs which use them. Developer-friendly not, that is.

Updates

As expected, some apps started rolling out enhanced support for Android 11's storage model after this section was first written. The notebox that follows hits the highlights of these upgrades, though none manage to resuscitate the broader utility of pre-11 Android. Also covered is a recent Android change which attempts to negate former uninstall perils—and largely falls short.

It's official: Samsung has now verified that the removal of access to USB-drive mount paths described earlier is by design in Android 11, and cannot be fixed. This unfairly penalizes legacy apps, and is regrettably a game ender for a host of tools on this platform. There simply is no good alternative to USB for getting substantial content on and off your phone. Especially in concert with 11's other demerits on shared-storage performance and app usability, this is a full stop for content-based activities.

Per its reply, Samsung cannot do anything about USB access in its code base, because it's a new policy mandated by Google as of Android 11. Specifically, /storage/xxxx-xxxx paths can no longer be set; USB drives may be mounted at /mnt/media_rw/xxxx-xxxx, but using that path requires special permissions or a rooted device, and it is not accessible to most programs and users. Hence, general USB access is no more, this isn't limited to Samsung devices, and it's up to Android to fix this if it wishes to be inclusive of programs that depend on traditional processing paradigms.

You can read the full reply about the USB path's demise from Samsung here. Although a definitive reply from Google on this issue is still pending at a parallel post on the Android Issue Tracker, the lack of USB-drive paths on the unrooted Pixel 4a suggests that this extends well beyond Samsung's code, and a new opinion-based decree from Google that wouldn't bend to the leading Android vendor seems unlikely to be reversed for a single developer As of this page's latest update, the decree remains in force in Android 12.

Impact

Assuming its policies stand, Android 11 has knowingly maimed scores of programs that use the portable POSIX API and rely on USB drives for non-trivial content collections, including the Mergeall backup/sync program. Users of such programs are left with snaillike connections, perilous rooting, or delicate microSD-card pulls—none of which is universally practical, supported, or remedial. And developers of such programs are left with broken products or costly rewrites to adopt Android's proprietary, dogmatically enforced, and Java-based APIs. Such rewrites are not an option for Mergeall, and unlikely elsewhere.

Eventually, Android 11's scoped-storage access restrictions described in the main doc may further limit what users can do with their programs, content, and devices. The full scope of these restrictions remains to be seen in Python-programming apps still loitering in an opt-out limbo that's partial at best; but they are already slowing the loiterers to a crawl, crippling apps brave enough to target the new Android, and well on their way to becoming the norm.

Ultimately, though, all it took to torpedo content-management programs like Mergeall was removal of direct USB-drive access. Between this, its poison-pill performance regressions, its newly honed glitches, and its looming access restrictions, Android 11 unfortunately does lock out many or most Python programs, and seems likely to be the sunset of this platform's general-purpose-computing ambitions. Prisons may be secure, but they're still prisons.

Conclusions

On a personal note, the Android 11 auto-update has functionally bricked this developer's phone, and seems the same as an auto maker creeping into your driveway and removing your car's engine. This device today can no longer be used for content backup and view—one of the main purposes for which it was purchased—and the fault squarely lies with the dubious agendas underlying Android 11. Though spun aggressively (and sometimes arrogantly) as enhanced security, the only thing 11's changes seem sure to secure is the revenue stream of cloud providers and mobile advertisers.

The future is of course unknown, and it's possible that Android 11 will still follow Windows 8's lead and reverse course over time. For the present, though, Android may have finally taken its place alongside the iPhone, as a deliberately closed and limited media portal for technically naive consumers. The mass market of exploitable customers which seems the true target of these devices' vendors probably won't care. But some developers will, especially those who craft programs beyond the trivial, and understand that "apps" has always been a derogatory term.

In the end, this page's takeaway is a simple one: they're only "just phones" if you let the auto-update happen. If you care about using Mergeall, tkinter GUIs, or your device in general, you should consider avoiding Android 11 and later unless and until this platform's many issues are addressed. This page may be revised again if improvements arise, and hopes they will; Android devices have great potential yearning to be unlocked.

But for now, this humble Python hacker won't be moving ahead with new development on this platform, and will be flashing a ROM to revert the Note20 to Android 10 very soon (learn how here, here, and here). A retreat, perhaps, but it's also a lot more usable, and a lot less fascist.

But that's not how the story ends.

In August 2021, Samsung released its newest flagship device, the foldable-screen Z Fold3, without support for a microSD card. This compounded the usability issues created by Android 11's removal of USB access for POSIX programs. Nevertheless, the Fold3 was compelling enough to inspire development of a new way to sync content onto its intentionally closed hardware.

The fix requires a proxy drive, and a new Mergeall script that creates change sets (a.k.a. deltas) which can be applied separately. It also requires a manual copy step along the way, and is not as direct or simple as the two-merge model that's used by command or GUI prior to Android 11. It does, however, qualify as a work-around for both Android's and Samsung's rude closures, and has been automated by scripts you can fetch and run on your devices. In fact, to sync any Android 11 or later device, you'll simply run one script on your PC, copy a deltas zipfile by file explorer, and run another script on your phone.

Because this document has grown long in the tooth, the rest of this Fold3 and deltas-fix sequel has been moved off page. Follow the continuing saga—and fight back against customer-antagonistic agendas—by clicking this link to visit the separate page:

Top

Media

Mergeall

Tkinter

News

Blog

Apps

Input

©M.Lutz