The Android 11 Updates Page

Published Jan-2021, updated Nov-2021

This page hosts updates concerning Android 11's implications for POSIX-based software, especially Python programs and Python/tkinter GUIs run as source-code in the Pydroid 3 and Termux apps. It began as an inline note at the end of this appendix—whose background context is effectively prerequisite reading here—but grew into content worthy of its own stand-alone page somewhere around update #3. Moving this outside the Mergeall-on-Android package also better accommodates future revisions, which seem about as certain in the Android world as death and taxes in the real one.

Main Contents

While this page is really just an extension of the Mergeall main doc and its tkinter cousin, you can view its own media as a set in this gallery.

The TL;DR Summary Box

As of early 2021, Android 11 is known to have revoked general access to USB drives; made programs that process files in shared storage glaringly slow; spawned new glitches in the Pydroid 3 and Termux apps; and reduced the functionality of apps like file explorers and web browsers. In fact, it's so bad that the Mergeall team opted to roll devices back to Android 10 after just a few weeks of testing. Your opting may vary, but Android seems destined for either computing irrelevance, or a Windows-8ish mea culpa that reverses 11's debilitating agendas. For more spoilers, see the alternative summary off page, or jump ahead here to the postscript. Better yet, read on. Given the hold these gadgets have quickly taken on the human species, this stuff does matter.

Prequel: Crisis Postponed till 11 (Apr-2019)

Android blinked in late April 2019—or at least called a temporary cease-fire with its users. Per the blog post here, enforcement of the new internal-storage permission constraints described in this appendix has been delayed until Android 11, the successor to 10 expected to appear in 2020 and become common on consumer devices by 2021 (and originally named R).

While welcome news, this really just postpones the tragedy by one year for developers and end users alike. A lot can happen in that year, but people in both camps are left hoping that its manifest won't include a crippling change which reduces Android devices to media-consumption portals, designed to turn private lives into profit centers. The odds are sadly slim in a world where billions seem happy with the Faustian bargain.

A New Hope: All Files Access (Oct-2020)

Per the latest Android 11 specs and docs in fall 2020, it now appears that programs which require broader read+write access to shared internal storage that survives app uninstalls may be able to acquire it with the new MANAGE_EXTERNAL_STORAGE manifest permission, plus per-app system settings made at users' discretion. Per current sources, some Java-based code may also be required to invoke or manage this permission at runtime, but the code could be self-contained, and seems possible in Python (see Pyjnius and others).

This new option, known as All files access, might be perfect for file managers, virus scanners, and incremental backup/sync programs like Mergeall that allow their users to process arbitrary folders on earlier Androids. It appears to grant access nearly as permissive as before, though other apps' Android/data folders may be off-limits. The new option also remains to be evaluated by the Mergeall team, but you can read about it for now at these pages (while they last; Android developer pages trend towards transience): Android 11 storage; the All files access exception; an FAQ on the subject; and Google Play edicts here and here.

Per the last of those links, using the new permission in apps uploaded to the Play store requires special approval—a point of control which hopefully won't be misused (see the recent Fortnite fiasco), but can be subverted in some contexts by Android app sideloading. Per the other links, the new permission may also make the Storage Access Framework unnecessary, but it's unknown if it will be available in Python-hosting apps like Termux and Pydroid 3, or require creation of fully standalone apps. Either way, apps using All files access cannot be uploaded to Google Play until early 2021, so you'll have to stay tuned for the conclusion to this saga (and appendix).

The Empire Strikes Back: 11's Damage (Jan-2021)

Thanks to an unexpected auto-update from Samsung, the Mergeall testing team has now had a chance to evaluate the apps and programs described in the main doc under Android 11. This update reflects initial testing with that platform on an unrooted Samsung Galaxy Note20 Ultra, as well as later results on a Pixel (per the update box below). While its findings may or may not apply to other devices, and are prone to change over time, users and developers alike are stakeholders in the Android 11 tale told here.

Contents

To tell this story efficiently and fairly, this section presents its results as a sort of incremental developer journal, which is amended as new facts emerge. Some of its coverage will make more sense if you have a basic understanding of Android storage categories and paths; for a quick intro, try this primer. Taking that background as a given, here is the wreckage uncovered while testing Android 11, and covered in this update:

  1. Pydroid 3: Shifted Keys Doubled and Keyboards MIA (not just in 11)
  2. Pydroid 3: Help and Other Activity-Manager Popups No Longer Work
  3. Termux: Permissions Spontaneously Lost, Requiring a Reinstall
  4. Global: USB Drives Can No Longer Be Accessed by Pathname or POSIX
  5. Global: Apps That Process Files in Shared Storage Are Radically Slower
  6. And So On: Scoped Storage’s Other Victims—and Other Shoe

Overview

The good news is that apps which still target earlier Androids get a temporary opt-out of scoped storage in Android 11, and seem, at least at first glance, to mostly work as they did before. In fact, permissions for such apps—now branded "legacy"—are in some ways more liberal than before. Here's the Android 11 permission stories captured in Termux and Pydroid 3 after their permission-granting steps. Compare these to behavior on Android 10 and earlier presented in the main doc; we'll revisit these results ahead (as well as new limits in file explorers), but these apps now can access microSD-card files more freely, and the root of internal shared storage at /sdcard and its equivalents is still fully usable for content storage and management.

The bad news is that the reprieve is nowhere near complete: as expected, Android 11 does break the Python programs mentioned in this document, in the multiple ways covered below. Most grievously, general access to USB drives has been revoked, even for "legacy" apps; and shared storage has been slowed nearly into oblivion, taking many long-running file-processing tasks with it. On top of this, Android 11 introduces new glitches to tkinter GUIs, trims the utility of tools like file explorers and web browsers, and bizarrely bungles permissions in some apps. In worst cases, the Android 11 breakages have the potential to render Python programs wholly unusable on this platform, in both source-code and standalone-app forms.

Please note up front that findings here mostly reflect an Android 11 upgrade rolled out to Samsung devices in late December 2020 and briefly tested over a few weeks. This upgrade has additional unrelated problems given scant or no coverage here, that seem to call its quality into question in general (e.g., twilight-zone states in file-explorer apps). To be sure, 11 still feels like a beta, and some of its ruinations noted below may be unintentional bugs which may be addressed soon in updates from apps, Samsung, or Android. Breakages which reflect deliberate functionality downgrades, however, may make the Android platform permanently out of scope for POSIX-based tools like Python.

Updates

Also keep in mind that many of this update's threads have evolved over time. While some issues' descriptions are being changed in place to reflect new findings, others are being amended in their blue-shaded end-of-section noteboxes as new info dribbles in. Be sure to see the updates in these boxes—including the first that follows—for full disclosure.

It’s Not Just Samsung: 11 Busts Pixels Too

After this page was written, all the glitches it describes were verified to also exist on a Google Pixel 4a running Android 11. On the Pixel too, USB-drive paths are nowhere to be found without rooting, shared storage is radically slower than app-specific/private storage, and all other issues are present. Hence, these are not unique to the Samsung Galaxy auto-upgrade casualty, and Android 11 has the same defects elsewhere. The descriptions that follow have been updated accordingly, and augmented for findings unique to the Pixel (e.g., rooting). As we'll see, Mergeall's GUI runs on both the Note20 and the Pixel, but it's not very useful with Android 11 on either.

Pydroid 3: Shifted Keys Doubled and Keyboards MIA (not just in 11)

First up on our damages tour is a new GUI glitch that applies to Pydroid 3 alone, and will hopefully be repaired in an app update soon. At present, though, tkinter text-input widgets are unusable in this app, because shifted input keys are doubled, and on-screen keyboards don't survive an app switch. Unlike all other breakages on this list, this one wasn't caused by Android 11 (it wound up being app bugs), but it hampers tkinter GUIs run on 11 nonetheless.

Details

In tkinter GUIs run by this app's current version on Android 11, all text-input widgets erroneously generate double characters when shifted. For example, shift+a generates AA in these widgets, and shift+@ (or just 2 on some keyboards) generates an @2 combo. The stuttered effect is captured in this GUI's screenshot. This key doubling happens on both Bluetooth and on-screen keyboards, and for all on-screen keyboards tested (Hacker's Keyboard, Samsung's and Microsoft's keyboards, and Google's Gboard), and it occurs before any extra keyboards are installed.

In addition, screen taps no longer restore on-screen keyboards, if the Pydroid 3 app is running a tkinter GUI and is temporarily left for another app or context. Switching to and from Recents, for example, leaves the app and its GUI without a keyboard. Your only input recourse is to restart the GUI, or summon Hacker's Keyboard via its permanent notification.

These glitches were observed on both a Samsung Galaxy Note20 Ultra and a Google Pixel 4a, in both cases running Pydroid 3 version 4.01_arm64; it's likely that they impact other devices too. The glitches do not occur on other Samsung devices running earlier versions of both Android and Pydroid 3, and versions 2 and 3 of Pydroid 3 are both free of this bug (though they have repaired others over time).

Prognosis

There are no known user or program work-arounds for these regressions. These are bugs somewhere in the Pydroid/SDLTk/SDL2 stack which implements tkinter in the Pydroid 3 app. They've been reported to Pydroid 3, and seem likely to be fixed soon. At present, though, they make most tkinter programs largely unusable for many or all devices running Android 11. Key doubling can be addressed by pressing Backspace after each shifted key, but it's preclusively painful; and lost keyboards can be avoided by not leaving the app, but this isn't ideal on a device built for multitasking. If you care about running tkinter GUIs on Android in Pydroid 3, stick with earlier releases until fixes appear.

Updates

This glitch's source and scope were initially unclear, but later testing proved that it wasn't caused by Android 11 or limited to Samsung devices. The description above now reflects this, but the box below details these later findings.

Stop the Presses: 10 Doubles Too

After writing this section's first draft, input-key doubling was also observed on Samsung devices running Android 10 (specifically, on a test device rolled back from 11 to 10). Given the timing of releases, this makes it likely that the glitch is a bug introduced by Pydroid 3's version 4, and not by Android itself. Hence, it's not new in Android 11, though it's still present there, and the bug parameters above still apply. Alas, tkinter text input apparently hasn't been right in Pydroid 3 since the reign of Android Pie, and badly needs to be fixed (though at least one user apparently didn't type uppercase in four months of 10 before 11).

And So Does the Pixel

The tkinter GUI glitches described in this section have now also been seen on a Pixel 4a, so it's not just a Samsung issue. They've been posted as a Play-store review too, because the app's email contact never replied to three emails over a month (and ignored others in the past). These bugs are still hopeful of a fix soon, but Pydroid 3's customer service seems as sketchy as its quality control; with any luck, both will be upgraded along with the app soon.

Partially Fixed in Pydroid 3 5.0

The doubled-key glitch was repaired in version 5.0 of Pydroid 3, released in late October 2021. After upgrading the app to this release, all tested tkinter widgets now properly generate a single upper-case character for a single upper-case key, as they did before the app's version 4.0. That's GGOOOODD news.

However, the onscreen keyboard is still lost when leaving and returning to the app, and the next section's activity-opening bug still exists. The former is a Pydroid 3 regression; the latter may have been triggered by Android 11 but exists in Pydroid 3 alone. That is, one of three reported Pydroid 3 bugs have been repaired in the app's 5.0; maybe the ratio will improve in 6.0?

Pydroid 3: Help and Other Activity-Manager Popups No Longer Work

Next up is a system-level glitch genuinely new in Android 11, and observed again only in Pydroid 3. In this app alone, activity-manager command lines used to spawn a web-page or text-file view intent no longer work on Android 11; the user's request appears to silently do nothing, as the command fails on permission errors. This will also hopefully be a quick fix for this app, but it renders media views in most tkinter GUIs broken today.

Details

In many of the tkinter GUIs covered in this doc and run by Pydroid 3 on Android, viewer popups for HTML Help files and other text content are generated by spawning Android am start activity-manager command lines with Python's os.system(). These commands launch a view intent that opens files by URL or path (demoed here); negate a Pydroid 3 hack which disables Python's webbrowser module for most files (as described here, the kludge special-cases file:// docs for Jupyter and breaks all others); and work well on Androids Oreo through 10.

As of the Android 11 upgrade, unfortunately, these activity-manager command lines now fail on permission errors on both Galaxy and Pixel devices. The failure is nearly mute, except for a clue left in the adb logcat output that notes "com.android.shell does not belong to uid=xxxx." To most users, though, the net effect is that GUI Help buttons, and other viewer popups, now silently do nothing.

Importantly, this occurs in Pydroid 3 only. On Android 11, the activity-manager command works in the Termux app—at both its terminal and in Python code it runs—but fails in both contexts in Pydroid 3. Here's the scene in Termux and Pydroid 3. Moreover, the same activity-manager command works in adb shell (see the log), and other shell commands (e.g., ls) run correctly in Pydroid 3 from both its terminal and Python code via os.system(), as before; only activity-manager commands fail, and only in Pydroid 3 on Android 11.

Prognosis

There are no known user work-arounds for this regression. It's unclear if it reflects Pydroid 3 permission choices, or a deliberate narrowing of permissions in Android 11. The fact that activity-manager commands work well in Termux, though, strongly suggests the former, which seems more hopeful of an early fix. If this instead stems from a new Android 11 constraint, it might still be addressed by changing programs to use the webbrowser module, but only if Pydroid 3's biased and limiting hack is ever backed out.

Still Not Fixed in Pydroid 3 5.0

Per the prior section's update, this bug was not repaired in Pydroid 3's 5.0 release of October 2021, despite being dutifully reported to all the app's contact points. Although some blame may ultimately lie with an Android 11 change, other apps are able to run the same activity-manager commands on this platform without error. Perhaps Pydroid 3 will catch up next year.

For the present, you can dig deeper by running a cat `which am` in Pydroid 3's Terminal to see its implementation of am. Also note that Python's webbrowser.open(URL) works in 5.0 on 11 for remote (i.e., HTTP) URLs, and qualifies as a workaround for some doc views; code on this site hasn't yet been updated for this advice, because it's already been burned too many times by the thrashing.

Termux: Permissions Spontaneously Lost, Requiring a Reinstall

Lest this page be accused of picking on Pydroid 3, let's move on to an issue seen only in Termux (so far). Though triggers are elusive, Android 11 has been seen to revoke file-access permissions from the Termux app, leaving it partly disabled. This happens only in 11, and feels like an unintentional glitch that's likely to be repaired soon; but it calls the robustness of Android 11's scoped-storage changes into question.

Details

As described both in the main doc and here, the Termux app requires its termux-setup-storage command to be run to trigger its permission-granting dialog. However, even after this is done—and for reasons disturbingly unknown—Termux may abruptly lose access to every internal-storage folder except its own app-specific/private folders. For example, an ls /sdcard works fine in the app initially, but then stops working suddenly. Just as strangely, reads of non-app folders are disabled after the switch, but writes are still allowed. This makes no sense, though it seems to have been confirmed in a related Termux dev thread about Android 11 storage permissions.

To date, this issue has been observed only in Termux, whose session logs before and after the access demotion capture its impact in full detail. In essence, this app is randomly downgraded to the sort of permissions it might hold without termux-setup-storage, and by default if it targeted Android 11 and its scoped-storage model. The latter seems unlikely for this app's tested version (0.101), given its September 2020 date.

Prognosis

This permission downgrade was seen twice in as many days with no apparent trigger on the Note20—and made a surprise reappearance during later testing on the Pixel 4a. This seems at least anecdotal evidence that the scoped-storage opt-out for legacy apps in Android 11 is still a bit flaky. Probably, the Termux app will require a new release to fully support Android 11's permissions model, though this may sacrifice more in the bargain. As this downgrade has not been seen in Pydroid 3, running Mergeall's GUI in that app suffices as a work-around for that program and others today.

Work-Around: Toggle Permission

By late 2021, this is now a well-known issue, and is described in the Termux Wiki page here. The upside is that the app's permission loss appears to happen only shortly after the app is installed, and the work-around is easy: simply toggle the app's storage permission off and on in Settings/Apps. An explanation for why the toggle trick works (if indeed one exists) is officially out of scope here.

Global: USB Drives Can No Longer Be Accessed by Pathname or POSIX

And now a potential endgame for a host of programs. As of Android 11, USB drives can no longer be accessed with their former /storage/xxxx-xxxx drive-ID mount paths, and the POSIX file API. Both worked well for years on unrooted Galaxy devices (specifically, Notes 9 through 20, running Androids Oreo through 10), but are unusable on Android 11. This is true for both Galaxy and Pixel devices, and seems likely elsewhere. If permanent, this makes it effectively impossible for portable POSIX-based programs like Mergeall to sync on-phone content to and from USB external drives, and leaves users who have non-trivial content collections with few good options on these devices, as we'll see here. For such programs and users, phones become "just phones"—which may be the bitter but true point of 11.

Details

This regression is best described by example. Given an attached microSD card and USB flashdrive with IDs 6C2A-1618 and 8145-191C, respectively, here is the Android 11 story in wireless adb; at terminals in Pydroid 3 and Termux, and again in Python code run in Pydroid 3 and Termux. As you can see, internal shared storage and the embedded microSD card are still addressable at /sdcard and /storage/6C2A-1618, respectively; the USB drive which should be at its former /storage/8145-191C is not, and seems the only loser in the brave new Android 11 world.

In contrast, most file explorer apps do provide access to USB-drive files under Android 11; here's the phone and one of its USB folders in Total Commander. Tellingly, though, file explorers now open these files with a jumble of indirect or specialized techniques that seems suggestive of toolset incoherence:

Regardless of the hacks, direct USB access by POSIX-style path seems to have been taken away from most programs altogether. This may mimic iOS's closed behavior, but it's a stark change for Android and its users.

This regression applies to source code run in both Termux and Pydroid 3; standalone Python Android apps built with tools like python-for-android and BeeWare; and any other program on Android that looks for USB content at its former mounted path. While it's relevant only to programs that make use of USB drives (and many do not), this is nearly a show-stopper for POSIX-based backup/sync programs like Mergeall; device-neutral pathnames are crucial to these programs' code, and the speed and convenience of USB drives themselves are essential to these programs' users.

Work-Arounds—or Not

Unlike other Android 11 issues in this update, there are a handful of candidate options for addressing this regression—including MTP, network drives, phone rooting, and microSD-card removal—but none qualify as universal solution, and all fall well short of external USB drives in terms of velocity, accuracy, and simplicity. There just is no substitute for USB drives today when it comes to content usability.

Because such broad claims demand evidence, the Mergeall team explored the handful of common alternatives to direct USB-drive access listed below. All were tested on an Android-11 Note20 Ultra with compare-only Mergeall runs against on-phone and on-PC copies of the same 124K-file, 10K-subfolder, and 178G content collection. This is a large but realistic use case, that's easy to manage with on-phone, on-demand syncs, thanks to Mergeall's incremental processing and fast USB drives. Like USB drives, all the alternatives below allow this content collection's copies to be addressed by pathname—the underlying requirement for both selection by GUI choosers or command-line arguments, and device-agnostic processing by portable POSIX-based programs.

Android's former USB-drive access met the pathname requirement perfectly; allowed Mergeall to be run directly on phones in Python apps; completed the same or similarly sized comparisons on phone in roughly 30 seconds on an Android-10 Note20 Ultra with an external SSD (and about a minute on an Android-Pie Note10+ with a slower flashdrive); and scored high marks on precision, reliability, and ease of use. Assuming this option has been forever locked down by Android 11, let's see how the challengers stack up:

Phone rooting
On phones that support rooting, it may still be possible to directly access a mounted USB drive, at its Android /mnt/media_rw/xxxx-xxxx pathname. If so, this would support on-phone syncs, and retain both the speed and convenience of the former /storage/xxxx-xxxx paths. More broadly, rooting grants super powers to access functionality taken away by vendors who seem to think that you are a security risk to your own phone. That's a viscerally Good Thing.

Regrettably, phone rooting is a complex task, may void warranties, and is out of scope for many or most users. It's also nearly impossible on many phones—including the Samsung US models in the Mergeall team's testing stable. (The later Pixel-rooting update at the end of this section provides more details and a counterargument, but rooting's scope is still drastically limited.) Let's see what else we can try.

Removable SD cards
On phones that support an embedded microSD card, pulling the card out is always a last-resort way to manage your on-phone content. In this mode, you remove the card; bring it up to date by running a sync to it on your PC; reinsert the card in your phone; and, optionally, run a sync on your phone from the card to faster internal storage. This mode can be used on any PC platform, and its on-phone syncs still work well on Android 11 using Mergeall's GUI or command-line interfaces. Moreover, the card is accessible by its usual pathname everywhere, and the syncs on both PC and phone are radically faster than all other options ahead on this list.

Unfortunately, microSD card ports are not available on all Android phones; juggling cards in and out of possibly encased phones this way is both inconvenient and fragile in the extreme; microSD is much slower than most USB 3.X drives available today; and it's possible that microSD cards may soon perish from this smartphone land altogether, rendering this option a nonstarter (at this writing, Samsung has just ditched them on its new S21). Really, using on-phone syncs to avoid card fumbling was a primary motivation for porting Mergeall to Android in the first place; let's see if we can do any better.

MTP over USB
MTP is what normally happens when you plug your phone into a PC by USB cable. It's the native interface provided by Android to access your phone's content from a PC, and was adopted by Android over its earlier USB mass storage, purportedly because it sidestepped filesystem and partition constraints, allowed access controls, and minimized user operations (though it also greatly diminished user options—prescient, perhaps, of Androids to come). MTP works reasonably well for the occasional copy/paste of a few files, and can also be used for broader syncs by accessing the attached phone's mount-point path on PCs.

When used for syncs, your phone is attached to your PC by a USB cable, and acts like an external drive; to sync to or from the phone, Mergeall (or similar) is run on the PC instead of the phone itself, using the phone's MTP pathname. Linux supports this out of the box at path /run/user/1000/gvfs/mtp:*, and mounted MTP drives can be accessed as /Volumes/Phone and drive letters on macOS (f.k.a. Mac OS) and Windows, respectively, with third-party extensions (Windows options were never tested, for reasons about to be disclosed).

Unfortunately, MTP syncs are painfully slow: the Mergeall compare-only run that formerly finished in some 30 seconds on phone by USB drive takes a whopping and fully impractical 53 minutes by MTP on an Ubuntu Linux machine, and about 43 minutes on two macOS boxes. MTP as tested also omitted all .* "hidden" files in content lists on Linux (they'll be removed by syncs from phone to PC, even if important); and one macOS driver returned MTP-origin timestamps a comical 15 years in the future, and did not support phone writes (the former kills syncs in general, and the latter precludes to-phone updates). The macOS driver fixed its timestamps in a later release, and Linux's built-in support always handled both timestamps and writes correctly, but MTP's speed problems makes all this moot.

Plainly, MTP is too slow for prime time. While some users with very small data sets and very large patience might be able to sync by MTP (or copy in full), this mode is clearly a mixed bag, and completely impractical for users who need to sync significant content collections on demand. Given all the hype behind cloud storage, it seems reasonable to assume that wireless options would be more honed and viable than MTP; let's take a look.

Samba servers
Samba is a free implementation of the SMB file-server protocol (which has a substantial Microsoft heritage). Like MTP, you can use Samba from PC clients for both tactical drag-and-drop copies, as well as general syncs based on mount-point paths, though the performance needs of the latter tend to be much more intense. When used for syncs, a Samba server app is run on your phone, your PC connects to it wirelessly as a client (which in this doc means by Wi-Fi on the same local network unless stated otherwise), and syncs are run on the PC using the Samba server's mounted path.

This scheme was unusable out of the gate. On both Linux and macOS, the Samba server proved impractically slow—and similar to MTP in speed. A compare on Linux using a cifs mount at /mnt/landrive, for example, took an unrealistic 36 minutes, and the same test on macOS by Finder connect at /Volumes/Share was so slow that the test was abandoned unfinished at nearly one hour in. These all compare with USB's 30-seconds showing.

The Samba story on Windows 10 was even worse: connection was nearly impossible, due to the combination of that platform's hard-coded ports (it supports only 445 or 13X), and Android's port-number restrictions (it makes ports < 1027 off-limits sans rooting). Third-party port forwarders and SMB mounters may help on Windows, but all those explored seemed subpar in terms of complexity, quality, and status; in fact, the Windows mounter recommended by one Android Samba-server app has ceased to be.

Obviously, Samba's speed and usability constraints disqualify it as a replacement for USB access on Android devices today when it comes to on-demand syncs (except for content collections so small that a simple drag-and-drop would suffice—and probably be simpler by MTP). Can other network-drive options do any better?

WebDAV servers
WebDAV is an open file-server protocol that extends HTTP, and boasts support in Apache. Like Samba, using this for syncs involves running a WebDAV server app on your phone, connecting to the phone wirelessly by PC, and running syncs on the PC using the server's mount path. This option seemed promising at first: the purchased server tested was much faster than both MTP and Samba, turning in a respectable 5-6 minute compare on both macOS (via Finder connect at /Volumes/192.168.1.2) and Linux (via davfs2 mount at /mnt/webdav; Nautilus was slower). Here's a shot of the server and a log from its Mergeall client.

Unfortunately, this scheme still ran much slower than USB's 30-second compares (10-12X, to be precise). Much worse, it yielded data corruption and transfer errors where supported at all. macOS's built-in client support produced glaring data corruption while viewing photos on the phone, choked on munged filenames per the footnote ahead, and was ruled out quickly. Linux's support fared better and transferred data properly when it worked; but it was also complex to configure, and in the end proved too unreliable to use: it sported frequent disconnects from the server during syncs (causing file errors and skips in Mergeall), as well as sporadic segfaults and system errors (of the sort Ubuntu wants to hear about).

Also like Samba, WebDAV is not supported by Windows 10 on non-default port numbers required by Android, and the third-party tool tested to work around this was both slow and inaccurate, in one test completely skipping some of the subfolders in trees being synced. While some Linux failures might be attributed to the server app used, and restarting Wi-Fi on the client PC or server phone sometimes helped temporarily, another WebDAV app on the Play store failed to start altogether after install, and the disconnects pose major usability hurdles.

Clearly, WebDAV's performance and reliability issues mean that it can't stand in for a USB drive on Android today either. While some options were untested and your mileage may naturally vary with any of them, this page's space and patience are now both nearly exhausted, Let's take a quick look at one last option.

FTP servers
When it comes to file transfers, FTP is the classic. In the name of completeness, an on-phone FTP-server app was also tested as an alternative to direct USB connection. When mounted as a PC pathname on macOS, this scheme ran the benchmark comparison in a promising 17 minutes by local Wi-Fi (its port loopback over USB cable was unusable). This is 34X slower than USB's 30 seconds, but might pass in some smaller use cases.

Sadly, though, the FTP server also botched modtimes badly: it returned them as 12:00 AM for every file older than 6 months, and arbitrarily bungled their seconds component for all others. The net effect is lethal to timestamp-based syncs, because every file registers as a difference. In addition, macOS mounted the server as read-only, which precludes to-phone propagation; and this scheme shared the macOS filenames and symlinks munges of the upcoming footnote, which can trigger file loss and errors. By contrast, USB compares correctly, and radically faster.

For FTP's complete results, see this logfile. While they may vary for other apps and platforms untested, finding workable USB alternatives now seems akin to hunting unicorns. Networking is innately slower than USB, and even if you stumble onto a scheme that meets your goals, it will likely come with downsides and complexities that push it well out of scope for all but the most hard-core phone users. For this page, the hunt has reached its end.

Per the brief look, though, FTP, like other server schemes and MTP, might suffice for tactical drag-and-drop copies, but isn't up to the task of larger-archive syncs. The latter demands the speed, accuracy, and simplicity of the direct USB access that's regrettably been yanked away in Android 11—a fate ironically doled out to FTP itself at the same hands.

Footnote: macOS FAT munges vs servers
Testing WebDAV servers also brought to light a subtle potential for filename mismatches when illegal characters are propagated from macOS on FAT32 and exFAT drives, and served elsewhere. This gotcha is covered off page here because it's not limited to Android 11. Briefly, spurious diffs can be caused by the Mac's mapping of illegals to/from Unicode privates (e.g., |\uf027), which may not be unmapped by servers running on other platforms. This further clouds the viability of WebDAV for macOS users. Although some Samba servers reverse the munge, its real fix lies in avoiding covert magic that works only in rarified Apple-only contexts. A fat chance indeed (and yes, the pun is a stretch).

Update: this munge was later seen to also affect both MTP and FTP servers used from macOS (filenames mapped by macOS won't match originals when served by these protocols to macOS); and a related macOS munge similarly triggers diffs for Unix symlinks written to Windows-filesystem drives by macOS (symlinks are written as simple files, which won't match the original items' types on macOS when served from elsewhere). In both cases, macOS's silent changes can lead to file loss or errors on later non-USB syncs. See the off-page note's updates for more details.

So there you have it: the desperate and failed quest for alternatives to the direct USB-drive access that Android 11 seems to have rudely ripped away from its users, for the sake a mandatory security policy that could have easily been left to the discretion of a user toggle. In sum, MTP and wireless network syncs are largely just as slow and error prone as expected; phone rooting isn't desirable for most users and impossible for many; and microSD cards are a slow and clumsy substitute that isn't available on many phones.

In other words, there is no work-around to be found here. While we can still hope that USB drives will make a comeback, the best bet today for users with non-trivial content collections seems grabbing a phone that supports a removable microSD card. While they last, at least; alas, some Android smartphones seem on the fast track to becoming content islands soon (yes, really).

Prognosis

This regression's source and scope are unknown, but the demise of USB paths has been reported to both Samsung and Android. If it's an unintended bug in either code base, it may be fixed soon. If it's a "by design" narrowing of functionality (and the file-opening behavior of file explorers suggests it may be), it makes Mergeall, and similar POSIX-based tools that rely on USB drives for data transfer, mostly unusable on Android 11.

Such tools have long relied on the fact that USB drives are accessed with the same sorts of pathnames as everything else. Without this device agnosticism, programs would need to allow for USB drives as a special case globally, and expand both file choosers and IO-related code to support them uniquely. This would be a massive rewrite effort required only for Android; would have to use Java-based and Java-biased APIs; and is unlikely to happen in portable software like Mergeall.

For device owners, the demise of addressable USB paths would mean having to either resort to painful microSD-ectomies, use impractically slow syncs over MTP or wireless protocols, or subscribe to cloud-based options eagerly promoted by device vendors but no better than the other alternatives we've explored here. As lamented previously, such narrowed utility would probably qualify as a death knell for many of the broader aspirations of the Android platform.

Updates

Or so ends the initial take. The rest of the story behind the demise of USB access unfolded over time with vendor feedback and new devices, so please see the updates box that follows (and the conclusion it references) for later discoveries and notes.

Other Sightings in the Wild

After writing this section, a web search turned up a thread on the Samsung developers forum which raises a USB issue that seems either related or identical to that mentioned here. If germane, this confirms that this extends beyond both Python apps and the Mergeall program.

The Pixel Lacks USB Too

Also after writing this section, USB-drive paths proved to be nonexistent on a Google Pixel 4a running Android 11 too, confirming that this isn't just a Samsung issue. To be fair, it's unknown if Pixels ever provided path-based USB access (alas, Mergeall's testing farm till recently has been limited to Samsung stock). Today, though, USB drives are nowhere to be found at /storage on the Pixel; /mnt/media_rw throws permission errors on unrooted phones; and file explorers treat USB content specially. The USB lockout is Android-wide.

Rooting Pixels Fixes USB, But...

The foregoing being said, rooting does allow the Pixel to access USB drives at /mnt/media_rw/xxxx-xxxx, as suggested above. Here's the fully functional post-rooting scene in Termux; at least in this app, using the /mnt path and a sudo or su command enables on-phone USB syncs with Mergeall as before, and similar access is available for other Python and POSIX-related code after rooting. A rooting how-to is beyond this doc's scope, but for pointers on the process, try the XDA intro; Magisk doc; XDA 4a page; Google flashing guide; and local search engine.

On first glance, rooting may seem to solve the USB-access problem on devices that can be altered this way (and some rootable Android phones have hardware that's largely on par with Galaxy flagships today: see the web). On closer inspection, though, this option is still too narrow to qualify as a general fix, for a number of reasons:

It's worth adding that custom ROMs may offer additional options beyond those of typical rooting; and Pydroid 3 might run with super-user permission if installed in /system/app as a system app (where still possible). Neither scheme was pursued, because both are even further afield than basic rooting, and easily out of scope for this doc. If you have to flash firmware and remount system partitions to use an app, there's probably something larger amiss on the target platform.

It's also worth adding that the future of Android rooting is currently under attack from new hardware attestation in Google's SafetyNet security layer. In short, some apps may not work on newer rooted phones (even some that seem to qualify as innocuous). While it remains to be seen if work-arounds can be found, the rooted Pixel 4a fails attestation today, putting super-user solutions on even shakier ground.

The Thrilling Conclusion?

Samsung has now definitively replied on the USB subject; at the risk of spoiling the surprise, be sure to stay tuned for the likely conclusion of this topic in this page's next (and perhaps final) major section.

Android 12: USB Is Still Inaccessible

Per testing on a Pixel 4a in late 2021, USB is still unavailable to POSIX programs in Android 12, the same as it was in Android 11. Here's a screenshot of a console session on 12 with USB paths failing near the end. Despite user feedback, Android did not resurrect general USB access in 12, but it did manage to float a glaring Google-Pay link to the top of the Notifications pulldown. It's reasonable to assume this has something to do with priorities.

Global: Apps That Process Files in Shared Storage Are Radically Slower

Finally, yet another breakage with potentially broad and dire consequences. Mergeall compare-only runs have now been clocked at 10 times slower on Android 11 than they were on Android 10 and earlier, and can be a colossal 100X slower on 11 in some contexts. This impacts usage in both Termux and Pydroid 3, and might make POSIX-based data-processing programs impractical on Android alone—whether they manage to work around 11's other constraints or not. And it's not just Mergeall: scans for sizes in all file explorers tested are also much slower on Android 11, and unusably so for large folders. The slowdowns ultimately reflect a deliberate performance downgrade for shared storage (at /sdcard and its ilk), which seems related to a filesystem change, and may trigger other throttling. But the full story requires a short empirical journey to see how deep the damage goes.

Initial Findings: File Scans Crawl

To measure this regression, Mergeall was run on-phone in compare-only mode, with both FROM and TO set to the path of the same large folder in internal storage shared by all apps. Compare-only runs like this primarily exercise per-folder directory listings and per-file stat calls, but do not update files. The test folder held 124K files and 10K subfolders in 178G space (this is the same use case as in the preceding section but without a USB drive), and was identical on the test devices: an Android-11 Note20 Ultra, and an Android-Pie Note10+. As noted earlier, this is a large data set, but it's easily manageable on any device with Mergeall's incremental processing.

The results: when run via Mergeall's GUI in Pydroid 3, this test took an average 104 seconds and 168 worst-case seconds on the Android-11 Note20, but just 15 average seconds and 11 best-case seconds on the Android-Pie Note10. Times for Mergeall command-line runs in Termux for the same folders were just as slow under Android 11—and frequently slower, often checking in at just over 200 seconds. For completeness, a two-year-old Oreo Note9 ran the test in 16 seconds, though the data set differed slightly.

In total, Android 11 posted a 7X average slowdown from earlier Androids in Pydroid 3 on these tests, and often 10X or worse. While the effect is nowhere near the prior note's massive 53 minutes logged for MTP (and cannot account for the slowness of other server options), it still qualifies as extreme. Moreover, the Note20 was already running comparable Mergealls 35-45% faster than the Note10 before the Note20 was forcibly upgraded to Android 11; hence, the true Android 11 sloth is larger than comparisons to the Note10 today suggest, as its slowdown also negates the former speedup.

Nor is Mergeall the only victim: all file explorer apps tested are impacted by this regression too. They appear to also be on the order of 10x slower when scanning folders to compute sizes on Samsung's Android 11. Their API choices are naturally unknown (they may or may not use POSIX calls), and their slowdowns haven't been formally quantified (i.e., app sloth exceeds tester patience). The slowdown is so obviously bad, though, that a Properties scan virtually hangs; this surely must have been noticed before this release pushed its way onto so many customers' phones.

Odder Data: Tapped App = Quicker Task

But it gets worse. Because the initial results we just met were so bad, a simple Python test script was coded to isolate and exercise the apparently slower IO operations used by Mergeall compare-only runs; see the test script here. For identical folder trees, testing showed this script to be a horrific 40X slower on Android 11 than Android Pie, which is substantially worse than 11's 7-10X slowdown for the full Mergeall program. In both Termux and Pydroid 3, Android 11 on the Note20 typically took 80 to 100 seconds to complete the script's tree walk, but Pie on the Note10 finished it in just 2 to 2.5.

And it gets weirder. Per later testing (really, fortuitous accident), it turns out that Android 11 runs the test script dramatically faster—sometimes finishing in just 10 seconds—if any other app is interacted with while the app running the script continues in the background. This happens in both Termux and Pydroid 3, and a simple scroll in Settings suffices to speed the running script tenfold. A similar speed boost is seen in Mergeall's Pydroid 3 GUI if it's swiped aside to use other apps while it's running.

And even weirder. On further inspection, it also turns out that a same or similar speedup occurs in the Termux and Pydroid 3 apps' terminals, as well as tkinter GUIs run in Pydroid 3, if you simply keep tapping an app's screen while it's busy with a long-running task—like the test script or a full Mergeall run. Presumably, the screen activity forces task-scheduling logic to count the app as still active and avoids lowering the app's priority, or the CPU's speed. Yes, really:

In truth, taps aren't strictly required—drawing imaginary circles on the app's display while it is running tasks will suffice. And no, it's not April Fool's day yet (though these apps seem perilously close to qualifying as comedy props).

Before you ask: yes, the battery was amply charged when all these observations were made (low-battery mode wasn't in play). And yes, there's proof: here's a Termux screen capture showing a 10X boost; its first two runs were not tapped, but the second two were. Untapped performance sometimes varies, but tapped runs are always much faster, and this phenomenon is system-wide per the last bullet above.

There is no explanation for this behavior yet, despite toggling every related setting on the device (spoiler: see the update ahead). But it strongly suggests that a new task throttling based on screen activity in Android 11 (or Samsung's flavor of it) is a major factor in slowdowns, and a likely accomplice of IO sloth. This throttling may be done in the name of CPU sharing, thermal management, or an overly ambitious battery optimization, and might be triggered by other factors or slowdowns (e.g., IO churn might aggravate it, or vice-versa). Some such app throttling may even be normal on phones, but it has assumed deadly proportions in 11, and makes many long-running tasks nearly unusable.

That being said, POSIX-based IO also appears slower in 11: tapped apps still don't match Pie's speed, and Android has already given notice about a deliberate 2X slowdown (more in the updates ahead). Moreover, the screen-activity throttling is either new or escalated in Android 11; is fairly labeled a bug which starves apps' long-running tasks of performance; and qualifies as a major regression and a likely dealbreaker for numerous programs on this platform.

Unfortunately, the Mergeall team has no tools to pursue this further. Performance analysis is murky at best on Android (especially now that per-app CPU usage seems off-limits to mere mortals), so this will have to await more-privileged parties' attention. Please watch the Android Issue Tracker bug report for updates. Technically speaking, neurotic tapping may be a kind of a work-around for this issue, but a fix in Android 11 (or Samsung's flavor of it) seems a much more rational resolution.

Prognosis

To recap: it appears that there are two major performance regressions in Android 11 that impact Mergeall—both a specific file-processing handicap, and a perhaps broader screen-activity throttling—and no reasonable user or program work-around is known for either. It's unclear where causes lie, but the issues have been reported to both Samsung and Android, and may also be fixed soon: Samsung has already promised a fix for some known Android 11 speed issues later in January, though it's unknown if the fix will address the performance issues covered here.

If the slowdowns are instead intentional consequences of Android 11's new mandates, they will severely paralyze (if not fully preclude) a host of POSIX-based and other software on Android 11, including all programs coded in Python that access files or perform non-trivial tasks normally, whether they are run as source-code or apps. In this scenario, program performance may have been deliberately sacrificed for the sake of Android 11's scoped-storage agenda. Regrettably, Android itself already seems to have suggested as much.

Time will tell if a half century of Unix practice is among the casualties of Android's newly found security obsession. All indicators today suggest it is, though only as long as developers enable such choices by rewarding this increasingly closed platform with grossly proprietary code. Dramatic to be sure, but this alarm badly needs sounding. The road to corporate tyranny is almost always paved with both customer complacency and developer complicity. Until we're told otherwise, though, we can still choose not to lay the bricks.

Updates

Like USB's demise, the rest of this speed-regression story filtered in over time, as vendors replied and new devices were tested. Be sure to see the updates box that follows for full closure on this thread.

Samsung Weighs In

Samsung, the Note20's maker, has now sent an initial reply, claiming that the performance issue is caused by Android's media provider scanning an external SD card, and suggesting that the remedy is to delete files. There are at least four problems with this:

There are clearly major performance regressions in Android 11; whatever their underlying causes, Samsung's reaction hopefully won't be to bury its head in the sand (and advertise ever more deceptively). This won't make customers happy, won't sell phones, and won't help Android reach its potential. Samsung support later replied with something irrelevant about thermal management not being a factor, but the thread was made moot by the Android response up next.

Android Weighs In

From the hate-to-say-I-told-you-so department, Android later also replied on this issue, verifying that it is indeed a bona fide and deliberate performance downgrade. While it lasts, see the terse but definitive AOSP doc page referenced by the Android Issue Tracker post. Here's a screenshot of the doc's important bits, lest they vanish in flux.

In short, the slowdown reflects a tradeoff which intentionally penalizes heavy users of files in shared storage—where heavy use apparently means anything over 100 files, and shared storage is any app-neutral location in /sdcard or its equivalents. Though its docs are thin, Android 11 seems to have moved such shared ("external") storage from the newer SDCardFS filesystem, to a FUSE filesystem of former infamy, paired with a Java-oriented MediaProvider handler. Per the docs, this was all for the sake of implementing scoped-storage access controls. The net effect, however, radically slows access to shared storage for programs unwilling or unable to adopt Android-only paradigms.

That is, Android 11 has been optimized for scoped storage's app-sandbox modelat the expense of everything else. This has major consequences for both users and programs:

Scoped storage's lack of file-explorer support today makes confirmation difficult, but a quick subset test seems to back up the claims: on the Android-11 Note20 Ultra, a Termux Mergeall run to compare a 13K-file 874-folder tree against itself finishes in 1 second or less for a copy in the app-specific folder, but typically 10 seconds for a copy in shared storage (and the Pixel's results up next are worse). As usual, the odd tapping trick inexplicably makes the shared-storage run faster, but never app-specific fast; shared storage has plainly been handicapped in 11.

While it's uncertain if Android's explanation accounts for all the throttling seen, it does qualify as sufficiently awful to call the game. The performance regressions are likely here to stay, and Mergeall and similar tools that process content in shared storage will regrettably run enormously slower on Android 11 than they did before. For this, we can thank Android 11's new "Privacy over performance" mantra, which, as good mantras do, has grown to eclipse the needs of actual users (many of whom will undoubtedly find "Dogma over usability" the apter tagline).

The Pixel Weighs In

Android 11's throttling of shared storage has now been verified and quantified on a non-Samsung device. Results at a Termux command line on a Google Pixel 4a running Android 11 are captured here. If you look close enough, you'll see that the same comparison in both app-private and app-specific storage takes about 0.13 seconds, but in shared storage requires 13 seconds. At least for this use case, this phone's shared storage is a full 100X slower!

Subtle bits: the degree of this throttling sometimes varies, but has nothing to do with the preceding link's folder, test, or app; it's the same elsewhere in shared storage, in the results of other scripts, and in Pydroid 3's Terminal. Moreover, this specific device may feel the hit worse than the Note20 Ultra (which uses later and faster UFS), and Mergeall compares are mostly directory listings and stat calls; shared storage is "only" 40X slower if programs read files along the way, per the Pixel's diffall results.

Still, the penalty for using open storage options in Android 11 is clearly extreme. By sharp contrast, the same tests with a larger data set on an Android 10 Note20 Ultra show no speed difference for shared storage at all (Android 11 tests are no longer possible on the Note20, for reasons revealed ahead). Moreover:

This issue has been reposted to the Android Issue Tracker with the new Pixel data in hope of a second look, but the outcome seems unlikely to differ. In the end, Android 11's new shared-storage selective sloth seems a poison pill for POSIX-based apps and will be a major inconvenience for many of their users, but it does not render devices wholly unusable; slowed tasks do finish eventually, and you can always deploy the silly-taps trick to speed these now-dysfunctional phones. The resolution of the other global Android 11 breakage, however, may be fully terminal, per the conclusion ahead.

More on Android CPU Throttling

Per later research, some CPU throttling for power management is indeed common and even normal in the Linux kernel underlying Android. In short: Android, via Linux scaling governors, reduces CPU cores' frequencies during idle states to conserve power and minimize heat, and increases them on screen touches with the assumption that user interaction is imminent. This works for interactive apps, but may fall short for non-interactive tasks like file processing, and cannot be changed without phone rooting.

To learn more about this throttling, try this Linux scaling-governor overview; this search for Android's flavor; this and this for more on scaling's status; and this later thread on Termux's GitHub forums for a similar Android-11 speed issue with shared memory, which qualifies as independent confirmation of this page's findings. You can also monitor your phone's CPU cores with the CPU-Z app, available here; watch the cores speed up on taps (or imaginary circles) in the app's SoC tab.

In addition to CPU scaling, Android, like all operating systems, implements process scheduling which can impact performance too, but this is far beyond our scope here (try the web as a first step). Both CPU scaling and process scheduling in Android can vary by version and vendor, which makes performance analysis on these devices challenging, if not nondeterministic.

That said, while all of this may account for the "magic tap" speed-up phenomenon, it does not explain why shared storage is radically slower than app storage. Tapped or not, app storage is still much faster than shared storage for the same tasks on Android 11 alone. Hence, this almost certainly has less to do with CPU or process throttling than 11's swap of shared-storage implementation. As of late 2021, we still await a remedy.

Android 12: Shared Storage Still Crawls

Per testing on a Pixel 4a in late 2021, shared storage is still just as slow in Android 12 as it was in Android 11. Here's a screenshot of a console session on 12 giving the relative speeds of app-private, app-specific and shared storage on 12. Despite numerous reports from developers and users, Android opted to ignore a glaring ~100X speed regression this time around. There's always 13 next year, of course, but in the present, GUI cosmetics seem to be more important than performance, and storage sloth is in danger of being posted to the permanent column soon.

And So On: Scoped Storage’s Other Victims—and Other Shoe

And beyond all this, of course, the full impact of Android 11's scoped-storage permission changes remains to be seen, if and when the "legacy" Python-programming apps leveraged in this guide target Android 11. Some apps are already impacted by the new chain-link fences:

Web browsers
Samsung's own web-browser app can no longer open file:// local-file URLs—which reduces users' options for viewing documents and website pages off-line. Firefox (version 84) was the only browser tested that still fully supported such off-line viewing with Android 11 on a Samsung Galaxy device—though not on a Pixel 4a, oddly, where Opera also fails to display on 11 what it can on 10. The web-server issues later in this list compound these lockdowns' impact.
File explorers
On unrooted phones, none of the seven file-explorer apps tested provide access to the app-specific folders of other apps at Android/data—which is where files ironically must be located for usably quick access. Here's the Pixel 4a running Google's own Files. It is unknown if such apps have yet adopted or been granted Android 11's All files access permission, and this is prone to change (see the updates box below for the latest details). But these apps are hog-tied today, leaving a Catch-22 for scoped storage which drastically limits content utility: how do you store and use files in faster folders that file explorers cannot access?
Network servers
As covered earlier, all of the file-server apps tested for wireless syncs were hampered by port-number restrictions and connection issues on Android 11. Web-server apps were untested on 11, and some are marginally usable for local website views on 10 (like this and this), but others are already suffering slow demise on this platform. It's unclear which version of Android began the closures, but its network support is clearly shrinking.

While the Python apps noted in this coverage have yet to emerge from their opt-out havens, their current stay of execution already comes with the visible barbs noted previously on this list: USB access is MIA, storage performance has been dialed back to zero, and glitches abound.

Again, please take this update in its context: it reflects just initial Android 11 testing on two devices; some its findings will likely be clarified or reversed over time; and this page may be updated as fixes and new information come in (watch the top-of-page date for revisions). For the present, though, Python and tkinter programs have clearly been harmed substantially by Android 11. Although a Return-of-the-Jedi update here is still slimly possible, it grows less likely with each new "aw crap" issued by the Mergeall testing team.

In closing this update, it's worth noting for fairness that other platforms also suffer frequent breakages at the hands of today's mosh-pit development model. Witness this site's coverage of recent regressions in Linux here and here; macOS here and here; and Windows here and here. While the ultimate goal behind Android 11's many regressions is a matter of speculation, its morph more clearly jives with the rude and reckless motifs now sadly common across the software field. Android has an especially clear pattern of both mandating languages and tools, and regularly breaking the programs which use them. Developer-friendly not, that is.

Updates

As expected, some apps started rolling out enhanced support for Android 11's storage model after this section was first written. The notebox that follows hits the highlights of these upgrades, though none manage to resuscitate the broader utility of pre-11 Android. Also covered is a recent Android change which attempts to negate former uninstall perils—and largely falls short.

File-Explorer Updates

As of mid-February '21, some file-explorer apps now support access to Android/data app-specific folders if granted permission. Among those tested anew: full access is provided by Cx File Explorer and Solid Explorer; partial access is present in Total Commander via a special link icon, though the folders are not accessible in some contexts (e.g., Copy To); and no access is (yet?) available in Google Files, Root Explorer, FX, and ASUS File Manager. Given that permissions were not to be rolled out until early 2021, be sure to check your explorers for their current state.

Where available, file-explorer access to these faster app-specific folders alleviates 11's shared-storage speed issue, but only in part: as covered earlier, because these folders may be automatically removed on app uninstalls, they are not safe places to keep your valued content. That is: you can put your stuff in quicker app storage and can now use it there in some explorers, but it might vaporize if you ever uninstall the hosting app. That's not a risk most users should be asked to take.

Furthermore, file explorers are only part of the content-access story. Many other apps won't be able to access files in app folders other than their own, which limits content in app-specific storage to just one role. Consider the two Python-programming apps referenced in this guide: if you wish to process content with programs in both Termux and Pydroid 3, it must be kept in slow shared storage, because neither app can access the other's app-specific folder. In other words, Android 11 forces you to choose between content dataflow and program speed. That's about as toxic to non-trivial computing as it can be.

In sum, even with file-explorer visibility, app storage's impermanence and inaccessibility make it a very poor work-around for shared-storage sloth, and leave Android devices with no good options for realistically scaled goals. It's also worth noting that rooting opens access to app-specific folders in tested file explorers too, but you should know by now that this isn't much of a solution either.

More File-Explorer Updates

In October '21, Android file-explorer support for app-specific storage was reevaluated for the scripts introduced at the end of this page. At the risk of stealing the conclusion's thunder, you can find this new coverage here. In short, app-specific storage can now be accessed by Cx, Solid, Total Commander (still sans Copy To), Root Explorer, and MiX, but not by Fx, Asus, or Samsung.

Disclaimer: if Unix permissions are propagated to a phone's app-specific storage, file explorers and the apps they open might be reduced to view only clients—they may be able to open content, but not change or delete it. On a Galaxy Z Fold3, for example, some content stored in Termux's app-specific storage can be modified by Termux alone, if it retains its Unix permissions. Allowing Android to instead create content with default permissions allows app-specific updates and removals by other apps, but this seems yet another convolution, if not glitch.

You can read more about app-specific access rules and the choice of Android 11 storage types in the follow-up doc here. If app-specific storage won't work for your use cases, your only Android 11 option for general and changeable content is shared storage—and its up-to-100X speed degradation. Whether by design or mistake, this pitfall is palpably limiting.

Web-Browser Updates

While working on another program in mid 2021, it was discovered that some browsers—namely, Chrome and Opera—can be tricked into viewing local HTML files on Android 11 by copying content to the browsers' own sandboxed folders with a file explorer that has permission to do so (more on the hack here). This isn't nearly the same as arbitrary local-file access, though, and your content will disappear if the browser is uninstalled or simply cleans its folder (and browsers seem very unlikely to adopt the manifest exception ahead for their own folders). For most 11 users and uses, local file views in web browsers are indeed defunct.

More Web-Browser Updates

Except things change: as of late 2021, some file explorers have found ways to make local-file views possible on Android 11 without the prior paragraph's trick. For example, Cx now automatically spawns an HTTP (web) server on the local host (phone) for viewing local HTML files opened in a browser; Solid instead uses a content-provider URI that seems to largely work; and other explorers may offer similar heroics. These are minimal views (e.g., folder listings fail), but suffice for much client-side content. Lesson: sometimes the best work-around to vendor flux is to simply wait for apps to catch up.

Retaining Content on Uninstalls

Though a weirdly well-kept secret, a new hasFragileUserData flag in app manifest files allows apps to give users the choice of retaining or deleting content on app uninstalls, via a new GUI checkbox. This retention applies to both app-private (/data/data) and app-specific (Android/data) folders, is now used by some popular apps, and can be explored on the web here and here. At least for instrumented apps and knowledgeable users, this may avoid the automatic-deletion peril on uninstalls, and appears at first glance to better support content in faster app storage.

On closer inspection, though, this seems of very limited scope. For one thing, app users must notice, understand, and click the GUI checkbox; else their content is still purged automatically on app uninstalls. For another, using the new switch requires a standalone app; Python code run by an app has no access to the app's manifest. Worst of all, this new switch does nothing about Android 11's access constraints; an app's data is still usable by that app alone, sans the extreme Android-only heroics discussed earlier. While an app's sandbox storage may now optionally persist on uninstalls, it remains useless to most other apps, and not a valid medium for cross-app content.

Really, this new flag seems meant only for retaining an app's own data across an uninstall-and-reinstall cycle, of the sort commonly used to fix issues. Though useful in such contexts, it does not restore the cross-app data flow severed by scoped storage, and it does not address the speed hit imposed by Android 11 on apps that process content in more-useful shared storage. Programs that freely share content are still relegated to shared storage, and penalized by its draconian sloth. Hobbling on-phone storage this way may or may not have been meant to boost demand for cloud-storage services, but this is surely among its likely consequences.

Retreat of the Jedi: USB is DOA (Feb-2021)

It's official: Samsung has now verified that the removal of access to USB-drive mount paths described earlier is by design in Android 11, and cannot be fixed. This unfairly penalizes legacy apps, and is regrettably a game ender for a host of tools on this platform. There simply is no good alternative to USB for getting substantial content on and off your phone. Especially in concert with 11's other demerits on shared-storage performance and app usability, this is a full stop for content-based activities.

Per its reply, Samsung cannot do anything about USB access in its code base, because it's a new policy mandated by Google as of Android 11. Specifically, /storage/xxxx-xxxx paths can no longer be set; USB drives may be mounted at /mnt/media_rw/xxxx-xxxx, but using that path requires special permissions or a rooted device, and it is not accessible to most programs and users. Hence, general USB access is no more, this isn't limited to Samsung devices, and it's up to Android to fix this if it wishes to be inclusive of programs that depend on traditional processing paradigms.

You can read the full reply about the USB path's demise from Samsung here. Although a definitive reply from Google on this issue is still pending at a parallel post on the Android Issue Tracker, the lack of USB-drive paths on the unrooted Pixel 4a suggests that this extends well beyond Samsung's code, and a new opinion-based decree from Google that wouldn't bend to the leading Android vendor seems unlikely to be reversed for a single developer As of this page's latest update, the decree remains in force in Android 12.

Impact

Assuming its policies stand, Android 11 has knowingly maimed scores of programs that use the portable POSIX API and rely on USB drives for non-trivial content collections, including the Mergeall backup/sync program. Users of such programs are left with snaillike connections, perilous rooting, or delicate microSD-card pulls—none of which is universally practical, supported, or remedial. And developers of such programs are left with broken products or costly rewrites to adopt Android's proprietary, dogmatically enforced, and Java-based APIs. Such rewrites are not an option for Mergeall, and unlikely elsewhere.

Eventually, Android 11's scoped-storage access restrictions described in the main doc may further limit what users can do with their programs, content, and devices. The full scope of these restrictions remains to be seen in Python-programming apps still loitering in an opt-out limbo that's partial at best; but they are already slowing the loiterers to a crawl, crippling apps brave enough to target the new Android, and well on their way to becoming the norm.

Ultimately, though, all it took to torpedo content-management programs like Mergeall was removal of direct USB-drive access. Between this, its poison-pill performance regressions, its newly honed glitches, and its looming access restrictions, Android 11 unfortunately does lock out many or most Python programs, and seems likely to be the sunset of this platform's general-purpose-computing ambitions. Prisons may be secure, but they're still prisons.

Conclusions

On a personal note, the Android 11 auto-update has functionally bricked this developer's phone, and seems the same as an auto maker creeping into your driveway and removing your car's engine. This device today can no longer be used for content backup and view—one of the main purposes for which it was purchased—and the fault squarely lies with the dubious agendas underlying Android 11. Though spun aggressively (and sometimes arrogantly) as enhanced security, the only thing 11's changes seem sure to secure is the revenue stream of cloud providers and mobile advertisers.

The future is of course unknown, and it's possible that Android 11 will still follow Windows 8's lead and reverse course over time. For the present, though, Android may have finally taken its place alongside the iPhone, as a deliberately closed and limited media portal for technically naive consumers. The mass market of exploitable customers which seems the true target of these devices' vendors probably won't care. But some developers will, especially those who craft programs beyond the trivial, and understand that "apps" has always been a derogatory term.

In the end, this page's takeaway is a simple one: they're only "just phones" if you let the auto-update happen. If you care about using Mergeall, tkinter GUIs, or your device in general, you should consider avoiding Android 11 and later unless and until this platform's many issues are addressed. This page may be revised again if improvements arise, and hopes they will; Android devices have great potential yearning to be unlocked.

But for now, this humble Python hacker won't be moving ahead with new development on this platform, and will be flashing a ROM to revert the Note20 to Android 10 very soon (learn how here, here, and here). A retreat, perhaps, but it's also a lot more usable, and a lot less fascist.

Postscript: Android 10 Rocks Again

The rollback to Android 10 went off without a hitch, and fully restored all the functionality Android 11 took away. In particular: USB drives are back again—the access revoked by Android 11 is available to Python programs on 10; and shared storage is blazingly fast again—the internal-to-internal 124K-file Mergeall compare that took 104 to 168 seconds on Android 11 finishes in just 8 to 9 seconds on 10. That is: yes!

That's great news, but also in some ways sad. Android 11's architectural changes are clearly designed to lock out everything except proprietary Java-based code that uses proprietary Java-based frameworks and will run nowhere else. This is a blatant and short-sighted affront to interoperability, which dramatically limits device utility and scope. POSIX-based tools like Python, and the decades of prior art they enabled, are not invited. This may reflect a culture clash of sorts, but the ramparts are being raised by Android alone.

With all the data in, Android 11 has indeed robbed its users of swaths of functionality in the name of a dodgy security rationale, and probably is the end of the line for these gadgets' flirtation with general-purpose computing. This seems nearly inevitable in hindsight. A platform which both excludes contributors and severs the flow of content between programs can only be doomed to a future of inane consumption and greedy coercion.

And yet, there is a bright side to close with here. By the time today's phone batteries die, Android will almost certainly be something completely different yet again—if not subsumed by friendlier alternatives altogether. That's a sequel which remains happily untold. Hope, after all, can never be deprecated.

The Force Awakens: Fold3 vs Deltas (Oct-2021)

But that's not how the story ends.

In August 2021, Samsung released its newest flagship device, the foldable-screen Z Fold3, without support for a microSD card. This compounded the usability issues created by Android 11's removal of USB access for POSIX programs. Nevertheless, the Fold3 was compelling enough to inspire development of a new way to sync content onto its intentionally closed hardware.

The fix requires a proxy drive, and a new Mergeall script that creates change sets (a.k.a. deltas) which can be applied separately. It also requires a manual copy step along the way, and is not as direct or simple as the two-merge model that's used by command or GUI prior to Android 11. It does, however, qualify as a work-around for both Android's and Samsung's rude closures, and has been automated by scripts you can fetch and run on your devices. In fact, to sync any Android 11 or later device, you'll simply run one script on your PC, copy a deltas zipfile by file explorer, and run another script on your phone.

Because this document has grown long in the tooth, the rest of this Fold3 and deltas-fix sequel has been moved off page. Follow the continuing saga—and fight back against customer-antagonistic agendas—by clicking this link to visit the separate page:

The story continues here



[Python Logo] Top Media Mergeall Tkinter News Blog Apps Input ©M.Lutz