The Android 11 Updates Page

Latest update: January 18, 2021

This page logs updates concerning Android 11's implications for Python programs and Python/tkinter GUIs, especially those run as source-code in the Pydroid 3 and Termux apps. It began as an inline note at the end of this appendix—whose background context is effectively prerequisite reading here—but grew into content worthy of its own stand-alone page somewhere around update #3. Moving this outside the Mergeall-on-Android package also better accommodates future revisions, which seem about as certain in the Android world as death and taxes in the real one.

Contents:

While this page is really just an extension of the Mergeall main doc and its tkinter cousin, you can also access its own media as a set in this gallery.

Crisis Postponed till 11 (Apr-2019)

Android blinked in late April 2019—or at least called a temporary cease-fire with its users. Per the blog post here, enforcement of the new internal-storage permission constraints described in this appendix has been delayed until Android 11, the successor to 10 expected to appear in 2020 and become common on consumer devices by 2021 (and originally named R).

While welcome news, this really just postpones the tragedy by one year for developers and end users alike. A lot can happen in that year, but people in both camps are left hoping that its manifest won't include a crippling change which reduces Android devices to media-consumption portals, designed to turn private lives into profit centers. The odds are sadly slim in a world where billions seem happy with the Faustian bargain.

A New Hope: All Files Access (Oct-2020)

Per the latest Android 11 specs and docs in fall 2020, it now appears that programs which require broader read+write access to shared internal storage that survives app uninstalls may be able to acquire it with the new MANAGE_EXTERNAL_STORAGE manifest permission, plus per-app system settings made at users' discretion. Per current sources, some Java-based code may also be required to invoke or manage this permission at runtime, but the code could be self-contained, and seems possible in Python (see Pyjnius and others).

This new option, known as All files access, might be perfect for file managers, virus scanners, and incremental backup/sync programs like Mergeall that allow their users to process arbitrary folders on earlier Androids. It appears to grant access nearly as permissive as before, though other apps' Android/data folders may be off-limits. The new option also remains to be evaluated by the Mergeall team, but you can read about it for now at these pages (while they last; Android developer pages trend towards transience): Android 11 storage; All files access; an FAQ on the subject; and Google Play edicts here and here.

Per the last of those links, using the new permission in apps uploaded to the Play store requires special approval—a point of control which hopefully won't be misused (see the recent Fortnite fiasco), but can be subverted in some contexts by Android app sideloading. Per the other links, the new permission may also make the Storage Access Framework unnecessary, but it's unknown if it will be available in Python-hosting apps like Termux and Pydroid 3, or require creation of fully standalone apps. Either way, apps using All files access cannot be uploaded to Google Play until early 2021, so you'll have to stay tuned for the conclusion to this saga (and appendix).

The Empire Strikes Back: First Failures (Jan-2021)

Thanks to an unexpected auto-update from Samsung, the Mergeall testing team has now had a chance to evaluate the apps and programs described in the main doc under Android 11. This update reflects initial testing with that platform on an unrooted Samsung Galaxy Note20 Ultra. While its findings may or may not apply to other devices, and seem likely to improve over time, users and developers alike are stakeholders in Android's current status.

Here are issues uncovered on Android 11 so far and covered below:

  1. Pydroid 3: Shifted Keys Now Produce Two Characters in tkinter
  2. Pydroid 3: Help and Other Activity-Manager Popups No Longer Work
  3. Termux: Permissions Spontaneously Lost, Requiring a Reinstall
  4. Global: USB Drives Can No Longer Be Accessed by Pathname or POSIX
  5. Global: Long-Running Tasks That Process Files Are Now Radically Slower
  6. And More to Come: Scoped Storage’s Other Shoe Has Yet to Drop

The good news is that apps which still target earlier Androids get a temporary opt-out of scoped storage in Android 11, and seem, at least at first glance, to mostly work as they did before. In fact, permissions for such apps (now branded "legacy") are in some ways more liberal than before. Here's the Android 11 permission stories captured in Termux and Pydroid 3 after their permission-granting steps. Compare these to the behavior on Android 10 and earlier presented in the main doc here; we'll revisit these results ahead, but these apps now can access microSD-card files more freely, and the root of internal storage at /sdcard is still fully usable.

The bad news is that the reprieve is nowhere near complete: as expected, Android 11 does break the Python programs mentioned in this document, in the multiple ways covered below. Most grievously, general access to USB drives has been revoked (even for "legacy" apps), and long-running tasks run much slower especially when processing files (though the true cause is a shocker). On top of this, Android 11 introduces new glitches to tkinter GUIs on Samsung devices, and bizarrely bungles permissions in some apps. In worst cases, the Android 11 breakages have the potential to render Python programs wholly unusable on this platform, in both source-code and standalone-app form.

Please note up front that findings here reflect an Android 11 upgrade rolled out to Samsung devices in late December 2020 and briefly tested over a few weeks. This upgrade has additional unrelated problems not noted here that seem to call its quality into question in general (e.g., twilight-zone states in file-explorer apps). To be sure, this feels like a beta, and some or all of its breakages noted below may be unintentional bugs which may be addressed very soon in updates from apps, Samsung, or Android. Breakages which reflect deliberate functionality downgrades, however, may make the Android platform permanently out of scope for POSIX-based tools like Python.

Pydroid 3: Shifted Keys Now Produce Two Characters in tkinter

First up on our Android 11 breakages tour is a new GUI glitch that applies to Pydroid 3 alone, and will hopefully be easy to repair in an app update soon. At present, though, tkinter text-input widgets are unusable in this app on Samsung devices (and perhaps others) running Android 11, because input keys are doubled.

Details

In tkinter GUIs run by this app on Android 11, all text-input widgets now erroneously generate double characters when shifted. For example, shift+"a" generates "AA" in these widgets, and shift+"2" generates a "@2" combo. This happens on both Bluetooth and on-screen keyboards, and for all on-screen keyboards tested (Hacker's Keyboard, Samsung's, and Google's Gboard). This was tested only on a Samsung device (the Galaxy Note20 Ultra) under Pydroid 3 (version 4.01_arm64), and it's unknown if it impacts any other brands or versions. However, this does not occur on any other Samsung devices running Androids 10, Pie, or Oreo, and was triggered by Samsung's Android 11 upgrade.

Prognosis

There are no known user or program work-arounds for this regression. This is a bug somewhere in the Pydroid/SDLTk/SDL2 stack which implements tkinter in the Pydroid 3 app. It has been reported to Pydroid 3, and seems likely to be fixed soon. At present, though, it renders most tkinter programs largely unusable for many (if not all) devices on Android 11; you can press Backspace after each shifted key, but it's preclusively painful. If you care about running tkinter GUIs on Android in Pydroid 3, stick with Android 10 until a fix appears.

Pydroid 3: Help and Other Activity-Manager Popups No Longer Work

Next up on Android 11's hit list is a new system-level glitch observed again only in Pydroid 3. In this app alone, activity-manager command lines used to spawn a web-page or text-file view intent no longer work on Android 11; the user's request appears to silently do nothing, as the command fails on permission errors. This will also hopefully be a quick fix for this app, but it renders media views in most tkinter GUIs broken today.

Details

In many of the tkinter GUIs covered in this doc and run by Pydroid 3 on Android, viewer popups for HTML Help files and other text content are generated by spawning Android am start activity-manager command lines with Python's os.system(). These commands launch a view intent that opens files by URL or path (demoed here); negate a Pydroid 3 hack which disables Python's webbrowser module for most files (described here); and work well on Androids Oreo through 10.

As of the Android 11 upgrade, unfortunately, these activity-manager command lines now fail on permission errors. The failure is nearly mute, except for a clue left in the adb logcat output that notes "com.android.shell does not belong to uid=xxxx." To most users, though, the net effect is that GUI Help buttons, and other viewer popups, now silently do nothing.

Importantly, this occurs in Pydroid 3 only. On Android 11, the activity-manager command works in the Termux app—at both its terminal and in Python code it runs—but fails in both contexts in Pydroid 3. Here's the scene in Termux and Pydroid 3. Moreover, the same activity-manager command works in adb shell (see the log), and other shell commands (e.g., ls) run correctly in Pydroid 3 from both its terminal and Python code via os.system(), as before; only activity-manager commands fail, and only in Pydroid 3 on Android 11.

Prognosis

There are no known user work-arounds for this regression. It's unclear if it reflects Pydroid 3 permission choices, or a deliberate narrowing of permissions in Android 11. The fact that activity-manager commands work well in Termux, though, strongly suggests the former, which seems more hopeful of an early fix. If this instead stems from a new Android 11 constraint, it might still be addressed by changing programs to use the webbrowser module, but only if Pydroid 3's limiting hack is ever backed out.

Termux: Permissions Spontaneously Lost, Requiring a Reinstall

Lest this page be accused of picking on Pydroid 3, let's move on to an issue seen only in Termux (so far). Though triggers are elusive, Android 11 has been seen to revoke file-access permissions from the Termux app, leaving it partly crippled. This feels like an unintentional glitch, and seems likely to be repaired soon if so; but it calls the robustness of Android 11's scoped-storage changes into question.

Details

As described both in the main doc and here, the Termux app requires its termux-setup-storage command to be run to trigger its permission-granting dialog. However, even after this is done—and for reasons disturbingly unknown—Termux may abruptly lose access to every internal-storage folder except its own app-specific/private folders. For example, an ls /sdcard works fine in the app initially, but then stops working suddenly. Just as strangely, reads of non-app folders are disabled after the switch, but writes are still allowed. This makes no sense, though it seems to have been confirmed in a related Termux dev thread about Android 11 storage permissions.

To date, this issue has been observed only in Termux, whose session logs before and after the access demotion capture its impact in full detail. In essence, this app is randomly downgraded to the sort of permissions it might hold without termux-setup-storage, and by default if it targeted Android 11 and its scoped-storage model. The latter seems unlikely for this app's tested version (0.101), given its September 2020 date.

Prognosis

This permission downgrade has been seen twice in as many days with no apparent trigger, and the only known remedy is to reinstall the app. This seems at least anecdotal evidence that the scoped-storage opt-out for legacy apps in Android 11 is still a bit flaky. Probably, the Termux app will require a new release to fully support Android 11's permissions model. As this downgrade has not been seen in Pydroid 3 (yet?), running Mergeall's GUI in that app suffices as a work-around for that program and others today.

Global: USB Drives Can No Longer Be Accessed by Pathname or POSIX

And now a potential endgame for a host of programs. USB drives can no longer be accessed in Android 11 with their former /storage/xxxx-xxxx drive-ID mount paths and the POSIX file API. Both worked well on Androids Oreo through 10 (and Notes 9 through 20), but are unusable on Android 11. If unrepaired, this makes it effectively impossible for portable POSIX-based programs like Mergeall to sync on-phone content to and from USB external drives, and leaves users who have non-trivial content collections with few good options on these devices.

Details

This regression is best described by example. Given an attached microSD card and USB flashdrive with IDs 6C2A-1618 and 8145-191C, respectively, here is the Android 11 story in wireless adb; at terminals in Pydroid 3 and Termux, and again in Python code run in Pydroid 3 and Termux. As you can see, internal storage and the embedded microSD card are still addressable at /sdcard and /storage/6C2A-1618, respectively; the USB drive which should be at /storage/8145-191C is not, and seems the only loser in the brave new Android 11 world.

In contrast, most file explorer apps do provide access to USB-drive files under Android 11; here's the phone and one of its USB folders in Total Commander. Tellingly, though, file explorers now open these files with a jumble of indirect or specialized techniques that seems suggestive of toolset incoherence:

Regardless of the hacks, direct USB access by POSIX-style path seems to have been taken away from most programs altogether.

This regression applies to source code run in both Termux and Pydroid 3; standalone Python Android apps built with tools like python-for-android and BeeWare; and any other program on Android that looks for USB content at its former mounted path. While it's relevant only to programs that make use of USB drives (and many do not), this is nearly a show-stopper for POSIX-based backup/sync programs like Mergeall; device-neutral pathnames are crucial to these programs' code, and the speed and convenience of USB drives themselves are essential to these programs' users.

Work-Arounds—or Not

Unlike other Android 11 issues in this update, there are a handful of candidate options for addressing this regression—including MTP, network drives, phone rooting, and microSD-card removal—but none qualify as universal solution, and all fall well short of external USB drives in terms of speed, reliability, and convenience. There simply is no substitute for USB drives today when it comes to content usability.

To verify such a broad claim, the following alternatives to direct USB-drive access were explored by the Mergeall team. All were tested with compare-only Mergeall runs against on-phone and on-PC copies of the same 124K-file, 10K-subfolder, and 178G content collection—a large but realistic use case, which led a happy on-phone life thanks to both Mergeall's incremental processing and fast USB drives. Crucially, all the USB-drive alternatives below also allow content collections to be addressed by pathname—the underlying requirement for selection by GUI choosers or command-line arguments, and processing by portable POSIX-based programs.

Direct USB-drive access met the pathname requirement perfectly; ran the Mergeall comparison in just 15 seconds; and scored high marks on reliability and convenience. Assuming this option has been forever locked down by Android 11, let's see how the challengers stack up:

Phone rooting
On phones that support rooting, it may still be possible to directly access a mounted USB drive, at its on-phone /mnt/media_rw/xxxx-xxxx Android pathname. If so, this would support on-phone syncs, and retain both the speed and convenience of the former /storage/xxxx-xxxx paths. More broadly, rooting grants super powers to access functionality taken away by vendors who seem to think that you are a security risk to your own phone. That's a viscerally Good Thing.

Regrettably, phone rooting is a complex task, may void warranties, and is out of scope for many or most users. It's also nearly impossible on many phones—including the Samsung US models in the Mergeall team's testing stable. Let's see what else we can try.

Removable SD cards
On phones that support an embedded microSD card, pulling the card out is always a last-resort way to manage your on-phone content. In this mode, you remove the card; bring it up to date by running a sync to it on your PC; reinsert the card in your phone; and, optionally, run a sync on your phone from the card to faster internal storage. This mode still works well with Mergeall on Android 11 in both its GUI and command-line modes, pathnames are as usual everywhere, and the syncs on both PC and phone are radically faster than all other options ahead on this list.

Unfortunately, microSD cards are not supported by many (and probably most) phones; juggling cards in and out of possibly encased phones this way is both inconvenient and fragile in the extreme; microSD is much slower than most USB 3.X drives available today; and it's possible that microSD cards may soon perish from this smartphone land altogether, rendering this option a nonstarter. Avoiding the card fumbling with on-phone syncs was also a primary motivation for porting Mergeall to Android in the first place; let's see if we can do any better.

MTP over USB
MTP is what normally happens when you plug your phone into a PC by USB cable. It's the native interface provided by Android to access to your phone's content from a PC, and was adopted by Android over USB mass storage, purportedly because it sidesteps Windows filesystem constraints, allows access controls, and minimizes user operations. MTP works reasonably well for the occasional copy/paste of a few files, and can also be used for broader syncs by accessing the attached phone's mount-point path on PCs.

When used for syncs, your phone is attached to your PC by a USB cable, and acts like an external drive; to sync to or from the phone, Mergeall (or similar) is run on the PC instead of the phone itself, using the phone's MTP pathname. Linux supports this out of the box at path /run/user/1000/gvfs/mtp:*, and mounted MTP drives can be accessed as /Volumes/Phone and drive letters on Mac OS and Windows, respectively, with third-party extensions (Windows options were never tested, for reasons about to be disclosed).

Unfortunately, MTP syncs are painfully slow: the Mergeall compare-only run that formerly finished in 15 seconds on phone by USB drive takes a whopping and fully impractical 53 minutes by MTP on an Ubuntu Linux machine, and about 43 minutes on two Mac OS boxes. Though largely moot, MTP as tested also omitted all .* "hidden" files in content lists on Linux (they'll be removed by syncs from phone to PC, even if important); and one Mac OS driver returned MTP-origin timestamps a comical 15 years in the future, and did not support phone writes or other updates (both killers for Mergeall, though Linux's built-in support handled timestamps and writes correctly).

Plainly, MTP is too slow for prime time. While some users with very small data sets and very large patience might be able to sync by MTP (or copy in full), this mode is clearly a mixed bag, and completely impractical for users who need to sync significant content collections. Given all the hype behind cloud storage, it seems reasonable to assume that wireless options would be more honed and viable than MTP; let's take a look.

Samba servers
Samba is a free implementation of the SMB file-server protocol (which has a substantial Microsoft heritage). Like MTP, you can use Samba from PC clients for both tactical drag-and-drop copies, as well as general syncs based on mount-point paths, though the performance needs of the latter tend to be much more intense. When used for syncs, a Samba server app is run on your phone, your PC connects to it wirelessly as a client (usually by Wi-Fi), and syncs are run on the PC using the Samba server's mounted path.

This scheme was unusable out of the gate. On both Linux and Mac OS, the Samba server proved impractically slow—and similar to MTP in speed. A compare on Linux using a cifs mount at /mnt/landrive, for example, ran a full 36 minutes; the same test on Mac OS by Finder connect at /Volumes/Share was so slow that the test was abandoned at nearly an hour in. These compare with USB's 15 seconds.

The Samba story on Windows 10 was even worse: connection was nearly impossible, due to the combination of that platform's hard-coded ports (it supports only 445 or 13X), and Android's port-number restrictions (it makes ports < 1027 off-limits sans rooting). Third-party port forwarders and SMB mounters may help on Windows, but all those explored seemed subpar in terms of complexity, quality, and status; in fact, the mounter recommended by one Android Samba app has ceased to be.

Obviously, Samba's speed and usability constraints disqualify it as a replacement for USB access on Android devices today (except for content collections so small that a simple drag-and-drop would suffice). Can other network-drive options do any better?

WebDAV servers
WebDAV is an open file-server protocol that extends HTTP, and boasts support in Apache. Like Samba, using this for syncs involves running a WebDAV server app on your phone, connecting to the phone wirelessly by PC, and running syncs on the PC using the server's mount path. This option seemed promising at first: the purchased server tested was much faster than both MTP and Samba, turning in a respectable 5-6 minute compare on both Mac OS (via Finder connect at /Volumes/192.168.1.2) and Linux (via davfs2 mount at /mnt/webdav; Nautilus was slower).

Unfortunately, this scheme still ran much slower than USB's 15-second compare, and yielded data corruption and transfer errors where supported at all. Mac OS's built-in client support produced glaring data corruption while viewing photos on the phone, and was ruled out quickly. Linux's support fared better and transferred data properly when it worked; but it was also complex to configure, and in the end proved too unreliable to use: it sported frequent disconnects from the server during syncs (generating file errors and skips in Mergeall), as well as sporadic segfaults and system errors (of the sort Ubuntu wants to hear about).

Also like Samba, WebDAV is not supported by Windows 10 on non-default port numbers required by Android, and the third-party tool tested to work around this was both slow and inaccurate, in one test completely skipping some of the subfolders in trees being synced. While some Linux failures might be attributed to the server app used, and restarting Wi-Fi on the client PC or server phone sometimes helped temporarily, another WebDAV app on the Play store failed to start altogether after install, and the disconnects pose major usability hurdles.

Clearly, WebDAV's performance and reliability issues mean that it can't stand in for a USB drive on Android today either. While some options were untested and your mileage may naturally vary with any of them, this page's list and space are now both exhausted.

Footnote: the Mac OS munge
WebDAV also brought to light a subtle potential for filename mismatches when illegal characters are propagated from Mac OS on FAT32 and exFAT drives, and served elsewhere. This gotcha is covered off-page here because it's not limited to Android 11. Briefly, spurious diffs can be caused by the Mac's mapping of illegals to/from Unicode privates, which may not be unmapped by servers running on other platforms. This further clouds the viability of WebDAV for Mac OS users. Although some Samba servers reverse the munge, its real fix lies in avoiding covert magic that works only in rarified Apple-only contexts (yes, a "fat chance" indeed).

So there you have it: the desperate and failed quest for alternatives to the direct USB-drive access that Android 11 seems to have rudely ripped away from its users, for the sake a mandatory security policy that could have easily been left to the discretion of a user toggle. In sum, MTP and wireless network syncs are largely just as slow and error prone as expected; phone rooting isn't desirable for most users and impossible for many; and microSD cards are a slow and clumsy substitute that isn't available on most phones.

In other words, there is no work-around to be found here. While we can still hope that USB drives will make a comeback, the best bet today for users with non-trivial content collections seems grabbing a phone that supports a removable microSD card. While they last, at least; alas, Android smartphones seem on the fast track to becoming content islands soon.

Prognosis

This regression's source and scope are unknown, but it has been reported to both Samsung and Android. If it's an unintended bug in either code base, it may be fixed soon. If it's a "by design" narrowing of functionality (and the file-opening behavior of file explorers suggests it may be), it renders Mergeall, and similar POSIX-based tools that rely on USB drives for data transfer, mostly unusable on Android 11.

Such tools have long relied on the fact that USB drives are accessed with the same sorts of pathnames as everything else. Without this device agnosticism, programs would need to allow for USB drives as a special case globally, and expand both file choosers and IO-related code to support them uniquely. This would be a massive rewrite effort required only for Android; would have to use Java-based and Java-biased APIs; and is unlikely to happen in portable software like Mergeall.

For device owners, the demise of addressable USB paths would mean having to either resort to painful microSD-ectomies, use impractically slow syncs over MTP or wireless protocols, or subscribe to cloud-based options eagerly promoted by device vendors but no better than the other alternatives we've explored here. As lamented previously, such narrowed utility would probably qualify as a death knell for many of the broader aspirations of the Android platform.

The Thrilling Conclusion

Update: after writing this section, a web search turned up a thread on the Samsung developers forum which raises a USB issue that seems either related or identical to that mentioned here; see the thread. If germane, this clearly extends beyond both Python apps and the Mergeall program. More importantly, Samsung has also now replied on the subject; at the risk of spoiling the surprise, be sure to see the likely conclusion of this topic in this page's next (and probably final) major section.

Global: Long-Running Tasks That Process Files Are Now Radically Slower

And finally, yet another breakage with potentially broad and dire consequences. Mergeall compare-only runs have now been clocked at 7 to 10 times slower on Android 11 than they were on Android 10 and earlier. This impacts both Termux and Pydroid 3, and might make POSIX-based data-processing programs impractical on Android alone—whether they manage to work around scoped-storage permission constraints or not. And it's not just Mergeall: scans for sizes in all file explorers tested are also much slower on Android 11, and unusably so for large folders. In fact, this appears to be a global performance regression that renders many long-running tasks impractical on Android 11, but we need to take a short empirical journey to see how deep the damage goes.

Initial Results: File Scans Crawl

To measure this regression, Mergeall was run on-phone in compare-only mode, with both FROM and TO set to the path of the same large folder in internal storage. Compare-only runs like this primarily exercise per-folder directory listings and per-file stat calls, but do not update files. The test folder held 124K files and 10K subfolders in 178G space (this is the same use case as in the preceding section), and was identical on the test devices: an Android-11 Note20, and an Android-Pie Note10. As noted earlier, this is a large data set, but it's easily manageable on any device with Mergeall's incremental processing.

The results: when run via Mergeall's GUI in Pydroid 3, this test took an average 104 seconds and 168 worst-case seconds on the Android-11 Note20, but just 15 average seconds and 11 best-case seconds on the Android-Pie Note10. Times for Mergeall command-line runs in Termux for the same folders were just as slow under Android 11—and frequently slower, often checking in at just over 200 seconds. For completeness, a two-year-old Oreo Note9 ran the test in 16 seconds, though the data set differed slightly.

In total, while nowhere near the prior note's massive 53 minutes logged for MTP, Android 11 still posted a 7X average slowdown from earlier Androids in Pydroid 3, and often 10X or worse. Moreover, the Note20 was already running comparable Mergealls 35-45% faster than the Note10 before the Note20 was forcibly upgraded to Android 11; hence, the true Android 11 sloth is larger than comparisons to the Note10 today suggest, as its slowdown also negates the former speedup.

Nor is Mergeall the only victim: all file explorer apps tested are impacted by this regression too. They appear to also be on the order of 10x slower when scanning folders to compute sizes on Samsung's Android 11. Their API choices are naturally unknown (they may or may not use POSIX calls), and their slowdowns haven't been formally quantified (i.e., app sloth exceeds tester patience). The slowdown is so obviously bad, though, that a Properties scan virtually hangs; this surely must have been noticed before this release pushed its way onto so many customers' phones.

The Bigger Hit: Tapped App = Quicker Task

Because the initial results we just met were so bad, a simple Python test script was coded to isolate and exercise the apparently slower IO operations used by Mergeall compare-only runs; see the test script here. For identical folder trees, testing showed this script to be a horrific 40X slower on Android 11 than Android Pie, which is substantially worse than 11's 7-10X slowdown for the full Mergeall program. In both Termux and Pydroid 3, Android 11 typically took 80 to 100 seconds to complete the script's tree walk, but Pie finished it in just 2 to 2.5.

But it gets weirder. Per later testing (well, really fortuitous accident), it turns out that Android 11 runs the test script dramatically faster—sometimes finishing in just 10 seconds—if any other app is interacted with while the app running the script continues in the background. This happens in both Termux and Pydroid 3, and a simple scroll in Settings suffices to speed the running script tenfold. A similar speed boost is seen in Mergeall's Pydroid 3 GUI if it's swiped aside to use other apps while it's running.

And even weirder. On further inspection, it also turns out that a same or similar speedup occurs in the Termux and Pydroid 3 apps' terminals, as well as tkinter GUIs run in Pydroid 3, if you simply keep tapping an app's screen while it's busy with a long-running task—like the test script or a full Mergeall run. Presumably, the screen activity forces task-scheduling logic to count the app as still active and avoids lowering the app's priority. Yes, really:

In truth, taps aren't strictly required—drawing imaginary circles on the app's display while it is running tasks will suffice. And no, it's not April Fool's day yet (though this is starting to look like a job for The Ministry of Silly Apps).

Before you ask: yes, the battery was amply charged when all these observations were made (low-battery mode wasn't in play). And yes, there's proof: here's a Termux screen capture showing a 10X boost; its first two runs were not tapped, but the second two were. Untapped performance sometimes varies, but tapped runs are always much faster, and this phenomenon is system wide per the last bullet above.

There is no explanation for this behavior yet (despite both toggling every related setting on the device, and searching every related term on the web). But it strongly suggests that a new task throttling based on screen activity in Android 11 (or Samsung's flavor of it) is a major factor in slowdowns, and easily overshadows POSIX IO sloth. This throttling may be done in the name of CPU sharing, or may be an overly aggressive battery optimization. Either way, it makes long-running tasks nearly unusable.

That being said, POSIX IO also appears slower in 11: tapped apps still don't match Pie's speed, and Android has already given notice about a deliberate 2X slowdown. Moreover, the screen-activity throttling is new in Android 11; is fairly labeled a bug which starves apps' long-running tasks of performance; and qualifies as a major regression and dealbreaker for many programs on this platform.

Unfortunately, the Mergeall team has no tools to pursue this further. Performance analysis is murky at best on Android (especially now that per-app CPU usage seems off-limits to mere mortals), so this will have to await more-privileged parties' attention. Please watch the Android Issue Tracker bug report for updates. Technically speaking, neurotic tapping may be a kind of a work-around for this issue, but a fix in Android 11 (or Samsung's flavor of it) seems a much more rational resolution.

Prognosis

To recap: it appears that there are two major performance regressions in Android 11 that impact Mergeall—both a specific POSIX-file handicap, and a broader screen-activity throttling—and no sane user or program work-around is known for either. It's unclear where causes lie, but the issues have been reported to both Samsung and Android, and may also be fixed soon: Samsung has already promised a fix for some known Android 11 speed issues later in January, though it's unknown if the fix will address the performance issues covered here.

If the slowdowns are instead intentional consequences of Android 11's new mandates, they will severely cripple (if not fully preclude) a host of POSIX-based and other software on Android 11, including all programs coded in Python that access files or perform non-trivial tasks normally, whether they are run as source-code or apps. In this scenario, program performance may have been deliberately sacrificed for the sake of Android 11's scoped-storage agenda. Regrettably, Android itself already seems to have suggested as much.

Time will tell if a half century of Unix practice is among the casualties of Android's newly found security obsession. All indicators today suggest it is, though only as long as developers enable such choices by rewarding this effectively closed platform with absurdly proprietary code. Dramatic to be sure, but this alarm needs sounding; the road to corporate tyranny is paved with customer complacency and developer complicity.

The Vendors Weigh In

Update: Samsung has now sent an initial reply, claiming that the performance issue is caused by Android's media scanner crawling an external SD card. The reply also suggests that the remedy is to delete files. There are at least four problems with this:

There are clearly major performance regressions in Android 11; whatever their underlying causes, Samsung's reaction hopefully won't be to bury its head in the sand. This won't make customers happy, won't sell phones, and won't help Android reach its potential.

Update: From the hate-to-say-I-told-you-so department, Android has now also replied on this issue, verifying that it is indeed a bona fide and deliberate performance downgrade; while they last, see the issue tracker post, and the AOSP doc page it references.

In short, the slowdown reflects an intentional tradeoff which penalizes heavy users of files in shared storage only. That is, Android 11 has been optimized for scoped storage at the expense of everything else. The only recourse for users is locating data in app-specific or app-private scoped storage, which has limited utility by design, and may be deleted along with the owning app. And the only remedy for POSIX file-processing programs is major rewrites to use the proprietary scoped-storage framework, which are off the table for most cross-platform code.

While it's unclear if this explanation accounts for all the throttling seen, it qualifies as both plausible and official. Hence, the game's up, performance regressions are likely here to stay, and Mergeall and similar tools that process content in shared storage will regrettably run much slower on Android 11. For this, we can thank the new "privacy over performance" mantra of Android 11, which is being pushed to the point of antagonism towards its users (many of whom will undoubtedly find "dogma over usability" the apter tagline).

In the end, Android 11's new selective sloth will be a major inconvenience for many, but it does not render devices wholly unusable (after all, you can always deploy the silly-taps trick to speed these now-dysfunctional phones). The resolution of the other global Android 11 breakage, however, may be fully terminal, per the final section ahead.

And More to Come: Scoped Storage’s Other Shoe Has Yet to Drop

And beyond all this, of course, the full impact of Android 11's scoped-storage permission changes remains to be seen, if and when Python-programming apps target Android 11. Some apps are already impacted (e.g., Samsung's own web browser app can no longer open file:// local-file URLs), but the Python apps noted in this coverage have yet to emerge from their opt-out havens. Even so, the current stay of execution already comes with the visible barbs noted above.

Again, please take this update in its context: it reflects just initial Android 11 testing on a single device; some its findings will likely be clarified or reversed over time; and this page may be updated as fixes and new information come in. For the present, though, Python and tkinter programs have clearly been harmed substantially by Android 11. While a Return-of-the-Jedi update here is still slimly possible, it grows less likely with each new "aw crap" issued by the Mergeall testing team.

In closing this update, it's worth noting for fairness that other platforms also suffer frequent breakages at the hands of today's mosh-pit development model. See the recent regressions in Linux here and here; Mac OS here and here; and Windows here and here. While the ultimate goal behind Android 11's many regressions is a matter of speculation, its morph more clearly jives with the rude and reckless motifs now sadly common in the software domain. Android has an especially clear pattern of both mandating languages and tools, and regularly breaking the programs which use them. Developer-friendly not, that is.

Retreat of the Jedi: USB is DOA (Jan-2021)

It's official: Samsung has now verified that the removal of USB-drive mount points described earlier is by design in Android 11, and cannot be fixed. This unfairly penalizes legacy apps, and is regrettably a game ender for a host of tools on this platform.

Per its reply, Samsung cannot do anything about this in its code base, because it's a new policy mandated by Google as of Android 11. Specifically, /storage/xxxx-xxxx paths can no longer be set; USB drives may be mounted at /mnt/media_rw/xxxx-xxxx, but using that path requires special permissions or a rooted device, and it is not accessible to most programs and users. Hence, general USB access is no more, this isn't limited to Samsung devices, and it's up to Android to fix this if it wishes to be inclusive of programs that depend on traditional processing paradigms.

You can read the full reply about the USB path's demise from Samsung here. Although a definitive reply from Google on this issue is still pending at a parallel post on the Android Issue Tracker, a new opinion-based decree from Google that wouldn't bend to the leading Android vendor seems unlikely to be reversed for a single developer.

Impact

Assuming its policies stand, Android 11 has knowingly maimed scores of programs that use the portable POSIX API and rely on USB drives for non-trivial content collections, including the Mergeall backup/sync program. Users of such programs are left with snaillike alternatives or microSD-card pulls, neither of which is universally practical or supported. And developers of such programs are left with broken products or costly rewrites to adopt Android's proprietary, dogmatically enforced, and Java-based APIs. Such rewrites are not an option for Mergeall, and seem unlikely elsewhere.

Eventually, Android 11's scoped-storage access restrictions described in the main doc will also greatly limit what users can do with their programs, content, and devices (sans the special permission and Java-based code described earlier). The full scope of these restrictions remains to be seen in Python-programming apps still loitering in an opt-out limbo that's partial at best; but they are already slowing the loiterers to a crawl, hobbling apps brave enough to target the new Android, and well on their way to becoming the norm.

Ultimately, though, all it took to torpedo content-management programs like Mergeall was removal of direct USB-drive access. Between this, its intentionally selective performance regressions, its new GUI glitches, and its looming access restrictions, Android 11 sadly does lock out many or most Python programs, and seems likely to be the sunset of this platform's general-purpose-computing ambitions.

Conclusions

On a personal note, the Android 11 auto-update has functionally bricked this developer's phone—and seems the same as an auto maker creeping into your driveway and removing your car's engine. This device today can no longer be used for one of the main purposes for which it was purchased (content backup and view), and the fault squarely lies with the dubious agendas underlying Android 11. While its downgrades are being aggressively spun as security "features," the rewards they seem most likely to secure are the revenue streams of cloud providers and mobile advertisers.

The future is of course unknown, and it's possible that Android 11 will still follow Windows 8's lead and reverse course over time. For the present, though, Android may have finally taken its place alongside the iPhone, as a deliberately closed and limited media portal for technically naive consumers. The mass market of exploitable customers which seems the true target of these devices' vendors probably won't care. But some developers will, especially those who craft programs beyond the trivial, and understand that "apps" has always been a derogatory term.

In the end, this page's takeaway is a simple one: If you care about using Mergeall, tkinter GUIs, or your device in general, you should consider avoiding an Android 11 update unless and until its many issues are addressed. This page may be revised again if improvements arise, and hopes they will; these devices have great potential yearning to be unlocked.

But for now, this humble Python hacker won't be moving ahead with new development on this platform, and will be flashing a ROM to revert to Android 10 very soon (learn how here and here). A retreat, perhaps, but it's also a lot more usable, and a lot less fascist.



[Python Logo] Top Media Mergeall Tkinter News Blog Apps Input ©M.Lutz