Latest update: June 12, 2021
This page hosts updates concerning Android 11's implications for POSIX-based software, especially Python programs and Python/tkinter GUIs run as source-code in the Pydroid 3 and Termux apps. It began as an inline note at the end of this appendix—whose background context is effectively prerequisite reading here—but grew into content worthy of its own stand-alone page somewhere around update #3. Moving this outside the Mergeall-on-Android package also better accommodates future revisions, which seem about as certain in the Android world as death and taxes in the real one.
While this page is really just an extension of the Mergeall main doc and its tkinter cousin, you can view its own media as a set in this gallery.
The TL;DR Summary Box
As of early 2021, Android 11 is known to have revoked general access to USB drives; made programs that process files in shared storage unusably slow; spawned new glitches in the Pydroid 3 and Termux apps; and reduced the functionality of apps like file explorers and web browsers. In fact, it's so bad that the Mergeall team opted to roll devices back to Android 10 after just a few weeks of testing. Your opting may vary, but Android seems destined for either computing irrelevance, or a Windows-8ish mea culpa that reverses 11's debilitating agendas. For more spoilers, see the alternative summary off page, or jump ahead here to the postscript. Better yet, read on. Given the hold these gadgets have quickly taken on the human species, this stuff does matter.
Android blinked in late April 2019—or at least called a temporary cease-fire with its users. Per the blog post here, enforcement of the new internal-storage permission constraints described in this appendix has been delayed until Android 11, the successor to 10 expected to appear in 2020 and become common on consumer devices by 2021 (and originally named R).
While welcome news, this really just postpones the tragedy by one year for developers and end users alike. A lot can happen in that year, but people in both camps are left hoping that its manifest won't include a crippling change which reduces Android devices to media-consumption portals, designed to turn private lives into profit centers. The odds are sadly slim in a world where billions seem happy with the Faustian bargain.
Per the latest Android 11 specs and docs in fall 2020,
it now appears that programs which require broader read+write access
to shared internal storage that survives app uninstalls may be
able to acquire it with the new
MANAGE_EXTERNAL_STORAGE manifest permission,
plus per-app system settings made at users' discretion.
Per current sources, some Java-based code may also be required
to invoke or manage this permission at runtime, but the code could
be self-contained, and seems possible in Python (see
This new option, known as All files access, might be perfect for
file managers, virus scanners, and incremental backup/sync programs like
that allow their users to process arbitrary folders on earlier Androids.
It appears to grant access nearly as permissive as before, though other apps'
Android/data folders may be off-limits.
The new option also remains to be evaluated by the Mergeall team, but you can read about it for now
at these pages (while they last; Android developer pages trend towards transience):
an FAQ on the
and Google Play edicts
Per the last of those links, using the new permission in apps uploaded to the Play store requires special approval—a point of control which hopefully won't be misused (see the recent Fortnite fiasco), but can be subverted in some contexts by Android app sideloading. Per the other links, the new permission may also make the Storage Access Framework unnecessary, but it's unknown if it will be available in Python-hosting apps like Termux and Pydroid 3, or require creation of fully standalone apps. Either way, apps using All files access cannot be uploaded to Google Play until early 2021, so you'll have to stay tuned for the conclusion to this saga (and appendix).
Thanks to an unexpected auto-update from Samsung, the Mergeall testing team has now had a chance to evaluate the apps and programs described in the main doc under Android 11. This update reflects initial testing with that platform on an unrooted Samsung Galaxy Note20 Ultra, as well as later results on a Pixel (per the update box below). While its findings may or may not apply to other devices, and are prone to change over time, users and developers alike are stakeholders in the Android 11 tale told here.
To tell this story efficiently and fairly, this section presents its results as a sort of incremental developer journal, which is amended as new facts emerge. Some of its coverage will make more sense if you have a basic understanding of Android storage categories and paths; for a quick intro, try this primer. Taking that background as a given, here is the wreckage uncovered while testing Android 11, and covered in this update:
The good news is that apps which still target earlier Androids get a temporary
of scoped storage in Android 11, and seem, at least at first glance, to mostly work
as they did before. In fact, permissions for such apps—now branded "legacy"—are
in some ways more liberal than before. Here's the Android 11 permission stories captured in
after their permission-granting steps. Compare these to behavior on Android 10 and
earlier presented in the
main doc; we'll revisit
these results ahead (as well as new limits in file explorers), but these apps
now can access microSD-card files more freely, and the root of internal shared storage
/sdcard and its
is still fully usable for content storage and management.
The bad news is that the reprieve is nowhere near complete: as expected, Android 11 does break the Python programs mentioned in this document, in the multiple ways covered below. Most grievously, general access to USB drives has been revoked, even for "legacy" apps; and shared storage has been slowed nearly into oblivion, taking many long-running file-processing tasks with it. On top of this, Android 11 introduces new glitches to tkinter GUIs, trims the utility of tools like file explorers and web browsers, and bizarrely bungles permissions in some apps. In worst cases, the Android 11 breakages have the potential to render Python programs wholly unusable on this platform, in both source-code and standalone-app forms.
Please note up front that findings here mostly reflect an Android 11 upgrade rolled out to Samsung devices in late December 2020 and briefly tested over a few weeks. This upgrade has additional unrelated problems given scant or no coverage here, that seem to call its quality into question in general (e.g., twilight-zone states in file-explorer apps). To be sure, 11 still feels like a beta, and some of its ruinations noted below may be unintentional bugs which may be addressed soon in updates from apps, Samsung, or Android. Breakages which reflect deliberate functionality downgrades, however, may make the Android platform permanently out of scope for POSIX-based tools like Python.
Also keep in mind that many of this update's threads have evolved over time. While some issues' descriptions are being changed in place to reflect new findings, others are being amended in their blue-shaded end-of-section noteboxes as new info dribbles in. Be sure to see the updates in these boxes—including the first that follows—for full disclosure.
It’s Not Just Samsung: 11 Busts Pixels Too
After this page was written, all the glitches it describes were verified to also exist on a Google Pixel 4a running Android 11. On the Pixel too, USB-drive paths are nowhere to be found without rooting, shared storage is radically slower than app-specific/private storage, and all other issues are present. Hence, these are not unique to the Samsung Galaxy auto-upgrade casualty, and Android 11 has the same defects elsewhere. The descriptions that follow have been updated accordingly, and augmented for findings unique to the Pixel (e.g., rooting). As we'll see, Mergeall's GUI runs on both the Note20 and the Pixel, but it's not very useful with Android 11 on either.
First up on our damages tour is a new GUI glitch that applies to Pydroid 3 alone, and will hopefully be repaired in an app update soon. At present, though, tkinter text-input widgets are unusable in this app, because shifted input keys are doubled, and on-screen keyboards don't survive an app switch. Unlike all other breakages on this list, this one wasn't caused by Android 11 (it wound up being app bugs), but it hampers tkinter GUIs run on 11 nonetheless.
In tkinter GUIs run by this app's current version on Android 11, all text-input
widgets erroneously generate double characters when shifted. For example,
AA in these widgets,
@ (or just
2 on some keyboards) generates an
@2 combo. The stuttered effect is captured in this GUI's
This key doubling happens on both Bluetooth and on-screen keyboards, and for all on-screen keyboards tested (Hacker's Keyboard, Samsung's and Microsoft's keyboards, and Google's Gboard), and it occurs before any extra keyboards are installed. In addition, screen taps no longer restore on-screen keyboards, if the app is running a tkinter GUI and is temporarily left for another app; your only input recourse is to restart the GUI.
These glitches were observed on both a Samsung Galaxy Note20 Ultra and a Google Pixel 4a, in both cases running Pydroid 3 version 4.01_arm64; it's likely that they impact other devices too. The glitches do not occur on other Samsung devices running earlier versions of both Android and Pydroid 3, and versions 2 and 3 of Pydroid 3 are both free of this bug (though they have repaired others over time).
There are no known user or program work-arounds for these regressions. These are bugs somewhere in the Pydroid/SDLTk/SDL2 stack which implements tkinter in the Pydroid 3 app. They've been reported to Pydroid 3, and seem likely to be fixed soon. At present, though, they make most tkinter programs largely unusable for many or all devices running Android 11. Key doubling can be addressed by pressing Backspace after each shifted key, but it's preclusively painful; and lost keyboards can be avoided by not leaving the app, but this isn't ideal on a device built for multitasking. If you care about running tkinter GUIs on Android in Pydroid 3, stick with earlier releases until fixes appear.
This glitch's source and scope were initially unclear, but later testing proved that it wasn't caused by Android 11 or limited to Samsung devices. The description above now reflects this, but the box below details these later findings.
Stop the Presses: 10 Doubles Too
After writing this section's first draft, input-key doubling was also observed on Samsung devices running Android 10 (specifically, on a test device rolled back from 11 to 10). Given the timing of releases, this makes it likely that the glitch is a bug introduced by Pydroid 3's version 4, and not by Android itself. Hence, it's not new in Android 11, though it's still present there, and the bug parameters above still apply. Alas, tkinter text input apparently hasn't been right in Pydroid 3 since the reign of Android Pie, and badly needs to be fixed (though at least one user apparently didn't type uppercase in four months of 10 before 11).
And So Does the Pixel
The tkinter GUI glitches described in this section have now also been seen on a Pixel 4a, so it's not just a Samsung issue. They've been posted as a Play-store review too, because the app's email contact never replied to three emails over a month (and ignored others in the past). These bugs are still hopeful of a fix soon, but to be blunt, Pydroid 3's customer service seems as poor as its quality control; with any luck, both will be upgraded along with the app.
Next up is a system-level glitch genuinely new in Android 11, and observed again only in Pydroid 3. In this app alone, activity-manager command lines used to spawn a web-page or text-file view intent no longer work on Android 11; the user's request appears to silently do nothing, as the command fails on permission errors. This will also hopefully be a quick fix for this app, but it renders media views in most tkinter GUIs broken today.
In many of the tkinter GUIs covered in
and run by Pydroid 3 on Android, viewer popups for HTML Help files and
other text content are generated by spawning Android
activity-manager command lines with Python's
commands launch a view intent that opens files by URL or path (demoed
negate a Pydroid 3 hack which disables Python's
webbrowser module for most
files (as described
the kludge special-cases
file:// docs for Jupyter and breaks all others);
and work well on Androids Oreo through 10.
As of the Android 11 upgrade, unfortunately, these activity-manager command lines
now fail on permission errors on both Galaxy and Pixel devices.
The failure is nearly mute, except for a clue left in the
that notes "com.android.shell does not belong to uid=xxxx." To most users, though, the net
effect is that GUI Help buttons, and other viewer popups, now silently do nothing.
Importantly, this occurs in Pydroid 3 only. On Android 11, the activity-manager
command works in the Termux app—at both its terminal and in Python code
it runs—but fails in both contexts in Pydroid 3. Here's the scene in
Moreover, the same activity-manager command works in
adb shell (see
shell commands (e.g.,
ls) run correctly in Pydroid 3 from both its
terminal and Python code via
os.system(), as before;
only activity-manager commands fail, and only in Pydroid 3 on Android 11.
There are no known user work-arounds for this regression. It's unclear if it
reflects Pydroid 3 permission choices, or a deliberate narrowing of permissions
in Android 11. The fact that activity-manager commands work well in Termux, though,
strongly suggests the former, which seems more hopeful of an early fix.
If this instead stems from a new Android 11 constraint, it might still be
addressed by changing programs to use the
webbrowser module, but
only if Pydroid 3's biased and limiting hack is ever backed out.
Lest this page be accused of picking on Pydroid 3, let's move on to an issue seen only in Termux (so far). Though triggers are elusive, Android 11 has been seen to revoke file-access permissions from the Termux app, leaving it partly disabled. This happens only in 11, and feels like an unintentional glitch that's likely to be repaired soon; but it calls the robustness of Android 11's scoped-storage changes into question.
As described both in the
the Termux app requires its
termux-setup-storage command to be run to
trigger its permission-granting dialog. However, even after this is done—and
for reasons disturbingly unknown—Termux may abruptly lose access
to every internal-storage folder except its own app-specific/private folders.
For example, an
ls /sdcard works fine in the app initially,
but then stops working suddenly. Just as strangely, reads of non-app folders
are disabled after the switch, but writes are still allowed. This makes no sense,
though it seems to have been confirmed in a related Termux
about Android 11 storage permissions.
To date, this issue has been observed only in Termux, whose session logs
the access demotion capture its impact in full detail.
In essence, this app is randomly
downgraded to the sort of permissions it might hold without
termux-setup-storage, and by default if it targeted Android 11 and its scoped-storage
model. The latter seems unlikely for this app's tested version (0.101),
given its September 2020 date.
This permission downgrade was seen twice in as many days with no apparent trigger on the Note20—and made a surprise reappearance during later testing on the Pixel 4a—and the only known remedy is to reinstall the app. This seems at least anecdotal evidence that the scoped-storage opt-out for legacy apps in Android 11 is still a bit flaky. Probably, the Termux app will require a new release to fully support Android 11's permissions model. As this downgrade has not been seen in Pydroid 3 (yet?), running Mergeall's GUI in that app suffices as a work-around for that program and others today.
And now a potential endgame for a host of programs. As of Android 11,
USB drives can no longer be accessed with their former
/storage/xxxx-xxxx drive-ID mount
paths, and the POSIX
Both worked well for years on unrooted Galaxy devices (specifically,
Notes 9 through 20, running Androids Oreo through 10), but are unusable on
This is true for both Galaxy and Pixel devices, and seems likely elsewhere.
If permanent, this makes it effectively impossible for portable
POSIX-based programs like
Mergeall to sync on-phone content to and from USB external drives,
and leaves users who have non-trivial content collections with
few good options on these devices, as we'll see here.
For such programs and users, phones become "just phones"—which may be the
bitter but true point of 11.
This regression is best described by example.
Given an attached microSD card and USB flashdrive with IDs
respectively, here is the Android 11 story in wireless
at terminals in
Pydroid 3 and
and again in Python code run in
Pydroid 3 and
As you can see, internal shared storage and the embedded microSD card are still addressable at
the USB drive which should be at its former
/storage/8145-191C is not,
and seems the only loser in the brave new Android 11 world.
In contrast, most file explorer apps do provide access to USB-drive files under Android 11; here's the phone and one of its USB folders in Total Commander. Tellingly, though, file explorers now open these files with a jumble of indirect or specialized techniques that seems suggestive of toolset incoherence:
content://com.android.externalstorage.*or similar URIs, which invoke content providers to access files indirectly
/storage/emulated/0/.localcache/usbcache folder, which hosts only files opened in this manner
/mnt/media_rw/xxxx-xxxxdrive-ID mount paths, which are inaccessible everywhere else, including
This regression applies to source code run in both Termux and Pydroid 3; standalone Python Android apps built with tools like python-for-android and BeeWare; and any other program on Android that looks for USB content at its former mounted path. While it's relevant only to programs that make use of USB drives (and many do not), this is nearly a show-stopper for POSIX-based backup/sync programs like Mergeall; device-neutral pathnames are crucial to these programs' code, and the speed and convenience of USB drives themselves are essential to these programs' users.
Unlike other Android 11 issues in this update, there are a handful of candidate options for addressing this regression—including MTP, network drives, phone rooting, and microSD-card removal—but none qualify as universal solution, and all fall well short of external USB drives in terms of velocity, accuracy, and simplicity. There just is no substitute for USB drives today when it comes to content usability.
Because such broad claims demand evidence, the Mergeall team explored the handful of common alternatives to direct USB-drive access listed below. All were tested on an Android-11 Note20 Ultra with compare-only Mergeall runs against on-phone and on-PC copies of the same 124K-file, 10K-subfolder, and 178G content collection. This is a large but realistic use case, that's easy to manage with on-phone, on-demand syncs, thanks to Mergeall's incremental processing and fast USB drives. Like USB drives, all the alternatives below allow this content collection's copies to be addressed by pathname—the underlying requirement for both selection by GUI choosers or command-line arguments, and device-agnostic processing by portable POSIX-based programs.
Android's former USB-drive access met the pathname requirement perfectly; allowed Mergeall to be run directly on phones in Python apps; completed the same or similarly sized comparisons on phone in roughly 30 seconds on an Android-10 Note20 Ultra with an external SSD (and about a minute on an Android-Pie Note10+ with a slower flashdrive); and scored high marks on precision, reliability, and ease of use. Assuming this option has been forever locked down by Android 11, let's see how the challengers stack up:
/mnt/media_rw/xxxx-xxxxpathname. If so, this would support on-phone syncs, and retain both the speed and convenience of the former
/storage/xxxx-xxxxpaths. More broadly, rooting grants super powers to access functionality taken away by vendors who seem to think that you are a security risk to your own phone. That's a viscerally Good Thing.
Regrettably, phone rooting is a complex task, may void warranties, and is out of scope for many or most users. It's also nearly impossible on many phones—including the Samsung US models in the Mergeall team's testing stable. (The later Pixel-rooting update at the end of this section provides more details and a counterargument, but rooting's scope is still drastically limited.) Let's see what else we can try.
Unfortunately, microSD card ports are not available on all Android phones; juggling cards in and out of possibly encased phones this way is both inconvenient and fragile in the extreme; microSD is much slower than most USB 3.X drives available today; and it's possible that microSD cards may soon perish from this smartphone land altogether, rendering this option a nonstarter (at this writing, Samsung has just ditched them on its new S21). Really, using on-phone syncs to avoid card fumbling was a primary motivation for porting Mergeall to Android in the first place; let's see if we can do any better.
When used for syncs, your phone is attached to your PC by a USB cable, and
acts like an external drive; to sync to or from the phone, Mergeall (or similar)
is run on the PC instead of the phone itself, using the phone's MTP pathname.
supports this out of the box at path
and mounted MTP drives can be accessed
/Volumes/Phone and drive letters on
(f.k.a. Mac OS) and
(Windows options were never tested, for reasons about to be disclosed).
Unfortunately, MTP syncs are painfully slow: the Mergeall
compare-only run that formerly finished in some 30 seconds on phone
takes a whopping and fully impractical
by MTP on an Ubuntu Linux
and about 43 minutes on two macOS
MTP as tested also omitted all
files in content lists on Linux (they'll be removed by syncs from phone to PC,
even if important); and one macOS driver returned MTP-origin timestamps a
comical 15 years in the future, and did not support phone writes
(the former kills syncs in general, and the latter precludes to-phone updates).
The macOS driver fixed its timestamps in a later
and Linux's built-in support always handled both timestamps and writes correctly,
but MTP's speed problems makes all this moot.
Plainly, MTP is too slow for prime time. While some users with very small data sets and very large patience might be able to sync by MTP (or copy in full), this mode is clearly a mixed bag, and completely impractical for users who need to sync significant content collections on demand. Given all the hype behind cloud storage, it seems reasonable to assume that wireless options would be more honed and viable than MTP; let's take a look.
This scheme was unusable out of the gate.
On both Linux and macOS,
the Samba server proved impractically slow—and similar to MTP in speed.
A compare on Linux using a
cifs mount at
for example, took an unrealistic 36 minutes, and the same test on macOS by Finder
/Volumes/Share was so slow that the test was
abandoned unfinished at nearly one hour in.
These all compare with USB's 30-seconds
The Samba story on Windows 10 was even worse: connection was nearly impossible, due to the combination of that platform's hard-coded ports (it supports only 445 or 13X), and Android's port-number restrictions (it makes ports < 1027 off-limits sans rooting). Third-party port forwarders and SMB mounters may help on Windows, but all those explored seemed subpar in terms of complexity, quality, and status; in fact, the Windows mounter recommended by one Android Samba-server app has ceased to be.
Obviously, Samba's speed and usability constraints disqualify it as a replacement for USB access on Android devices today when it comes to on-demand syncs (except for content collections so small that a simple drag-and-drop would suffice—and probably be simpler by MTP). Can other network-drive options do any better?
/Volumes/192.168.1.2) and Linux (via davfs2 mount at
/mnt/webdav; Nautilus was slower). Here's a shot of the server and a log from its Mergeall client.
Unfortunately, this scheme still ran much slower than USB's 30-second compares (10-12X, to be precise). Much worse, it yielded data corruption and transfer errors where supported at all. macOS's built-in client support produced glaring data corruption while viewing photos on the phone, choked on munged filenames per the footnote ahead, and was ruled out quickly. Linux's support fared better and transferred data properly when it worked; but it was also complex to configure, and in the end proved too unreliable to use: it sported frequent disconnects from the server during syncs (causing file errors and skips in Mergeall), as well as sporadic segfaults and system errors (of the sort Ubuntu wants to hear about).
Also like Samba, WebDAV is not supported by Windows 10 on non-default port numbers required by Android, and the third-party tool tested to work around this was both slow and inaccurate, in one test completely skipping some of the subfolders in trees being synced. While some Linux failures might be attributed to the server app used, and restarting Wi-Fi on the client PC or server phone sometimes helped temporarily, another WebDAV app on the Play store failed to start altogether after install, and the disconnects pose major usability hurdles.
Clearly, WebDAV's performance and reliability issues mean that it can't stand in for a USB drive on Android today either. While some options were untested and your mileage may naturally vary with any of them, this page's space and patience are now both nearly exhausted, Let's take a quick look at one last option.
Sadly, though, the FTP server also botched modtimes badly: it returned them as 12:00 AM for every file older than 6 months, and arbitrarily bungled their seconds component for all others. The net effect is lethal to timestamp-based syncs, because every file registers as a difference. In addition, macOS mounted the server as read-only, which precludes to-phone propagation; and this scheme shared the macOS filenames and symlinks munges of the upcoming footnote, which can trigger file loss and errors. By contrast, USB compares correctly, and radically faster.
For FTP's complete results, see this logfile. While they may vary for other apps and platforms untested, finding workable USB alternatives now seems akin to hunting unicorns. Networking is innately slower than USB, and even if you stumble onto a scheme that meets your goals, it will likely come with downsides and complexities that push it well out of scope for all but the most hard-core phone users. For this page, the hunt has reached its end.
Per the brief look, though, FTP, like other server schemes and MTP, might suffice for tactical drag-and-drop copies, but isn't up to the task of larger-archive syncs. The latter demands the speed, accuracy, and simplicity of the direct USB access that's regrettably been yanked away in Android 11—a fate ironically doled out to FTP itself at the same hands.
\uf027), which may not be unmapped by servers running on other platforms. This further clouds the viability of WebDAV for macOS users. Although some Samba servers reverse the munge, its real fix lies in avoiding covert magic that works only in rarified Apple-only contexts. A fat chance indeed (and yes, the pun is a stretch).
Update: this munge was later seen to also affect both MTP and FTP servers used from macOS (filenames mapped by macOS won't match originals when served by these protocols to macOS); and a related macOS munge similarly triggers diffs for Unix symlinks written to Windows-filesystem drives by macOS (symlinks are written as simple files, which won't match the original items' types on macOS when served from elsewhere). In both cases, macOS's silent changes can lead to file loss or errors on later non-USB syncs. See the off-page note's updates for more details.
So there you have it: the desperate and failed quest for alternatives to the direct USB-drive access that Android 11 seems to have rudely ripped away from its users, for the sake a mandatory security policy that could have easily been left to the discretion of a user toggle. In sum, MTP and wireless network syncs are largely just as slow and error prone as expected; phone rooting isn't desirable for most users and impossible for many; and microSD cards are a slow and clumsy substitute that isn't available on many phones.
In other words, there is no work-around to be found here. While we can still hope that USB drives will make a comeback, the best bet today for users with non-trivial content collections seems grabbing a phone that supports a removable microSD card. While they last, at least; alas, some Android smartphones seem on the fast track to becoming content islands soon.
This regression's source and scope are unknown, but the demise of USB paths has been reported to both Samsung and Android. If it's an unintended bug in either code base, it may be fixed soon. If it's a "by design" narrowing of functionality (and the file-opening behavior of file explorers suggests it may be), it makes Mergeall, and similar POSIX-based tools that rely on USB drives for data transfer, mostly unusable on Android 11.
Such tools have long relied on the fact that USB drives are accessed with the same sorts of pathnames as everything else. Without this device agnosticism, programs would need to allow for USB drives as a special case globally, and expand both file choosers and IO-related code to support them uniquely. This would be a massive rewrite effort required only for Android; would have to use Java-based and Java-biased APIs; and is unlikely to happen in portable software like Mergeall.
For device owners, the demise of addressable USB paths would mean having to either resort to painful microSD-ectomies, use impractically slow syncs over MTP or wireless protocols, or subscribe to cloud-based options eagerly promoted by device vendors but no better than the other alternatives we've explored here. As lamented previously, such narrowed utility would probably qualify as a death knell for many of the broader aspirations of the Android platform.
Or so ends the initial take. The rest of the story behind the demise of USB access unfolded over time with vendor feedback and new devices, so please see the updates box that follows (and the conclusion it references) for later discoveries and notes.
Other Sightings in the Wild
After writing this section, a web search turned up a thread on the Samsung developers forum which raises a USB issue that seems either related or identical to that mentioned here. If germane, this confirms that this extends beyond both Python apps and the Mergeall program.
Also after writing this section, USB-drive
paths proved to be nonexistent on a Google Pixel 4a running Android 11 too,
confirming that this isn't just a Samsung issue. To be fair, it's unknown
if Pixels ever provided path-based USB access (alas, Mergeall's testing
farm till recently has been limited to Samsung stock). Today, though, USB drives
are nowhere to be found at
/storage on the Pixel;
throws permission errors on unrooted phones; and file explorers treat USB
content specially. The USB lockout is Android-wide.
The foregoing being said, rooting does allow the Pixel to access USB drives
/mnt/media_rw/xxxx-xxxx, as suggested
above. Here's the fully functional post-rooting scene
at least in this app, using the
/mnt path and a
su command enables on-phone USB syncs with Mergeall as before, and similar
access is available for other Python and POSIX-related code after rooting.
A rooting how-to is beyond this doc's scope, but for pointers on the
process, try the
XDA 4a page;
Google flashing guide;
and local search engine.
On first glance, rooting may seem to solve the USB-access problem on devices that can be altered this way (and some rootable Android phones have hardware that's largely on par with Galaxy flagships today: see the web). On closer inspection, though, this option is still too narrow to qualify as a general fix, for a number of reasons:
su -ccommand lines, but no way has yet been found to run tkinter GUIs in this app with the enhanced permission. The app's Terminal cannot start GUIs,
suor not; running
suin GUIs is a no-op; spawning GUIs from shim scripts via
su -copens Terminal and fails; and an
suopens the app but loses
su. In short, tkinter is IDE-only in Pydroid 3, and the IDE seems to lack
su. This remains TBD, but if the constraint is firm, rooting is a USB fix for Mergeall's command lines only; its GUI is still unable to access USB on 11.
data/folders is wildly faster than shared storage. Since users should generally store content in the latter to avoid accidental removals on uninstalls (and allow cross-app processing where helpful), many programs will still crawl even if they gain access to USB by rooting. We need to move on to see why.
It's worth adding that
may offer additional options beyond those of typical rooting;
and Pydroid 3 might run with super-user permission if installed in
/system/app as a
Neither scheme was pursued, because both are even further afield than basic rooting,
and easily out of scope for this doc.
If you have to flash firmware and remount system partitions to use an app,
there's probably something larger amiss on the target platform.
It's also worth adding that the future of Android rooting is currently under attack from new hardware attestation in Google's SafetyNet security layer. In short, some apps may not work on newer rooted phones (even some that seem to qualify as innocuous). While it remains to be seen if work-arounds can be found, the rooted Pixel 4a fails attestation today, putting super-user solutions on even shakier ground.
The Thrilling Conclusion
Most importantly, Samsung has also now definitively replied on the USB subject; at the risk of spoiling the surprise, be sure to stay tuned for the likely conclusion of this topic in this page's last (and probably final) section.
Finally, yet another breakage with potentially broad and dire consequences. Mergeall compare-only runs have now been clocked at 10 times slower on Android 11 than they were on Android 10 and earlier, and can be a colossal 100X slower on 11 in some contexts. This impacts usage in both Termux and Pydroid 3, and might make POSIX-based data-processing programs impractical on Android alone—whether they manage to work around 11's other constraints or not. And it's not just Mergeall: scans for sizes in all file explorers tested are also much slower on Android 11, and unusably so for large folders. The slowdowns ultimately reflect a deliberate performance downgrade for shared storage, which seems related to a filesystem change, and may trigger other throttling; but the full story requires a short empirical journey to see how deep the damage goes.
To measure this regression, Mergeall was run on-phone in
compare-only mode, with
both FROM and TO set to the path of the same large folder in internal storage
shared by all apps.
Compare-only runs like this primarily exercise per-folder directory listings
stat calls, but do not update files.
The test folder held 124K files and 10K subfolders in 178G space (this is the
same use case as in the preceding section but without a USB drive), and was
identical on the test devices: an Android-11 Note20 Ultra, and an Android-Pie Note10+.
As noted earlier, this is a large data set, but it's easily manageable on any
device with Mergeall's incremental processing.
The results: when run via Mergeall's GUI in Pydroid 3, this test took an average 104 seconds and 168 worst-case seconds on the Android-11 Note20, but just 15 average seconds and 11 best-case seconds on the Android-Pie Note10. Times for Mergeall command-line runs in Termux for the same folders were just as slow under Android 11—and frequently slower, often checking in at just over 200 seconds. For completeness, a two-year-old Oreo Note9 ran the test in 16 seconds, though the data set differed slightly.
In total, Android 11 posted a 7X average slowdown from earlier Androids in Pydroid 3 on these tests, and often 10X or worse. While the effect is nowhere near the prior note's massive 53 minutes logged for MTP (and cannot account for the slowness of other server options), it still qualifies as extreme. Moreover, the Note20 was already running comparable Mergealls 35-45% faster than the Note10 before the Note20 was forcibly upgraded to Android 11; hence, the true Android 11 sloth is larger than comparisons to the Note10 today suggest, as its slowdown also negates the former speedup.
Nor is Mergeall the only victim: all file explorer apps tested are impacted by this regression too. They appear to also be on the order of 10x slower when scanning folders to compute sizes on Samsung's Android 11. Their API choices are naturally unknown (they may or may not use POSIX calls), and their slowdowns haven't been formally quantified (i.e., app sloth exceeds tester patience). The slowdown is so obviously bad, though, that a Properties scan virtually hangs; this surely must have been noticed before this release pushed its way onto so many customers' phones.
But it gets worse. Because the initial results we just met were so bad, a simple Python test script was coded to isolate and exercise the apparently slower IO operations used by Mergeall compare-only runs; see the test script here. For identical folder trees, testing showed this script to be a horrific 40X slower on Android 11 than Android Pie, which is substantially worse than 11's 7-10X slowdown for the full Mergeall program. In both Termux and Pydroid 3, Android 11 on the Note20 typically took 80 to 100 seconds to complete the script's tree walk, but Pie on the Note10 finished it in just 2 to 2.5.
And it gets weirder. Per later testing (really, fortuitous accident), it turns out that Android 11 runs the test script dramatically faster—sometimes finishing in just 10 seconds—if any other app is interacted with while the app running the script continues in the background. This happens in both Termux and Pydroid 3, and a simple scroll in Settings suffices to speed the running script tenfold. A similar speed boost is seen in Mergeall's Pydroid 3 GUI if it's swiped aside to use other apps while it's running.
And even weirder. On further inspection, it also turns out that a same or similar speedup occurs in the Termux and Pydroid 3 apps' terminals, as well as tkinter GUIs run in Pydroid 3, if you simply keep tapping an app's screen while it's busy with a long-running task—like the test script or a full Mergeall run. Presumably, the screen activity forces task-scheduling logic to count the app as still active and avoids lowering the app's priority. Yes, really:
In truth, taps aren't strictly required—drawing imaginary circles on the app's display while it is running tasks will suffice. And no, it's not April Fool's day yet (though these apps seem perilously close to qualifying as comedy props).
Before you ask: yes, the battery was amply charged when all these observations were made (low-battery mode wasn't in play). And yes, there's proof: here's a Termux screen capture showing a 10X boost; its first two runs were not tapped, but the second two were. Untapped performance sometimes varies, but tapped runs are always much faster, and this phenomenon is system-wide per the last bullet above.
There is no explanation for this behavior yet, despite both toggling every related setting on the device, and searching every related term on the web. But it strongly suggests that a new task throttling based on screen activity in Android 11 (or Samsung's flavor of it) is a major factor in slowdowns, and perhaps overshadows IO sloth. This throttling may be done in the name of CPU sharing, thermal management, or an overly ambitious battery optimization, and might be triggered by other factors or slowdowns (e.g., IO churn might aggravate it, or vice-versa). Some such app throttling may even be normal on phones, but it has assumed deadly proportions in 11, and makes many long-running tasks nearly unusable.
That being said, POSIX-based IO also appears slower in 11: tapped apps still don't match Pie's speed, and Android has already given notice about a deliberate 2X slowdown (more in the updates ahead). Moreover, the screen-activity throttling is either new or escalated in Android 11; is fairly labeled a bug which starves apps' long-running tasks of performance; and qualifies as a major regression and a likely dealbreaker for numerous programs on this platform.
Unfortunately, the Mergeall team has no tools to pursue this further. Performance analysis is murky at best on Android (especially now that per-app CPU usage seems off-limits to mere mortals), so this will have to await more-privileged parties' attention. Please watch the Android Issue Tracker bug report for updates. Technically speaking, neurotic tapping may be a kind of a work-around for this issue, but a fix in Android 11 (or Samsung's flavor of it) seems a much more rational resolution.
To recap: it appears that there are two major performance regressions in Android 11 that impact Mergeall—both a specific file-processing handicap, and a perhaps broader screen-activity throttling—and no reasonable user or program work-around is known for either. It's unclear where causes lie, but the issues have been reported to both Samsung and Android, and may also be fixed soon: Samsung has already promised a fix for some known Android 11 speed issues later in January, though it's unknown if the fix will address the performance issues covered here.
If the slowdowns are instead intentional consequences of Android 11's new mandates, they will severely paralyze (if not fully preclude) a host of POSIX-based and other software on Android 11, including all programs coded in Python that access files or perform non-trivial tasks normally, whether they are run as source-code or apps. In this scenario, program performance may have been deliberately sacrificed for the sake of Android 11's scoped-storage agenda. Regrettably, Android itself already seems to have suggested as much.
Time will tell if a half century of Unix practice is among the casualties of Android's newly found security obsession. All indicators today suggest it is, though only as long as developers enable such choices by rewarding this increasingly closed platform with grossly proprietary code. Dramatic to be sure, but this alarm badly needs sounding. The road to corporate tyranny is almost always paved with both customer complacency and developer complicity. Until we're told otherwise, though, we can still choose not to lay the bricks.
Like USB's demise, the rest of this speed-regression story filtered in over time, as vendors replied and new devices were tested. Be sure to see the updates box that follows for full closure on this thread.
Samsung Weighs In
Samsung, the Note20's maker, has now sent an initial reply, claiming that the performance issue is caused by Android's media provider scanning an external SD card, and suggesting that the remedy is to delete files. There are at least four problems with this:
.nomediato its internal storage, and letting it run overnight; and it's still chronic after three weeks of heavy 11 usage.
There are clearly major performance regressions in Android 11; whatever their underlying causes, Samsung's reaction hopefully won't be to bury its head in the sand (and advertise ever more deceptively). This won't make customers happy, won't sell phones, and won't help Android reach its potential. Samsung support later replied with something irrelevant about thermal management not being a factor, but the thread was made moot by the Android response up next.
From the hate-to-say-I-told-you-so department, Android later also replied on this issue, verifying that it is indeed a bona fide and deliberate performance downgrade. While it lasts, see the terse but definitive AOSP doc page referenced by the Android Issue Tracker post. Here's a screenshot of the doc's important bits, lest they vanish in flux.
In short, the slowdown reflects a tradeoff which intentionally penalizes
heavy users of files in shared storage—where heavy use apparently
means anything over 100 files, and shared storage is any app-neutral location in
/sdcard or its
Though its docs are thin, Android 11 seems to have moved such shared
("external") storage from the newer
filesystem to a
filesystem of former
for the sake of implementing scoped-storage access controls.
Unless programs are greatly changed to use proprietary Java APIs,
the net effect slows shared-storage access radically.
That is, Android 11 has been optimized for scoped storage's app-sandbox model—at the expense of everything else. This has major consequences for both users and programs:
data/data), which have limited utility by design; are not accessible to some file explorers today; may never be accessible to other types of apps; and might be deleted accidentally along with the owning app. Content kept in more permanent and useful shared storage incurs the disastrous speed hit, even for "legacy" apps.
Scoped storage's lack of file-explorer support today makes confirmation difficult, but a quick subset test seems to back up the claims: on the Android-11 Note20 Ultra, a Termux Mergeall run to compare a 13K-file 874-folder tree against itself finishes in 1 second or less for a copy in the app-specific folder, but typically 10 seconds for a copy in shared storage (and the Pixel's results up next are worse). As usual, the odd tapping trick inexplicably makes the shared-storage run faster, but never app-specific fast; shared storage has plainly been handicapped in 11.
While it's uncertain if Android's explanation accounts for all the throttling seen, it does qualify as sufficiently awful to call the game. The performance regressions are likely here to stay, and Mergeall and similar tools that process content in shared storage will regrettably run enormously slower on Android 11 than they did before. For this, we can thank Android 11's new "Privacy over performance" mantra, which, as good mantras do, has grown to eclipse the needs of actual users (many of whom will undoubtedly find "Dogma over usability" the apter tagline).
Android 11's throttling of shared storage has now been verified and quantified on a non-Samsung device. Results at a Termux command line on a Google Pixel 4a running Android 11 are captured here. If you look close enough, you'll see that the same comparison in both app-private and app-specific storage takes about 0.13 seconds, but in shared storage requires 13 seconds. This phone's shared storage is a full 100X slower!
Subtle bits: the degree of this throttling sometimes varies, but has nothing to
do with the preceding link's folder, app, or test; it's the same elsewhere in shared
and in Pydroid 3's
Moreover, this specific device may feel the hit worse than the Note20 Ultra
(which uses later and faster
and Mergeall compares are mostly directory listings and
shared storage is "only" 40X slower if programs read files along the way,
per the Pixel's
Still, the penalty for using open storage options in Android 11 is clearly extreme. By sharp contrast, the same tests with a larger data set on an Android 10 Note20 Ultra show no speed difference for shared storage at all (Android 11 tests are no longer possible on the Note20, for reasons revealed ahead). Moreover:
This issue has been reposted to the Android Issue Tracker with the new Pixel data in hope of a second look, but the outcome seems unlikely to differ. In the end, Android 11's new shared-storage selective sloth seems a poison pill for POSIX-based apps and will be a major inconvenience for many of their users, but it does not render devices wholly unusable; slowed tasks do finish eventually, and you can always deploy the silly-taps trick to speed these now-dysfunctional phones. The resolution of the other global Android 11 breakage, however, may be fully terminal, per the conclusion ahead.
And beyond all this, of course, the full impact of Android 11's scoped-storage permission changes remains to be seen, if and when the "legacy" Python-programming apps leveraged in this guide target Android 11. Some apps are already impacted by the new chain-link fences:
file://local-file URLs—which reduces users' options for viewing documents and website pages off-line. Firefox (version 84) was the only browser tested that still fully supported such off-line viewing with Android 11 on a Samsung Galaxy device—though not on a Pixel 4a, oddly, where Opera also fails to display on 11 what it can on 10. The web-server issues later in this list compound these lockdowns' impact.
Android/data—which is where files ironically must be located for usably quick access. Here's the Pixel 4a running Google's own Files. It is unknown if such apps have yet adopted or been granted Android 11's All files access, and this is prone to change (see the updates box below for the latest details). But these apps are hobbled today, leaving a Catch-22 for scoped storage which drastically limits content utility: how do you store and use files in faster folders that file explorers cannot access?
While the Python apps noted in this coverage have yet to emerge from their opt-out havens, their current stay of execution already comes with the visible barbs noted previously on this list: USB access is MIA, storage performance has been dialed back to zero, and glitches abound.
Again, please take this update in its context: it reflects just initial Android 11 testing on two devices; some its findings will likely be clarified or reversed over time; and this page may be updated as fixes and new information come in (watch the top-of-page date for revisions). For the present, though, Python and tkinter programs have clearly been harmed substantially by Android 11. Although a Return-of-the-Jedi update here is still slimly possible, it grows less likely with each new "aw crap" issued by the Mergeall testing team.
In closing this update, it's worth noting for fairness that other platforms also suffer frequent breakages at the hands of today's mosh-pit development model. Witness this site's coverage of recent regressions in Linux here and here; macOS here and here; and Windows here and here. While the ultimate goal behind Android 11's many regressions is a matter of speculation, its morph more clearly jives with the rude and reckless motifs now sadly common across the software field. Android has an especially clear pattern of both mandating languages and tools, and regularly breaking the programs which use them. Developer-friendly not, that is.
As expected, some apps started rolling out enhanced support for Android 11's storage model after this section was first written. The notebox that follows hits the highlights of these upgrades, though none manage to resuscitate the broader utility of pre-11 Android.
As of mid-February '21, some file-explorer apps now support access to
Android/data app-specific folders if granted permission.
Among those tested anew: full access is provided by Cx File Explorer
and Solid Explorer; partial access is present in Total Commander via a special
though the folders are not accessible in some contexts (e.g., Copy to); and
no access is (yet?) available in Google Files, Root Explorer, FX, and ASUS File Manager.
Given that permissions were not to be rolled out until
be sure to check your explorers for their current state.
Where available, file-explorer access to these faster app-specific folders alleviates 11's shared-storage speed issue, but only in part: as covered earlier, because these folders are automatically removed on app uninstall, they are not safe places to keep your valued content. That is: you can put your stuff in quicker app storage and can now use it there in some explorers, but it will vaporize if you ever uninstall the hosting app. That's not a risk most users should be asked to take.
Furthermore, file explorers are only part of the content-access story. Most other apps won't be able to access files in app folders other than their own, which limits content in app-specific storage to just one role. Consider the two Python-programming apps referenced in this guide: if you wish to process content with programs in both Termux and Pydroid 3, it must be kept in slow shared storage, because neither app can access the other's app-specific folder. In other words, Android 11 forces you to choose between content dataflow and program speed. That's about as toxic to non-trivial computing as it can be.
In sum, even with file-explorer visibility, app storage's impermanence and inaccessibility make it a very poor work-around to shared-storage sloth, and leave Android devices with no good options for realistically scaled goals. It's also worth noting that rooting opens access to app-specific folders in tested file explorers too, but you should know by now that this isn't much of a solution either.
It's official: Samsung has now verified that the removal of access to USB-drive mount paths described earlier is by design in Android 11, and cannot be fixed. This unfairly penalizes legacy apps, and is regrettably a game ender for a host of tools on this platform. Especially in concert with 11's other demerits on performance and usability, this is a full stop for content-based activities.
Per its reply, Samsung cannot do anything about USB access in its code base, because it's
a new policy mandated by Google as of Android 11. Specifically,
/storage/xxxx-xxxx paths can no longer be set; USB drives may
be mounted at
but using that path requires special permissions or a rooted device, and it is not accessible
to most programs and users.
Hence, general USB access is no more, this isn't limited to Samsung devices,
and it's up to Android to fix this if it wishes to be inclusive of
programs that depend on traditional processing paradigms.
You can read the full reply about the USB path's demise from Samsung here. Although a definitive reply from Google on this issue is still pending at a parallel post on the Android Issue Tracker, the lack of USB-drive paths on the unrooted Pixel 4a suggests that this extends well beyond Samsung's code, and a new opinion-based decree from Google that wouldn't bend to the leading Android vendor seems unlikely to be reversed for a single developer.
Assuming its policies stand, Android 11 has knowingly maimed scores of programs that use the portable POSIX API and rely on USB drives for non-trivial content collections, including the Mergeall backup/sync program. Users of such programs are left with snaillike connections, perilous rooting, or delicate microSD-card pulls—none of which is universally practical, supported, or remedial. And developers of such programs are left with broken products or costly rewrites to adopt Android's proprietary, dogmatically enforced, and Java-based APIs. Such rewrites are not an option for Mergeall, and unlikely elsewhere.
Eventually, Android 11's scoped-storage access restrictions described in the main doc may further limit what users can do with their programs, content, and devices. The full scope of these restrictions remains to be seen in Python-programming apps still loitering in an opt-out limbo that's partial at best; but they are already slowing the loiterers to a crawl, crippling apps brave enough to target the new Android, and well on their way to becoming the norm.
Ultimately, though, all it took to torpedo content-management programs like Mergeall was removal of direct USB-drive access. Between this, its poison-pill performance regressions, its newly honed glitches, and its looming access restrictions, Android 11 unfortunately does lock out many or most Python programs, and seems likely to be the sunset of this platform's general-purpose-computing ambitions.
On a personal note, the Android 11 auto-update has functionally bricked this developer's phone, and seems the same as an auto maker creeping into your driveway and removing your car's engine. This device today can no longer be used for content backup and view—one of the main purposes for which it was purchased—and the fault squarely lies with the dubious agendas underlying Android 11. Though spun aggressively (and sometimes arrogantly) as enhanced security, the only thing 11's changes seem sure to secure is the revenue stream of cloud providers and mobile advertisers.
The future is of course unknown, and it's possible that Android 11 will still follow Windows 8's lead and reverse course over time. For the present, though, Android may have finally taken its place alongside the iPhone, as a deliberately closed and limited media portal for technically naive consumers. The mass market of exploitable customers which seems the true target of these devices' vendors probably won't care. But some developers will, especially those who craft programs beyond the trivial, and understand that "apps" has always been a derogatory term.
In the end, this page's takeaway is a simple one: they're only "just phones" if you let the auto-update happen. If you care about using Mergeall, tkinter GUIs, or your device in general, you should consider avoiding Android 11 and later unless and until this platform's many issues are addressed. This page may be revised again if improvements arise, and hopes they will; Android devices have great potential yearning to be unlocked.
But for now, this humble Python hacker won't be moving ahead with new development on this platform, and will be flashing a ROM to revert the Note20 to Android 10 very soon (learn how here, here, and here). A retreat, perhaps, but it's also a lot more usable, and a lot less fascist.
Postscript: Android 10 Rocks Again
The rollback to Android 10 went off without a hitch, and fully restored all the functionality Android 11 took away. In particular: USB drives are back again—the access revoked by Android 11 is available to Python programs on 10; and shared storage is blazingly fast again—the internal-to-internal 124K-file Mergeall compare that took 104 to 168 seconds on Android 11 finishes in just 8 to 9 seconds on 10. That is: yes!
That's great news, but also in some ways sad. Android 11's architectural changes are clearly designed to lock out everything except proprietary Java-based code that uses proprietary Java-based frameworks and will run nowhere else. This is a blatant and short-sighted affront to interoperability, which dramatically limits device utility and scope. POSIX-based tools like Python, and the decades of prior art they enabled, are not invited. This may reflect a culture clash of sorts, but the ramparts are being raised by Android alone.
With all the data in, Android 11 has indeed robbed its users of swaths of functionality in the name of a dodgy security obsession, and probably is the end of the line for these gadgets' flirtation with general-purpose computing. This seems nearly inevitable in hindsight. A platform which both excludes contributors and severs the flow of content between programs can only be doomed to a future of inane consumption and greedy coercion.
And yet, there is a bright side to close with here. By the time today's phone batteries die, Android will almost certainly be something completely different yet again—if not subsumed by friendlier alternatives altogether. That's a sequel which remains happily untold. Hope, after all, can never be deprecated.