Pydroid 3 Loses Shared Storage — and Scope

September 2023, lutz@learning-python.com

This page covers a major downgrade in version 6 of the Pydroid 3 Android app. The downgrade was first spotted in summer 2023 after upgrading to version 6.4, but may have appeared earlier in the app's 6.X line, which spans 2023. Release blurbs and sideloads suggest it originated in 6.1, but there is no change log for this app. To avoid inaccuracy, this change is tagged 6.X here.

This page's goal is not to document Pydroid 3, but to amend and retract former advice at this website. Pydroid 3 has been promoted by this site for years as a way to run its Python tkinter GUI programs on Android (e.g., see the Android-tkinter doc). While the app still works in limited roles, its 6.X release makes it unusable for many or most realistic tasks. The point here is to call out these 6.X impacts for users of programs both at this site and elsewhere. Along the way, this page also explores Android storage, and alternatives for running Python-coded GUIs on Android.

Tap Contents below for this page's section links, and the Top button after scrolls to return here. You can also browse a small gallery that collects screenshots referenced in this page.

Contents

Overview

In short, Python programs run by version 6.X of the Pydroid 3 app can no longer access any content (i.e., files of data, media, etc.) in general shared storage. Instead, content can now be read and written only in specific folders accessible to the app. This makes it impossible to use Pydroid 3 to process content shared with other apps without special handling and extra steps. The net effect is onerous enough to move many formerly usable programs out-of-scope for this app.

The app's new limitations stem from the scoped-storage rules introduced in Android 11. These rules have been lamented widely both on this site and the web, so we'll skip the backstory here. (TL;DR: Android has grown determined to lockdown storage on what are fundamentally shared-storage machines—an arguably short-sighted initiative that comes with both radical reductions in utility and speed, and thin security rationales easily refuted by the platform's own prior success.)

While Android is ultimately to blame, however, the Pydroid 3 app's 6.X release also chose to conform to the new closures for the sake of its Play store presence and revenue, rather than adopting less restrictive venues that support broader options (e.g., F-Droid). The unfortunate result is reduced utility for both users and developers of Python programs on Android. This downgrade seems especially grievous for an app which is mostly a bundle of free-and-open-source code.

To be fair, Pydroid 3 is still usable in learning roles, which has always been its primary stated goal. You can use it to explore Python basics without a PC just as before, and both trivial scripts and self-contained use cases may still work well. But for more practical goals that rely on toolchains of programs or multiple-step workflows, you will generally have to import and export any content used or produced by this app with manual copies. While file explorers and sync tools support most such copies, the extra steps convolute tasks badly enough to make some pre-6.X roles impractical.

This rest of this page details the storage changes in the latest Pydroid 3, and presents both implications and alternatives for Python programs on the Android platform. As we'll see, some users burned by 6.X's new limitations might be able to limp along with either manual content copies or an earlier version of the app, but building standalone apps that do not depend on Pydroid 3 seems the best option for most new Python development on Android.

Storage Options and Tradeoffs in 6.X

Prior to 6.X, Pydroid 3 used an earlier Android permission scheme, which allowed Python code to freely access content anywhere in shared storage. For example, content could be stored in a folder simultaneously accessible to file-explorer apps, backup and sync tools, the Termux command-line app, and tkinter GUIs run in Pydroid 3. As of 6.X, however, programs run by Pydroid 3 can access content only in a small handful of special locations. The app itself lists these in tips like this and this. The sections that follow describe these options in more detail than you'll find in the app.

The App's Own App-Private Folder

Exception: phone rooting might ease some of this folder's access restrictions, but is impossible on most devices, and too rare to merit coverage here in any event. Let's move on to more practical options.

The App's Own App-Specific Folder

Perhaps worse, because this folder is not generally accessible to most other apps, it requires manual duplication of files: shared content must be explicitly imported to and exported from this folder by performing copies outside the app. That alone may disqualify this folder for many former Pydroid 3 roles.

In its defense, this folder today may be faster than shared storage at large, and can be accessed by most file-explorer apps for copies and opens. But few if any sync tools will support it, and file explorers may lose access to it in the near future if Android tightens a known loophole. Moreover, its speed advantage is subject to operation mixes, Android changes, and the vagaries of Android performance at large (both this platform and its individual vendors may throttle apps arbitrarily, for power, heat, memory, or any sort of bias in general).

In combination, this folder's auto-deletes, extra import/export steps, and tenuous status make it a dubious medium for shared content. Most users would be better off using the next option we'll meet on this list, after the following app-specific fine points.

Privacy caution: some content in app-specific folders may also be automatically uploaded to your Google Drive account, if you have enabled backups. The uploads will be automatically downloaded when the app is reinstalled (more details here). This is normal for apps and may seem a feature to some, and the backups are limited to just 25M—barely enough for 10 typical photos. But it's also a privacy concern if your shared content is personal or sensitive in nature: you can't control which files may be uploaded, and content really shouldn't be copied anywhere without your knowledge and consent. Don't use this folder if you care.

Path footnote: for newcomers to Android, the path root folder /sdcard is the same as /storage/emulated/0 on most devices, and is simply a synonym for it (technically, it's a link, and may differ on some phones due to Android's variability). Whatever it's called, though, this root is shared storage, but some parts of it, like app-specific subfolders, are mounted and managed differently from the rest. For more info, try the Android path primer here, and the storage overview here.

The Pydroid3 "Public" Shared-Storage Folder

Beside this broader access, this folder comes with a number of bonuses. Unlike app folders, its content is not auto-deleted on uninstalls and clears. Unlike SAF access discussed ahead, this folder naturally supports multi-file programs and local files using POSIX file-path tools. And unlike app-specific folders, file explorers are more likely to support this folder both now and in a less tolerant Android future.

In fact, this option may be your best—and only viable—choice for shared content in 6.X and later. Shared storage is usually slower than app storage today, but this is a defect in Android subject to improvement (see the bug report), and the auto-deletions and other limitations of other 6.X schemes make them difficult to recommend. Furthermore, you can still run programs in this folder using home-screen shortcuts, by using the change-directories trick discussed in the next section.

On the other hand, this folder still adds extra steps to content workflows: because this is the only shared-storage folder that Pydroid 3 now supports, the code you run here cannot access files anywhere else. Hence, for most use cases, you still must manually import and export shared content here with copies or syncs run in other apps. And you must do so on every change to content shared with programs run in Pydroid 3; for many goals, that will be tedious enough to qualify as a showstopper.

Undocumented step: this folder supports shared-content processing only if you force Pydroid 3 to open a system chooser dialog to grant access, by trying to open or save a file in this folder from its files menu (the UI looks like this and this). Else, the app can access only files in this folder that it creates itself, which is the default behavior of Android11+'s scoped storage. Pydroid 3 partly eases this default by obtaining permission to this one specific folder, but skips broader access options we'll meet ahead.

Privacy caution: the app's 6.X release tersely mentions "synchronizing" content in this folder, and there was indeed a lengthy delay after granting the app access to content copied there, during which files started appearing in the app's file chooser one by one, and initially renamed with an odd .iiec_sync extension (here was the scene in the UI). What happens during these pauses is nowhere explained. Is your content copied elsewhere? Uploaded to your Google Drive account? Somewhere else? And is your content intact after the process? What about its timestamps and other metadata?

Data-usage monitoring suggests the app is not uploading content placed in its folders in full (transfer rates are far smaller than content size, even allowing for compression). But this lack of transparency for an app that holds the full-network-access permission is a flag on the play for privacy, and makes placing personal or sensitive content in any of the app's folders a use-at-your-own-risk proposition. An app which won't tell you what it does with your stuff may not be an app worth trusting with your stuff (especially given its other abuses).

The Storage Access Framework (SAF) Protocol

This Android-only scheme will be used if you open a script in Pydroid 3 using either a file-explorer app, a file-explorer home-screen shortcut, or the app's own SAF picker. All these contexts use content:// content URIs and Android APIs, instead of /folder/file file paths and portable POSIX calls in the underlying Linux system. Crucially, SAF access by default supports just single-file programs with neither imported modules nor local files. Although this handles trivial scripts, it obviously won't work for most of the Python programs you'll write or run.

Undocumented trick: the foregoing being said, SAF access can be forced to support both modules and local files by adding a Python os.chdir() call at program startup, to change to the program's actual folder. That is, in the script being run, insert this call and pass to it the absolute path of the folder containing the script. As an example, the following call makes modules and local files accessible for SAF launches of the Frigcal program covered ahead:

os.chdir('/sdcard/Download/Frigcal-source')

In Frigcal's case, the path used above will work for accessing modules and files in Pydroid before 6.X, but not later. Starting with 6.X, SAF launches must use one of the few folders now supported by the app:

os.chdir('/sdcard/Documents/Pydroid3/Frigcal-source')

All of which implies that both code and files must be located in the same place—with implications called out in the next section.

Impacts on Programs in 6.X

So where does that leave Pydroid 3 users, then? Of all the foregoing options, the Documents/Pydroid3 folder seems the only reasonable choice for those boldly going forward with Pydroid 3's latest releases. This option adds manual steps in 6.X, but it's at least better than content that vaporizes silently on app uninstall—a draconian byproduct of Android's growing obsession with security. Because other options are so poor, we'll assume Documents/Pydroid3 going forward in this doc.

Even with this best option, though, you can't do as much in Pydroid 3 as you could before. For one thing, you now must install and run multi-file programs only in a folder specific to this app. For another, you now almost certainly must manually import and export content shared with other apps, by copying it in a Pydroid 3-accessible folder with file explorers or sync tools. These extra import/export steps convolute any workflow that relies on content created or used outside this app, and are tedious enough to be preclusive for many prior Pydroid 3 roles.

To back up that claim, the rest of this section describes how two programs formerly used in Pydroid 3 are impacted. As we'll see, neither example has much of a future in the app's 6.X and later.

Frigcal: Now with Manual Calendar Copies

First up on the chopping block, Frigcal is a personal calendar/journal program, whose Android usage is described on this site's Android-tkinter page. Its tkinter GUI ran well in earlier Pydroid 3 versions, but is likely impractical on later releases given the new storage constraints of 6.X.

The tale of Frigcal's fate in 6.X is both illustrative and mixed. Like most, this is a multi-file program which must be unzipped in the app's Documents/Pydroid3 to be run in 6.X (Downloads no longer works as a program host because it's shared storage). Once so unzipped, though, the program folder is visible to the app for both module imports and local files. This constraint isn't too burdensome, because installs are a one-time task.

Moreover, file-explorer shortcuts to the program's main script still work well, as long as you add os.chdir() calls to change to the unzip folder per the coverage earlier. In frigcal.py , add this both after initial imports and the preexisting call. Else, the SAF open on tap will not provide access to the program's imported modules in Pydroid 3; here's the error before the directory change is live. This is extra work, but it's required only for SAF launches (not manual opens in the app's IDE), and it's a one-time job that's not exactly cause for dismissal.

Using calendar files, however, poses a much larger problem in 6.X. By design, Frigcal is a cross-platform program that relies on portable ICS calendar files for interoperability with other apps, devices, and platforms. Given that these files can be freely used anywhere and by both Frigcal and other calendar programs, they will normally reside in a shared-storage folder on your phone that's external to Pydroid 3 and synced with other computers and phones. Prior to 6.X, their path is easily configured once for use by the program like this:

"""
Where Frigcal finds the portable ICS calendar files it uses.
This is configured in Documents/Pydroid3/frigcal_configs.py,
and references a synced folder in general shared storage.
/storage/emulated/0 = /sdcard, usually: shared-storage root.
"""

# FAILS -- cannot access other shared folders as of 6.X
icspath = '/storage/emulated/0/MY-STUFF/Code/frigcal/Calendars'

Because the app's 6.X release can no longer access these files in general shared storage, though, this configuration no longer works; here's the error it generates. Instead, 6.X+ users will regrettably have to set the configuration as follows, and manually copy their calendar files to and from the app's Documents/Pydroid3 folder—and manually recopy after every sync or change:

"""
The 6.X+ solution... as long as you also unzip Frigcal in the source
folder, and manually copy your calendar files here on every change.
"""

# WORKS -- with tedious copies
icspath = '/storage/emulated/0/Documents/Pydroid3/Frigcal-source/Calendars'

In other words, Pydroid 3 6.X adds an extra post-sync step to normal usage. This does work, and file explorers and sync tools are generally up to the task, but the copies are extra steps that can be easily missed, and may be understandably skipped.

Finally, keep in mind that the ICS files you would be copying into Pydroid 3-accessible storage this way are personal and potentially sensitive calendar/journal data. Given that this app does not fully disclose its handling of files there as noted earlier, this may expose you to more privacy and integrity risks than you care to take. Together with the extra calendar import/export steps, this probably makes running the program in Pydroid 3 impractical as of 6.X. Alas, Frigcal will likely have to employ one of the workarounds we'll meet after the next section's 6.X casualty cohort.

Mergeall: There's Access, and Then There's Access

Next on the hit list, Mergeall is a portable backup/sync tool for content folders. It was initially usable on Android with command-line scripts run in Termux, and later with its tkinter GUI run in Pydroid 3. In Pydroid 3 6.X, its GUI still runs, but it's been rendered useless for most prior roles.

Mergeall was already severely impaired by Android 11's removal of USB-drive access: removable USB storage is always unavailable by default to POSIX file-path code run on Android 11 and later, regardless of apps' API target or shared-storage status. But Mergeall is further hobbled by Pydroid 3's 6.X changes: without access to arbitrary content folders, a sync app is largely pointless.

Really, the notion of "access" in Android's brave new scoped-storage world is poorly defined and unevenly implemented. The tkinter GUI of the Mergeall system, for example, bizarrely appears at first glance to work in the 6.X release of Pydroid 3, because it happily marches down folder trees in areas of shared storage that are no longer supposed to be accessible to the app.

On closer inspection, though, this is just an illusion: Mergeall is simply listing subfolders in the trees, and does not see any files residing in those subfolders. A simple test script run in Pydroid 3 6.X proves the point:

"""
Save and run this in Pydroid 3's Documents/Pydroid3 "public" folder.
The sharedfolder path is in general shared storage and outside PD3's 
access, though folder-only listings and stat calls work anyhow - weirdly.
Mergeall appears to work because of listdir(), but it sees no files.
"""
import os

sharedfolder = '/storage/emulated/0/MY-STUFF/Code/frigcal/Calendars'
sharedfile   = sharedfolder + '/' + 'frigcal-default-calendar.ics'

# WORKS -- BUT returns only subfolders, and no files
print(os.listdir(sharedfolder))

# WORKS -- metadata access: size, modtimes, etc.
print(os.stat(sharedfile))

# FAILS -- no content access outside Documents/Pydroid3
print(open(sharedfile))

Per this script, code run in Pydroid 3 6.X can list folders anywhere in shared storage with Python's os.listdir(), but the listings for folders outside the app's permissions contain only subfolders, and no files. Detail-minded users can verify this in Mergeall's run summary too: it tallies all folders, but always zero files (here's Mergeall again, versus an alternative we'll meet in the next section). The net result is an inane traversal of the folder tree's skeleton, with no file inspections along the way.

Stranger still, Python's os.stat() works to access metadata (e.g., sizes) of files anywhere in shared storage, but the crucial open() call fails to access the very same files' contents with permission exceptions. Contents is more sensitive than metadata, of course, but the implementation of privacy seems to have left some doors unlocked.

In sum, Mergeall's curious behavior shows that shared storage is still partially accessible: apps can list folders but not files, and metadata is freely available; but the contents of files is off-limits except where allowed by the app's new permission rules. For Pydroid 3, this means you still must place content in an accessible folder, because the rest of shared storage is forbidden—where it counts.

And for Mergeall specifically, this means your only recourse is to locate all your general content in Pydroid 3's folder in order to sync it with programs run by this app. As a direct consequence, you'll also have to trust that Pydroid 3's opaque "synchronization" steps noted earlier won't mutate or expose the content you copy to this folder. This seems a lot to ask when it comes to your digital property. Loss of USB access may have put Mergeall on life support, but Pydroid 3's new hurdles seem to pull the plug in full.

Happily, there is a full solution for Mergeall's fall: although its original tkinter GUI is no longer useful in Pydroid 3, you can still leverage Mergeall's main sync code on Android in the PC-Phone USB Sync standalone app. That app wraps Mergeall in a Kivy GUI which obtains permissions that restore both the general shared-storage access dropped by Pydroid 3 6.X, as well as the file-path USB-drive access cut by Android 11. To see how this app negates the closures, let's move on.

Alternatives for Python on Android

Now that we've seen the impacts of Pydroid 3's storage restrictions in 6.X, it's time to explore workarounds. Some users, of course, may never use content shared with the world outside the Pydroid 3 app, and others might be able to get by with manual imports and exports as needed. If that set includes you, you should be able to install 6.X with few ill effects.

For the rest of us, there are two ways to address the new lockdowns, as the next sections will explain.

Disclaimer: this section is not an exhaustive guide to using Python on Android. It simply shows how to run Python-coded GUIs on this platform using two schemes known to work well. There are other ways to run non-GUI Python code on Android (e.g., Termux), and other ways to build standalone Python apps for Android (e.g., BeeWare), but they are not on the agenda here. For other ideas, try this wiki or the web.

Using an Older Version of Pydroid 3

If Pydroid 3 6.X's access rules are impractical for the programs you use or code, it is possible to skirt the issue in full by running an earlier version of Pydroid 3. Either don't upgrade the app to version 6.X, or sideload a prior version fetched outside the Play store. For example, APKMirror hosts Pydroid 3 version 5—your best bet to avoid the downgrade; download its APK to your phone and install by opening in a file-explorer app (this is sideloading). Per testing, older versions at APKMirror also respect prior purchases to remove ads in the app.

Both fixes come with caveats. Keeping a prior install will naturally work only until you purchase a new phone. And sideloading prior versions is not without perils either: you must vet and trust the source, and must uninstall any newer Pydroid 3 first (Android refuses to downgrade apps, even when its users ask it to). In addition, both schemes will work only until Android prohibits the older code, and you'll miss out on future Pydroid 3 upgrades (though new does not necessarily imply improved: see 6.X).

Given the tradeoffs of staying in the past, adopting Pydroid 3's new and limited functionality will regrettably be the only path forward for some users of existing programs. For new Python development on Android that wishes to be inclusive of all users, the next section offers an alternative that removes Pydroid 3 from the equation altogether.

Building Standalone Apps

Although manual copies and prior versions may suffice in some contexts, they're subpar for programs that aim for broader audiences. For these, developers will find richer solutions in Python-focused tools that build standalone apps.

The available toolset in this domain is evolving and beyond this page's scope, but you can sample the options with a web search. All the options package your Python code as a normal Android app, and most include interfaces for accessing Android's Java API from Python if and when required. Combined with Python's by-design portability, the net result allows you to deliver idiomatic programs on Android with code that can work on other platforms too.

Portability requires some effort, of course, and building such apps is more complex than running code in Pydroid 3. But the benefits are many. Unlike code run in another app, standalone apps:

• Install and run normally on Android
• May be uploaded to app stores for distribution or sale
• Are immune to changes in languages and libraries they use
• Are not dependent on a hosting app's choices and status
• Can use permissions to reinstate storage access lost in Android 11 and now Pydroid 3

The last of these is most important here. In particular, standalone apps are able to obtain the Android 11+ All Files Access (a.k.a. MANAGE_EXTERNAL_FILES) permission, which restores access to both shared storage and USB drives. You can read more about this permission here and here. With it, content in shared storage and removable USB drives can be processed with traditional file paths and POSIX tools like those native to Python. For example, Python's open() call and os-module file tools work on such content normally with this permission in hand.

The chief downside of All Files Access is that some types of apps using it are not allowed on the Play store; read the store's rules here. Despite the heavy-handed language on that page, most apps with valid use cases for this permission are allowed on the store. Moreover, other app stores likely won't be so rigid, and this permission is always available for apps coded and sideloaded for in-house or personal use. All Files Access may also pose performance tradeoffs, but so do each of Android's proprietary APIs, and this is an ever-morphing metric on the platform.

More to the point: unlike code run in Pydroid 3, a standalone app is not subject to the permission choices made by a hosting app, and can leverage any storage option available on Android. This includes both the broad All Files Access permission for shared and removable storage, as well as the more limited scheme that Pydroid 3 now uses to gain access to folder Documents/Pydroid3. As an app, your code is a first-class citizen, constrained only by Android itself.

PC-Phone USB Sync: An All-Files-Access Story

Finally, as an example of what's possible with standalone apps, the Python-coded PC-Phone USB Sync backup/sync app noted earlier uses the All Files Access permission to gain access to arbitrary shared and removable storage folders. Here's the app's main tab in action, along with its folder chooser at work in shared storage and an attached USB drive; thanks to its broader permission, both storage areas are fully available to this app's users. Simply put, the app is able to offer crucial functionality that Pydroid 3 no longer can.

To go standalone, this app also uses:

• Kivy to code a portable GUI, which wraps the Mergeall system described above for easy use, and works and looks the same on both Android and all PCs
• Buildozer to build the program as a standalone app, which works on Android phones and tablets, and can be either sideloaded or posted to the Play store and others
• PyInstaller to create standalone apps and executables for Windows, macOS, and Linux PCs, which can be distributed and installed as usual

Like tkinter, Kivy enables GUIs that look and run the same on Android and PCs (more examples here). Kivy generally requires more advanced programming skills than tkinter and is not without warts, but it should be accessible to most Python developers. Buildozer and PyInstaller similarly have steeper learning curves than Pydroid 3's IDE, but their rewards are products which are much more seamless, and proportionately more powerful.

All of the tools listed above are free and open source, and allow you to develop programs in portable Python code that is mostly the same on Android and PCs. That's a huge win for software either developed on any other platform than Android, or intentionally coded to run across disparate devices from a common code base.

For more info on the PC-Phone USB Sync app specifically, visit its website or tap Note at its picture here.

All that being said, it's worth adding that All Files Access may be overkill in some apps. For example, if the Frigcal program discussed above ever morphs into an app, it wouldn't require such broad permissions. Instead, the single-folder permission scheme used by Pydroid 3 may suffice for calendar files (see Android docs). While All Files Access works well, and is even required by some apps, you should naturally choose the right tool for each job.

Conclusion

Hopefully, this page has given you a clearer picture of the storage constraints introduced by Pydroid 3's 6.X release. This website has no connection with this app—apart from the fact that some of its programs rely on Pydroid 3 as a host on Android, and won't work the same going forward. If you use these programs, or have programs of your own in the same boat, this page may help you adopt workarounds of your own.

It's unfortunate that Pydroid 3 opted to limit itself, when freer options are available. The Termux app, for example, still supports shared-storage in full today, and likely will in the future. Its current support reflects targeting a former Android level, but its future support is poised to be based on All Files Access. Because Termux is distributed on a different app store, it's not constrained by the Play store's storage rules. Perhaps Pydroid 3 will someday adopt similar policies and reward its users with broader utility.

Until then, you can still use Pydroid 3 to explore Python and tkinter basics when a PC is not available. But it is no longer recommended for realistic programs that share content among the components of a larger workflow or toolchain. For these, developers may be better served by Python-focused options that build standalone apps.