File: pyedit-products/unzipped/fixfrozenpaths.py
""" ================================================================================ fixfrozenpaths.py: setup path/cwd context (part of the PyEdit [3.0]) Import-path setup and resources folder path for various distribution contexts: source, app, and executable. Setup differs per freeze tools for apps and exes. This file's code has been propagated to mergeall, frigcal, PyMailGUI, PPUS. [4.0] Much of the following is legacy docs - skip ahead to code for latest info. Also, now uses sys.executable instead is sys.argv[0] for frozen exe's path: usually the same, but argv[0] is symlink location when used, not its result. Orignal notes: This code was initially added to allow PyEdit to be run standalone in the PyMailGUI package. When shipped with PyMailGUI's source-code release, the PP4E folder is in a custom location and not on PYTHONPATH: add its path here manually, so we can run PyEdit standalone too. We could reuse PyMailGUI's SharedNames.fixTkBMP() this way too, but that sets up too much coupling and dependency here - copy it. Later expansion: This has now grown more (really, wickedly) complex with the addition of app, executable, and standalone-source distributions. In these cases, the PP4E package is nested alongside this script or "baked in" to the frozen package, but these self-contained versions also need access to: Module: textConfig.py user configs, shipped separately in source form, everywhere. Located on sys.path when imported (and must be user-visible and editable). Data: The /icons folder for Tk program-window icons, on Windows and Linux. Located via cwd or dirname(__file__), in that order (client or own). Data: UserGuide.html, and icons+images in ./docetc that it uses, everywhere. Located via the dirname(__file__) directory, only. [4.0] This is now online only and no longer shipped in app/exe packages. Data: The Help dialog also uses an image in ./docetc (like UserGuide.html), and later used the package's README file. [4.0] Not so - Help's image now fetched from icons/ (not omitted docetc/). Data: Auto-saves may rely on cwd for some save paths, including the config file's default of ./__pyedit-autosaves__, relative to the install dir. Proxy: The Capture Run-Code mode uses __file__ to locate its proxy script which may not be in "." in embedded-component mode (e.g., when used in PyMailGUI). For PyInstaller standalone executables (but not py2app bundles), the proxy script is also frozen and in '.', as sys.executable is PyEdit, not Python. See build-app-exe/windows/build.py for additional notes on this case. [4.0] proxy now in fetchMyInstallDir() result, and is an executable or .py in Content/Resources in macOS PyInstaller app and the install/unzip dir itself in Windows exe; the .py is used for both source code runs of PyEdit and a configured local Python (whether PyEdit is frozen or source), and is in the same folder as the main script for source-code runs (only). More details: [4.0] Windows exe and macOS app now BOTH use PyInstaller, Linux is dropped. Mac apps currently use py2app, and Windows and Linux exes use PyInstaller. The build scripts or the installer commands they run (in build-app-exe) exclude the textConfig module so it is not frozen, and arrange for the above items to appear at accessible places: - ON WINDOWS AND LINUX, these items are copied by build scripts into the same folder as the single-file PyInstaller executable. This shipped folder shows up in dirname(sys.argv[0]), but not in dirname(__file__) which is always empty and maps to an arbitrary temp cwd (sys._MEIPASS) used to unzip the bundle. The executable's install folder is also not on sys.path automatically, and sys.executable is the exe, not Python. In fact, no Python executable is shipped, just its frozen library; each script must be a separately-frozen executable, with all its requirements embedded. For more backround, try: https://pyinstaller.org/en/stable/runtime-information.html - ON MAC, these items are automatically included in the app bundle's folder Contents/Resources, to which py2app's Contents/MacOS bootstrapping code auto cds, and in which the main script (this file's importer) runs. This dir has both the main script and a zipfile with all PP4E package modules' bytecode. However, neither this folder nor '.' is on sys.path for imports automatically. sys.executable still refers to a real Python, whose executable is included in the app bundle, and which has all modules and packages named at build time "baked in." App scripts can thus be run as source with the app's Python. The sketchy docs are here: https://py2app.readthedocs.io/en/latest/index.html. In source-code packages, these items reside naturally in the source-code folder. Here, we need to set the cwd on some platforms for data, and must set the import path to see textConfig.py on all platforms, while taking care to allow for the portable/simpler source-code based distribution schemes. On Windows and Linux, the frozen subprocess proxy must do similar for textConfig imports, and must arrange for temporary folders to be pruned on non-normal exits. The cwd and sys.path are program-wide globals; __file__ is a module global, so changing it here would not impact this file's importer (not done). See also this file's __main__ code: Windows exes must patch multiprocessing, and Mac apps must catch Apple opendoc events manually (py2app argv emulation is broken, and PyInstaller fails for ActiveState Tk 8.5 or more). Running user code from a frozen app/exe is also problematic, and resolved by forcing all stdlib modules and allowing configs to name a Python: see subprocproxy.py. ---- FORMER CAVEAT: INSTALL PATHS AND UTILITIES For Windows and Linux PyInstaller exectuables, an os.chdir(exedir) was used in prior codings to make the empty __file__ dir map to the install dir in sys.argv[0] (instead of the _MEIPASS temp unzip folder). This allowed the __file__ to be used to portably derive the install folder. Unfortunately, this also precluded using relative paths in command-line arguments. This was not an issue in PyEdit or PyMailGUI (they have no or rarely-used args), but utility scripts in both frigcal and mergeall process file paths by nature. Omitting the chdir almost worked, but programs then could not find their icons, help, and spawned scripts if run from elsewhere via a command line instead of direct click. FIXED: the chdir is no longer run; instead, clients call fetchMyInstallDir() to select the install path explicitly per deployment mode -- from __file __ for source and Mac apps, and sys.argv[0] for PyInstaller exes. The cwd is unchanged, so relative paths work in all frozen executables' command lines. Exception: some programs (in frigcal) manually os.chdir() to the install dir returned from fetchMyInstallDir() early, for easy access to their data items relative to "."; this works only if no command-line path arguments are used. Here, the fix means the optional filename command-line argument can be relative to the folder where a command line is run. It's also possible to include some items in the unzip folder as Analysis data, but not items that must be user-visible -- including help and config modules. The new function also now allows for use in componnt mode: it expects a nested folder in the executable's dir for PyEdit data, only when running in component mode as a PyIstaller one-file executable (e.g., in PyMailGUI). ---- CONTEXT: WHY FREEZE? So why the bother? Apps and exes best support icons and desktop presence, and fully support associating pyedit to open text files automatically on clicks (.bat files work but flash on Windows, and Mac flatly requires an app bundle). Associations are especially important for a tool like PyEdit, which can open other files. They're made by simple right/ctrl-clicks on both Windows and Mac: - Mac: rightclick+Open With; or rightclick+Get Info+Change All for many - Windows: rightclick+Open With+Choose+Browse; or Control Panel+Defaults Linux, as common, is more complex (e.g., editing Ubuntu mime-list and .desktop files): https://duckduckgo.com/?q=associate+program+with+file+ubuntu Apps and exes also better support icons and drag-and-drop and right-clicks; do not require users to install Python too; and embed a specific Python version which makes future Python changes moot. Apps and exes can also be run by simple clicks instead of command lines (though this also works for source-code files on Windows, and via right-click to the auto-installed Python Launcher app on Mac). Further, on Mac apps can be drug to Launchpad (/Applications) for easy access. Downsides: source is demoted, and Mac apps badly obscure user config modules. The app/exe model also may not apply very well to decoupled/spawned scripts as in mergeall and PyMailGUI (the spawnee need not be an app, but can be frozen). Single-file executables on Windows also require a patch to be used with multiprocessing for Grep (see multiprocessing_exe_patch.py and __main__ for more details), pose unique hurdles for some subprocessing module use cases, and can convolute code substantially in non-trivial programs - per the next section. ---- POSTMORTEM FREEZE SUMMARY: A NET WIN? On final reflection, Python's frozen-executable support reflects clearly noble efforts, but seems chaotic, if not unrealistic, and significantly convolutes an application's code. There are multiple very-different freezers; some have not been maintained as well as others; some fail completely on major platforms; some generate executables with very-long startup times or very-large folders; and most are riddled with special cases requiring wildly-hackish workarounds that can obscure the actual purpose of application code. Real-world tools like multiprocessing and subprocess likely won't work without coding heroics. Running arbitrary user code requires major help, due to the core model of minimal library inclusions. And the entire freezing paradigm is painfully similar to C development and makefiles, if not fundamentally at odds with both source-based scripting languages and Python's dynamic nature. In the end, PyEdit was able to make freezes work acceptably, and this does help much for icons, associations, and Python dependencies. Freezing effectively promotes scripts to first-class program status. But for non-trivial use cases this comes at a heavy cost in extra code convolution and development tasks, whose net you'll have to judge for yourself. This much is clear: placating unreasonable command-line phobia alone may be inadequate justification. For this developer's $0.02: py2app app bundles on the Mac seem more than justified, given their much-improved user experience on that platform. PyInstaller executables, though, seem to require too many code changes and bug workarounds to warrant the minor user-interface upgrades they provide, and other freeze tools seemed altogether orphaned or broken on Windows. On the other hand, insulating code from Python changes is an absolute win. ================================================================================ """ import sys, os, time DebugContext = False # trace before/after settings? # For platform-specific choices RunningOnMacOS = sys.platform.startswith('darwin') # all macOS (f.k.a. OS X) RunningOnWindows = sys.platform.startswith('win') # all Windows RunningOnLinux = sys.platform.startswith('linux') # all Linux, incl Android # [4.0] additions RunningOnAndroid = hasattr(sys, 'getandroidapilevel') # Android only (py 3.7+) RunningOnLinuxOnly = RunningOnLinux and not RunningOnAndroid # non-Android Linux #=============================================================================== # Debugging paths #=============================================================================== def logit(*args): logfile = open('/Users/me/pelog.txt', 'a') # edit me... print(*args, file=logfile) logfile.close() def showRunContext(info=logit, env=True, msg=None): # info=logit | print| (lambda *args: None) """ display this distribution's runtime context; for analyzing options and changes made here; caveat: __file__ is this mod's, not importer's; SEE ALSO: docetc/examples/RunCode/contextdump.py to display context in GUI spawned via Run Code; [4.0] beef up, and log to file for app/exe; """ info('='*90) info(time.asctime()) if msg: info(msg) else: info('PyEdit cwd: ', os.getcwd()) info('PyEdit __file__: ', __file__) info('PyEdit sys.argv[0]: ', sys.argv[0]) info('PyEdit sys.executable:', sys.executable) info('PyEdit sys.frozen: ', getattr(sys, 'frozen', '*notset*')) info('PyEdit sys.argv[1:]: ', sys.argv[1:]) info('PyEdit sys.path:') for path in sys.path: info(' ', path) if env: info('PyEdit os.environ:') for key in sorted(os.environ): info(key, '=>\n\t', os.environ[key]) #=============================================================================== # Set global import-path context to find imported modules. #=============================================================================== # # In PyEdit, this enables importing: # - textConfig.py, which is user changeable # # All other modules are baked into the frozen executable, and all data # resources (icons, proxy, etc.) are located via fetchMyInstallDir() below. # # Don't add '__pyedit-component-data__' subdir here: textConfig.py is in # the embedding app's folder, not in the component resources subfolder, # and sys.executable is the embeeding app in this mode (not PyEdit). #=============================================================================== if DebugContext: showRunContext() if hasattr(sys, 'frozen') and sys.frozen == 'macosx_app': # # 1) Mac py2app app-bundle folder distribution # Frozen importer's bootloader is in PyEdit.app's Content/MacOS dir. # Add '.' for importing config module in app's Content/Resources dir. # dirname(__file__) and cwd work for icons, UserGuide.html, scripts, # and imports: code is source files, run by the app's bundled Python. # Ok to use cwd here: py2app cds to the data dir by default anyhow. # assert RunningOnMacOS sys.path.append(os.getcwd()) # for textConfig.py import elif hasattr(sys, 'frozen') and (RunningOnWindows or RunningOnLinux): # # 2) Windows and Linux PyInstaller single-file executable distribution # Use exe's path (not temp _MEI* unzip) for config module and all data. # The config module cannot be in PyInstaller's auto-unzip Temp dir. # DROPPED os.chdir(): this made empty __file__ dir map to install dir, # but precluded any relative paths in cmdline args to frozen exes; see # fetchMyInstallDir() below for the later explicit-install-dir scheme. # exepath = sys.executable # [4.0] sys.argv[0]=symlink exedir = os.path.dirname(os.path.abspath(exepath)) sys.path.append(exedir) # for textConfig.py import ##os.chdir(exedir) # NOT for extras: fetchMyInstallDir! elif hasattr(sys, 'frozen') and (RunningOnMacOS): # # 3) [4.0] macOS app now built by Pyinstaller, not py2app, so this clause # supercedes #1 above. In this mode, PyEdit is a PyEdit.app/ folder with # both Contents/MacOS for exes and binary libs, and Contents/Resources for # text data and modules. See MACOS SPECIAL CASE ahead for more info. # exepath = sys.executable # [4.0] sys.argv[0]=symlink exedir = os.path.dirname(os.path.abspath(exepath)) sys.path.append(os.path.join(exedir, '..', 'Resources')) # for textConfig.py import else: # # 4) Portable Source code distributions - nested within PP4E package or not: # - Run in self-contained PyEdit distribution with PP4E package nested here # - Imported or run in a client program, which nests PP4E, which nests me # - Imported by multiprocessing for grep producer in either PP4E scheme # Must use dir(__file__), not '.', to support relative pathname arguments. # # [4.0] THIS CASE ALSO INCLUDES ANDROID (source-code pkg in Pydroid 3 app). # normal source-code case for when file opened in app itself, but __file__ # is /data/user/0/.../files/temp_iiec_codefile.py when running as a homescreen # shortcut or explorer open-with and accessed by SAF file provider: use CWD # and assume os.chdir() line enabled in PyEdit.py. A horrible hack indeed, # but PyEdit's script name will never clash with the app's temp file. # Not required in Frigcal, because the top-level script (in /data) imports # another module (in CWD) that imports ffp.py; in PyEdit, top-level imports. # if RunningOnAndroid and __file__.endswith('temp_iiec_codefile.py'): # [4.0] special case for android shortcut/explorer srcdir = os.path.abspath(os.getcwd()) else: # android direct opens and all other source contexts srcdir = os.path.dirname(os.path.abspath(__file__)) if 'PP4E' in os.listdir(srcdir): # # 4a) Srcdir already on sys.path: no action required for imports. # Data will be located by fetchMyInstallDir() below. # This might also check for the package in each sys.path dir. # pass else: # # 4b) Assume this file is nested in an enclosing PP4E package. # Data will be located by fetchMyInstallDir() below. # Use __file__ to allow rel path args. Harmless if wrong. # upfromhere = os.path.join(srcdir, '..', '..', '..') sys.path.append(upfromhere) if DebugContext: showRunContext() #=============================================================================== # Return path where program resource files reside (some are user changeable). #=============================================================================== # # In PyEdit, the result is used to access installed: # # - Window icon for Windows and Linux # - README and image for Help # - Subprocproxy exe or script for Run Code # # It's also used to create and write a Run Code nag file (if permissions allow). #=============================================================================== def fetchMyInstallDir(__file__): # not global __file__ """ -------------------------------------------------------- call this to fetch folder where extra items reside; use to access installed icon, readme, proxy script; replaces former os.chdir() which precluded rel paths; the return value is always an absolute pathname; pass importer's __file__ to __file__ arg: for frozen Mac p2app, this module's dir(__file__) is in a zipfile, and differs from the importer's dir(__file__); they're the same for source code, and unused for Win/Lin exes; -------------------------------------------------------- """ if hasattr(sys, 'frozen') and (RunningOnWindows or RunningOnMacOS): # # PyInstaller executable: from sys.executable = exe's dir; # __file__ dir is empty (or worse) and cwd may be any user # folder if the importer is run from a command line elsewhere; # both are unusable in frozen app/exe in general (cwd='/'); # # for pyedit only, data is in a component dir iff running # as a component in the pymailgui program: pyedit subdir # data is copied by build scripts to this nested folder; # # TBD: componentdir is at END of result here (not in!); # it's not quite correct to assume that result is only a # PyEdit dir: clicking a filename in Windows sets argv[0] # to that filename's folder, even for a frozen exe - if # that folder happens to have a .ico or README, ... # # [4.0] dropped RunningOnLinux in test: we're no longer # building its exe becaus Linux lib deps are a nightmare. # # [4.0] MACOS SPECIAL CASE # The macOS app now built by Pyinstaller, not by py2app. # Its resources folder is a .app/Contents/MacOS sibling so # this must use ../Resources. The Pyedit app's build moves # resources manually instead of using the PI --add-data arg # like Frigcal; that arg creates links in Contents/Frameworks # that make the manual ../Resources here unnecessary in Frigcal # (in PI --onedir, __file__ is .app/Contents/Frameworks, and # sys.exe is /MacOS). See Frigcal's version for more details. # exepath = sys.executable # [4.0] sys.argv[0]=symlink exedir = os.path.dirname(os.path.abspath(exepath)) componentdir = '__pyedit-component-data__' if componentdir in os.listdir(exedir): return os.path.join(exedir, componentdir) else: if RunningOnWindows: return exedir elif RunningOnMacOS: return os.path.abspath(os.path.join(exedir, '..', 'Resources')) # abs else: # # Mac py2app bundle or source-code: from __file__ as usual; # cwd is anywhere: return importing file's install folder; # this mod's __file__ is *not* ok to use here for frozen # Mac apps: it's in a zipfile, not importer's Resources/; # # [4.0] THIS CASE ALSO INCLUDES ANDROID # See same comment above for explanation of special case here; # srcpath = __file__ if RunningOnAndroid and srcpath.endswith('temp_iiec_codefile.py'): # [4.0] special case for android shortcut/explorer srcdir = os.path.abspath(os.getcwd()) else: # android direct opens and all other source contexts srcdir = os.path.dirname(os.path.abspath(srcpath)) return srcdir if DebugContext: showRunContext(msg=f'{fetchMyInstallDir(__file__)=}')