[Feb-2017] Summary=> Demo AppleDouble ._* resource-fork files under Python's POSIX toolset. In all cases, these files appear for data files stored on non-Mac filesystems only; the Mac's own filesystems store resources natively. Finder automatically creates resource-fork files on non-Mac drives, and binds them to the data file logically - operations on a data file are applied to the resource fork too (e.g., deletes, moves). Python's POSIX-based toolset is more mixed: open() does not silently add a resource fork for new data files, and os.remove() for a resource file doesn't delete the data file. When run on the Mac, these behaviors are crucial for mergeall's -skipcruft (which copies data files only), and the nuke-cruft-files script (which deletes resource files only). On the other hand, Python's os.remove() and os.rename() for a data file on a non-Mac drive are automatically applied by the Mac's system libraries to the resource fork file too (just like Finder). This can potentially lead to errors in some programs that process a folder's files individually. For more background: https://en.wikipedia.org/wiki/AppleSingle_and_AppleDouble_formats https://en.wikipedia.org/wiki/Resource_fork #------------------------------------------------------------------------------------- # Preliminaries (assumed by all tests ahead) #------------------------------------------------------------------------------------- # Copy file off Mac drive - via main data fork filename >>> f = open('/MY-STUFF/Sheets/cashflow.xlsb', 'rb') >>> b = f.read() >>> f.close() # There is no resource fork file on this Mac drive >>> [x for x in os.listdir('/MY-STUFF/Sheets') if 'cashflow.xlsb' in x] ['cashflow.xlsb'] >>> import glob >>> glob.glob('/MY-STUFF/Sheets/*cashflow.*') ['/MY-STUFF/Sheets/cashflow.xlsb', '/MY-STUFF/Sheets/endyr02-cashflow.XLS'] >>> glob.glob('/MY-STUFF/Sheets/.*cashflow.*') # see glob note ahead [] #------------------------------------------------------------------------------------- # MAC DRIVE (HFS+) #------------------------------------------------------------------------------------- Summary=> On native Mac filesystem drives, no ._* AppleDouble files are created or special-cased (these pertain to non-Mac drives only). >>> os.listdir('/Users/blue/Desktop/temp/test') ['.DS_Store'] >>> f = open('/Users/blue/Desktop/temp/test/cashflow.xlsb', 'wb') >>> f.write(b) 397387 >>> f.close() # Python POSIX write creates data file only on Mac HFS+ drive >>> os.listdir('/Users/blue/Desktop/temp/test') ['.DS_Store', 'cashflow.xlsb'] >>> os.path.getmtime('/Users/blue/Desktop/temp/test/cashflow.xlsb') 1480791888.0 # Open to view only: creates Excel temp, but no ._* resource fork on HFS >>> os.listdir('/Users/blue/Desktop/temp/test') ['.DS_Store', 'cashflow.xlsb', '~$cashflow.xlsb'] >>> os.path.getmtime('/Users/blue/Desktop/temp/test/cashflow.xlsb') 1480791888.0 # Close, don't save: removes Excel temp >>> os.listdir('/Users/blue/Desktop/temp/test') ['.DS_Store', 'cashflow.xlsb'] >>> os.path.getmtime('/Users/blue/Desktop/temp/test/cashflow.xlsb') 1480791888.0 # Open and save: still no distinct resource fork file on HFS+ >>> os.listdir('/Users/blue/Desktop/temp/test') ['.DS_Store', 'cashflow.xlsb'] >>> os.path.getmtime('/Users/blue/Desktop/temp/test/cashflow.xlsb') 1480792056.0 # Delete .xlsb file: just a single file here >>> os.listdir('/Users/blue/Desktop/temp/test') ['.DS_Store'] #------------------------------------------------------------------------------------- # WINDOWS DRIVE (FAT32) #------------------------------------------------------------------------------------- Summary=> AppleDouble ._* files are silently created on non-Mac drives when used by Mac apps, even if not copied over from Mac sources. # Create data file only, from Python >>> os.listdir('/Volumes/KINGSTONFAT/test') [] >>> f = open('/Volumes/KINGSTONFAT/test/cashflow.xlsb', 'wb') >>> f.write(b) 397387 >>> f.close() # Python POSIX write creates data-fork file only on FAT. # *EXAMPLE*: This is what happens for mergeall's -skipcruft. >>> os.listdir('/Volumes/KINGSTONFAT/test') ['cashflow.xlsb'] >>> os.path.getmtime('/Volumes/KINGSTONFAT/test/cashflow.xlsb') 1480791672.0 # Open to view in Excel: suffices to create a ._* resource fork on FAT (!). # The 1 file has become 4: data + temp, and ._* AppleDoubles of each. >>> os.listdir('/Volumes/KINGSTONFAT/test') ['cashflow.xlsb', '~$cashflow.xlsb', '._~$cashflow.xlsb', '._cashflow.xlsb'] >>> os.path.getmtime('/Volumes/KINGSTONFAT/test/cashflow.xlsb') 1480791672.0 # Close, don't save: removes Excel ~* temp and its double, but not ._* resource fork. # Note that Finder never shows the ._* AppleDouble resource fork files (despite a # defaults setting to enable this), but they are always returned by os.listdir(). >>> os.listdir('/Volumes/KINGSTONFAT/test') ['cashflow.xlsb', '._cashflow.xlsb'] >>> os.path.getmtime('/Volumes/KINGSTONFAT/test/cashflow.xlsb') 1480791672.0 # Open and save: but a write is not needed to create the resource ._* (!) >>> os.listdir('/Volumes/KINGSTONFAT/test') ['cashflow.xlsb', '._cashflow.xlsb'] >>> os.path.getmtime('/Volumes/KINGSTONFAT/test/cashflow.xlsb') 1480791788.0 # Delete .xlsb file in Finder: automatically removes the truly-hidden ._* too (!) >>> os.listdir('/Volumes/KINGSTONFAT/test') [] #------------------------------------------------------------------------------------- # WARNING: glob() doesn't return .*, but os.listdir() does #------------------------------------------------------------------------------------- Summary=> Be cautious with glob() results on Unix; they differ from os.listdir(). # After write file, open in Excel >>> os.listdir('/Volumes/KINGSTONFAT/test') ['cashflow.xlsb', '._cashflow.xlsb'] >>> >>> import glob >>> glob.glob('/Volumes/KINGSTONFAT/test/*') ['/Volumes/KINGSTONFAT/test/cashflow.xlsb'] >>> >>> glob.glob('/Volumes/KINGSTONFAT/test/*cashflow.*') ['/Volumes/KINGSTONFAT/test/cashflow.xlsb'] >>> >>> glob.glob('/Volumes/KINGSTONFAT/test/.*cashflow.*') ['/Volumes/KINGSTONFAT/test/._cashflow.xlsb'] >>> >>> [x for x in os.listdir('/Volumes/KINGSTONFAT/test') if 'cashflow.xlsb' in x] ['cashflow.xlsb', '._cashflow.xlsb'] #------------------------------------------------------------------------------------- # WINDOWS DRIVE: what happens when files are deleted from Python instead of Finder? #------------------------------------------------------------------------------------- Summary=> Mac POSIX libs auto-delete ._* files with data file just like Finder, but not vice-versa. This may cause some programs failures. # Create and remove data file only: normal >>> os.listdir('/Volumes/KINGSTONFAT/test') [] >>> f = open('/Volumes/KINGSTONFAT/test/cashflow.xlsb', 'wb') >>> f.write(b) 397387 >>> f.close() >>> os.listdir('/Volumes/KINGSTONFAT/test') ['cashflow.xlsb'] >>> os.remove('/Volumes/KINGSTONFAT/test/cashflow.xlsb') >>> os.listdir('/Volumes/KINGSTONFAT/test') [] # Delete *data file* after resource fork created by opening in Excel >>> f = open('/Volumes/KINGSTONFAT/test/cashflow.xlsb', 'wb') >>> f.write(b) 397387 >>> f.close() >>> os.listdir('/Volumes/KINGSTONFAT/test') ['cashflow.xlsb'] # open in Excel, close but don't save file >>> os.listdir('/Volumes/KINGSTONFAT/test') ['cashflow.xlsb', '._cashflow.xlsb'] # automatically removes reource fork too (!) >>> os.remove('/Volumes/KINGSTONFAT/test/cashflow.xlsb') >>> os.listdir('/Volumes/KINGSTONFAT/test') [] # Delete *resource file* after it is created by Excel open >>> f = open('/Volumes/KINGSTONFAT/test/cashflow.xlsb', 'wb') >>> f.write(b) 397387 >>> f.close() >>> os.listdir('/Volumes/KINGSTONFAT/test') ['cashflow.xlsb'] # open and close in Excel (save or not) >>> os.listdir('/Volumes/KINGSTONFAT/test') ['cashflow.xlsb', '._cashflow.xlsb'] # does not remove data file if resource file removed (!) # *EXAMPLE*: This is what happens in nuke-cruft-files.py >>> os.remove('/Volumes/KINGSTONFAT/test/._cashflow.xlsb') >>> os.listdir('/Volumes/KINGSTONFAT/test') ['cashflow.xlsb'] # clean up data file: the ._* special-casing seems too implicit and error-prone >>> os.remove('/Volumes/KINGSTONFAT/test/cashflow.xlsb') >>> os.listdir('/Volumes/KINGSTONFAT/test') [] [Why it can matter: if a script runs through an os.listdir() result to delete files on a non-Mac filesystem drive, some ._* resource fork files may no longer be present when reached - and will fail.] #------------------------------------------------------------------------------------- # WINDOWS DRIVE: renames (moves) bring the ._* along too (and even rename them!) #------------------------------------------------------------------------------------- Summary=> Mac POSIX libs auto-move ._* files with data file just like Finder. This is true even if the data file's name is changed in the process. # Moves bring along AppleDouble automatically >>> f = open('/Volumes/KINGSTONFAT/test/cashflow.xlsb', 'wb') >>> f.write(b) 397387 >>> f.close() >>> os.listdir('/Volumes/KINGSTONFAT/test') # after initial Python write ['cashflow.xlsb'] >>> os.listdir('/Volumes/KINGSTONFAT/test') # after open in Excel ['cashflow.xlsb', '._cashflow.xlsb'] >>> os.listdir('/Volumes/KINGSTONFAT/test2') # target dir is empty [] >>> os.rename('/Volumes/KINGSTONFAT/test/cashflow.xlsb', '/Volumes/KINGSTONFAT/test2/cashflow.xlsb') >>> os.listdir('/Volumes/KINGSTONFAT/test') [] >>> os.listdir('/Volumes/KINGSTONFAT/test2') # brings AppleDouble along... ['cashflow.xlsb', '._cashflow.xlsb'] # Ditto, even if moving to a new name (!) >>> f = open('/Volumes/KINGSTONFAT/test/cashflow.xlsb', 'wb') >>> f.write(b) 397387 >>> f.close() >>> os.listdir('/Volumes/KINGSTONFAT/test') # post Python write ['cashflow.xlsb'] >>> os.listdir('/Volumes/KINGSTONFAT/test') # post Excel open ['cashflow.xlsb', '._cashflow.xlsb'] >>> os.listdir('/Volumes/KINGSTONFAT/test2') [] >>> os.rename('/Volumes/KINGSTONFAT/test/cashflow.xlsb', '/Volumes/KINGSTONFAT/test2/SPAM.xlsb') >>> os.listdir('/Volumes/KINGSTONFAT/test') [] >>> os.listdir('/Volumes/KINGSTONFAT/test2') # buy one, get one free? ['SPAM.xlsb', '._SPAM.xlsb'] #------------------------------------------------------------------------------------- # AND Windows and Linux may or may not handle ._* file the same way... (test me) #------------------------------------------------------------------------------------- #------------------------------------------------------------------------------------- # RESOURCE FORKS ROUND-TRIPPED to non-Mac drives (use dot_clean if not merged auto) #------------------------------------------------------------------------------------- Summary=> Some programs may copy AppleDouble resource-fork files to Mac drives, but not merge them to the main data fork file (including mergeall). If this happens, Mac's dot_clean can merge the two in an entire tree. Note that mergeall doesn't make "._*" files: this is only possible if they have been added to an archive by Mac apps/Finder in other ways (e.g., merging from a non-Mac drive used as mergeall's FROM). >>> macdrive = '/Users/blue/Desktop/temp/test/' # HFS+ (Mac Extended) >>> fatdrive = '/Volumes/KINGSTONFAT/test/' # FAT32 (flashdrive) # Copy a file to a macdrive folder (and open in excel there or not) >>> f = open('/MY-STUFF/Sheets/cashflow.xlsb', 'rb') >>> b = f.read() >>> f.close() >>> >>> f = open(macdrive + 'cf.xlsb', 'wb') >>> f.write(b) 397387 >>> f.close() >>> >>> os.listdir(macdrive) ['.DS_Store', 'cf.xlsb'] >>> os.listdir(fatdrive) [] # Copy from macdrive to fatdrive in Finder >>> os.listdir(fatdrive) ['cf.xlsb'] # Open/view on fatdrive in Excel: makes AppleDouble resource fork >>> os.listdir(fatdrive) ['cf.xlsb', '._cf.xlsb'] # Remove from fatdrive in Finder or Python: both files gone! (see above) >>> os.remove(fatdrive + 'cf.xlsb') >>> os.listdir(fatdrive) [] # Copy from macdrive to fatdrive in python >>> os.listdir(macdrive) ['.DS_Store', 'cf.xlsb'] >>> os.listdir(fatdrive) [] >>> f = open(macdrive + 'cf.xlsb', 'rb') >>> b = f.read() >>> f.close() >>> >>> f = open(fatdrive + 'cf.xlsb', 'wb') >>> f.write(b) 397387 >>> f.close() >>> >>> os.listdir(fatdrive) ['cf.xlsb'] >>> os.listdir(macdrive) ['.DS_Store', 'cf.xlsb'] # Open on fatdrive in Excel: makes "._*" >>> os.listdir(fatdrive) ['cf.xlsb', '._cf.xlsb'] >>> os.listdir(macdrive) ['.DS_Store', 'cf.xlsb'] # Delete from macdrive in Finder: one file only >>> os.listdir(fatdrive) ['cf.xlsb', '._cf.xlsb'] >>> os.listdir(macdrive) ['.DS_Store'] # Copy from fatdrive to macdrive in Finder >>> os.listdir(fatdrive) ['cf.xlsb', '._cf.xlsb'] >>> os.listdir(macdrive) # copies resource too (presumably!) ['.DS_Store', 'cf.xlsb'] # Delete from macdrive in Python: one file only >>> os.remove(macdrive + 'cf.xlsb') >>> os.listdir(fatdrive) ['cf.xlsb', '._cf.xlsb'] >>> os.listdir(macdrive) ['.DS_Store'] # Copy to macdrive in python (EXAMPLE: what mergeall does) >>> for file in os.listdir(fatdrive): ... f = open(macdrive + file, 'wb') ... f.write(open(fatdrive + file, 'rb').read()) ... f.close() ... print(file) ... 397387 cf.xlsb 4096 ._cf.xlsb >>> >>> os.listdir(fatdrive) # <= BOTH files copied over, not merged! ['cf.xlsb', '._cf.xlsb'] >>> os.listdir(macdrive) ['._cf.xlsb', '.DS_Store', 'cf.xlsb'] >>> ^D # Merge from the Terminal shell via Mac's dot_merge ~$ ls -a /Volumes/KINGSTONFAT/test . .. ._cf.xlsb cf.xlsb ~$ ls -a /Users/blue/Desktop/temp/test . .. .DS_Store ._cf.xlsb cf.xlsb ~$ dot_clean -m /Users/blue/Desktop/temp/test # Still split on fatdrive, but merged on macdrive ~$ ls -a /Volumes/KINGSTONFAT/test . .. ._cf.xlsb cf.xlsb ~$ ls -a /Users/blue/Desktop/temp/test . .. .DS_Store cf.xlsb # CAUTION: will run on fatdrive too, but resource is silently dropped ~$ dot_clean -m /Volumes/KINGSTONFAT/test # <= Don't do this ~$ ls -a /Volumes/KINGSTONFAT/test . .. cf.xlsb ~$ ls -a /Users/blue/Desktop/temp/test . .. .DS_Store cf.xlsb # # Bonus: how to inspect a resource fork from the shell.. # ~$ ls -l /MY-STUFF/Sheets/cashflow.xlsb -rw-r--r--@ 1 blue staff 397387 Dec 2 10:02 /MY-STUFF/Sheets/cashflow.xlsb ~$ ls -l /MY-STUFF/Sheets/cashflow.xlsb/..namedfork/rsrc -rw-r--r-- 1 blue staff 0 Dec 2 10:02 /MY-STUFF/Sheets/cashflow.xlsb/..namedfork/rsrc ~$ cat /MY-STUFF/Sheets/cashflow.xlsb/..namedfork/rsrc ~$ [end]