![[Home page]](../PythonPowered.gif "Home page"){kind=small}
| Books | Code | Blog | Python | Author | Train | Find | ©M.Lutz |
File: mergeall-products/unzipped/docetc/miscnotes/mac-chflags-error22.txt
Mac's chflags() seems to throw error 22=Invalid Argument, when trying to
copy a file's flags from a Mac HFS+ drive to a non-Mac (exFAT) drive.
The flags were added by TextEdit; they record encoding type, and cause
every file edited in TextEdit to trigger an error message in mergeall,
even though they copy correctly in both content and timestamps. This
fails on an uncaught exception in the very last step of shutil.copystat(),
(used by cpall.py) after all else has been copied correctly.
Solution: redefine os.chflags() to try original, catch and ignore
errno.EINVAL on Macs only, and allow all others to pass. Could
delete os.chflags() completely, but may be useful when TO is a
Mac drive. Could also verify that times were copied over
correctly and reraise if not, but this seems too much work.
Update: Mac extended attributes can also arise in other contexts,
so this is a general concern; another example from PyEdit:
$ ls -l@ Whitepaper.html
-rwxrwxrwx@ 1 blue wheel 90072 May 30 18:46 Whitepaper.html
com.apple.quarantine 29
$ xattr Whitepaper.html
com.apple.quarantine
$ xattr -p com.apple.quarantine Whitepaper.html
0002;5928a690;Microsoft Word;
-------------------------------------------------------------------------------
After this mergeall error was reported:
-------------------------------------------------------------------------------
# force difference...
$ rm /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
# mergeall report...
*Resolving tree differences
Skipping system cruft (metadata) files in FROM folders
**Error copying FROM file: skipped /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
<class 'OSError'> [Errno 22] Invalid argument: '/Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt'
Phase runtime: 0.9569850449915975
-------------------------------------------------------------------------------
Files same in content and modtimes, but extended flags differ
-------------------------------------------------------------------------------
$ ls -l /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
-rwxrwxrwx@ 1 blue wheel 5295 Dec 5 11:36 /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
-rwxrwxrwx 1 blue staff 5295 Dec 5 11:36 /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
$
$ diff /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
$
$ xattr /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
com.apple.TextEncoding
$ xattr /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
$
$ xattr -p com.apple.TextEncoding /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
utf-8;134217984
$ xattr -p com.apple.TextEncoding /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
xattr: /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt: No such xattr: com.apple.TextEncoding
$
$ ls -lO@ /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
-rwxrwxrwx@ 1 blue wheel - 5295 Dec 5 11:36 /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
com.apple.TextEncoding 15
l$ ls -lO@ /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
-rwxrwxrwx 1 blue staff - 5295 Dec 5 11:36 /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
$
-------------------------------------------------------------------------------
Fails in the shell too, with same error -- BUT seems to copy flags anyhow...
-------------------------------------------------------------------------------
$ cp -p /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
cp: fchflags: /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt: Invalid argument
$
$ xattr /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
com.apple.TextEncoding
$ xattr /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
com.apple.TextEncoding
$
$ xattr -p com.apple.TextEncoding /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
utf-8;134217984
$ xattr -p com.apple.TextEncoding /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
utf-8;134217984
$
-------------------------------------------------------------------------------
What Python shutil.copystat() fails on: does not catch errorno 22 = EINVAL
-------------------------------------------------------------------------------
$ py3
Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:47:25)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, stat
>>> st = os.stat('/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt')
>>> os.chflags('/Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt', st.st_flags)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: [Errno 22] Invalid argument: '/Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt'
>>>
>>> os.chflags('/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt', st.st_flags) # but Mac drive works
>>> ^D
$
$ xattr -p com.apple.TextEncoding /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
utf-8;134217984
$ xattr -p com.apple.TextEncoding /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
utf-8;134217984
------------------------------------------------------------------------------
The error code in Python
------------------------------------------------------------------------------
>>> import errno
>>> dir(errno)
['E2BIG', 'EACCES', 'EADDRINUSE', 'EADDRNOTAVAIL', 'EAFNOSUPPORT', 'EAGAIN', 'EALREADY', 'EAUTH', 'EBADARCH', 'EBADEXEC', 'EBADF', 'EBADMACHO', 'EBADMSG', 'EBADRPC', 'EBUSY', 'ECANCELED', 'ECHILD', 'ECONNABORTED', 'ECONNREFUSED', 'ECONNRESET', 'EDEADLK', 'EDESTADDRREQ', 'EDEVERR', 'EDOM', 'EDQUOT', 'EEXIST', 'EFAULT', 'EFBIG', 'EFTYPE', 'EHOSTDOWN', 'EHOSTUNREACH', 'EIDRM', 'EILSEQ', 'EINPROGRESS', 'EINTR', 'EINVAL', 'EIO', 'EISCONN', 'EISDIR', 'ELOOP', 'EMFILE', 'EMLINK', 'EMSGSIZE', 'EMULTIHOP', 'ENAMETOOLONG', 'ENEEDAUTH', 'ENETDOWN', 'ENETRESET', 'ENETUNREACH', 'ENFILE', 'ENOATTR', 'ENOBUFS', 'ENODATA', 'ENODEV', 'ENOENT', 'ENOEXEC', 'ENOLCK', 'ENOLINK', 'ENOMEM', 'ENOMSG', 'ENOPOLICY', 'ENOPROTOOPT', 'ENOSPC', 'ENOSR', 'ENOSTR', 'ENOSYS', 'ENOTBLK', 'ENOTCONN', 'ENOTDIR', 'ENOTEMPTY', 'ENOTSOCK', 'ENOTSUP', 'ENOTTY', 'ENXIO', 'EOPNOTSUPP', 'EOVERFLOW', 'EPERM', 'EPFNOSUPPORT', 'EPIPE', 'EPROCLIM', 'EPROCUNAVAIL', 'EPROGMISMATCH', 'EPROGUNAVAIL', 'EPROTO', 'EPROTONOSUPPORT', 'EPROTOTYPE', 'EPWROFF', 'ERANGE', 'EREMOTE', 'EROFS', 'ERPCMISMATCH', 'ESHLIBVERS', 'ESHUTDOWN', 'ESOCKTNOSUPPORT', 'ESPIPE', 'ESRCH', 'ESTALE', 'ETIME', 'ETIMEDOUT', 'ETOOMANYREFS', 'ETXTBSY', 'EUSERS', 'EWOULDBLOCK', 'EXDEV', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'errorcode']
>>> errno.errorcode[22]
'EINVAL'
>>> errno.EINVAL
22
------------------------------------------------------------------------------
A fuller example: WITHOUT (BEFORE CPALL.PY FIX)
------------------------------------------------------------------------------
$ rm /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
# rerun mergeall via its GUI, same error as above: files+times same, flags NOT copied
*Resolving tree differences
Skipping system cruft (metadata) files in FROM folders
**Error copying FROM file: skipped /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
<class 'OSError'> [Errno 22] Invalid argument: '/Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt'
$ ls -lO@ /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
-rwxrwxrwx@ 1 blue wheel - 5295 Dec 5 11:36 /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
com.apple.TextEncoding 15
$ ls -lO@ /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
-rwxrwxrwx 1 blue staff - 5295 Dec 5 11:36 /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
# recreate error interctively: same results
$ py3
Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:47:25)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, stat
>>> st = os.stat('/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt')
>>> os.chflags('/Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt', st.st_flags)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: [Errno 22] Invalid argument: '/Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt'
>>> ^D
$ ls -lO@ /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
-rwxrwxrwx@ 1 blue wheel - 5295 Dec 5 11:36 /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
com.apple.TextEncoding 15
$ ls -lO@ /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
-rwxrwxrwx 1 blue staff - 5295 Dec 5 11:36 /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
$
$ xattr -p com.apple.TextEncoding /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
utf-8;134217984
$ xattr -p com.apple.TextEncoding /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
xattr: /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt: No such xattr: com.apple.TextEncoding
$
# same error at shell: BUT flags seem to be copied over anyhow (unlike Python)
$ cp -p /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
cp: fchflags: /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt: Invalid argument
$
$ ls -lO@ /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
-rwxrwxrwx@ 1 blue wheel - 5295 Dec 5 11:36 /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
com.apple.TextEncoding 15
$ ls -lO@ /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
-rwxrwxrwx@ 1 blue staff - 5295 Dec 5 11:36 /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
com.apple.TextEncoding 15
$ xattr -p com.apple.TextEncoding /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
utf-8;134217984
$ xattr -p com.apple.TextEncoding /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
utf-8;134217984
$
------------------------------------------------------------------------------
POST CPALL.PY FIX
------------------------------------------------------------------------------
$ rm /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
*Resolving tree differences
Skipping system cruft (metadata) files in FROM folders
copied new FROM file, /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
Phase runtime: 0.015111149987205863
$ ls -l /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
-rwxrwxrwx@ 1 blue wheel 5295 Dec 5 11:36 /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
-rwxrwxrwx 1 blue staff 5295 Dec 5 11:36 /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
$
$ diff /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
$
$ ls -lO@ /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
-rwxrwxrwx@ 1 blue wheel - 5295 Dec 5 11:36 /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
com.apple.TextEncoding 15
$ ls -lO@ /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
-rwxrwxrwx 1 blue staff - 5295 Dec 5 11:36 /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
$
$ xattr -p com.apple.TextEncoding /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
utf-8;134217984
$ xattr -p com.apple.TextEncoding /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
xattr: /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt: No such xattr: com.apple.TextEncoding
$
$ xattr /MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
com.apple.TextEncoding
$ xattr /Volumes/SAVAGEX256G/MY-STUFF/__more__/Gadgets/mac/CHEAT-SHEET.txt
$