Large built-in toolset:
sys, os, os.path, socket, signal, select, thread, threading, glob, shutil, tempfile, multiprocessing, subprocess, asyncio…
► See Python library manual for the full story
Large third-party domain:
PySerial, PyUSB, Pexpect, PySoap, Twisted, CORBA ORBs, …
► See the web for specific domains
Python tools: sys
● Python-related exports
● path: for imports, initialized from PYTHONPATH, changeable
● platform: for nonportable code, ‘win32’, ‘linux’, ‘sunos’, etc.
● sys.exit(N), sys.exc_info(), sys.executable, sys.version, sys.modules …
>>> import sys
>>> sys.path # first means CWD
['.', '/usr/local/lib/python', …etc…]
>>> sys.path
['C:/Python25', 'C:\\Python25\\Lib\\idlelib', 'C:\\WINDOWS\\system32\\python25.zip', …etc…
>>> sys.path
['', 'C:\\Users\\mark\\AppData\\Local\\Programs\\Python\\Python35\\python35.zip', …etc…
>>> sys.platform # once upon a time…
'sunos4'
>>> if sys.platform[:3] == 'win': # or .startswith('win'), .startswith('linux')
print 'on Windows'
on Windows
>>> sys.executable
'C:\\Python25\\pythonw.exe'
>>> sys.executable
'C:\\Users\\mark\\AppData\\Local\\Programs\\Python\\Python35\\python.exe'
>>> sys.version
'2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)]'
>>> sys.version
'3.5.0 (v3.5.0:374f501f4567, Sep 13 2015, 02:27:37) [MSC v.1900 64 bit (AMD64)]'
>>> if float(sys.version[:3]) >= 3.5: print('recent')
...
recent
>>> sys.version_info
sys.version_info(major=3, minor=5, micro=0, releaselevel='final', serial=0)
>>> sys.modules.keys()
…loaded module names…
System tools: os
● POSIX bindings: operating system exports
● ~200+ attributes on some platforms, + nested “os.path” module
● Mostly portable, some calls absent on standard Windows Python
● Cygwin Python adds full UNIX call set on Windows
Content survey
► Shell environment variables
os.environ
► Running shell commands, programs:
os.system, os.popen, os.startfile,
os.popen2/3/4 (2.X) (+subprocess)
► Spawning processes:
os.fork, os.pipe, os.exec,
os.waitpid, os.kill
► Descriptor files, with locking:
os.open, os.read, os.write
► File processing:
os.remove, os.rename,
os.mkfifo, os.mkdir, os.rmdir, os.removedirs
► Administrative tools:
os.getcwd, os.chdir,
os.chmod, os.getpid
► Directory tools:
os.listdir, os.scandir
(3.5+), os.walk
► Portability tools:
os.sep, os.pathsep,
os.curdir, os.path.split, os.path.join
► os.path nested submodule: pathname tools
os.path.exists('filepathname')
os.path.isdir('filepathname')
os.path.getsize('filepathname')
os.path.getmtime('filepathname')
>>> import os, sys
>>> lister = 'dir /B' if sys.platform.startswith('win') else 'ls' # folder listing command
>>> listing = os.popen('%s *.py' % lister).readlines() # or: glob.glob('*.py')
>>> for name in listing: print(name, end='') # 2.X: print name,
...
classtools.py
coroutine.py
person.py
>>> editor = 'notepad' if sys.platform.startswith('win') else 'vi' # edit each in listing
>>> for name in listing: os.system(editor + ' ' + name)
...
os.popen('cmd', 'w').write('data')
(i, o) = os.popen2('cmd') # popen2/3/4 removed in 3.X: use subprocess
(i, o, e) = os.popen3('cmd')
(i, o_e) = os.popen4('cmd')
os.startfile('file') # Windows only: registry
► See “pty” module,
“pexpect” extension, to automate
interactive programs without deadlocks
► See ahead: new “subprocess”
module in 2.5+ for more low-level control over streams
Example: testing command-line scripts (advanced sessions)
Shell environment variables: os
● os.environ: read/write access to shell variables
● normal dictionary interface
>>> import os
>>> os.environ['USER']
'mlutz'
>>> os.environ['USER'] = 'Bob' # changes for process and its children
Arguments and streams: sys
● sys.argv: command-line arguments
● sys.stdin/stdout/stderr: standard stream files
● sys.executable is path to running interpreter
% type play.py
#!/usr/local/bin/python
import sys
print sys.argv
sys.stdout.write("ta da!\n") # same as: print 'ta da!'
% python play.py -x -i spammify
['play.py', '-x', '-i', 'spammify']
ta da!
See getopt, optparse,
argparse modules for parsing complex
command lines
♦ Built-in file objects
For most file applications
♦ Processing binary and text data
Binary: use “rb” and “wb” in open() to suppress line-end translations on Windows
Binary: use bytes (b’…’) for data on 3.X, use str (‘…’) on 2.X
Text, ASCII: use str and open() on both 2.X and 3.X
Text, Unicode: use str and open() on 3.X, use unicode (u’…’) and codecs.open() on 2.X (see final unit)
♦ Module os descriptor-based file tools
For special/advanced file processing modes
♦ Module os filename tools
Deletions, renamings, etc.
♦ Module os.path tools
File existence, directory tests, size, etc.
♦ See also
Sockets, pipes, fifos, shelves, DBM files
Single directories
C:\temp>python
>>> import os
>>> os.popen('dir /B').readlines()
['about-pp.html\n', 'python1.5.tar.gz\n', 'about-pp2e.html\n', # \n was \012 in the past
'about-ppr2e.html\n', 'newdir\n']
>>> os.popen('ls C:\PP2ndEd').readlines()
['README.txt\n', 'cdrom\n', 'chapters\n', 'etc\n', 'examples\n',
'examples.tar.gz\n', 'figures\n', 'shots\n']
>>> import glob
>>>
glob.glob('C:\PP2ndEd\*')
['C:\\PP2ndEd\\examples.tar.gz',
'C:\\PP2ndEd\\README.txt',
'C:\\PP2ndEd\\shots',
'C:\\PP2ndEd\\figures', 'C:\\PP2ndEd\\examples',
'C:\\PP2ndEd\\etc',
'C:\\PP2ndEd\\chapters', 'C:\\PP2ndEd\\cdrom']
>>> os.listdir('C:\PP2ndEd')
['examples.tar.gz', 'README.txt', 'shots', 'figures', 'examples', 'etc',
'chapters', 'cdrom']
>>> os.listdir('.')
['summer.out', 'summer.py', 'table1.txt', ... ]
>>> dirents = os.scandir('.')
>>> for dirent in dirents:
... print(dirent.name, dirent.path, dirent.is_file()) # see mergeall use case: 5x~10x faster
...
classtools.py
.\classtools.py True
classtools.pyc
.\classtools.pyc True
coroutine.py
.\coroutine.py True
__pycache__ .\__pycache__ False
Directory trees
>>> import os
>>> def lister(dummy, dirname, filesindir):
... print '[' + dirname + ']'
... for fname in filesindir:
... print os.path.join(dirname, fname) # handle one file
...
>>> os.path.walk('.', lister, None)
[.]
.\about-pp.html
.\python1.5.tar.gz
.\about-pp2e.html
.\about-ppr2e.html
.\newdir
[.\newdir]
.\newdir\temp1
.\newdir\temp2
.\newdir\temp3
.\newdir\more
[.\newdir\more]
.\newdir\more\xxx.txt
.\newdir\more\yyy.txt
>>> import os
>>> for (thisDir, dirsHere, filesHere) in os.walk('.'):
print thisDir, '=>'
for filename in filesHere:
print '\t', filename
. =>
w9xpopen.exe
py.ico
pyc.ico
README.txt
NEWS.txt
…
C:\temp>python
>>> import find
>>> find.find('*')
['.\\about-pp.html', '.\\about-pp2e.html', '.\\about-ppr2e.html',
'.\\newdir', '.\\newdir\\more', '.\\newdir\\more\\xxx.txt',
'.\\newdir\\more\\yyy.txt', '.\\newdir\\temp1', '.\\newdir\\temp2',
'.\\newdir\\temp3', '.\\python1.5.tar.gz']
# list files in dir tree by recursion
import sys, os
def mylister(currdir):
print '[' + currdir + ']'
for file in os.listdir(currdir): # list files here
path = os.path.join(currdir, file) # add dir path back
if not os.path.isdir(path):
print path
else:
mylister(path) # recur into subdirs
if __name__ == '__main__':
mylister(sys.argv[1]) # dir name in cmdline
Example: finding large files (advanced sessions)
>>> import glob, string, os
>>> glob.glob("*.py")
['cheader1.py', 'finder1.py', 'summer.py']
>>> for name in glob.glob("*.py"):
... os.rename(name, string.upper(name))
...
>>> glob.glob("*.PY")
['FINDER1.PY', 'SUMMER.PY', 'CHEADER1.PY']
#!/usr/bin/python
"""
Return all files matching a filename pattern at and below a root directory;
custom version of the now deprecated find module in the standard library:
import as "PP4E.Tools.find"; like original, but uses os.walk loop, has no
support for pruning subdirs, and is runnable as a top-level script;
find() is a generator that uses the os.walk() generator to yield just
matching filenames: use findlist() to force results list generation;
"""
import fnmatch, os
def find(pattern, startdir=os.curdir):
for (thisDir, subsHere, filesHere) in os.walk(startdir):
for name in subsHere + filesHere:
if fnmatch.fnmatch(name, pattern):
fullpath = os.path.join(thisDir, name)
yield fullpath
def findlist(pattern, startdir=os.curdir, dosort=False):
matches = list(find(pattern, startdir))
if dosort: matches.sort()
return matches
if __name__ == '__main__':
import sys
namepattern, startdir = sys.argv[1], sys.argv[2]
for name in find(namepattern, startdir): print(name)
♦ Spawns (copies) a program
♦ Parent and child run independently
♦ fork spawns processes; system/popen spawn commands
♦ Not available on standard Windows Python today:
►use threads, spawnv, multiprocessing, subprocess, Cygwin (or coroutines?)
# starts programs until you type 'q'
import os
parm = 0
while 1:
parm = parm+1
pid = os.fork()
if pid == 0: # copy process
os.execlp('python', 'python', 'child.py', str(parm)) # overlay program
assert 0, 'error starting program' # shouldn't return
else:
print 'Child is', pid
if raw_input() == 'q': break
See other process examples ahead in IPC section
♦ Runs function calls in parallel, share global (module) memory
♦ Portable: runs on Windows, Solaris, any with pthreads
♦ Global interpreter lock: one thread running code at a time
♦ Thread switches on bytecode counter and long-running calls (later 3.X: on timeouts)
♦ Must still synchronize concurrent updates with thread locks
♦ C extensions release and acquire global lock too
import
thread # _thread in 3.X
def
counter(myId, count):
# synchronize stdout access to avoid multi
prints on 1 line
for i in range(count):
mutex.acquire()
print '[%s] => %s' % (myId, i)
mutex.release()
mutex
= thread.allocate_lock()
for
i in range(10):
thread.start_new(counter, (i, 100))
import
time
time.sleep(10)
print
'Main thread exiting.'
Output
.
.
.
[3] => 98
[4] => 98
[5] => 98
[7] => 98
[8] => 98
[0] => 99
[9] => 98
[6] => 99
[1] => 99
[2] => 99
[3] => 99
[4] => 99
[5] => 99
[7] => 99
[8] => 99
[9] => 99
Main thread exiting.
Locking concurrent updaters
Fails:
# fails on windows due to concurrent updates;
# works if check-interval set higher or lock
# acquire/release calls made around the adds
import thread, time
count = 0
def adder():
global count
count = count + 1 # update shared global
count = count + 1 # thread swapped out before returns
for i in range(100):
thread.start_new(adder, ()) # start 100 update threads
time.sleep(5)
print count
Works:
import thread, time
mutex = thread.allocate_lock()
count = 0
def adder():
global count
mutex.acquire()
count = count + 1 # update shared global
count = count + 1 # thread swapped out before returns
mutex.release()
for i in range(100):
thread.start_new(adder, ()) # start 100 update threads
time.sleep(5)
print count
Also Works:
thread
lock context manager (auto acquire/release, like close for files)
import thread, time
mutex = thread.allocate_lock()
count = 0
def adder(lock):
global count
time.sleep(0.10)
with lock:
count = count + 1 # update shared global
count = count + 1
for i in range(100):
thread.start_new_thread(adder, (mutex,)) # start 100 update threads
time.sleep(11)
print count
See also:
“threading” module’s class-based interface
“Queue” module’s thread-safe queue get/put
import threading
class
mythread(threading.Thread): # subclass
Thread object
def __init__(self, myId, count):
self.myId = myId
self.count = count
threading.Thread.__init__(self)
def run(self): # run provides thread
logic
for i in range(self.count): # still synch stdout access
stdoutmutex.acquire()
print '[%s] => %s' % (self.myId,
i)
stdoutmutex.release()
stdoutmutex =
threading.Lock() # same as
thread.allocate_lock()
thread = mythread()
thread.start()
thread.join() # wait for exit
Coding Options for threading
import threading, _thread
def action(i):
print(i ** 32)
# subclass with state
class
Mythread(threading.Thread):
def __init__(self, i):
self.i = i
threading.Thread.__init__(self)
def run(self): #
redefine run for action
print(self.i ** 32)
Mythread(2).start() # start
invokes run()
# pass action in
thread =
threading.Thread(target=(lambda: action(2)))
# run invokes target
thread.start()
# same but no lambda wrapper for
state
threading.Thread(target=action,
args=(2,)).start() # callable plus
its args
# basic thread module
thread.start_new_thread(action,
(2,)) # all-function
interface
Thread Queue
"producer and consumer threads communicating with a shared queue"
numconsumers = 2 # how many consumers to start
numproducers = 4 # how many producers to start
nummessages = 4 # messages per producer to put
import _thread as thread, queue, time
safeprint = thread.allocate_lock() # else prints may overlap
dataQueue = queue.Queue() # shared global, infinite size
def producer(idnum):
for msgnum in range(nummessages):
time.sleep(idnum)
dataQueue.put('[producer id=%d, count=%d]' % (idnum, msgnum))
def consumer(idnum):
while True:
time.sleep(0.1)
try:
data = dataQueue.get(block=False)
except queue.Empty:
pass
else:
with safeprint:
print('consumer', idnum, 'got =>', data)
if __name__ == '__main__':
for i in range(numconsumers):
thread.start_new_thread(consumer, (i,))
for i in range(numproducers):
thread.start_new_thread(producer, (i,))
time.sleep(((numproducers-1) * nummessages) + 1)
print('Main thread exit.')
C:\...\PP4E\System\Threads> queuetest.py
consumer 1 got => [producer id=0, count=0]
consumer 0 got => [producer id=0, count=1]
consumer 1 got => [producer id=0, count=2]
consumer 0 got => [producer id=0, count=3]
consumer 1 got => [producer id=1, count=0]
consumer 1 got => [producer id=2, count=0]
consumer 0 got => [producer id=1, count=1]
consumer 1 got => [producer id=3, count=0]
consumer 0 got => [producer id=1, count=2]
consumer 1 got => [producer id=2, count=1]
consumer 1 got => [producer id=1, count=3]
consumer 1 got => [producer id=3, count=1]
consumer 0 got => [producer id=2, count=2]
consumer 1 got => [producer id=2, count=3]
consumer 1 got => [producer id=3, count=2]
consumer 1 got => [producer id=3, count=3]
Main thread exit.
Other Examples: Queue module, on CD Extras\Code\pp3e
Shell commands: os.system, os.popen, subprocess
# testexit_sys.py
def later():
import sys
print('Bye sys world')
sys.exit(42)
print('Never reached')
if __name__ == '__main__': later()
Basic
os module tools
C:\...\PP4E\System\Exits> python
>>> os.system('python testexit_sys.py')
Bye sys world
42
>>> pipe = os.popen('python testexit_sys.py')
>>> pipe.read()
'Bye sys world\n'
>>> pipe.close()
42
subprocess
gives more control: 3 ways
C:\...\PP4E\System\Exits> python
>>> from subprocess import Popen, PIPE, call
>>> pipe = Popen('python testexit_sys.py', stdout=PIPE)
>>> pipe.stdout.read()
b'Bye sys world\r\n'
>>> pipe.wait()
42
>>> call('python testexit_sys.py')
Bye sys world
42
>>> pipe = Popen('python testexit_sys.py', stdout=PIPE)
>>> pipe.communicate()
(b'Bye sys world\r\n', None)
>>> pipe.returncode
42
Sending input
>>> pipe = Popen('python hello-in.py', stdin=PIPE)
>>> pipe.stdin.write(b'Pokey\n')
>>> pipe.stdin.close()
>>> pipe.wait()
0
Both: input and output
>>> pipe = Popen('python reader.py', stdin=PIPE, stdout=PIPE)
>>> pipe.stdin.write(b'Lumberjack\n')
>>> pipe.stdin.write(b'12\n')
>>> pipe.stdin.close()
>>> output = pipe.stdout.read()
>>> pipe.wait()
0
>>> output
b'Got this: "Lumberjack"\r\nThe meaning of life is 12 24\r\n'
Tying programs’ streams together with pipes
>>> p1 = Popen('python writer.py', stdout=PIPE)
>>> p2 = Popen('python reader.py', stdin=p1.stdout, stdout=PIPE)
>>> output = p2.communicate()[0]
>>> output
b'Got this: "Help! Help! I\'m being repressed!"\r\nThe meaning of life is 42 84\r\n'
>>> p2.returncode
0
Multiprocessing: processes with threading API
+: Portability of threads + parallel performance of processes
-: Pickleability constraints (bound methods), not freely shared state
# Example 5-29. PP4E\System\Processes\multi1.py
"""
multiprocess basics: Process works like threading.Thread, but
runs function call in parallel in a process instead of a thread;
locks can be used to synchronize, e.g. prints on some platforms;
starts new interpreter on windows, forks a new process on unix;
"""
import os
from multiprocessing import Process, Lock
def whoami(label, lock):
msg = '%s: name:%s, pid:%s'
with lock:
print(msg % (label, __name__, os.getpid()))
if __name__ == '__main__':
lock = Lock()
whoami('function call', lock)
p = Process(target=whoami, args=('spawned child', lock))
p.start()
p.join()
for i in range(5):
Process(target=whoami, args=(('run process %s' % i), lock)).start()
with lock:
print('Main process exit.')
C:\...\PP4E\System\Processes> multi1.py
function call: name:__main__, pid:8752
spawned child: name:__main__, pid:9268
Main process exit.
run process 3: name:__main__, pid:9296
run process 1: name:__main__, pid:8792
run process 4: name:__main__, pid:2224
run process 2: name:__main__, pid:8716
run process 0: name:__main__, pid:6936
# Example 5-30. PP4E\System\Processes\multi2.py
"""
Use multiprocess anonymous pipes to communicate. Returns 2 connection
object representing ends of the pipe: objects are sent on one end and
received on the other, though pipes are bidrectional by default
"""
import os
from multiprocessing import Process, Pipe
def sender(pipe):
"""
send object to parent on anonymous pipe
"""
pipe.send(['spam'] + [42, 'eggs'])
pipe.close()
def talker(pipe):
"""
send and receive objects on a pipe
"""
pipe.send(dict(name='Bob', spam=42))
reply = pipe.recv()
print('talker got:', reply)
if __name__ == '__main__':
(parentEnd, childEnd) = Pipe()
Process(target=sender, args=(childEnd,)).start() # spawn child with pipe
print('parent got:', parentEnd.recv()) # receive from child
parentEnd.close() # or auto-closed on gc
(parentEnd, childEnd) = Pipe()
child = Process(target=talker, args=(childEnd,))
child.start()
print('parent got:', parentEnd.recv()) # receieve from child
parentEnd.send({x * 2 for x in 'spam'}) # send to child
child.join() # wait for child exit
print('parent exit')
C:\...\PP4E\System\Processes> multi2.py
parent got: ['spam', 42, 'eggs']
parent got: {'name': 'Bob', 'spam': 42}
talker got: {'ss', 'aa', 'pp', 'mm'}
parent exit
# Example 5-32. PP4E\System\Processes\multi4.py
"""
Process class can also be subclassed just like threading.Thread;
Queue works like queue.Queue but for cross-process, not cross-thread
"""
import os, time, queue
from multiprocessing import Process, Queue # process-safe shared queue
# queue is a pipe + locks/semas
class Counter(Process):
label = ' @'
def __init__(self, start, queue): # retain state for use in run
self.state = start
self.post = queue
Process.__init__(self)
def run(self): # run in newprocess on start()
for i in range(3):
time.sleep(1)
self.state += 1
print(self.label ,self.pid, self.state) # self.pid is this child's pid
self.post.put([self.pid, self.state]) # stdout file is shared by all
print(self.label, self.pid, '-')
if __name__ == '__main__':
print('start', os.getpid())
expected = 9
post = Queue()
p = Counter(0, post) # start 3 processes sharing queue
q = Counter(100, post) # children are producers
r = Counter(1000, post)
p.start(); q.start(); r.start()
while expected: # parent consumes data on queue
time.sleep(0.5) # this is essentially like a GUI,
try: # though GUIs often use threads
data = post.get(block=False)
except queue.Empty:
print('no data...')
else:
print('posted:', data)
expected -= 1
p.join(); q.join(); r.join() # must get before join putter
print('finish', os.getpid(), r.exitcode) # exitcode is child exit status
C:\...\PP4E\System\Processes> multi4.py
start 6296
no data...
no data...
@ 8008 101
posted: [8008, 101]
@ 6068 1
@ 3760 1001
posted: [6068, 1]
@ 8008 102
posted: [3760, 1001]
@ 6068 2
@ 3760 1002
posted: [8008, 102]
@ 8008 103
@ 8008 -
posted: [6068, 2]
@ 6068 3
@ 6068 -
@ 3760 1003
@ 3760 -
posted: [3760, 1002]
posted: [8008, 103]
posted: [6068, 3]
posted: [3760, 1003]
finish 6296 0
# Example 5-33. PP4E\System\Processes\multi5.py
"Use multiprocessing to start independent programs, os.fork or not"
import os
from multiprocessing import Process
def runprogram(arg):
os.execlp('python', 'python', 'child.py', str(arg))
if __name__ == '__main__':
for i in range(5):
Process(target=runprogram, args=(i,)).start()
print('parent exit')
Anonymous pipes
import os, time
def child(pipeout):
zzz = 0
while True:
time.sleep(zzz) # make parent wait
msg = ('Spam %03d' % zzz).encode() # pipes are binary bytes
os.write(pipeout, msg) # send to parent
zzz = (zzz+1) % 5 # goto 0 after 4
def parent():
pipein, pipeout = os.pipe() # make 2-ended pipe
if os.fork() == 0: # copy this process
child(pipeout) # in copy, run child
else: # in parent, listen to pipe
while True:
line = os.read(pipein, 32) # blocks until data sent
print('Parent %d got [%s] at %s' % (os.getpid(), line, time.time()))
parent()
Example: cross-linking streams
♦ Input → connect stdin to program’s stdout: raw_input
♦ Output → connect stdout to program’s stdin: print
♦ Also see: os.popen2 call in Python 2.X
file: ipc.py
import os
def spawn(prog, args):
pipe1 = os.pipe() # (parent input, child output)
pipe2 = os.pipe() # (child input, parent output)
pid = os.fork() # make a copy of this process
if pid:
# in parent process
os.close(pipe1[1]) # close child ends here
os.close(pipe2[0])
os.dup2(pipe1[0], 0) # sys.stdin = pipe1[0]
os.dup2(pipe2[1], 1) # sys.stdout = pipe2[1]
else:
# in child process
os.close(pipe1[0]) # close parent ends here
os.close(pipe2[1])
os.dup2(pipe2[0], 0) # sys.stdin = pipe2[0]
os.dup2(pipe1[1], 1) # sys.stdout = pipe1[1]
cmd = (prog,) + args
os.execv(prog, cmd) # overlay new program
Named pipes (fifos)
"""
named pipes; os.mkfifo is not available on Windows (without Cygwin);
there is no reason to fork here, since fifo file pipes are external
to processes--shared fds in parent/child processes are irrelevent;
"""
import os, time, sys
fifoname = '/tmp/pipefifo' # must open same name
def child():
pipeout = os.open(fifoname, os.O_WRONLY) # open fifo pipe file as fd
zzz = 0
while True:
time.sleep(zzz)
msg = ('Spam %03d\n' % zzz).encode() # binary as opened here
os.write(pipeout, msg)
zzz = (zzz+1) % 5
def parent():
pipein = open(fifoname, 'r') # open fifo as text file object
while True:
line = pipein.readline()[:-1] # blocks until data sent
print('Parent %d got "%s" at %s' % (os.getpid(), line, time.time()))
if __name__ == '__main__':
if not os.path.exists(fifoname):
os.mkfifo(fifoname) # create a named pipe file
if len(sys.argv) == 1:
parent() # run as parent if no args
else: # else run as child process
child()
[C:\...\PP4E\System\Processes] $ python pipefifo.py # parent window
Parent 8324 got "Spam 000" at 1268003696.07
Parent 8324 got "Spam 001" at 1268003697.06
Parent 8324 got "Spam 002" at 1268003699.07
Parent 8324 got "Spam 003" at 1268003702.08
Parent 8324 got "Spam 004" at 1268003706.09
Parent 8324 got "Spam 000" at 1268003706.09
Parent 8324 got "Spam 001" at 1268003707.11
...etc: Ctrl-C to exit...
[C:\...\PP4E\System\Processes]$ file /tmp/pipefifo # child window
/tmp/pipefifo: fifo (named pipe)
[C:\...\PP4E\System\Processes]$ python pipefifo.py -child
...Ctrl-C to exit...
Sockets (a first look: see Internet unit)
"""
sockets for cross-task communication: start threads to communicate over sockets;
independent programs can too, because sockets are system-wide, much like fifos;
see the GUI and Internet parts of the book for more realistic socket use cases;
some socket servers may also need to talk to clients in threads or processes;
sockets pass byte strings, but can be pickled objects or encoded Unicode text;
caveat: prints in threads may need to be synchronized if their output overlaps;
"""
from socket import socket, AF_INET, SOCK_STREAM # portable socket api
port = 50008 # port number identifies socket on machine
host = 'localhost' # server and client run on same local machine here
def server():
sock = socket(AF_INET, SOCK_STREAM) # ip addresses tcp connection
sock.bind(('', port)) # bind to port on this machine
sock.listen(5) # allow up to 5 pending clients
while True:
conn, addr = sock.accept() # wait for client to connect
data = conn.recv(1024) # read bytes data from this client
reply = 'server got: [%s]' % data # conn is a new connected socket
conn.send(reply.encode()) # send bytes reply back to client
def client(name):
sock = socket(AF_INET, SOCK_STREAM)
sock.connect((host, port)) # connect to a socket port
sock.send(name.encode()) # send bytes data to listener
reply = sock.recv(1024) # receive bytes data from listener
sock.close() # up to 1024 bytes in message
print('client got: [%s]' % reply)
if __name__ == '__main__':
from threading import Thread
sthread = Thread(target=server)
sthread.daemon = True # don't wait for server thread
sthread.start() # do wait for children to exit
for i in range(5):
Thread(target=client, args=('client%s' % i,)).start()
C:\...\PP4E\System\Processes> socket_preview.py
client got: [b"server got: [b'client1']"]
client got: [b"server got: [b'client3']"]
client got: [b"server got: [b'client4']"]
client got: [b"server got: [b'client2']"]
client got: [b"server got: [b'client0']"]
"""
same socket, but talk between independent programs too, not just threads;
server here runs in a process and serves both process and thread clients;
sockets are machine-global, much like fifos: don't require shared memory
"""
from socket_preview import server, client # both use same port number
import sys, os
from threading import Thread
mode = int(sys.argv[1])
if mode == 1: # run server in this process
server()
elif mode == 2: # run client in this process
client('client:process=%s' % os.getpid())
else: # run 5 client threads in process
for i in range(5):
Thread(target=client, args=('client:thread=%s' % i,)).start()
C:\...\PP4E\System\Processes> socket-preview-progs.py 1 # server window
C:\...\PP4E\System\Processes> socket-preview-progs.py 2 # client window
client got: [b"server got: [b'client:process=7384']"]
C:\...\PP4E\System\Processes> socket-preview-progs.py 2
client got: [b"server got: [b'client:process=7604']"]
C:\...\PP4E\System\Processes> socket-preview-progs.py 3
client got: [b"server got: [b'client:thread=1']"]
client got: [b"server got: [b'client:thread=2']"]
client got: [b"server got: [b'client:thread=0']"]
client got: [b"server got: [b'client:thread=3']"]
client got: [b"server got: [b'client:thread=4']"]
C:\..\PP4E\System\Processes> socket-preview-progs.py 3
client got: [b"server got: [b'client:thread=3']"]
client got: [b"server got: [b'client:thread=1']"]
client got: [b"server got: [b'client:thread=2']"]
client got: [b"server got: [b'client:thread=4']"]
client got: [b"server got: [b'client:thread=0']"]
C:\...\PP4E\System\Processes> socket-preview-progs.py 2
client got: [b"server got: [b'client:process=6428']"]
Signals
"""
catch signals in Python; pass signal number N as a command-line arg,
use a "kill -N pid" shell command to send this process a signal; most
signal handlers restored by Python after caught (see network scripting
chapter for SIGCHLD details); on Windows, signal module is available,
but it defines only a few signal types there, and os.kill is missing;
"""
import sys, signal, time
def now(): return time.asctime() # current time string
def onSignal(signum, stackframe): # python signal handler
print('Got signal', signum, 'at', now()) # most handlers stay in effect
signum = int(sys.argv[1])
signal.signal(signum, onSignal) # install signal handler
while True: signal.pause() # wait for signals (or: pass)
♦ For starting programs on Windows
♦ Spawnv like fork+exec for Unix
♦ See also: os.system(“start file.py”)
############################################################
# do something simlar by forking process
instead of threads
# this doesn't currently work on Windows,
because it has no
# os.fork call; use os.spawnv to start
programs on Windows
# instead; spawnv is roughly like a
fork+exec combination;
############################################################
import os, sys
for i in range(10):
if sys.platform[:3] == 'win':
path = r'C:\program files\python\python.exe'
os.spawnv(os.P_DETACH, path,
('python',
'thread-basics6.py'))
else:
pid = os.fork()
if pid != 0:
print 'Process %d spawned' % pid
else:
os.execlp('python', 'python', 'thread-basics6.py')
print 'Main process exiting.'
site-forward.py
#######################################################
# Create forward link pages for relocating
a web site.
# Generates one page for every existing
site file;
# upload the generated files to your old
web site.
#######################################################
import os, string
uploaddir = 'rmi-forward' # where to store forward files
servername = 'starship.python.net' # where site is relocating to
homedir = '~lutz/home' # where site will be rooted
sitefilesdir = 'public_html' # where site files live locally
templatename = 'template.html' # template for generated pages
template
= open(templatename).read()
sitefiles = os.listdir(sitefilesdir) # filenames, no dir prefix
count = 0
for filename in sitefiles:
fwdname = os.path.join(uploaddir, filename) # or +os.sep+filename
print 'creating', filename, 'as', fwdname
filetext = string.replace(template, '$server$', servername)
filetext = string.replace(filetext, '$home$', homedir)
filetext = string.replace(filetext, '$file$', filename)
open(fwdname, 'w').write(filetext)
count = count + 1
print 'Last file =>\n', filetext
print 'Done:', count, 'forward files
created.'
template.html
<HTML><BODY>
<H1>This page has moved</H1>
<P>This page now lives at this address:
<P><A
HREF="http://$server$/$home$/$file$">
http://$server$/$home$/$file$</A>
<P>Please click on the new address to jump to
this page, and
update any links accordingly.
</P>
</BODY></HTML>
♦ pack1 puts files in a single file, with separator lines
♦ unpack1 recreates original files from a ‘pack1’ file
♦ unmore unpacks result of a “more <files>” command
file: pack1.py
#!/usr/local/bin/python
import sys # load the system module
marker = ':'*6
for name in sys.argv[1:]: # for all command arguments
input = open(name, 'r') # open the next input file
print marker + name # write a separator line
print input.read(), # write the file's contents
file: unpack1.py
#!/usr/local/bin/python
import sys
marker = ':'*6
for line in sys.stdin(): # for all input lines
if line[:6] != marker:
print line, # write real lines
else:
sys.stdout = open(line[6:-1], 'w')
Unmore: scripts, functions, classes
► ‘more’ writes 3 lines before each file
::::::::::::::
filename
::::::::::::::
<file text>
file unmore.py
#!/usr/local/bin/python
# unpack result of "more x y z > f"
# usage: "% unmore.py f" or "% unmore.py < f"
# uses simple top-level script logic
import sys
marker = ':'*14
try:
input = open(sys.argv[1], "r")
except:
input = sys.stdin
output = sys.stdout
while 1:
line = input.readline()
if not line: # end of file?
break
elif line[:14] != marker: # text line?
output.write(line)
else: # file prefix
fname = input.readline()[:-1] # strip eoln ('\n')
print 'creating', `fname`
output = open(fname, "w") # next output
line = input.readline() # end of prefix
if line[:14] != marker:
print "OOPS!"; sys.exit(1)
print 'Done.'
Adding a functional interface
file: unmore2.py
#!/usr/local/bin/python
# unpack result of "more x y z > f"
# usage: "unmore2.py f" or "unmore2.py < f"
# packages unpacking logic as an importable function
import sys
marker = ':'*14
def unmore(input):
output = sys.stdout
while 1:
line = input.readline()
if not line: # end of file?
break
elif line[:14] != marker: # text line?
output.write(line)
else: # file prefix
fname = input.readline()[:-1] # strip eoln
print 'creating', `fname`
output = open(fname, "w") # next output
if input.readline()[:14] != marker:
print "OOPS!"; sys.exit(1)
if __name__ == '__main__':
if len(sys.argv) == 1:
unmore(sys.stdin) # unmore2.py < f
else:
unmore(open(sys.argv[1], 'r')) # unmore2.py f
print 'Done.'
% unmore2.py t.more
creating 't1.txt'
creating 't2.txt'
Done.
% python
>>> from unmore2 import unmore
>>> unmore(open("t.more", "r"))
creating 't1.txt'
creating 't2.txt'
These are suggested reading, if you are looking for something to do during the lab session.
File: fixeoln_one.py
#########################################################
# Use:
"python fixeoln_one.py [tounix|todos] filename".
#
convert end-lines in the single text file whose name
# is
passed in on the command line, to the target form
# (unix
or dos). The _one, _dir, and _all
converters
#
resuse the convert function here; we could implement
# this
by inspecting command-line argument patterns
#
instead of writing 3 separate scripts, but that can
#
become complex for the user.
convertEndlines changes
#
endlines only if necessary--lines that are already in
# the
target format are left unchanged, so it's okay to
#
convert a file > once in any of the 3 fixeoln scripts.
#########################################################
def
convertEndlines(format, fname): # convert one file
newlines = [] #
todos: \n => \r\n
for line in open(fname,
'r').readlines(): # tounix:
\r\n => \n
if format == 'todos':
if line[-1:] == '\n' and
line[-2:-1] != '\r':
line = line[:-1] + '\r\n'
elif format == 'tounix': # avoids IndexError
if line[-2:] == '\r\n': # slices are scaled
line = line[:-2] + '\n'
newlines.append(line)
open(fname, 'w').writelines(newlines)
if
__name__ == '__main__':
import sys
errmsg = 'Required arguments missing:
["todos"|"tounix"] filename'
assert (len(sys.argv) == 3 and sys.argv[1]
in ['todos', 'tounix']), errmsg
convertEndlines(sys.argv[1], sys.argv[2])
print 'Converted', sys.argv[2]
File: fixeoln_dir.py
#########################################################
# Use:
"python fixeoln_dir.py [tounix|todos] patterns?".
#
convert end-lines in all the text files in the current
#
directory (only: does not recurse to subdirectories).
#
Resuses converter in the single-file _one version.
#########################################################
import
sys, glob
from
fixeoln_one import convertEndlines
listonly
= 0
patts =
['*.py', '*.txt', '*.c', '*.cxx', '*.h', '*.i', 'makefile*', 'output*']
if
__name__ == '__main__':
errmsg = 'Required first argument missing:
"todos" or "tounix"'
assert (len(sys.argv) >= 2 and
sys.argv[1] in ['todos', 'tounix']), errmsg
if len(sys.argv) > 2: # glob anyhow: '*' not applied
on dos
patts = sys.argv[2:] # though not really needed on
linux
filelists = map(glob.glob, patts) # name matches in this dir only
count = 0
for list in filelists:
for fname in list:
print count+1, '=>', fname
if not listonly:
convertEndlines(sys.argv[1],
fname)
count = count + 1
print 'Converted %d files' % count
File: fixeoln_all.py
#########################################################
# Use:
"python fixeoln_all.py [tounix|todos] patterns?".
# find
and convert end-of-lines in all text files
# at
and below the directory where this script is
# run
(the dir you are in when you type 'python').
# If
needed, tries to use the Python find.py lib
#
module, else reads the output of a unix-style
# find
executable command; we could also use a
# find
-exec option to spawn a coverter script.
# Uses
default filename patterns list if absent.
#
#
Example:
# cd Html\Examples
# python ..\..\Tools\fixeoln_all.py tounix
#
#
converts any DOS end-lines to UNIX end-lines, in
# all
text files in and below the Examples directory
#
(i.e., all source-code files). Replace
"tounix" with
#
"todos" to convert any UNIX end-lines to DOS form
#
instead. This script only changes files
that need to
# be
changed, so it's safe to run brute-force from a
#
root-level directory to force platform conformance.
#
"python ..\..\Tools\fixeoln_all.py tounix *.txt"
#
converts just .txt files (quote on UNIX: "*.txt").
# See
also: fixeoln_one and fixeoln_dir versions.
#########################################################
import
os, sys, string
debug = 0
pyfind = 0
# force py find
listonly
= 0
def
findFiles(patts, debug=debug, pyfind=pyfind):
try:
if sys.platform[:3] == 'win' or pyfind:
print 'Using Python find'
import find # use python
lib find.py
matches = map(find.find,
patts) # start dir default =
'.'
else:
print 'Using find executable'
matches = []
for patt in patts:
findcmd = 'find . -name
"%s" -print' % patt # run find
command
lines =
os.popen(findcmd).readlines() #
remove endlines
matches.append(map(string.strip,
lines)) # lambda x: x[:-1]
except:
assert 0, 'Sorry - cannot find files'
if debug: print matches
return matches
if
__name__ == '__main__':
from fixeoln_dir import patts
from fixeoln_one import convertEndlines
errmsg = 'Required first argument missing:
"todos" or "tounix"'
assert (len(sys.argv) >= 2 and
sys.argv[1] in ['todos', 'tounix']), errmsg
if len(sys.argv) > 2: # quote in unix shell
patts = sys.argv[2:] # else tries to expand
matches = findFiles(patts)
count = 0
for matchlist in matches: # a list of lists
for fname in matchlist: # one per pattern
print count+1, '=>', fname
if not listonly:
convertEndlines(sys.argv[1],
fname)
count = count + 1
print 'Converted %d files' % count
Click here to go to
lab exercises
Click here to go to
exercise solutions
Click here to go to
solution source files
Click here to go to
lecture example files