We're back after a server migration that caused effbot.org to fall over a bit harder than expected. Expect some glitches.

How do I run a subprocess with pipes connected to both input and output?

Use the subprocess or popen2 modules. An example:

import popen2
fromchild, tochild = popen2.popen2("command")
tochild.write("input\n")
tochild.flush()
output = fromchild.readline()

Warning: in general it is unwise to do this because you can easily cause a deadlock where your process is blocked waiting for output from the child while the child is blocked waiting for input from you. This can be caused because the parent expects the child to output more text than it does, or it can be caused by data being stuck in stdio buffers due to lack of flushing. The Python parent can of course explicitly flush the data it sends to the child before it reads any output, but if the child is a naive C program it may have been written to never explicitly flush its output, even if it is interactive, since flushing is normally automatic.

Note that a deadlock is also possible if you use popen3 to read stdout and stderr. If one of the two is too large for the internal buffer (increasing the buffer size does not help) and you read() the other one first, there is a deadlock, too.

Note on a bug in popen2: unless your program calls wait() or waitpid(), finished child processes are never removed, and eventually calls to popen2 will fail because of a limit on the number of child processes. Calling os.waitpid with the os.WNOHANG option can prevent this; a good place to insert such a call would be before calling popen2 again.

In many cases, all you really need is to run some data through a command and get the result back. Unless the amount of data is very large, the easiest way to do this is to write it to a temporary file and run the command with that temporary file as input. The standard tempfile exports a mktemp() function to generate unique temporary file names.

import tempfile
import os
class Popen3:
   """
   This is a deadlock-safe version of popen that returns
   an object with errorlevel, out (a string) and err (a string).
   (capturestderr may not work under windows.)
   Example: print Popen3('grep spam','\n\nhere spam\n\n').out
   """
   def __init__(self,command,input=None,capturestderr=None):
       outfile=tempfile.mktemp()
       command="( %s ) > %s" % (command,outfile)
       if input:
           infile=tempfile.mktemp()
           open(infile,"w").write(input)
           command=command+" <"+infile
       if capturestderr:
           errfile=tempfile.mktemp()
           command=command+" 2>"+errfile
       self.errorlevel=os.system(command) >> 8
       self.out=open(outfile,"r").read()
       os.remove(outfile)
       if input:
           os.remove(infile)
       if capturestderr:
           self.err=open(errfile,"r").read()
           os.remove(errfile)

Note that many interactive programs (e.g. vi) don’t work well with pipes substituted for standard input and output. You will have to use pseudo ttys (“ptys”) instead of pipes. Or you can use a Python interface to Don Libes’ “expect” library. A Python extension that interfaces to expect is called “expy” and available from http://expectpy.sourceforge.net. A pure Python solution that works like expect is pexpect http://pexpect.sourceforge.net.

CATEGORY: library

CATEGORY: cleanup