Python Subprocess


The subprocess module enables you to start new applications from your Python program. How cool is that?

Related course:
Complete Python Masterclass

Start a process in Python:

You can start a process in Python using the Popen function call. The program below starts the unix program ‘cat’ and the second parameter is the argument. This is equivalent to ‘cat test.py’.  You can start any program with any parameter.

#!/usr/bin/env python 
 
from subprocess import Popen, PIPE
 
process = Popen(['cat', 'test.py'], stdout=PIPE, stderr=PIPE)
stdout, stderr = process.communicate()
print stdout

The process.communicate() call reads input and output from the process.  stdout is the process output. stderr will be written only if an error occurs.  If you want to wait for the program to finish you can call Popen.wait().

Subprocess call():

Subprocess has a method call() which can be used to start a program. The parameter is a list of which the first argument must be the program name. The full definition is:

subprocess.call(args, *, stdin=None, stdout=None, stderr=None, shell=False)
# Run the command described by args. 
# Wait for command to complete, then return the returncode attribute.

In the example below the full command would be “ls -l”

#!/usr/bin/env python 
 
import subprocess
subprocess.call(["ls", "-l"])

Save process output (stdout)

We can get the output of a program and store it in a string directly using check_output. The method is defined as:

 subprocess.check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False)
 # Run command with arguments and return its output as a byte string.

Example usage:

#!/usr/bin/env python 
import subprocess
 
s = subprocess.check_output(["echo", "Hello World!"])
print("s = " + s)
Back
Next

14 thoughts on “Python Subprocess

  1. Frank
    KEDARNATH GRANDHE - March 8, 2018

    Hi Frank,

    I want to use the subprocess module to perform ssh connection to a linux server (using username and password) and I would like to navigate to a specific path and access (perform various operations on) the data/file in that path.

    Can you please help me with some info on how I can do this?

    1. Frank
      Frank - March 12, 2018

      You can send shell commands directly to the ssh process like so:

      import subprocess
      import sys
       
      HOST="127.0.0.1"
      ssh = subprocess.Popen(["ssh",
                              "%s" % HOST],
                              stdin=subprocess.PIPE,
                              stdout=subprocess.PIPE,
                              stderr=subprocess.PIPE,
                              universal_newlines=True,
                              bufsize=0)
       
      # send ssh commands to stdin
      ssh.stdin.write("ls .\n")
      ssh.stdin.write("uname -a\n")
      ssh.stdin.write("uptime\n")
      ssh.stdin.close()
       
      # fetch output
      for line in ssh.stdout:
          print(line),

      If you want to keep a ssh session open for a while, you could use the pxssh module.

  2. Frank
    Lukasz Szczepaniak - October 30, 2017

    Hi,

    I am struggle with provide a pipe ‘|’ between command execution. Let’s say I try to make “ifconfig ixgbe | grep ether”. How to provide pipe to the code? When I try something like this:

    from subprocess import Popen, PIPE
    ...
            if interface==ixgbe0:
                interface = Popen(['ifconfig', 'ixgbe0'], stdout=PIPE, stderr=PIPE)
                stdout, stderr = interface.communicate()

    It works like ‘ifconfig ixgbe’ but when I add pipe, like this:

            if interface==ixgbe0:
                interface = Popen(['ifconfig', 'ixgbe0',  '|', 'grep', 'ether], stdout=PIPE, stderr=PIPE)
                stdout, stderr = interface.communicate()

    It does not show any error, but there is no output for ether.

    1. Frank
      Frank - November 4, 2017

      I could use the pipe operator with ifconfig using this way (I used wlp7s0 instead of ixgbe0):

      from subprocess import check_output
       
      out = check_output('ifconfig wlp7s0 | grep ether', shell=True)
      print(out)

      You can also do it this way:

      from subprocess import PIPE,Popen
       
      p1 = Popen(["ifconfig", "wlp7s0"],stdout=PIPE)
      p2 = Popen(["grep", "ether"],stdin=p1.stdout,stdout=PIPE,stderr=PIPE)
      p1.stdout.close()
      out, err = p2.communicate()
      print(out)

      We link p1 and p2.
      (Be sure to change the device to ixgbe0).

  3. Frank
    M.d Rehman - July 10, 2017

    Hi frank. I wanted to know if i could store many values using (check_output). I want to use ping operation in cmd as subprocess and store the ping statistics in the variable to use them.

    1. Frank
      Frank - July 11, 2017

      Hi Rehman, you can store the entire output like this:

      #!/usr/bin/env python
      import subprocess
       
      s = subprocess.check_output(["ping", "-c 1", "google.com"])
      print("s = " + str(s))

      Python 3 returns this as bytes, so we convert it to string with the function decode.
      Then we split on the newline character:

      #!/usr/bin/env python
      import subprocess
       
      s = subprocess.check_output(["ping", "-c 4", "google.com"])
      output = s.decode("utf-8")
       
      lines = output.split('\n')
       
      for line in lines:
          print(line)

      Now we have a few strings. We can parse the line or lines we want. We’ll split that string into parts.
      I’ll take the statistics summary.

      #!/usr/bin/env python
      import subprocess
       
      s = subprocess.check_output(["ping", "-c 4", "google.com"])
      output = s.decode("utf-8")
       
      lines = output.split('\n')
       
      statistics = (lines[7])
      values = statistics.split(", ")
      for v in values:
          print(v)

      All variables are stored in the list values. You can then store them in variables if you want. v[0] contains the first variable of the line.

  4. Frank
    Hina - January 11, 2016

    Hi Frank,
    i have found your website very useful as i am python learner. please if you could guide me the way to read pdf file in python. Secondly, i want tutorials about natural language processing in python. can you help in this regard.
    Many Thanks

    1. Frank
      Frank - January 28, 2016

      Hi Hina,
      Python has no standard method to read pdf. To read pdf you need to use a module. You can find related projects here: https://github.com/search?l=Python&q=pdf&type=Repositories&utf8=%E2%9C%93
      Alternatively you can convert the pdf document to a txt method, using the Pdftotext tool: https://en.wikipedia.org/wiki/Pdftotext

  5. Frank
    Benben - September 1, 2015

    Hi,Frank

    Could tell me why?

    I’ve copy it and run it . it show me:

    from subprocess import Popen, PIPE
     
    process = Popen(['cat', 'test.py'], stdout=PIPE, stderr=PIPE)
    stdout, stderr = process.communicate()
    print stdout
     
    Traceback (most recent call last):
      File "C:\Python27\learn\subprocess.py", line 5, in 
        process = Popen(['cat', 'test.py'], stdout=PIPE, stderr=PIPE)
      File "C:\Python27\lib\subprocess.py", line 710, in __init__
        errread, errwrite)
      File "C:\Python27\lib\subprocess.py", line 958, in _execute_child
     
     
       startupinfo)
    WindowsError: [Error 2]
    1. Frank
      Frank - September 1, 2015

      Hi,
      The first parameter of Popen() is ‘cat’, this is a unix program.
      I’m not sure if this program is available on windows, try changing it to notepad.exe

      1. Frank
        Francisco Javier Martin - May 30, 2017

        If you start notepad.exe as a windowed app then python will not get the output.
        The MSDOS command similar to “cat” is “type”.

        process = Popen([‘type’, ‘test.py’], stdout=PIPE, stderr=PIPE)

        1. Frank
          Frank - May 31, 2017

          Thanks Francisco!

  6. Frank
    Kheenan - July 16, 2015

    I just want to say These are the best tutorials of any anywhere. I use tutorials all the time and these are just so concise and effective. Thanks a lot and keep at it!

    1. Frank
      Frank - July 16, 2015

      Thanks Kheenan!

Leave a Reply