5 Best Ways to Use the Subprocess Module in Python

by

💡 Problem Formulation: Python developers often need to interact with the system shell or external commands and processes. For example, you might need to ping a server and process the result within a Python script. The subprocess module in Python is designed to facilitate these tasks, providing a powerful interface for spawning new processes, connecting to their input/output/error pipes, and obtaining their return codes.

Method 1: Using run() to Execute Commands

The subprocess.run() function is a versatile method for running external commands. It blocks until the command completes and returns a CompletedProcess instance. This function is recommended for new code as it is more straightforward than older alternatives like os.system().

Here’s an example:

import subprocess

completed = subprocess.run(["echo", "Hello from subprocess"], capture_output=True, text=True)
print(completed.stdout)

Output:

Hello from subprocess

This code runs the echo command and captures its output. We use the capture_output=True argument to capture stdout and stderr, and text=True to work with string data instead of bytes. The variable completed holds a CompletedProcess object with information about the executed process.

Method 2: Managing Processes with Popen()

The subprocess.Popen() class is used for more complex process management. Unlike run(), Popen() does not wait for the process to finish; it provides a way to start a process and continue with other tasks. You can later synchronize with the process using methods like wait().

Here’s an example:

import subprocess

process = subprocess.Popen(["sleep", "2"])
process.wait()
print("Process finished")

Output:

Process finished

The code above starts a process that simply sleeps for 2 seconds. The call to process.wait() blocks until the process completes. After the sleep period, the message “Process finished” is printed to the console.

Method 3: Capturing Standard Output and Error Streams

To capture standard output and error streams from a subprocess, subprocess.Popen() can be used with the stdout and stderr parameters. This method is essential for capturing command output and errors for logging or processing purposes.

Here’s an example:

import subprocess

with subprocess.Popen(["ls", "-l", "/nonexistentdirectory"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) as proc:
    stdout, stderr = proc.communicate()
print("STDOUT:", stdout.decode())
print("STDERR:", stderr.decode())

Output:

STDOUT: 
STDERR: ls: cannot access '/nonexistentdirectory': No such file or directory

This snippet attempts to list the contents of a nonexistent directory. By setting stdout and stderr to subprocess.PIPE, it captures the output and error streams. The .communicate() method reads both streams, and the decode method converts bytes to strings for printing.

Method 4: Running Shell Commands

Sometimes it’s necessary to run commands through the shell interpreter, especially for commands that utilize shell features like globbing or pipelines. For this, set the shell parameter to True within methods like run() or Popen().

Here’s an example:

import subprocess

result = subprocess.run('echo $HOME', shell=True, capture_output=True, text=True)
print(result.stdout.strip())

Output:

/Users/yourusername

By running the command with shell=True, environment variables such as $HOME can be interpreted correctly by the shell. The output is captured and printed as a string without newline characters due to the .strip() function.

Bonus One-Liner Method 5: Using check_output() for Quick Output Capture

The subprocess.check_output() function is a convenient one-liner to quickly capture the output of a command. It’s handy for simple use-cases when you only need the command’s output and are okay with an exception being thrown on errors.

Here’s an example:

import subprocess

output = subprocess.check_output(["echo", "Quick output capture"], text=True)
print(output.strip())

Output:

Quick output capture

This line of code captures the output of the echo command using check_output(). A CalledProcessError exception will be raised if the command returns a non-zero exit status. The text=True parameter ensures the output is a string, not bytes.

Summary/Discussion

  • Method 1: run(). The preferred way in modern Python to run simple commands. Blocks until completion. No direct process control.
  • Method 2: Popen(). For advanced process management. Non-blocking with asynchronous capabilities. Slightly more complex usage.
  • Method 3: Capturing Streams. Essential for obtaining command output and errors for further processing. Requires handling of the streams and subprocess lifecycle.
  • Method 4: Shell Commands. Convenient for shell-specific features. Be cautious of shell injection vulnerabilities when using user input.
  • Bonus Method 5: check_output(). Quick and easy output capture for simple scenarios. No control over process and defaults to raising exceptions on errors.