subprocess

Resources

run

When scripting a task it is common to need to run a external process, for instance a program to do a particular data analysis. This external process will be a subprocess launched by our Python running process. The subprocess module includes the run function to run external processes.

Let’s image that we want to run the ls command (dir in windows).

from subprocess import run

cmd = ['ls']
run(cmd)
comprehensions.qmd
counter.html
counter.qmd
enumerate.qmd
everyday_python.html
everyday_python.qmd
lambda.qmd
paths.qmd
range.qmd
shutil.qmd
subprocess.qmd
subprocess.quarto_ipynb
typing.qmd
zip.qmd
CompletedProcess(args=['ls'], returncode=0)

Run, by default, expects a list of strings, not a string with the command. For instance, image that our command includes a parameter.

from subprocess import run

cmd = ['ls', '-l']
run(cmd)
total 352
-rw-r--r--  1 jose  staff   4361 13 sept. 09:52 comprehensions.qmd
-rw-r--r--@ 1 jose  staff  72152 14 sept. 09:07 counter.html
-rw-r--r--  1 jose  staff   3719 13 sept. 09:52 counter.qmd
-rw-r--r--  1 jose  staff   1786 13 sept. 09:52 enumerate.qmd
-rw-r--r--@ 1 jose  staff  43219 14 sept. 09:07 everyday_python.html
-rw-r--r--  1 jose  staff    640 13 sept. 09:52 everyday_python.qmd
-rw-r--r--  1 jose  staff   2191 13 sept. 09:52 lambda.qmd
-rw-r--r--  1 jose  staff   4601 13 sept. 16:06 paths.qmd
-rw-r--r--  1 jose  staff   1657 13 sept. 09:52 range.qmd
-rw-r--r--  1 jose  staff   2885 13 sept. 09:52 shutil.qmd
-rw-r--r--  1 jose  staff   3762 13 sept. 09:52 subprocess.qmd
-rw-r--r--@ 1 jose  staff   6899 14 sept. 09:07 subprocess.quarto_ipynb
-rw-r--r--  1 jose  staff   2450 13 sept. 09:52 typing.qmd
-rw-r--r--  1 jose  staff   1356 13 sept. 09:52 zip.qmd
CompletedProcess(args=['ls', '-l'], returncode=0)

In any case the run function will launch the external process and will also wait for the process to finish, and only then will the function return a CompletedProcess object.

from subprocess import run

cmd = ['ls', '-l']
process = run(cmd)
print(process.returncode)
total 352
-rw-r--r--  1 jose  staff   4361 13 sept. 09:52 comprehensions.qmd
-rw-r--r--@ 1 jose  staff  72152 14 sept. 09:07 counter.html
-rw-r--r--  1 jose  staff   3719 13 sept. 09:52 counter.qmd
-rw-r--r--  1 jose  staff   1786 13 sept. 09:52 enumerate.qmd
-rw-r--r--@ 1 jose  staff  43219 14 sept. 09:07 everyday_python.html
-rw-r--r--  1 jose  staff    640 13 sept. 09:52 everyday_python.qmd
-rw-r--r--  1 jose  staff   2191 13 sept. 09:52 lambda.qmd
-rw-r--r--  1 jose  staff   4601 13 sept. 16:06 paths.qmd
-rw-r--r--  1 jose  staff   1657 13 sept. 09:52 range.qmd
-rw-r--r--  1 jose  staff   2885 13 sept. 09:52 shutil.qmd
-rw-r--r--  1 jose  staff   3762 13 sept. 09:52 subprocess.qmd
-rw-r--r--@ 1 jose  staff   6899 14 sept. 09:07 subprocess.quarto_ipynb
-rw-r--r--  1 jose  staff   2450 13 sept. 09:52 typing.qmd
-rw-r--r--  1 jose  staff   1356 13 sept. 09:52 zip.qmd
0

Launching without waiting

return code

Every process once is finished return a return code or exit status. This return code is an integer and the standard is to return 0 when everything has been fine or any other number in the event of an error happening in the subprocess. You can access to the exit code of the subprocess.

If you want the run function to fail in the event of the called process having any problem you could use the check argument.

stdout and stdin

You can store the result of stdout and stderr as properties of the completed process object.

from subprocess import run

cmd = ['ls', '/hello']
process = run(cmd, capture_output=True)
print(process.stdout)
print(process.stderr)
b''
b'ls: /hello: No such file or directory\n'

Be aware that, by default, the standard output streams will be binary objects, if you want them to be strings you have to provide an encoding.

from subprocess import run

cmd = ['ls', '/hello']
process = run(cmd, capture_output=True, encoding='utf-8')
print(process.stdout)
print(process.stderr)

ls: /hello: No such file or directory

Popen

The run function will wait for the subprocess to finnish before returning. If you just want to launch the process, but not wait for it to finish you can use the Popen class.

The use of Popen is very similar to the use of run, the main difference being that Popen will return a Popen object immediately, without waiting for it to finish.

Once you have that object, you could check if the process has already finished or you could also wait for the process to finish.

from subprocess import Popen

cmd = ['ls']
process = Popen(cmd)
print(process.poll())
print(process.wait())
print(process.returncode)
None
comprehensions.qmd
counter.html
counter.qmd
enumerate.qmd
everyday_python.html
everyday_python.qmd
lambda.qmd
paths.qmd
range.qmd
shutil.qmd
subprocess.qmd
subprocess.quarto_ipynb
typing.qmd
zip.qmd
0
0