Shell Command¶
The standard libary’s subprocess
modules provides a convenient way to
invoke arbitrary subprocesses. However, it is currently written primarily
from an application developer’s point of view: it views invocation of the
system shell as something risky that leaves you open to shell injection
attacks, rather than the normal, unexceptional operation it is when using
Python to automate system administration tasks.
This module aims to take over where subprocess
leaves off, providing
convenient, low-level access to the system shell, that automatically handles
filenames and paths containing whitespace, as well as protecting naive code
from shell injection vulnerabilities.
Significantly, it allows system administrators to divide responsibility appropriately, using Python for its superior data structures and flow control syntax, while using the underlying system shell normally for command invocation and pipeline manipulation.
A couple of basic examples:
>>> from shell_command import shell_call
>>> shell_call("ls *.py")
setup.py shell_command.py test_shell_command.py
0
>>> shell_call("ls -l *.py")
-rw-r--r-- 1 ncoghlan ncoghlan 391 2011-12-11 12:07 setup.py
-rw-r--r-- 1 ncoghlan ncoghlan 7855 2011-12-11 16:16 shell_command.py
-rwxr-xr-x 1 ncoghlan ncoghlan 8463 2011-12-11 16:17 test_shell_command.py
0
Now with some string interpolation (protected from shell injection attacks by
default - use !u
to override):
>>> from shell_command import shell_call
>>> shell_call("ls {}", "*.py")
ls: cannot access *.py: No such file or directory
2
>>> shell_call("ls {!u}", "*.py")
setup.py shell_command.py test_shell_command.py
0
And a slightly more complex example:
>>> from shell_command import shell_output, iter_shell_output
>>> print(shell_output("ls -l *.py"))
-rw-r--r-- 1 ncoghlan ncoghlan 391 2011-12-11 12:07 setup.py
-rw-r--r-- 1 ncoghlan ncoghlan 7855 2011-12-11 16:16 shell_command.py
-rwxr-xr-x 1 ncoghlan ncoghlan 8463 2011-12-11 16:17 test_shell_command.py
>>> for line in iter_shell_output("ls -l *.py | tee {}", "example file.txt"):
... print(line)
...
-rw-r--r-- 1 ncoghlan ncoghlan 391 2011-12-11 12:07 setup.py
-rw-r--r-- 1 ncoghlan ncoghlan 7855 2011-12-11 16:16 shell_command.py
-rwxr-xr-x 1 ncoghlan ncoghlan 8463 2011-12-11 16:17 test_shell_command.py
>>> print(open("example file.txt").read())
-rw-r--r-- 1 ncoghlan ncoghlan 391 2011-12-11 12:07 setup.py
-rw-r--r-- 1 ncoghlan ncoghlan 7855 2011-12-11 16:16 shell_command.py
-rwxr-xr-x 1 ncoghlan ncoghlan 8463 2011-12-11 16:17 test_shell_command.py
String Interpolation¶
This module uses a custom string interpolation mechanism based on
str.format()
. By default, all interpolated arguments are coerced to
strings and quoted to escape any whitespace and shell metacharacters. This
quoting can be bypassed with the !u
conversion specifier as shown in the
examples above.
Convenience API¶
The module level convenience API consists of four functions:
-
shell_call
(*args, **kwds)[source]¶ A subprocess.call() variant for shell command invocation.
args and kwds are both passed though to the string formatting call. Refer to String Interpolation for details of the implicit quoting behaviour.
-
check_shell_call
(*args, **kwds)[source]¶ A subprocess.check_call() variant for shell command invocation.
args and kwds are both passed though to the string formatting call. Refer to String Interpolation for details of the implicit quoting behaviour.
-
shell_output
(*args, **kwds)[source]¶ A subprocess.check_output() variant for shell command invocation.
args and kwds are both passed though to the string formatting call. Refer to String Interpolation for details of the implicit quoting behaviour.
For a successful call, a trailing newline (if any) will be removed from the result.
Use shell redirection (2>&1) to capture stderr in addition to stdout.
As with subprocess.check_output(), this returns encoded bytes by default in Python 3. Passing
universal_newlines=True
in the constructor will also automatically decode the output to text with the UTF-8 codec. Alternatively, the result may be explicitly decoded after the call.
-
iter_shell_output
(*args, **kwds)[source]¶ An alternative to shell_output() that yields output data as it becomes available.
args and kwds are both passed though to the string formatting call. Refer to String Interpolation for details of the implicit quoting behaviour.
Since lines are made available as they are produced, the final line will still contain its terminating newline (if any).
This operation relies on the use of
select.select()
on subrocess pipes, and hence is known to fail on Windows.
ShellCommand¶
The ShellCommand
class implements the actual functionality of the
module. By creating instances of this class directly, it is possible to
override the default arguments to subprocess.Popen
used by the
convenience functions.
-
class
ShellCommand
(command, **subprocess_kwds)[source]¶ ShellCommand accepts a command string and Popen constructor arguments.
When initialised with an existing ShellCommand object, a new copy is made with the original Popen arguments updated with any new arguments.
Method arguments are interpolated into the command string using str.format() style processing. All method arguments are coerced to strings and escaped using shlex.quote() by default, use the custom conversion specifier ”!u” (for “unquoted”) or any of the standard conversion specifiers (such as ”!s”) to bypass this quoting process.
As brace characters (‘{‘ and ‘}’) in the command string are used to indicate interpolated fields, they must either be included in an interpolated value or else doubled (i.e. ‘{{‘ and ‘}}’) in the format string in order to be passed to the underlying shell.
The “shell” argument to Popen is enabled be default, but this can be overridden by explicitly setting it to False. In Python 3, the “universal_newlines” option is also enabled by default.
-
check_shell_call
(*args, **kwds)[source]¶ A subprocess.check_call() variant for shell command invocation.
Refer to ShellCommand for details of the implicit quoting behaviour.
-
format
(*args, **kwds)[source]¶ A str.format() variant for shell command interpolation.
Refer to ShellCommand for details of the implicit quoting behaviour.
-
format_map
(mapping)[source]¶ A str.format_map() variant for shell command interpolation.
Refer to ShellCommand for details of the implicit quoting behaviour.
-
iter_shell_output
(*args, **kwds)[source]¶ An alternative to shell_output() that yields output data as it becomes available.
Since lines are made available as they are produced, the final line will still contain its terminating newline (if any).
This operation relies on the use of select.select() on subrocess pipes, and hence is known to fail on Windows.
-
shell_call
(*args, **kwds)[source]¶ A subprocess.call() variant for shell command invocation.
Refer to ShellCommand for details of the implicit quoting behaviour.
-
shell_output
(*args, **kwds)[source]¶ A subprocess.check_output() variant for shell command invocation.
Refer to ShellCommand for details of the implicit quoting behaviour.
Use shell redirection (2>&1) to capture stderr in addition to stdout A trailing newline (if any) will be removed from the result
As with subprocess.check_output(), this returns encoded bytes by default in Python 3. Passing “universal_newlines=True” in the constructor will also automatically decode the output to text with the UTF-8 codec. Alternatively, the result may be explicitly decoded after the call.
-
Obtaining the Module¶
This module can be installed directly from the Python Package Index with pip:
pip install shell_command
Alternatively, you can download and unpack it manually from the shell_command PyPI page.
There are no operating system or distribution specific versions of this
module - it is a pure Python module that should work on all platforms (aside
from :func:iter_shell_output
being known not to work on Windows).
Supported Python versions are 2.7 and 3.2+.
Development and Support¶
Shell Command is developed and maintained on BitBucket. Problems and suggested improvements can be posted to the issue tracker.