{{Header}}
{{title|title=
/bin/bash - Proper Whitespace Handling - Whitespace Safety - End-of-Options Parameter Security
}}
{{#seo:
|description=Supporting multiple command line parameters with spaces in wrapper scripts, safe output handling, and use of the end-of-options parameter (--).
}}
{{coding_style_mininav}}
{{intro|
Supporting multiple command line parameters with spaces in wrapper scripts, safe output handling, and use of the end-of-options parameter (<code>--</code>) for better safety.
}}

= Summary =
* <u>Shell options:</u> Use <code>set -o errexit</code>, <code>set -o nounset</code>, <code>set -o errtrace</code>, and <code>set -o pipefail</code>.
* <u>Quoted expansions:</u> Quote variable expansions and prefer <code>${variable}</code> style.
* <u>Array-based command building:</u> Build command lines with arrays. On POSIX <code>sh</code>, use <code>set --</code>.
* <u>End-of-options marker:</u> Use the end-of-options parameter <code>--</code> where supported, after options and before positional parameters.
* <u>Long options:</u> Prefer long option names over short flags when sensible.
* <u>Safe output:</u> Do not use <code>echo</code>. Use <code>printf</code> with an explicit format string, or <code>stecho</code> for terminal output.
* <u>Safe line input:</u> Use <code>IFS= read -r</code> for line-oriented input.
* <u>Checking unset variables:</u> Under <code>nounset</code>, check explicitly whether variables exist.
* <u>Local variable declaration:</u> Declare <code>local</code> variables first, then assign to them on a separate line.
* <u>Dynamic scoping awareness:</u> Remember that Bash <code>local</code> uses dynamic scoping, and do not <code>local</code>ize <code>BASH_REMATCH</code>.
* <u>Loop subshell avoidance:</u> Avoid piping into <code>while read</code> loops, because that can create a subshell.
* <u>Stdin protection:</u> Prevent stdin stealing inside loops by using a separate file descriptor when needed.
* <u>NUL-delimited input:</u> Use NUL-delimited input where required, for example with <code>find -files0-from</code>.
* <u>Pipefail caution:</u> Be careful with <code>pipefail</code> when piping into early-exiting consumers such as <code>grep --quiet</code>.
* <u>Readable project style:</u> Prefer explicit, readable, whitespace-safe code and project helper functions where appropriate.

= Safe ways to print =
For this style guide, do not use <code>echo</code>. Use <code>printf</code> with an explicit format string instead.

* {{VideoLink
|videoid=lq98MM2ogBk
|text=bash's echo command is broken
}}
* {{VideoLink
|videoid=ft0_cw54qak
|text=echo is broken: a follow-up video
}}

<code>shellcheck</code> bug reports:

* [https://github.com/koalaman/shellcheck/issues/2674 Warn on echo "$var" when $var might be -e #2674]
* [https://github.com/koalaman/shellcheck/issues?q=is%3Aissue+is%3Aopen+echo+in%3Atitle Open shellcheck issues related to echo]

Please note that <code>printf</code> does not have a default format specifier. The first positional parameter is always treated as the format string. When the format is omitted, untrusted data can be interpreted as format directives or backslash escapes. It is always recommended to be explicit about the format being used.

Normally, there is no need to interpret escape sequences from a variable. Therefore, use the <code>printf</code> format specifier <code>%s</code> when the data is not printed to a terminal:

{{CodeSelect|code=
var="$(printf '%s' "${untrusted_text}")"
}}

<code>printf '%s\n' "message here"</code> is the usual replacement for <code>echo "message here"</code>.

If you require escapes to be interpreted, interpret them on a per-need basis:

{{CodeSelect|code=
red="$(printf '%b' "\e[31m")"    # red=$'\e[31m'   # printf -v red     '%b' "\e[31m"
nocolor="$(printf '%b' "\e[m")"  # nocolor=$'\e[m' # printf -v nocolor '%b' "\e[m"
}}

Escapes that are already interpreted can then be printed with <code>%s</code>:

{{CodeSelect|code=
var="$(printf '%s' "${red} ${untrusted_text} ${nocolor}")"
}}

And this is why you should use <code>stecho</code> when printing to the terminal, because it sanitizes unsafe characters ([[unicode]]). Simply using <code>printf '%s'</code> is not sufficient when escapes are already interpreted:

{{CodeSelect|code=
stecho "${red} ${untrusted_text} ${nocolor}"
printf '%s' "${red} ${untrusted_text} ${nocolor}" {{!}} stecho
printf '%s' "${red} ${untrusted_text} ${nocolor}" {{!}} stecho {{!}} less -R
}}

'''Rule of thumb''':

* <code>echo</code>: Never.
* <code>printf</code>: Whenever the printed data is not used by a terminal.
** Format <code>%b</code>: Only for trusted data or fixed literals.
** Format <code>%s</code>: With any data.
* <code>stecho</code>: Whenever the printed data is used by a terminal.
** When not using <code>stecho</code>: When <code>stecho</code> cannot reasonably be considered available, such as during early build steps when building Kicksecure from source code using derivative-maker.

Resources:

* [https://github.com/anordal/shellharden/blob/master/how_to_do_things_safely_in_bash.md#echo--printf shellharden: echo / printf]
* [https://unix.stackexchange.com/questions/65803/why-is-printf-better-than-echo Unix & Linux Stack Exchange: Why is printf better than echo?]
* [https://pubs.opengroup.org/onlinepubs/9799919799/utilities/echo.html POSIX <code>echo</code> specification]

= Bash Proper Whitespace Handling =

* Quote variables.
* Build parameters using arrays.
* Enforce <code>nounset</code>.
* Use end-of-options.
* Style: use long option names.

<pre>
#!/bin/bash

## https://yakking.branchable.com/posts/whitespace-safety/

#set -x
set -o errexit
set -o nounset
set -o errtrace
set -o pipefail

lib_dir="/tmp/test/lib/program with space/something spacy"
main_app_dir="/tmp/test/home/user/folder with space/abc"

mkdir --parents -- "${lib_dir}"
mkdir --parents -- "${main_app_dir}"

declare -a cmd_list

cmd_list+=("cp")
cmd_list+=("--recursive")
cmd_list+=("--")
cmd_list+=("${lib_dir}")
cmd_list+=("${main_app_dir}/")

printf '%s\n' "cmd_list has ${#cmd_list[@]} items"

## Execution example.
"${cmd_list[@]}"

## 'for' loop example.
for cmd_item in "${cmd_list[@]}"; do
    printf '%s\n' "cmd_item: '$cmd_item'"
done

## Alternative.
cmd_alt_list=(
    cp               ## program
    --recursive      ## recursive
    --               ## stop option parsing (protects against paths that begin with '-')
    "$lib_dir"       ## source directory
    "$main_app_dir/" ## destination
)

## 'for' loop example.
for cmd_alt_item in "${cmd_alt_list[@]}"; do
    printf '%s\n' "cmd_alt_item: '$cmd_alt_item'"
done
</pre>

= Why <code>nounset</code> =

Without <code>nounset</code>, an unset variable silently expands to an empty string. That can turn a dangerous path into something unintended.

<pre>
rm -- "/$UNSET_VAR"
</pre>

If <code>UNSET_VAR</code> is unset and <code>nounset</code> is disabled, this becomes:

<pre>
rm -- "/"
</pre>

On many systems that will fail with an error such as:

<pre>
rm: cannot remove '/': Is a directory
</pre>

That specific command happens to fail here, but the pattern is still unsafe. With <code>set -o nounset</code>, the shell aborts earlier before running <code>rm</code>.

Setting <code>UNSET_VAR=""</code> would not solve the general problem either. Variables that may intentionally be empty should be handled explicitly.

= local =
== Error swallowing ==
<u>Note:</u>

{{CodeSelect|code=
local testvar=$(false)
}}

<u>Expected:</u> error

<u>Actual:</u> no error

When declaration and assignment are combined on the same line, <code>local</code> itself returns success and masks the failing command substitution.

Better:

{{CodeSelect|code=
local testvar
testvar=$(false)
}}

== Dynamic scoping ==
<code>local</code> variables in Bash use dynamic scoping. That means nested function calls can still read and modify them unless they declare their own <code>local</code> variable.

<u>Example:</u>

{{CodeSelect|code=
fn_01 () {
  local myvar
  myvar='supposedly local'
  printf '%s\n' "in fn_01, myvar is $myvar"
  fn_02
  printf '%s\n' "in fn_01, myvar is now $myvar"
}

fn_02 () {
  printf '%s\n' "in fn_02, myvar is $myvar"
  myvar='not so local after all'
  printf '%s\n' "in fn_02, myvar is now $myvar"
}

fn_01
}}

<u>Output:</u>

<pre>
in fn_01, myvar is supposedly local
in fn_02, myvar is supposedly local
in fn_02, myvar is now not so local after all
in fn_01, myvar is now not so local after all
</pre>

To avoid problems from this, declare all function-local variables as <code>local</code> at the head of a function. For example:

{{CodeSelect|code=
fn_01 () {
  local myvar
  myvar='local to fn_01'
  printf '%s\n' "in fn_01, myvar is $myvar"
  fn_02
  printf '%s\n' "in fn_01, myvar is now $myvar"
}

fn_02 () {
  local myvar
  myvar='local to fn_02'
  printf '%s\n' "in fn_02, myvar is $myvar"
}

fn_01
}}

<u>Output:</u>

<pre>
in fn_01, myvar is local to fn_01
in fn_02, myvar is local to fn_02
in fn_01, myvar is now local to fn_01
</pre>

== BASH_REMATCH ==
Do not <code>local -a BASH_REMATCH</code>!

{{quotation
|quote=Note specifically: <code>Bash sets BASH_REMATCH in the global scope; declaring it as a local variable will lead to unexpected results.</code>
|context=[https://www.gnu.org/software/bash/manual/bash.html GNU Bash manual]
}}

= POSIX array =

On a POSIX shell, positional parameters provide the portable array-like container. <code>$@</code> has different scope per function or main script. You can build it with <code>set --</code>:

Add items to the array:
<pre>
set -- a b c
</pre>

Add items to the beginning or end of the array:
<pre>
set -- b
set -- a "$@" c
</pre>

= Use of End-of-Options Parameter (--) =

The end-of-options parameter "<code>--</code>" is important because otherwise inputs might be mistaken for command options. This can even become a security issue. Here are examples using the <code>sponge</code> command:

{{CodeSelect|code=
sponge -a testfilename </dev/null
}}

Result: OK. This works because "<code>testfilename</code>" does not look like an option.

{{CodeSelect|code=
sponge -a --testfilename </dev/null
}}

Result: Fail. The command interprets "<code>--testfilename</code>" as options:

<pre>
sponge: invalid option -- '-'
sponge: invalid option -- 't'
sponge: invalid option -- 'e'
...
</pre>

{{CodeSelect|code=
sponge -a -- --testfilename </dev/null
}}

Result: OK. The <code>--</code> signals that "<code>--testfilename</code>" is a filename, not an option.

Conclusion:

* The <code>--</code> parameter marks the end of command options.
* Place <code>--</code> after all command options and before filenames or other positional parameters, where the command supports it.
* This technique is applicable to many Unix/Linux commands, not just <code>sponge</code>.
* It is especially useful when input may begin with <code>-</code>.

= nounset - Check if Variable Exists =

<pre>
#!/bin/bash

set -o errexit
set -o nounset
set -o errtrace
set -o pipefail

## Enable for testing.
#unset HOME

if [ -z "${HOME+x}" ]; then
    printf '%s\n' "Error: HOME is not set." >&2
    exit 1
fi

printf '%s\n' "$HOME"
</pre>

= Safely Using Find with NUL-Delimited Input =
Example:

<u>Note:</u> The variable could be different. It could, for example, be <code>--/usr</code>.

{{CodeSelect|code=
folder_name="/usr"
}}

{{CodeSelect|code=
printf '%s\0' "${folder_name}" {{!}} find -files0-from - -perm /u=s,g=s -print0
}}

Do not use <code>stecho</code> or <code>stprint</code> here, because <code>find -files0-from</code> requires NUL-delimited input.

<code>NUL</code> ("<code>\0</code>") is required because:

{{quotation
|quote=The starting points in file have to be separated by ASCII NUL characters. Two consecutive NUL characters, i.e., a starting point with a Zero-length file name is not allowed and will lead to an error diagnostic followed by a non-Zero exit code later.
|context=[https://manpages.debian.org/unstable/findutils/find.1.en.html Debian <code>find</code> man page]
}}

A single trailing <code>NUL</code> is normal. Two consecutive <code>NUL</code> bytes would mean an empty file name entry, which is invalid.

= loops =
== subshells created by pipelines ==
Avoid piping data into a loop. This spawns a subshell even without using <code>$()</code> syntax. Bad code example:

<pre>
str="abc
def
ghi"
line_count=0

printf '%s\n' "${str}" | while IFS= read -r line; do
  ((line_count += 1))
done

printf '%s\n' "${line_count}"

## Expected result: 3
## Actual result: 0
</pre>

Instead, redirect command output into the loop. Good code example:

<pre>
str="abc
def
ghi"
line_count=0

while IFS= read -r line; do
  ((line_count += 1))
done < <(printf '%s\n' "${str}")

printf '%s\n' "${line_count}"

## Result: 3
</pre>

== stdin stealing ==
Commands that read from stdin can swallow data that was supposed to be processed by the <code>read</code> component of a <code>while read</code> loop. <code>qrexec-client-vm</code> is one example, and <code>vim</code> is another. Bad code example:

<pre>
str="abc
def
ghi"

while IFS= read -r line; do
  vim -- "$line"
done < <(printf '%s\n' "${str}")

## Output:
##
## Vim: Warning: Input is not from a terminal
## Vim: Error reading input, exiting...
## Vim: preserving files...
## Vim: Finished.
</pre>

Work around this by using alternative file descriptors and redirection. Good code example:

<pre>
str="abc
def
ghi"

while IFS= read -r line 0<&3; do
  vim -- "$line"
done 3< <(printf '%s\n' "${str}")

## Result: Opens "abc", then "def", then "ghi" in Vim.
</pre>

= misc =
<pre>
base_name="${file_name##*/}"
file_extension="${base_name##*.}"
</pre>

= coding style =
* no workarounds for older Bash versions. Assume the Bash version of Debian {{Stable project version based on Debian codename}}.
* prefer explicit, readable, whitespace-safe code over compact shell tricks
* use long options rather than short options when sensible, for example use <code>cp --recursive</code> instead of <code>cp -r</code>
* no trailing whitespaces allowed in source code files
* all source code files must have a newline at the end
* no git style symlinks ([[Git#git_symlinks|git symlinks]]) (text file without newline at the end) because of past [https://security.snyk.io/vuln/SNYK-UNMANAGED-GITGIT-2372015 git symlink CVE]
* avoid [[unicode]] whenever possible. See also [[unicode-show]].
* use:
** <code>shellcheck</code>
** avoid <code>rm</code> when <code>safe-rm</code> is appropriate <ref>
* https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/issues/123
* https://github.com/valvesoftware/steam-for-linux/issues/3671
</ref>
** avoid <code>wget</code> and <code>curl</code> in project code, prefer <code>scurl</code> ([[Secure Downloads]])
** avoid <code>grep</code> for simple string matching in project code, use <code>str_match</code>
** <code>str_replace</code>
** <code>append-once</code>
** <code>overwrite</code>
* use <code>${variable}</code> style
* use shell options

<pre>
set -o errexit
set -o nounset
set -o errtrace
set -o pipefail
</pre>

* do not use:
** <code>which</code>, use <code>command -v</code> instead. This is because <code>which</code> is an external binary, whereas <code>command</code> is a shell built-in.
* file name extensions:
** POSIX <code>sh</code> libraries: <code>.sh</code>
** Bash libraries: <code>.bsh</code>
** Executables: no file name extension
** (executables = scripts that can be run but cannot be sourced, libraries = scripts that can be sourced but may optionally be run as well)

= pipefail and early-exiting consumers =
This combination can be an issue because the consumer may exit early and the producer may then receive <code>SIGPIPE</code> (broken pipe).

<pre>
#!/bin/bash

set -o errexit
set -o nounset
set -o errtrace
set -o pipefail

for i in {1..10000}; do
  printf '%s\n' "0"
done | grep --quiet -- "0"
</pre>

This can fail even though <code>grep --quiet</code> finds a match. <code>grep --quiet</code> exits as soon as it has enough input, while the producer may still be writing. With <code>pipefail</code> enabled, the producer's non-zero exit status can then make the whole pipeline fail.

Guideline:

* Avoid <code>producer {{!}} grep --quiet -- pattern</code> when <code>pipefail</code> is enabled.
* Prefer matching directly against a variable or file when possible.
* In project code, prefer helper functions such as <code>str_match</code> where they fit the use case.
* If an early-exit consumer is intentional, handle exit statuses explicitly instead of assuming the pipeline is harmless.

= Improved Error Handler =
Inspired by [https://github.com/pottmi/stringent.sh stringent.sh]

{{CodeSelect|code=
if (( "$BASH_SUBSHELL" >= 1 )); then
  kill "$$"
fi
}}

Usually not needed. When a subshell detects an error due to <code>errexit</code> and <code>errtrace</code>, it returns a non-zero exit status and the parent shell also sees the failure. Preventing the error handler from running twice is only useful in rare cases.

= Resources =
* [https://github.com/anordal/shellharden/blob/master/how_to_do_things_safely_in_bash.md shellharden: How to do things safely in bash]
* [https://dwheeler.com/essays/fixing-unix-linux-filenames.html David A. Wheeler: Fixing Unix/Linux/POSIX filenames]
* use with care: {{VideoLink
|videoid=DvDu8_A2uhs
|text=Seat Belts and Airbags for bash
}}
** use with care: [https://github.com/pottmi/stringent.sh stringent.sh]

= See Also =
* [[Dev/coding style]]

= Footnotes =
<references />
{{Footer}}
[[Category:Design]]