Przejdź do treści

🧩 Advanced Shell Expansions (Full POSIX + Bash Edition)

🧠 Overview

Shell expansions are the core transformation engine of every POSIX‑style shell. Before the shell executes anything, it rewrites the input through a deterministic pipeline of expansion phases. These phases interact in subtle, sometimes surprising ways — especially when mixing POSIX semantics with Bash‑specific features.

This module provides a complete, exhaustive, production‑grade reference for expansions in:

  • POSIX shells (dash, ash, busybox, ksh)
  • Bash (full feature set)
  • Zsh (differences noted where relevant)

This is the definitive expansions module for advanced engineers, SREs, DevOps, and anyone writing automation that must be correct, safe, and predictable.

🎓 Who this is for

  • Senior engineers writing production‑grade shell scripts
  • DevOps/SRE building CI/CD pipelines, container entrypoints, automation
  • Engineers debugging quoting, splitting, globbing, or substitution issues
  • People who want to understand the shell as a deterministic compiler
  • Anyone who has ever said “why the hell did the shell do THAT?”

🧩 Role in the Ecosystem

Expansions interact with:

If you misunderstand expansions, you misunderstand the shell.

1. Expansion Pipeline (POSIX + Bash Differences)

🧩 1.1 POSIX Expansion Order

POSIX defines the following strict order:

  1. Brace expansion (Bash-only; POSIX ignores)
  2. Tilde expansion
  3. Parameter expansion
  4. Command substitution
  5. Arithmetic expansion
  6. Word splitting
  7. Pathname expansion (globbing)
  8. Quote removal

This order is not negotiable. Every shell script bug involving quoting or splitting comes from misunderstanding this pipeline.

🧩 1.2 Bash Expansion Order

Bash adds:

  • brace expansion
  • arrays
  • associative arrays
  • extended globbing
  • case modification
  • indirect expansion
  • name references
  • process substitution

Bash expansion order:

  1. Brace expansion
  2. Tilde expansion
  3. Parameter expansion
  4. Command substitution
  5. Arithmetic expansion
  6. Word splitting
  7. Globbing
  8. Quote removal

Same as POSIX, but with more features inside each phase.

🧩 1.3 Zsh Differences (important!)

Zsh:

  • performs globbing before splitting (opposite of POSIX/Bash)
  • has recursive globbing (**)
  • has glob qualifiers ((.), (N), (D), (/))
  • has extended globbing enabled by default
  • has brace expansion always enabled
  • has different rules for arrays and splitting

This module focuses on POSIX + Bash, but Zsh differences are noted where they matter.

🧩 1.4 Expansion Pipeline Visualized

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
RAW INPUT
   ↓
Brace Expansion (Bash)
   ↓
Tilde Expansion
   ↓
Parameter Expansion
   ↓
Command Substitution
   ↓
Arithmetic Expansion
   ↓
Word Splitting
   ↓
Globbing
   ↓
Quote Removal
   ↓
EXECUTION

🧩 1.5 Expansion Pipeline Invariants

  • Expansions happen before execution.
  • Expansions happen left-to-right, but in phases.
  • Quoting controls splitting and globbing, not parameter expansion.
  • Command substitution always runs in a subshell.
  • Word splitting is the most dangerous phase.
  • Globbing happens after splitting.
  • Quote removal happens last.

2. Parameter Expansion (POSIX + Bash)

Parameter expansion is the most powerful and most misused expansion phase.

🧩 2.1 Basic Forms

1
2
$VAR
${VAR}

{} are required when:

  • concatenating: ${VAR}suffix
  • using operators: ${VAR:-default}

🧩 2.2 Defaulting Operators

1
2
${VAR:-default}   # use default if VAR is unset or empty
${VAR-default}    # use default if VAR is unset (not empty)

🧩 2.3 Assignment Operators

1
2
${VAR:=default}   # assign default if VAR is unset or empty
${VAR=default}    # assign default if VAR is unset

🧩 2.4 Alternate Value Operators

1
2
${VAR:+alt}       # alt if VAR is set
${VAR+alt}        # alt if VAR is set (even if empty)

🧩 2.5 Error Operators

1
2
${VAR:?error message}
${VAR?error message}

If VAR is unset (or empty with :?), the shell prints the message and exits.

🧩 2.6 Length Operator

1
${#VAR}

Length in characters (Bash: bytes unless using Unicode mode).

🧩 2.7 Substring Extraction

1
2
${VAR:offset}
${VAR:offset:length}

Negative offsets allowed in Bash:

1
${VAR: -2}   # last two chars

🧩 2.8 Pattern Removal

From the end:

1
2
${VAR%pattern}   # shortest match
${VAR%%pattern}  # longest match

From the beginning:

1
2
${VAR#pattern}   # shortest match
${VAR##pattern}  # longest match

Example:

1
2
3
file="backup.tar.gz"
echo "${file%.gz}"   # backup.tar
echo "${file%%.*}"   # backup

🧩 2.9 Pattern Substitution

1
2
3
4
${VAR/pat/repl}     # first match
${VAR//pat/repl}    # all matches
${VAR/#pat/repl}    # prefix only
${VAR/%pat/repl}    # suffix only

🧩 2.10 Case Modification (Bash-only)

1
2
3
4
${VAR^^}   # uppercase all
${VAR,,}   # lowercase all
${VAR^}    # uppercase first
${VAR,}    # lowercase first

🧩 2.11 Indirect Expansion (Bash-only)

1
2
name="USER"
echo "${!name}"   # expands to value of $USER

🧩 2.12 Name References (Bash-only)

1
2
declare -n ref=VAR
ref="hello"   # sets VAR

🧩 2.13 Expansion in Assignment Context

1
2
x="$y"
arr[0]="$value"

Assignments do not undergo word splitting or globbing.

🧩 2.14 Expansion in Arithmetic Context

1
i=$(( j + k ))

Arithmetic context has its own expansion rules.

3. Command Substitution

Command substitution runs a command and replaces the substitution with its output.

🧩 3.1 POSIX Rules

1
2
$(cmd)
`cmd`
  • runs in a subshell
  • strips trailing newlines
  • performs parameter expansion inside

🧩 3.2 Bash Rules

Bash adds:

  • nested command substitution
  • command substitution inside arrays
  • command substitution inside arithmetic

🧩 3.3 Newline Stripping

This is one of the most dangerous behaviors:

1
2
3
printf "a\nb\n" | cat > file
x=$(printf "a\nb\n")
printf "%s" "$x"   # prints "ab"

🧩 3.4 Nested Command Substitution

1
result=$(echo "$(date)")

🧩 3.5 Command Substitution in Pipelines

1
x=$(cmd1 | cmd2)

Runs the entire pipeline in a subshell.

🧩 3.6 Command Substitution in Redirections

1
cmd >"$(mktemp)"

🧩 3.7 Command Substitution in Arrays (Bash)

1
arr=($(cmd))

Dangerous — splitting + globbing occur.

🧩 3.8 Performance Considerations

  • each substitution = fork
  • avoid in tight loops
  • prefer <file over $(cat file)

4. Arithmetic Expansion

🧩 4.1 POSIX Arithmetic

1
$(( expression ))

Integer-only.

🧩 4.2 Bash Arithmetic

Supports:

  • base prefixes (0x, 08, 2#1010)
  • variables without $
  • side effects (i++, ++i)

🧩 4.3 Integer Rules

  • always integers
  • overflow behavior depends on shell

🧩 4.4 Base Prefixes

1
2
$(( 0xFF ))
$(( 2#1010 ))

🧩 4.5 Side Effects

1
2
((i++))
((++i))

🧩 4.6 Arithmetic in Loops

1
for ((i=0; i<10; i++)); do

🧩 4.7 Arithmetic in Arrays

1
arr[$((i+1))]=value

5. Word Splitting (POSIX + Bash)

Word splitting is one of the most dangerous and misunderstood phases of the expansion pipeline. It happens after parameter/command/arithmetic expansion, and before globbing.

🧩 5.1 POSIX Word Splitting Rules

Word splitting occurs when:

  • the result of expansion is unquoted, and
  • the expansion produced unquoted whitespace, and
  • $IFS contains characters that match that whitespace.

Default IFS:

1
<space><tab><newline>

🧩 5.2 Bash Word Splitting Rules

Bash follows POSIX, but adds:

  • $'...' quoting (no splitting)
  • arrays (no splitting when expanding "${arr[@]}")
  • mapfile (no splitting)
  • read -a (splitting into array)

🧩 5.3 IFS Semantics

IFS has three classes:

  • Whitespace characters (space, tab, newline)
  • Non-whitespace characters
  • Unset IFS

Whitespace IFS:

  • consecutive whitespace collapses into a single delimiter
  • leading/trailing whitespace is ignored

Non-whitespace IFS:

  • each character is a delimiter
  • consecutive delimiters produce empty fields

Example:

1
2
IFS=:
echo $PATH

Splits on :.

🧩 5.4 Empty IFS

1
IFS=""

→ disables splitting entirely.

🧩 5.5 IFS=$'\n\t'

This is a common safe mode:

1
IFS=$'\n\t'

Removes splitting on spaces.

🧩 5.6 Word Splitting in Arrays (Bash)

1
arr=($output)

Dangerous — splitting + globbing occur.

Safe:

1
mapfile -t arr < <(cmd)

🧩 5.7 Word Splitting in Command Substitution

1
files=$(ls)

Splits on whitespace → never use ls for parsing.

🧩 5.8 Word Splitting in eval

1
eval "cmd $var"

→ expansion → splitting → globbing → eval → execution Four layers of danger.

🧩 5.9 Word Splitting in for Loops

1
for x in $list; do

Splits on IFS.

Safe:

1
for x in "${arr[@]}"; do

6. Pathname Expansion (Globbing)

Globbing happens after word splitting.

🧩 6.1 POSIX Globbing

Supported patterns:

  • * — any string
  • ? — any single character
  • [...] — character class

🧩 6.2 Bash Globbing

Bash adds:

  • extglob
  • globstar (**)
  • dotglob
  • nullglob
  • failglob
  • nocaseglob

🧩 6.3 nullglob, failglob, dotglob

nullglob

1
shopt -s nullglob

If no match → expands to empty list.

failglob

1
shopt -s failglob

If no match → error.

dotglob

1
shopt -s dotglob

Includes dotfiles.

🧩 6.4 extglob

Enable:

1
shopt -s extglob

Patterns:

  • @(pattern-list) — one of
  • !(pattern-list) — anything except
  • ?(pattern-list) — zero or one
  • *(pattern-list) — zero or more
  • +(pattern-list) — one or more

Example:

1
rm !(*.txt)

Deletes everything except .txt.

🧩 6.5 Glob Qualifiers (Zsh)

Examples:

1
2
3
*.log(.m0)   # files modified 0 days ago
*.log(D)     # include dotfiles
*.log(N)     # nullglob

🧩 6.6 Globbing in Arrays

1
arr=( *.log )

🧩 6.7 Globbing in Redirections

1
cmd > *.log

If multiple matches → error.

🧩 6.8 Globbing in Pipelines

1
cat *.log | grep ERROR

7. Brace Expansion (Bash-only)

Brace expansion happens before all other expansions.

🧩 7.1 Basic Forms

1
echo {a,b,c}

🧩 7.2 Numeric Sequences

1
2
3
echo {1..5}
echo {01..10}
echo {1..10..2}

🧩 7.3 Nested Braces

1
echo {a,b}{1,2}

🧩 7.4 Comma Lists

1
echo {foo,bar,baz}.txt

🧩 7.5 Brace Expansion vs Globbing

Brace expansion is pure text generation, not pattern matching.

🧩 7.6 Brace Expansion vs Parameter Expansion

1
2
echo ${var}     # parameter expansion
echo {var}      # literal {var}

8. Quote Removal

Quote removal happens after all expansions except brace expansion.

🧩 8.1 POSIX Rules

  • double quotes preserve splitting/globbing suppression
  • single quotes preserve everything literally
  • backslash escapes characters

🧩 8.2 Bash Rules

Bash adds:

  • ANSI C quoting: $'...'
  • locale-aware escapes
  • $'' inside arrays

🧩 8.3 Interaction with Splitting

1
2
"$var"   # no splitting
$var     # splitting

🧩 8.4 Interaction with Globbing

1
"*"

→ literal *

1
*

→ globbing

9. Arrays (Bash-only)

Arrays are one of Bash’s most powerful features.

🧩 9.1 Indexed Arrays

1
2
arr=(a b c)
echo "${arr[1]}"

🧩 9.2 Associative Arrays

1
2
declare -A map
map[key]="value"

🧩 9.3 Array Slicing

1
echo "${arr[@]:1:2}"

🧩 9.4 Array Expansion Rules

"${arr[@]}"

→ expands to N separate words

"${arr[*]}"

→ expands to one word, joined by IFS

🧩 9.5 Arrays in Command Substitution

1
arr=($(cmd))

Dangerous — splitting + globbing.

🧩 9.6 Arrays in Arithmetic

1
(( arr[i]++ ))

🧩 9.7 Arrays in Redirections

1
cmd >"${arr[0]}"

10. Here-Docs & Here-Strings

Here-docs and here-strings are powerful input mechanisms with their own expansion rules.

🧩 10.1 Here-Doc Expansion Rules

Unquoted delimiter:

1
2
3
cat <<EOF
Hello $USER
EOF

→ expansions occur inside the body.

Quoted delimiter:

1
2
3
cat <<'EOF'
Hello $USER
EOF

→ no expansions occur.

🧩 10.2 Quoted vs Unquoted Delimiters

Delimiter Expansions Interpretation
<<EOF yes normal expansions
<<'EOF' no literal text
<<"EOF" yes but quotes removed

🧩 10.3 Here-Strings

1
grep foo <<< "$text"

Equivalent to:

1
echo "$text" | grep foo

🧩 10.4 Here-Docs in Pipelines

1
2
3
4
cat <<EOF | grep foo
line1
line2
EOF

Runs in a subshell in POSIX shells.

🧩 10.5 Here-Docs in Functions

Here-docs inside functions follow the same expansion rules.

11. Process Substitution (Bash/Zsh)

Process substitution creates temporary FIFOs or /dev/fd entries.

🧩 11.1 <(cmd)

Replaces with a filename:

1
diff <(sort a) <(sort b)

🧩 11.2 >(cmd)

Sends output to a process:

1
tee >(grep ERROR > errors.log)

🧩 11.3 FD Behavior

  • creates pipes or FIFOs
  • child processes inherit FDs
  • can cause hangs if FDs leak

🧩 11.4 Subshell Behavior

Each substitution runs in a subshell.

🧩 11.5 Interaction with Pipelines

1
cmd | tee >(process1) >(process2)

Creates multiple parallel subshells.

🧩 11.6 Interaction with Expansions

Process substitution happens during command substitution phase.

12. Expansions in Redirections

Redirections undergo:

  1. parameter expansion
  2. command substitution
  3. arithmetic expansion
  4. pathname expansion

Example:

1
cmd > "$dir/$(date +%F).log"

13. Expansions in Pipelines

Each pipeline segment may run in a subshell:

1
echo "$x" | while read line; do ...; done

→ while runs in a subshell in POSIX shells.

14. Expansions in Functions

Functions expand arguments exactly like commands.

1
myfunc "$var" "$(cmd)"

15. Expansions in eval

eval performs:

  1. expansions
  2. splitting
  3. globbing
  4. evaluation
  5. expansions again
  6. execution

This is why eval is dangerous.

16. Expansions in Test Conditions

🧩 16.1 [ ] (POSIX)

  • expansions occur
  • splitting occurs
  • globbing occurs

Example:

1
[ $x = *.txt ]

→ globbing happens!

🧩 16.2 [[ ]] (Bash)

  • no splitting
  • no globbing unless intended
  • safer

Example:

1
[[ $x == *.txt ]]

→ pattern match, not globbing.

17. Expansions in Dockerfile ENTRYPOINT

ENTRYPOINT uses shell form:

1
ENTRYPOINT echo $HOME

→ expansions occur in /bin/sh -c.

Exec form:

1
ENTRYPOINT ["echo", "$HOME"]

→ no expansions.

18. Expansions in Kubernetes Manifests

Kubernetes does not expand shell variables. Only the shell inside the container does.

Example:

1
command: ["sh", "-c", "echo $HOSTNAME"]

19. Expansions in CI/CD Runners

CI runners often:

  • use dash (not bash)
  • disable brace expansion
  • disable arrays
  • disable process substitution

Scripts must be portable.

20. Expansions in SSH Remote Commands

1
ssh host "echo $HOME"

→ expansion happens on the remote host.

1
ssh host echo $HOME

→ expansion happens locally.

21. Expansions in cron/systemd

cron runs with:

  • minimal PATH
  • no user environment
  • no shell rc files

Always use:

1
2
#!/bin/sh
PATH=/usr/bin:/bin

22. Security Pitfalls

🧩 22.1 Unquoted Variables

1
rm -rf $DIR/*

🧩 22.2 eval Injection

1
eval "cmd $user_input"

🧩 22.3 Command Substitution Injection

1
var=$(cat "$file")

If $file contains backticks → code execution.

🧩 22.4 Globbing Injection

1
for f in $list; do

23. Performance Engineering

🧩 23.1 Avoid command substitution in loops

Bad:

1
for f in $(ls); do

Good:

1
for f in *; do

🧩 23.2 Prefer <file over $(cat file)

🧩 23.3 Avoid globbing in tight loops

🧩 23.4 Use arrays instead of splitting

🧩 23.5 Use mapfile for bulk reads

24. Debugging Expansions

🧩 24.1 Use set -x

Shows expanded commands.

🧩 24.2 Use PS4 for deep debugging

1
2
export PS4='+ ${BASH_SOURCE}:${LINENO}:${FUNCNAME[0]}: '
set -x

🧩 24.3 Use printf '%q' to inspect quoting

🧩 24.4 Use declare -p to inspect variables

🧩 24.5 Use strace to inspect execve boundaries

25. Real‑World Failures

🧩 25.1 CI job deletes root directory

1
rm -rf "$WORKSPACE"/*

🧩 25.2 JSON corruption due to newline stripping

🧩 25.3 Word splitting breaks arguments

🧩 25.4 Globbing expands unexpectedly

🧩 25.5 eval injection in production

26. Patterns

🧩 Quote everything by default

🧩 Use arrays for lists

🧩 Use printf for safe output

🧩 Validate variables before expansion

🧩 Avoid eval

🧩 Prefer [[ ]] over [ ]

🧩 Use process substitution for parallelism

🧩 Use here-docs for structured input

27. Anti‑Patterns

  • Using echo for structured data
  • Relying on implicit splitting
  • Unquoted globs
  • Using ls for parsing
  • Using eval with user input
  • Using arrays via arr=($(cmd))
  • Using for x in $list

🧠 Summary

Shell expansions form a deterministic, multi-phase compilation pipeline that transforms raw input into executable commands. Understanding expansions is essential for:

  • correctness
  • safety
  • performance
  • portability
  • CI/CD reliability
  • container entrypoints
  • automation
  • debugging

This module provides the complete, extended, production-grade reference for POSIX + Bash expansions.

Mastering expansions means mastering the shell.