🧩 Advanced Shell Expansions

🧠 Overview

Shell expansions are the heart of the shell execution model. Before any command runs, the shell rewrites the input through a deterministic but complex sequence of transformations: parameter expansion, command substitution, arithmetic expansion, pathname expansion, word splitting, and quote removal. Most subtle bugs, security issues, and unexpected behaviors originate here.

---

🎓 Who this is for

• Engineers writing production‑grade shell scripts.
• DevOps/SRE dealing with CI/CD, containers, and automation.
• Anyone debugging quoting, variable behavior, or unexpected command output.
• People who want to understand the shell as a transformation engine.

---

🧩 Internals / Mechanics

🧩 The expansion order (POSIX)

Shell expansions occur in this strict order:

1. Brace expansion (Bash only)
2. Tilde expansion
3. Parameter expansion ($VAR, ${VAR}, ${VAR:-default})
4. Command substitution ($(...), `...`)
5. Arithmetic expansion ($(( ... )))
6. Word splitting
7. Pathname expansion (globbing: *, ?, [...])
8. Quote removal

Understanding this order is essential for predicting behavior.

🧩 Parameter expansion

Examples:

${VAR:-default}   # default if unset or empty
${VAR:=default}   # assign default if unset or empty
${VAR:+alt}       # alt if VAR is set
${VAR:?error}     # error if unset or empty

🧩 Command substitution

Runs in a subshell:

files=$(ls)

Always strips trailing newlines.

🧩 Arithmetic expansion

i=$((i+1))

Uses C‑style integer arithmetic.

🧩 Word splitting

Controlled by $IFS. Happens after parameter/command/arithmetic expansion.

🧩 Pathname expansion (globbing)

*.log

Expands to matching filenames unless nullglob or failglob is set.

---

🔧 Techniques

🔧 Always quote variables unless you want splitting

"$var"

🔧 Use arrays to avoid splitting issues (Bash)

arr=("$@")

🔧 Use printf instead of echo

printf '%s\n' "$var"

🔧 Use ${var%pattern} and ${var#pattern} for trimming

file="backup.tar.gz"
echo "${file%.gz}"   # backup.tar

---

⚠️ Pitfalls

⚠️ Unquoted variables causing splitting

rm -rf $DIR/*

If $DIR is empty → expands to rm -rf /*.

⚠️ Command substitution strips newlines

lines=$(printf "a\nb\n")
printf "%s" "$lines"   # prints "ab"

⚠️ Globs expanding unexpectedly

for f in *; do
  ...
done

If no files exist, behavior depends on shell options.

⚠️ Brace expansion confusion

echo {1..3}.txt

Bash expands this, POSIX shells do not.

---

🚨 Real‑World Failures

🚨 Failure: CI job deletes root directory

rm -rf "$WORKSPACE"/*

If $WORKSPACE is empty → expands to rm -rf /*.

🚨 Failure: Command substitution breaks JSON

json=$(cat file.json)
echo "$json"   # newlines stripped → invalid JSON

🚨 Failure: Word splitting breaks arguments

grep $pattern file

If $pattern contains spaces → multiple arguments.

---

🛠️ Patterns

🛠️ Pattern: Quote everything by default

"$var"
"${arr[@]}"

🛠️ Pattern: Use arrays for lists

files=( *.log )

🛠️ Pattern: Use printf for safe output

🛠️ Pattern: Validate variables before expansion

: "${WORKSPACE:?must be set}"

---

❌ Anti‑Patterns

❌ Anti‑pattern: Using echo for structured data

echo is not reliable for JSON, XML, or binary data.

❌ Anti‑pattern: Relying on implicit splitting

for x in $list; do

❌ Anti‑pattern: Unquoted globs

rm -rf $DIR/*

---

🔍 Debugging

🔍 Use set -x to trace expansions

Shows expanded commands before execution.

🔍 Use printf '%q' (Bash) to inspect quoting

printf '%q\n' "$var"

🔍 Use declare -p to inspect variables

---

⚙️ Performance

⚙️ Avoid unnecessary command substitutions

var=$(cat file)   # slow
var=$(<file)      # fast

⚙️ Avoid globbing in tight loops

Use arrays or find.

---

🧵 Process Control

Expansions themselves do not spawn processes, but command substitution does:

result=$(cmd)   # always forks

---

🐳 Containers

🐳 Expansion issues in entrypoints

Environment variables may be empty or unset. Always validate before expansion.

---

🛰️ CI/CD

🛰️ Fail fast on unset variables

set -u

🛰️ Validate all inputs before expansion

---

🧠 Summary

Shell expansions are a deterministic but complex transformation pipeline. Mastering them prevents:

• data corruption
• security issues
• accidental deletions
• broken CI pipelines

Expansions are where most shell bugs originate — understanding them is essential for writing safe, predictable scripts.