🔍 Advanced Shell Debugging

🧠 Overview

Debugging shell scripts is fundamentally different from debugging traditional programming languages. The shell is a runtime orchestrator with:

implicit forks
subshells
complex expansion rules
hidden state
asynchronous pipelines
signal propagation
file descriptor manipulation

This module teaches how to debug shell behavior at the level of processes, signals, FDs, and execution flow — not just syntax.

🎓 Who this is for

DevOps/SRE diagnosing failures in CI/CD, containers, or automation.
Engineers writing complex orchestration scripts or entrypoints.
Anyone dealing with unexpected behavior in pipelines, subshells, or expansions.
People who want to understand why the shell behaves the way it does.

🧩 Internals / Mechanics

🧩 What debugging actually means in the shell

Debugging requires observing:

Parsing — how the shell interprets the script.
Expansion — how variables, globs, and substitutions transform input.
Execution — which commands fork, which run as builtins.
Process control — how jobs, groups, and signals behave.
File descriptors — how redirections and pipes are wired.

🧩 What `set -x` really shows

set -x prints:

expanded commands
redirections
pipeline stages
evaluated arguments

It does not show:

forks
exec calls
file descriptor operations
signal delivery
subshell boundaries

For that, you need system‑level tools.

🔧 Techniques

🔧 Enhanced tracing with PS4

export PS4='+ ${BASH_SOURCE}:${LINENO}:${FUNCNAME[0]}: '
set -x

This adds:

file name
line number
function name

to every traced command.

🔧 Use `strace` to observe syscalls

strace -f -e trace=process,desc sh script.sh

Reveals:

fork/exec
open/close
dup2
pipe creation
read/write
signal handling

🔧 Inspect open file descriptors

lsof -p "$pid"

Useful for debugging:

leaked FDs
pipe wiring
unexpected redirections

🔧 Inspect process trees

ps f
pstree -p

Shows:

subshells
pipeline processes
orphaned children

🔧 Debug pipelines

strace -f -e trace=pipe,read,write sh script.sh

🔧 Debug subshells

Add:

echo "PID=$$" >&2

to see where forks occur.

⚠️ Pitfalls

⚠️ `set -x` does not show command substitution failures

result=$(failing_cmd)

The failure is hidden unless explicitly checked.

⚠️ `set -e` does not propagate into subshells

( set -e; failing_cmd )
echo $?   # often 0

⚠️ Pipelines hide errors without `pipefail`

⚠️ Pipe buffer deadlocks (~64 KB)

Producer blocks when consumer is slow.

⚠️ Traps do not fire in subshells

( trap 'echo hi' EXIT )

Runs only inside the subshell.

🚨 Real‑World Failures

🚨 Failure: CI pipeline hangs with no logs

producer | consumer

If consumer is slow → pipe fills → producer blocks → pipeline freezes.

Debug:

strace -f -e trace=write,read producer

🚨 Failure: Command substitution hides critical error

json=$(jq . config.json)

If jq fails → $json becomes empty → deployment breaks silently.

🚨 Failure: Zombie processes in containers

Shell as PID 1 does not reap children.

Debug:

1	`ps -o pid,ppid,stat,cmd`

🛠️ Patterns

🛠️ Pattern: Debug mode flag

DEBUG=${DEBUG:-0}
((DEBUG)) && set -x

🛠️ Pattern: Structured logging

log() { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*" >&2; }

🛠️ Pattern: PID tracing

echo "PID=$$ PPID=$PPID" >&2

🛠️ Pattern: FD tracing

ls -l /proc/$$/fd >&2

❌ Anti‑Patterns

❌ Debugging with `echo` everywhere

Breaks JSON, corrupts pipelines, pollutes logs.

❌ Using `echo` instead of `printf`

❌ Ignoring `$?`

❌ Debugging without checking process trees

🔍 Debugging Tools Summary

Tool	Purpose
`set -x`	trace expansions & commands
`PS4`	enhanced tracing metadata
`strace`	syscalls, forks, FD, signals
`lsof`	open file descriptors
`ps`, `pstree`	process trees
`pgrep`, `pkill`	process discovery
`trap`	signal debugging
`printf '%q'`	inspect quoting
`declare -p`	inspect variables

⚙️ Performance Considerations

Debugging slows execution:

set -x → 2–10× slower
strace → 10–100× slower
heavy logging → I/O bottlenecks

Use selectively.

🧵 Process Control

Debugging signals

strace -f -e trace=signal sh script.sh

Debugging job control

set -m
jobs -l

🐳 Containers

🐳 Debugging entrypoints