🧵 Advanced Shell Process Control

🧠 Overview

Process control is where the shell stops being “a command runner” and becomes a process orchestrator. This includes:

• process groups
• job control
• signal delivery
• foreground/background execution
• subshells
• zombies and reaping
• traps
• PID 1 behavior in containers

This is one of the most misunderstood areas of shell behavior — and one of the most critical for production systems.

---

🎓 Who this is for

• DevOps/SRE managing long‑running scripts, daemons, or containers
• Engineers writing orchestration logic or supervising child processes
• Anyone debugging zombies, hanging pipelines, or broken signal handling
• People who want deterministic, production‑grade process behavior

---

🧩 Role in the Ecosystem

Process control is the backbone of:

• Subshells & Environment
• Traps & Signals
• Advanced Pipelines
• container entrypoints
• CI/CD runners
• daemon supervision
• job control in interactive shells

Without understanding process groups, sessions, and signal propagation, shell scripts behave unpredictably under load, in pipelines, or inside containers.

---

🧩 Internals / Mechanics

🧩 The shell as a process controller

A shell manages:

• processes (PIDs)
• process groups (PGIDs)
• sessions
• terminal control
• signal routing
• job tables

When you run:

cmd1 | cmd2 &

the shell:

1. creates pipes
2. forks children
3. assigns them to a process group
4. optionally puts the group in the background
5. tracks them in the job table

---

🧩 Foreground vs background

• Foreground job owns the terminal → receives SIGINT, SIGQUIT, etc.
• Background job does NOT own the terminal → signals must be sent manually.

---

🧩 Process groups

A pipeline typically shares a process group:

cmd1 | cmd2 | cmd3

All three commands receive SIGINT when you press Ctrl‑C.

---

🧩 Subshells and isolation

Subshells:

• have their own PID
• do NOT share variable state
• inherit environment
• inherit file descriptors unless redirected

See also: Subshells & Environment

---

🔧 Techniques

🔧 Use wait to reap children

cmd &
pid=$!
wait "$pid"

Prevents zombies.

---

🔧 Use trap for clean shutdown

trap 'cleanup; exit 0' SIGINT SIGTERM

---

🔧 Use process substitution to avoid unnecessary subshells

diff <(sort a) <(sort b)

---

🔧 Use set -m (job control) only in interactive shells

Never enable job control in scripts.

---

⚠️ Pitfalls

⚠️ Zombie processes from unreaped children

cmd &
# no wait → zombie

---

⚠️ Ctrl‑C not stopping pipelines

If the shell is not managing process groups correctly, only the foreground process receives SIGINT.

---

⚠️ Traps not firing in subshells

( trap 'echo hi' EXIT )

The trap runs in the subshell, not the parent.

---

⚠️ Using kill -9 as a default

SIGKILL prevents cleanup and can corrupt state.

---

🚨 Real‑World Failures

🚨 Failure: Shell script used as PID 1 leaks zombies

In Docker:

CMD ["sh", "-c", "run-app.sh"]

sh becomes PID 1 → does NOT reap children → zombies accumulate.

Fix:

• use tini or dumb-init
• or implement a SIGCHLD handler + wait

---

🚨 Failure: CI job hangs due to orphaned background process

long_task &
exit 0

The background process keeps running → CI never finishes.

Fix:

trap 'kill 0' EXIT

---

🚨 Failure: Ctrl‑C doesn’t stop a pipeline

cmd1 | cmd2

If the shell doesn’t set a unified process group, only cmd2 receives SIGINT.

---

🛠️ Patterns

🛠️ Pattern: Explicit signal handling

trap 'echo stopping; kill 0; exit' SIGINT SIGTERM

---

🛠️ Pattern: Use wait for all children

pids=()
for x in {1..5}; do
  worker "$x" &
  pids+=("$!")
done

for pid in "${pids[@]}"; do
  wait "$pid"
done

---

🛠️ Pattern: Use a minimal init in containers

tini or dumb-init solves:

• zombie reaping
• signal forwarding
• predictable shutdown

---

❌ Anti‑Patterns

❌ Using shell as a process supervisor

Shell is not systemd. Avoid:

• long‑running loops
• manual restarts
• complex signal routing

---

❌ Ignoring SIGCHLD

Leads to zombie accumulation.

---

❌ Running background jobs without cleanup

cmd &
exit

---

🔍 Debugging

🔍 Inspect process tree

ps f
pstree -p

---

🔍 Inspect process groups

ps -o pid,pgid,comm

---

🔍 Trace signals

strace -e trace=signal -f sh script.sh

---

⚙️ Performance

⚙️ Avoid excessive forking

Use builtins where possible.

---

⚙️ Avoid long‑running background loops

They consume CPU and complicate shutdown.

---

⚙️ Use wait -n (Bash) for efficient worker pools

---

🐳 Containers

🐳 Shell as PID 1

PID 1 has special semantics:

• ignores some signals by default
• must reap children
• must forward signals

---

🐳 Use an init wrapper

ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["run.sh"]

---

🛰️ CI/CD

🛰️ Ensure deterministic shutdown

CI runners kill jobs with SIGTERM → scripts must handle it.

---

🛰️ Avoid background jobs unless necessary

They often outlive the job and cause hangs.

---

🧪 Testing

ps -o pid,pgid,sid,tty,comm

( sleep 1 & )
ps -el | grep Z

trap 'echo TERM; exit' TERM
sleep 100 &
kill -TERM $$

---

🧠 Summary

Process control is the backbone of reliable shell scripting. Mastering:

• process groups
• signals
• job control
• subshells
• reaping
• PID 1 behavior

…is essential for writing safe, predictable, production‑grade automation.