Przejdź do treści

🧵 Redirections & File Descriptors (Advanced Internals)

🧠 Overview

This module explains how the shell manages file descriptors (FDs) and redirections, which is the foundation of:

  • pipelines
  • logging
  • background workers
  • process supervision
  • container entrypoints
  • CI/CD orchestration
  • avoiding hangs caused by open FDs

Most “mysterious” shell bugs — hanging pipelines, missing logs, truncated output, zombie‑like behavior — come from misunderstanding FD inheritance and redirection order.

🎓 Who this is for

  • DevOps/SRE working with CI/CD, containers, and automation
  • Engineers writing entrypoints, wrappers, or supervisors
  • Anyone debugging:
  • stuck pipelines
  • missing output
  • FD leaks
  • weird buffering
  • People who want deterministic, production‑grade shell behavior

🧩 Role in the Ecosystem

Redirections & FD mechanics underpin:

If you don’t understand FD inheritance and redirection order, you’re guessing when debugging pipeline hangs or missing output.

🧩 Internals / Mechanics

🧩 What is a file descriptor?

A file descriptor is an integer index into a per‑process table:

  • 0 — stdin
  • 1 — stdout
  • 2 — stderr

Everything else (pipes, sockets, files) is just another FD: 3, 4, 5, …

FDs survive fork(), and survive execve() unless marked CLOEXEC.

🧩 How redirections work

Examples:

1
2
3
4
5
cmd > out.txt
cmd >> out.txt
cmd 2> err.txt
cmd > out.txt 2>&1
cmd &> all.txt

Mechanically:

  1. The shell forks a child.
  2. In the child, before execve():
  3. it opens files,
  4. duplicates FDs (dup2()),
  5. closes unused FDs.
  6. Then it calls execve().

Redirections happen before exec, not inside the program.

🧩 Order matters

These two are NOT the same:

1
2
cmd > file 2>&1
cmd 2>&1 > file

Why?

  • In the first:
  • stdout → file

  • stderr → stdout → file

  • In the second:

  • stderr → stdout (terminal)
  • stdout → file

This is one of the most common shell bugs.

🧩 FD duplication (>&)

1
cmd 3>&1

Means:

  • duplicate FD 1 (stdout) into FD 3
  • FD 3 now points to the same underlying file/pipe/socket

Useful for:

  • capturing original stdout
  • redirecting temporarily
  • restoring later

🧩 Closing FDs

1
cmd 3>&-

Closes FD 3 in the child before exec.

If a pipeline hangs, it’s often because:

  • some FD was not closed
  • some process inherited a write end of a pipe

🧩 Redirecting both stdout and stderr

1
2
3
cmd > file 2>&1
cmd &> file
cmd 1>file 2>file

All valid, but with different portability and ordering semantics.

🧩 Here‑documents and here‑strings

1
2
3
4
5
cat <<EOF
hello
EOF

grep foo <<< "$text"

Mechanically:

  • the shell creates a pipe or temp file
  • writes the here‑doc content into it
  • connects the read end to stdin of the command

🧩 FD inheritance across exec

If you run:

1
some-wrapper.sh

and inside:

1
exec my-app

Then:

  • my-app inherits all open FDs of the wrapper
  • unless they were closed or marked CLOEXEC

This is why:

  • apps sometimes keep ports open
  • pipelines hang
  • logs get duplicated
  • CI jobs never finish

🔧 Techniques

🔧 Capture stdout and stderr separately

1
cmd >out.log 2>err.log

🔧 Capture both together

1
cmd >all.log 2>&1

🔧 Capture stderr but keep stdout on terminal

1
cmd 2>err.log

🔧 Temporarily redirect stdout

1
2
3
4
5
6
7
8
{
  echo "before"
  exec 3>&1
  exec 1>out.log
  echo "this goes to file"
  exec 1>&3
  echo "back to terminal"
}

🔧 Debug FD usage

1
ls -l /proc/$$/fd

Shows:

  • which FDs are open
  • where they point
  • which pipes are still alive

⚠️ Pitfalls

⚠️ Wrong ordering of redirections

1
cmd 2>&1 >file

stderr goes to terminal, not file.

⚠️ FD leaks causing pipeline hangs

If any process keeps a pipe write end open:

  • readers never see EOF
  • pipeline hangs forever

Common causes:

  • parent shell not closing FDs
  • tools that fork internally
  • missing CLOEXEC

⚠️ Redirecting inside subshells unexpectedly

1
( cmd > file )

Only affects the subshell, not the parent.

⚠️ Overwriting logs accidentally

1
2
cmd > log
cmd >> log

Use >> for append.

🚨 Real‑world failures

🚨 Failure: CI job hangs forever

Cause:

  • a child process inherited a pipe FD
  • kept the write end open
  • uploader waited for EOF → never came

Fix:

  • close unused FDs
  • use CLOEXEC
  • inspect /proc/$pid/fd

🚨 Failure: stderr printed to terminal instead of file

Cause:

1
cmd 2>&1 >file

Fix:

1
cmd >file 2>&1

🚨 Failure: logs missing in production

Cause:

  • script used > instead of >>
  • logs overwritten on each run

🛠️ Patterns

🛠️ Pattern: Explicit FD management

1
2
3
4
exec 3>&1
exec 1>log.txt
run
exec 1>&3

🛠️ Pattern: Use CLOEXEC in non‑shell tools

In Python/Go/Rust:

  • set CLOEXEC on internal pipes
  • prevents FD leaks across exec

🛠️ Pattern: Structured logging via redirection

1
run-app 1>app.log 2>app.err

❌ Anti‑patterns

  • assuming redirection order doesn’t matter
  • relying on implicit FD behavior
  • ignoring inherited FDs
  • redirecting inside subshells unintentionally
  • using > when you meant >>

🔍 Debugging

🔍 Inspect open FDs

1
ls -l /proc/$$/fd

🔍 Trace FD operations

1
strace -f -e trace=desc sh script.sh

Shows:

  • open
  • close
  • dup
  • dup2
  • pipe

🔍 Visualize pipeline FD graph

1
strace -f -e trace=process,desc sh script.sh

🧠 Summary

Redirections & file descriptors are the plumbing layer of the shell:

  • redirections happen before exec
  • FD inheritance is real and dangerous
  • ordering matters
  • CLOEXEC prevents leaks
  • pipelines are FD graphs, not syntax

Once you understand FD mechanics, you can design:

  • non‑hanging pipelines
  • predictable logging
  • safe entrypoints
  • robust CI/CD scripts
  • clean process supervision