🔗 Extended Pipelines and Process Substitution

Pipelines chain commands. Process substitution treats command output as files. Together they enable powerful data flows.

🧭 Pipeline Fundamentals

command1 | command2 | command3

Data flows left to right: - stdout of command1 → stdin of command2 - stdout of command2 → stdin of command3

Exit code reflects last command (unless pipefail is set).

---

🧪 Advanced Pipeline Patterns

Tee — Branch Output

Save to file AND continue pipeline:

command | tee output.log | further_processing

Append mode:

command | tee -a output.log | filter

Combining Stdout and Stderr

command 2>&1 | grep ERROR   # Both streams to grep
command |& grep ERROR       # Bash shorthand for 2>&1 |

Named Pipes (FIFOs)

Create persistent communication channel:

mkfifo mypipe

# Producer (runs in background)
while true; do
    echo "data $(date)" > mypipe
    sleep 1
done &

# Consumer
while IFS= read -r line; do
    echo "Received: $line"
done < mypipe

Named pipes persist until both ends are closed.

---

🧠 Process Substitution

Syntax

command <(subcommand)
command >(subcommand)

Practical Examples

Compare sorted outputs:

diff <(sort file1.txt) <(sort file2.txt)

Feed generated data:

while IFS= read -r line; do
    process "$line"
done < <(generate_data)

Parallel processing:

paste <(cut -d',' -f1 data.csv) <(cut -d',' -f2 data.csv)

---

🧪 Complex Pipeline Chains

Multi-Stage Processing

cat access.log \
  | grep "POST /api" \
  | awk '{print $1, $4, $7}' \
  | sort -k2 -n \
  | uniq -c \
  | sort -nr \
  | head -10

Stages: 1. Filter POST requests 2. Extract IP, timestamp, endpoint 3. Sort by timestamp 4. Count unique entries 5. Sort by count 6. Show top 10

Parallel Branches

{
    echo "Header"
    cat part1.txt
    cat part2.txt
} | sort | tee >(wc -l > count.txt) | gzip > archive.gz

Explanation: - Merge header + two files - Sort merged stream - Branch 1: Count lines → count.txt - Branch 2: Compress → archive.gz

---

🧠 Buffering Issues

Pipelines use buffers — this affects timing:

Buffer Type	Size	Trigger
Full	4-8 KB	Buffer fills
Line	N/A	Newline encountered
None	0	Immediate flush

Force line buffering:

stdbuf -oL command | while read line; do
    echo "Got: $line"
done

Or use unbuffer (from expect package):

unbuffer command | process

---

🧪 Performance Optimization

Minimize Pipeline Stages

Each | spawns a subprocess:

# ❌ 4 processes
cat file | grep pattern | wc -l

# ✅ 1 process (faster)
grep -c pattern file

Use Built-ins When Possible

# ❌ External commands
cat file | while read line; do echo "$line"; done

# ✅ Built-in (no subshell)
while IFS= read -r line; do echo "$line"; done < file

---

🧾 Portability Notes

Feature	POSIX	Bash	Zsh
Basic pipes \|	✅	✅	✅
Process substitution	❌	✅	✅
|& (combined)	❌	✅	❌
Named pipes mkfifo	✅	✅	✅
stdbuf	❌	✅	❌

---

🧾 Summary

• Pipelines chain commands efficiently.
• Use tee to branch output.
• Process substitution treats commands as files.
• Be aware of buffering — use stdbuf when needed.
• Minimize subprocesses for performance.
• Prefer built-ins over external commands.

👉 Continue to: Filesystem Operations