Przejdź do treści

🛰️ Advanced Shell CI/CD

🧠 Overview

Shell scripts are the backbone of CI/CD systems. They glue together:

  • build tools
  • test runners
  • artifact pipelines
  • deployment logic
  • container tooling
  • cloud CLIs

But CI/CD environments are non‑interactive, ephemeral, strict, and unforgiving. This module teaches how to write deterministic, idempotent, fail‑fast, production‑grade shell scripts for CI/CD pipelines.

🎓 Who this is for

  • DevOps/SRE building or maintaining CI/CD pipelines.
  • Engineers writing build/test/deploy automation.
  • Anyone dealing with flaky pipelines, silent failures, or inconsistent behavior.
  • People who want predictable, reproducible automation.

🧩 Internals / Mechanics

🧩 CI/CD is a hostile environment

CI/CD runners typically have:

  • no interactive shell
  • no aliases
  • no user dotfiles
  • minimal PATH
  • strict timeouts
  • unpredictable parallelism
  • ephemeral filesystems
  • limited logging

Scripts must assume nothing about the environment.

🧩 CI/CD shells must be deterministic

Determinism requires:

  • explicit environment validation
  • explicit exit behavior
  • explicit dependencies
  • explicit paths
  • explicit cleanup

🧩 CI/CD pipelines rely heavily on exit codes

  • 0 → success
  • non‑zero → fail the job
  • pipelines without pipefail hide failures
  • command substitution hides failures
  • subshells isolate failures

🔧 Techniques

🔧 Always enable strict mode

1
set -euo pipefail

🔧 Validate all required environment variables

1
2
: "${GIT_SHA:?missing GIT_SHA}"
: "${ENVIRONMENT:?missing ENVIRONMENT}"

🔧 Validate required tools

1
2
3
4
command -v docker >/dev/null 2>&1 || {
  printf '%s\n' "docker not installed" >&2
  exit 1
}

🔧 Use absolute paths

1
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"

🔧 Log everything to stderr

1
log() { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*" >&2; }

🔧 Use trap for cleanup

1
trap 'cleanup; exit 1' ERR

⚠️ Pitfalls

⚠️ Silent failures in pipelines

1
docker build . | tee build.log

⚠️ Unset variables breaking deployments

1
echo "Deploying to $ENV"

If $ENV is empty → deploys to wrong environment.

⚠️ Race conditions in parallel CI jobs

⚠️ Using interactive commands

  • read
  • sudo
  • prompts
  • editors

⚠️ Relying on user dotfiles

CI shells do not load:

  • .bashrc
  • .profile
  • .zshrc

🚨 Real‑World Failures

🚨 Failure: Build passes despite failing tests

1
npm test | tee test.log

npm test fails → tee succeeds → pipeline exit = 0.

Fix:

1
2
set -o pipefail
npm test | tee test.log

🚨 Failure: Deployment script deploys to production accidentally

1
kubectl apply -f deploy.yaml

Missing environment validation → wrong cluster.

Fix:

1
2
: "${ENVIRONMENT:?missing ENVIRONMENT}"
kubectl --context="cluster-$ENVIRONMENT" apply -f deploy.yaml

🚨 Failure: CI job hangs due to background process

1
2
long_task &
exit 0

Background process keeps running → job never finishes.

Fix:

1
trap 'kill 0' EXIT

🛠️ Patterns

🛠️ Pattern: One script per responsibility

  • build.sh
  • test.sh
  • package.sh
  • deploy.sh

🛠️ Pattern: Idempotent scripts

Scripts must be safe to re‑run.

🛠️ Pattern: Explicit inputs, explicit outputs

Avoid implicit state.

🛠️ Pattern: Use JSON for structured output

Avoid parsing human‑readable logs.

🛠️ Pattern: Fail fast, fail loud

1
set -euo pipefail

❌ Anti‑Patterns

❌ Anti‑pattern: Using echo for structured data

❌ Anti‑pattern: Relying on implicit PATH

❌ Anti‑pattern: Running interactive commands

❌ Anti‑pattern: Silent failure swallowing

❌ Anti‑pattern: Using pipelines for state mutation

🔍 Debugging

🔍 Enable debug mode only when needed

1
2
DEBUG=${DEBUG:-0}
((DEBUG)) && set -x

1
env | sort >&2

🔍 Trace syscalls

1
strace -f -e trace=process,desc sh script.sh

🔍 Inspect CI logs for FD issues

⚙️ Performance

⚙️ Avoid slow loops

Use xargs, parallel, or find -exec … +.

⚙️ Avoid unnecessary cloning or scanning

Cache aggressively.

⚙️ Use parallelism carefully

Avoid race conditions.

🧵 Process Control

🧵 Ensure proper signal handling

CI runners send SIGTERM on timeout.

🧵 Avoid background jobs unless necessary

🧵 Use wait to avoid zombies

🐳 Containers

🐳 Validate environment before running containerized commands

🐳 Use exec in entrypoints

1
exec app "$@"

🐳 Avoid long‑running loops in CI containers

🧠 Summary

CI/CD shell scripting requires:

  • strict mode
  • deterministic behavior
  • explicit validation
  • predictable exit codes
  • safe pipelines
  • structured logging
  • idempotency
  • no interactive assumptions

Mastering these techniques produces reliable, reproducible, production‑grade pipelines.