📦 System Services & Daemons

Shell scripts frequently interact with system services and daemons such as web servers, databases, background workers, and system-level components. Understanding how to manage, monitor, and integrate with these services is essential for reliable automation and production‑grade operations.

🎓 Who This Is For

DevOps/SRE engineers maintaining Linux servers
Sysadmins managing systemd or legacy init systems
Developers writing deployment or provisioning scripts
Anyone integrating shell automation with long‑running services

🧩 Role in the Ecosystem

Linux services are managed by:

systemd (modern, dominant)
SysV init (legacy)
OpenRC, runit, s6 (specialized systems)

Shell scripts often:

Start/stop/restart services
Check service health
Read logs
Reload configuration
Manage environment files
Integrate with daemons during deployment

🧩 Key Concepts

1. systemd Units

Systemd uses unit files to define services:

.service — daemons
.timer — scheduled tasks
.socket — socket activation
.target — grouping

2. Service Lifecycle

Common operations:

systemctl start service
systemctl stop service
systemctl restart service
systemctl reload service

3. Logs

Systemd logs are centralized:

1	`journalctl -u service`

4. Environment Files

Services can load environment variables from:

/etc/default/service
/etc/sysconfig/service
EnvironmentFile=/etc/service.env

5. Exit Codes

Service status is determined by exit codes and systemd state.

🔧 Techniques

Start/Stop/Restart Services

systemctl restart nginx
systemctl stop postgresql
systemctl start docker

Check Service Status

systemctl is-active nginx
systemctl is-enabled nginx
systemctl status nginx

Reload Configuration Without Restart

1	`systemctl reload nginx`

Follow Logs

journalctl -u nginx -f

Run Commands as a Service User

sudo -u www-data command

Enable/Disable Services at Boot

systemctl enable ssh
systemctl disable cups

Timers (systemd cron replacement)

List timers:

1	`systemctl list-timers`

Run timer immediately:

1	`systemctl start backup.timer`

Legacy SysV Init

service nginx start
service nginx status
/etc/init.d/nginx restart

Check Open Ports

1	`ss -tulnp`

Check Daemon Health

systemctl is-active --quiet nginx || echo "nginx down"

⚠️ Limitations & Pitfalls

systemd behavior varies across distributions
EnvironmentFile paths differ (Debian vs RHEL)
Reload vs restart semantics vary by service
Some services ignore reload
SysV scripts may not support all operations
systemd may restart services automatically (Restart=always)
Logs may rotate or be rate‑limited

🧠 When to Use Shell for Service Management

Deployment scripts
CI/CD pipelines
Health checks
Configuration reloads
Log inspection
Service orchestration on single hosts

❌ When Not to Use Shell

Complex orchestration (use Ansible, Salt, Puppet)
Multi‑node coordination (use Kubernetes, Nomad, systemd‑run)
High‑availability failover logic
Long‑running monitoring loops (use systemd timers or proper daemons)

✅ Best Practices

Use systemctl instead of legacy service
Check exit codes explicitly
Use systemctl is-active for health checks
Use journalctl -f for real‑time logs
Avoid hard‑coding paths to unit files
Use systemctl daemon-reload after modifying units
Use environment files for configuration
Prefer timers over cron for systemd systems
Document service dependencies

🧪 Testing Service Integration

Check service state:

1	`systemctl status ssh`

Check logs:

journalctl -u ssh -n 50

Test restart:

systemctl restart ssh
echo $?   # 0 = success

Test health:

systemctl is-active --quiet ssh && echo ok || echo fail

🧠 Summary

Shell scripts integrate naturally with system services through systemd, SysV init, and related tools. Understanding service lifecycle, logs, environment files, and health checks is essential for reliable automation and production‑grade operations.