⚙️ Linux Init Systems and Services

Linux has evolved through multiple init systems, each offering different approaches to service management, system boot, and process supervision.

🧭 Evolution of Linux Init Systems

SysV Init (Traditional)

The original Unix System V init system, still used in some enterprise environments:

# Service management (SysV)
sudo service apache2 start
sudo service apache2 stop
sudo service apache2 restart
sudo service apache2 status

# Enable/disable at boot
sudo chkconfig apache2 on
sudo chkconfig apache2 off

# List all services
chkconfig --list

Upstart (Ubuntu 6.10-15.04)

Event-driven init system developed by Ubuntu:

# Upstart commands
sudo start service-name
sudo stop service-name
sudo restart service-name
initctl list  # List all jobs

systemd (Modern Standard)

Contemporary init system adopted by most major distributions:

# systemd service management
sudo systemctl start service-name
sudo systemctl stop service-name
sudo systemctl restart service-name
sudo systemctl status service-name

# Enable/disable at boot
sudo systemctl enable service-name
sudo systemctl disable service-name

# List units
systemctl list-units
systemctl list-unit-files

---

🧪 systemd Deep Dive

Unit File Structure

Systemd uses unit files to define services, sockets, devices, and other system components:

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application Service
After=network.target
Wants=postgresql.service

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp
Restart=always
RestartSec=10
Environment=NODE_ENV=production

# Security settings
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/lib/myapp

[Install]
WantedBy=multi-user.target

Service Types

# Common service types
Type=simple     # Service runs directly
Type=forking    # Service forks background process
Type=oneshot    # One-time execution
Type=notify     # Service notifies systemd when ready
Type=idle       # Delay until all jobs finish

Service Management Commands

# Basic service operations
systemctl start nginx
systemctl stop nginx
systemctl restart nginx
systemctl reload nginx
systemctl status nginx

# Service enablement
systemctl enable nginx    # Enable at boot
systemctl disable nginx   # Disable at boot
systemctl is-enabled nginx # Check enablement
systemctl is-active nginx  # Check active state

# Service masking (prevent starting)
systemctl mask nginx      # Completely disable
systemctl unmask nginx    # Re-enable

---

🧠 Advanced systemd Features

Targets (Runlevels)

# System targets (equivalent to runlevels)
systemctl list-units --type=target

# Common targets
poweroff.target     # Power off system
rescue.target       # Single-user mode
multi-user.target   # Multi-user CLI
graphical.target    # Multi-user GUI
reboot.target       # Reboot system

# Change target
sudo systemctl isolate multi-user.target
sudo systemctl set-default multi-user.target

Journal Management

# systemd journal commands
journalctl              # View all logs
journalctl -f           # Follow logs (like tail -f)
journalctl -u nginx     # Logs for specific unit
journalctl --since "1 hour ago"
journalctl --since "2023-01-01" --until "2023-01-02"

# Log severity levels
journalctl -p emerg     # Emergency
journalctl -p alert     # Alerts
journalctl -p crit      # Critical
journalctl -p err       # Errors
journalctl -p warning   # Warnings
journalctl -p notice    # Notices
journalctl -p info      # Informational
journalctl -p debug     # Debug

# Export logs
journalctl --output=json > logs.json

Service Dependencies

# Service dependency management
[Unit]
Description=Web Application
After=network.target postgresql.service
Requires=postgresql.service
Wants=redis.service
Conflicts=apache.service

# After - Start after specified units
# Requires - Hard dependency (fail if not available)
# Wants - Soft dependency (try but don't fail)
# Conflicts - Prevent running with specified units

---

🧪 Service Configuration Management

Custom Service Creation

# Creating a custom service
create_service() {
    local service_name="$1"
    local exec_path="$2"
    local description="${3:-Custom Service}"

    local service_file="/etc/systemd/system/${service_name}.service"

    # Validate inputs
    if [ -z "$service_name" ] || [ -z "$exec_path" ]; then
        echo "Usage: create_service <name> <executable> [description]" >&2
        return 1
    fi

    if [ ! -x "$exec_path" ]; then
        echo "Executable not found or not executable: $exec_path" >&2
        return 1
    fi

    # Create service file
    sudo tee "$service_file" > /dev/null <<EOF
[Unit]
Description=$description
After=network.target

[Service]
Type=simple
ExecStart=$exec_path
Restart=always
RestartSec=3
User=$(whoami)
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF

    # Set proper permissions
    sudo chmod 644 "$service_file"

    # Reload systemd configuration
    sudo systemctl daemon-reload

    # Enable service
    sudo systemctl enable "$service_name"

    echo "Service $service_name created and enabled"
    echo "Start with: sudo systemctl start $service_name"
}

# Usage example
# create_service "myapp" "/opt/myapp/bin/server" "My Custom Application"

Service Template Units

# Template services for multiple instances
# /etc/systemd/system/webapp@.service
[Unit]
Description=Web Application Instance %i
After=network.target

[Service]
Type=simple
ExecStart=/opt/webapp/bin/server --instance=%i
Restart=always
User=webapp

[Install]
WantedBy=multi-user.target

# Usage
sudo systemctl start webapp@instance1
sudo systemctl start webapp@instance2
sudo systemctl enable webapp@instance1

---

🧠 Service Monitoring and Troubleshooting

Health Checks

# Service health monitoring script
monitor_service_health() {
    local service_name="$1"
    local check_interval="${2:-30}"

    echo "Monitoring service: $service_name (interval: ${check_interval}s)"

    while true; do
        if systemctl is-active --quiet "$service_name"; then
            echo "[$(date)] $service_name: RUNNING"
        else
            echo "[$(date)] $service_name: STOPPED - attempting restart"
            sudo systemctl start "$service_name"

            if systemctl is-active --quiet "$service_name"; then
                echo "[$(date)] $service_name: RESTARTED SUCCESSFULLY"
            else
                echo "[$(date)] $service_name: FAILED TO RESTART"
                # Send alert here
            fi
        fi

        sleep "$check_interval"
    done
}

# Usage
# monitor_service_health nginx 60

Log Analysis

# Advanced log analysis
analyze_service_logs() {
    local service_name="$1"
    local hours_back="${2:-24}"

    echo "=== Log Analysis for $service_name ==="
    echo "Period: Last $hours_back hours"
    echo

    # Error count
    local error_count
    error_count=$(journalctl -u "$service_name" --since "$hours_back hours ago" -p err | wc -l)
    echo "Errors: $error_count"

    # Warning count
    local warn_count
    warn_count=$(journalctl -u "$service_name" --since "$hours_back hours ago" -p warning | wc -l)
    echo "Warnings: $warn_count"

    # Recent errors
    echo
    echo "Recent Errors:"
    journalctl -u "$service_name" --since "$hours_back hours ago" -p err --no-pager | tail -10

    # Service status
    echo
    echo "Current Status:"
    systemctl status "$service_name" --no-pager

    # Resource usage
    echo
    echo "Resource Usage:"
    systemctl show "$service_name" --property=CPUUsage,MemoryCurrent, TasksCurrent
}

# Usage
# analyze_service_logs nginx 24

Performance Monitoring

# Service performance monitoring
monitor_service_performance() {
    local service_name="$1"

    echo "=== Performance Monitoring for $service_name ==="

    # Get main PID
    local main_pid
    main_pid=$(systemctl show --property MainPID --value "$service_name")

    if [ "$main_pid" -eq 0 ]; then
        echo "Service not running"
        return 1
    fi

    echo "Main PID: $main_pid"
    echo

    # Memory usage
    echo "Memory Usage:"
    ps -p "$main_pid" -o pid,vsz,rss,pmem,comm

    # CPU usage
    echo
    echo "CPU Usage:"
    top -b -n 1 -p "$main_pid" | tail -1

    # Thread count
    echo
    echo "Thread Information:"
    ps -p "$main_pid" -L -o pid,lwp,nlwp,comm | head -10

    # File descriptors
    echo
    echo "File Descriptors:"
    if [ -d "/proc/$main_pid/fd" ]; then
        echo "Open FDs: $(ls "/proc/$main_pid/fd" | wc -l)"
        echo "FD Details:"
        ls -la "/proc/$main_pid/fd" | head -10
    fi
}

---

🧪 Legacy Init System Management

SysV Init Scripts

# Example SysV init script structure
#!/bin/bash
# /etc/init.d/myservice

### BEGIN INIT INFO
# Provides:          myservice
# Required-Start:    $local_fs $network
# Required-Stop:     $local_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: My Custom Service
# Description:       Longer description of the service
### END INIT INFO

. /lib/lsb/init-functions

DAEMON=/usr/local/bin/myservice
NAME=myservice
DESC="My Custom Service"
PIDFILE=/var/run/myservice.pid

test -x $DAEMON || exit 0

set -e

case "$1" in
  start)
    log_daemon_msg "Starting $DESC" "$NAME"
    start-stop-daemon --start --quiet --pidfile $PIDFILE \
      --exec $DAEMON -- $DAEMON_OPTS
    log_end_msg 0
    ;;
  stop)
    log_daemon_msg "Stopping $DESC" "$NAME"
    start-stop-daemon --stop --quiet --pidfile $PIDFILE \
      --exec $DAEMON
    rm -f $PIDFILE
    log_end_msg 0
    ;;
  restart|force-reload)
    $0 stop
    $0 start
    ;;
  status)
    if [ -f $PIDFILE ]; then
      echo "$NAME is running"
    else
      echo "$NAME is not running"
    fi
    ;;
  *)
    echo "Usage: $0 {start|stop|restart|force-reload|status}" >&2
    exit 1
    ;;
esac

exit 0

Init System Detection

# Detect current init system
detect_init_system() {
    if [ -d "/run/systemd/system" ]; then
        echo "systemd"
    elif [ -d "/etc/init" ] && command -v initctl >/dev/null 2>&1; then
        echo "upstart"
    elif [ -x "/sbin/init" ]; then
        local init_version
        init_version=$(/sbin/init --version 2>/dev/null)
        if echo "$init_version" | grep -q "upstart"; then
            echo "upstart"
        elif echo "$init_version" | grep -q "systemd"; then
            echo "systemd"
        else
            echo "sysvinit"
        fi
    else
        echo "unknown"
    fi
}

# Usage
# echo "Current init system: $(detect_init_system)"

---

🧠 Service Security Best Practices

Secure Service Configuration

# Secure systemd service template
[Unit]
Description=Secure Web Service
After=network.target
RequiresMountsFor=/var/lib/myapp

[Service]
# Process isolation
User=myapp
Group=myapp
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ReadOnlyPaths=/
ReadWritePaths=/var/lib/myapp /var/log/myapp

# Resource limits
LimitNOFILE=65536
LimitNPROC=1024
MemoryMax=512M
CPUQuota=50%

# Security features
CapabilityBoundingSet=
AmbientCapabilities=
DevicePolicy=closed
IPAddressDeny=any
IPAddressAllow=192.168.0.0/16

# Execution
WorkingDirectory=/var/lib/myapp
ExecStart=/opt/myapp/bin/server
Restart=always
RestartSec=5

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

[Install]
WantedBy=multi-user.target

Service Hardening Script

# Service hardening automation
harden_service() {
    local service_name="$1"
    local service_file="/etc/systemd/system/${service_name}.service"

    if [ ! -f "$service_file" ]; then
        echo "Service file not found: $service_file" >&2
        return 1
    fi

    echo "Hardening service: $service_name"

    # Backup original file
    sudo cp "$service_file" "${service_file}.backup.$(date +%s)"

    # Add security directives
    sudo tee -a "$service_file" > /dev/null <<EOF

# Security Hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/lib/$service_name /var/log/$service_name
CapabilityBoundingSet=
DevicePolicy=closed
EOF

    # Reload configuration
    sudo systemctl daemon-reload

    echo "Service hardened. Please review and restart: sudo systemctl restart $service_name"
}

---

🧾 Init System Comparison

Feature	SysV Init	Upstart	systemd
Startup	Sequential	Event-driven	Parallel
Dependencies	Simple	Event-based	Complex graph
Service Files	Shell scripts	Configuration files	Unit files
Logging	syslog	Built-in	Journal
Resource Mgmt	Basic	Moderate	Advanced
Container Support	None	Limited	Excellent
Adoption	Legacy	Ubuntu only	Most distros

Migration Considerations

When migrating between init systems: 1. Service file conversion - Translate configurations 2. Dependency mapping - Recreate service relationships 3. Logging adaptation - Adjust monitoring tools 4. Startup timing - Validate boot sequence 5. Security policies - Update hardening rules

---

🧾 Summary

Linux init systems have evolved from simple sequential SysV init to sophisticated parallel systemd:

Key Benefits of Modern Init Systems

• Faster boot times - Parallel service startup
• Better dependency management - Complex service graphs
• Enhanced logging - Structured journal system
• Improved security - Built-in hardening features
• Resource control - Fine-grained limits
• Container integration - Native support

systemd Advantages

• Unified management - Services, devices, sockets
• Advanced monitoring - Real-time metrics
• Flexible configuration - Templates and overrides
• Security features - Sandboxing and isolation
• Developer tools - Comprehensive debugging

Best Practices

1. Use systemd for new deployments - Industry standard
2. Implement proper service dependencies - Reliable startup
3. Configure resource limits - Prevent resource exhaustion
4. Enable security features - Protect against exploits
5. Monitor service health - Proactive maintenance
6. Document service configurations - Easy troubleshooting

👉 Continue to: Linux Namespaces and Cgroups