🐧 Linux Debugging Tools

Linux offers a rich ecosystem of debugging tools that help diagnose and troubleshoot system issues, performance bottlenecks, and application problems. This guide covers both built-in and third-party utilities for effective Linux debugging.

---

🔍 Core Debugging Utilities

strace – System Call Tracing

Tracks system calls made by a process, helping identify I/O issues, hangs, or unexpected behavior.

# Trace all system calls of a program
strace ls

# Save trace to file
strace -o trace.log my_program

# Follow child processes
strace -f my_daemon

💡 Tip: Use -e trace=file to focus on file-related syscalls.

---

ltrace – Library Call Tracing

Monitors dynamic library calls instead of kernel-level syscalls.

# Trace shared library calls
ltrace ./my_app

# Filter by symbol name
ltrace -e malloc ./memory_hungry_app

---

gdb – GNU Debugger

A powerful interactive debugger for compiled programs (especially C/C++).

# Start GDB session
gdb ./program

# Run with arguments
(gdb) run arg1 arg2

# Set breakpoint
(gdb) break main

# Backtrace stack
(gdb) bt

🛠️ Requires debug symbols (-g flag during compilation).

---

📊 Performance Analysis Tools

top / htop – Real-Time Process Monitoring

Displays live resource usage per process.

# Interactive top view
htop

# Sort by CPU usage
top -o %CPU

---

perf – Performance Profiling

Advanced profiling tool for analyzing CPU cycles, cache misses, branch predictions.

# Record profile data
perf record -g ./my_program

# View report
perf report

# Flame graph generation (requires extra tools)
perf script | stackcollapse-perf.pl | flamegraph.pl > perf.svg

---

vmstat, iostat, sar – System Metrics

Part of the sysstat package, these tools provide detailed metrics over time.

# Memory and swap statistics every 2 seconds
vmstat 2

# Disk I/O stats
iostat -x 5

# Collect historical data
sar -u 1 10

---

🗃️ Filesystem & Storage Diagnostics

lsof – List Open Files

Identify which processes have files open or are listening on ports.

# Show open files by process
lsof -p 1234

# Find who's using port 80
lsof -i :80

# Display network connections
lsof -iTCP -sTCP:LISTEN

---

iotop – Per-Process I/O Monitoring

Shows real-time disk I/O per process.

sudo iotop

---

df / du – Disk Space Usage

Monitor filesystem space and directory sizes.

# Human-readable disk usage summary
df -h

# Size of directories recursively
du -sh /var/log/*

---

🌐 Network Troubleshooting

ss / netstat – Socket Statistics

Check active network connections and socket states.

# Show TCP sockets in listen state
ss -tuln

# Equivalent with netstat
netstat -tulnp

---

tcpdump – Packet Sniffing

Capture and analyze raw packets on the wire.

# Capture traffic on eth0
sudo tcpdump -i eth0

# Filter HTTP traffic
sudo tcpdump -i any port 80

---

ping / traceroute / mtr – Connectivity Testing

Basic connectivity checks and path tracing.

# Test reachability
ping google.com

# Trace hop-by-hop path
traceroute google.com

# Continuous traceroute with latency stats
mtr google.com

---

🧠 Memory Debugging

valgrind – Memory Error Detection

Detect memory leaks, invalid reads/writes, and other errors.

# Check for memory leaks
valgrind --tool=memcheck --leak-check=yes ./my_program

# Generate suppression file for false positives
valgrind --gen-suppressions=all ./my_program

---

/proc/meminfo – Kernel Memory Info

Access low-level memory info directly from procfs.

cat /proc/meminfo | grep -E "(MemTotal|MemFree|Buffers|Cached)"

---

🕵️‍♂️ Logs and Journaling

journalctl – systemd Journal Viewer

Query logs managed by systemd journal.

# View recent logs
journalctl -n 100

# Tail logs like tail -f
journalctl -f

# Filter logs for service
journalctl -u nginx.service

---

dmesg – Kernel Ring Buffer

View kernel boot messages and hardware diagnostics.

# Recent kernel messages
dmesg | tail

# Highlight errors
dmesg -T --level=err,warn

---

🧰 Advanced Toolkits

BCC (BPF Compiler Collection)

Use eBPF probes for advanced tracing without modifying code.

Example: trace slow file opens:

opensnoop -p PID

Install via package manager:

sudo apt install bpfcc-tools

---

htop vs atop – Enhanced Monitoring

While htop gives visual clarity, atop stores historical snapshots for post-mortem analysis.

# Run atop interactively
atop

# Read previous day's snapshot
atop -r /var/log/atop/atop_20250404

---

🧾 Summary Table

Tool	Purpose
strace	Syscall tracing
ltrace	Library call tracing
gdb	Interactive source-level debugger
perf	CPU profiling
htop	Live process viewer
lsof	Open files and ports
tcpdump	Packet capture
valgrind	Memory leak detection
journalctl	systemd log inspection
dmesg	Kernel diagnostic output

---

🧠 Best Practices

✅ Always verify your assumptions about system behavior ✅ Prefer structured logging in applications for easier diagnosis ✅ Combine multiple tools for layered insights ✅ Use containers/namespaces to isolate experiments ✅ Store logs securely and rotate regularly

---

🧾 See Also

• Linux Proc and FS Semantics
• Linux Namespaces and CGroups
• Patterns: Logging and Telemetry
• Anti-Patterns: Silent Failures

---