🦉 OpenBSD Debugging Tools

OpenBSD emphasizes security and simplicity, providing a focused set of debugging tools that align with its "secure by default" philosophy. These utilities offer essential diagnostics while maintaining the system's minimalist approach.

---

🔍 Core Debugging Utilities

ktrace / kdump – Kernel-Level Tracing

OpenBSD's primary tracing mechanism for monitoring system calls and signals.

# Start tracing a process
ktrace ./my_app

# View trace results
kdump

# Trace specific system calls only
ktrace -t c ./network_program

# Follow child processes
ktrace -f ./daemon_program

---

strace – System Call Tracing¹

Third-party utility similar to Linux's strace (available via packages).

# Trace system calls of a command
strace ls

# Save trace output to file
strace -o trace.log ./my_program

¹ Install via: pkg_add strace

---

gdb – GNU Debugger

Standard debugger for analyzing compiled programs with debug symbols.

# Load program into GDB
gdb ./program

# Run with arguments
(gdb) run arg1 arg2

# Set breakpoint at main
(gdb) break main

# Show backtrace
(gdb) bt

---

📊 Performance Analysis Tools

top – Process Activity Monitor

Built-in tool showing real-time system resource usage.

# Interactive top view
top

# Batch mode for scripting
top -b -n 1

---

vmstat – Virtual Memory Statistics

Reports virtual memory, process scheduling, disk I/O, and interrupt statistics.

# Display every 2 seconds
vmstat 2

# Show disk-specific stats
vmstat -i

---

systat – Dynamic System Statistics

Interactive system monitoring with multiple display modes.

# Start systat
systat

# Switch views (e.g., network, disk, memory)
:network
:disk
:malloc

---

🗃️ Filesystem & Storage Diagnostics

fstat – File Status Information

Shows which files are currently open by processes.

# Show all open files
fstat

# Filter by process ID
fstat -p 1234

# Show network connections
fstat -n

---

iostat – I/O Statistics

Monitors disk I/O performance and device utilization.

# Extended device statistics
iostat -x

# Update every 5 seconds
iostat 5

---

df / du – Disk Space Usage

Monitor filesystem space and directory sizes.

# Human-readable disk usage summary
df -h

# Size of directories recursively
du -sh /var/log/*

---

🌐 Network Troubleshooting

sockstat – Socket Statistics

Lists active sockets and associated processes.

# Show listening sockets
sockstat -l

# All IPv4 connections
sockstat -4

# Specific protocol
sockstat -P tcp

---

netstat – Network Statistics

Displays network connections, routing tables, and interface statistics.

# Active connections
netstat -an

# Listening ports
netstat -an | grep LISTEN

# Interface statistics
netstat -i

---

tcpdump – Packet Sniffer

Captures and analyzes network traffic packets.

# Capture on default interface
tcpdump

# Filter HTTP traffic
tcpdump port 80

# Write capture to file
tcpdump -w capture.pcap

---

🧠 Memory Debugging

valgrind – Memory Error Detector¹

Detects memory leaks and invalid memory accesses.

# Check for memory errors
valgrind --tool=memcheck ./my_program

# Generate suppressions
valgrind --gen-suppressions=all ./my_program

¹ Available via packages: pkg_add valgrind

---

/proc/*/status – Process Memory Info

Access process memory information directly from procfs (if mounted).

# Check process memory usage
cat /proc/1234/status | grep VmRSS

---

🕵️‍♂️ Logs and System Messages

dmesg – Kernel Ring Buffer

Views boot-time messages and runtime kernel diagnostics.

# Recent kernel messages
dmesg | tail

# Highlight errors/warnings
dmesg -T | grep -E "(error|warning)"

---

syslogd + logger – System Logging

Standard Unix logging facilities supported in OpenBSD.

# Send custom log message
logger "Test log entry"

# View recent logs
tail /var/log/messages

---

🧰 Advanced Diagnostic Tools

DDB – In-Kernel Debugger

Built-in kernel debugger for live system analysis or crash dump examination.

Enable with kernel option:

options DDB

Common commands:

# Show current backtrace
bt

# Continue execution
continue

# Show registers
show registers

# Examine memory
x/10x 0xffffffff80000000

---

crash – Crash Dump Analyzer²

Analyzes kernel crash dumps for post-mortem debugging.

# Analyze crash dump
crash /var/crash/kern.core

² Requires crash dump configuration in /etc/rc.conf.local

---

🧾 Summary Table

Tool	Purpose
ktrace	Kernel-level process tracing
strace	System call tracing (external)
gdb	Source-level debugging
fstat	Open file descriptor tracking
sockstat	Socket and connection monitoring
tcpdump	Network packet capture
valgrind	Memory error detection
dmesg	Kernel diagnostic messages
DDB	In-kernel debugger

---

🧠 Best Practices

✅ Use ktrace/kdump as primary tracing mechanism ✅ Leverage fstat for identifying file handle leaks ✅ Monitor system resources with vmstat and iostat ✅ Regularly review dmesg for hardware compatibility issues ✅ Combine tools for cross-layer diagnostics ✅ Keep logs organized for audit purposes

---

🧾 See Also

• OpenBSD Pledge and Unveil
• OpenBSD RC and Daemons
• OpenBSD Security Model
• OpenBSD Shell Basics
• Reference: BSD vs GNU Tools
• Patterns: Logging and Telemetry

---