🦉 FreeBSD Debugging Tools

FreeBSD provides a comprehensive suite of debugging and diagnostic tools that allow administrators and developers to inspect system behavior, troubleshoot performance issues, and analyze application problems effectively.

---

🔍 Core Debugging Utilities

truss – System Call Tracing

FreeBSD's equivalent of Linux's strace, used for tracing system calls made by processes.

# Trace system calls of a command
truss ls

# Trace a running process
truss -p 1234

# Save output to file
truss -o trace.log ./my_program

💡 Tip: Use -f to follow child processes and -D to show syscall durations.

---

ktrace / kdump – Kernel-Level Tracing

More detailed than truss, allows tracing multiple processes and kernel activities.

# Start tracing a process
ktrace ./my_app

# View trace results
kdump

# Trace specific system calls only
ktrace -t c ./network_program

---

gdb – GNU Debugger

Same as on Linux, used for debugging compiled programs with debug symbols.

# Load program into GDB
gdb ./program

# Run with arguments
(gdb) run arg1 arg2

# Set breakpoint at main
(gdb) break main

# Show backtrace
(gdb) bt

🛠️ Ensure binary was compiled with -g flag for debug info.

---

📊 Performance Analysis Tools

top – Process Activity Monitor

Built-in tool showing real-time system resource usage.

# Interactive top view
top

# Batch mode (for scripting)
top -b -n 1

---

procstat – Process Information Utility

Provides detailed statistics about running processes including memory maps, file descriptors, and threads.

# Show basic info for PID
procstat 1234

# List open files
procstat -f 1234

# Memory mapping details
procstat -v 1234

---

vmstat – Virtual Memory Statistics

Reports virtual memory statistics, process scheduling, disk I/O, and more.

# Display every 2 seconds
vmstat 2

# Show disk-specific stats
vmstat -i

---

🗃️ Filesystem & Storage Diagnostics

fstat – File Status Information

Shows which files are currently open by processes.

# Show all open files
fstat

# Filter by process ID
fstat -p 1234

# Show network connections
fstat -n

---

iostat – I/O Statistics

Monitors disk I/O performance and utilization.

# Extended device statistics
iostat -x

# Update every 5 seconds
iostat 5

---

geom – GEOM Subsystem Control

Manages and queries the GEOM (Geometry) subsystem for disk devices.

# List all disks and partitions
geom disk list

# Show disk labels
geom label status

# View RAID configurations
geom mirror status

---

🌐 Network Troubleshooting

sockstat – Socket Statistics

Lists active sockets and associated processes.

# Show listening sockets
sockstat -l

# All IPv4 connections
sockstat -4

# Specific protocol (e.g., TCP)
sockstat -P tcp

---

tcpdump – Packet Sniffer

Captures and analyzes network traffic packets.

# Capture on default interface
tcpdump

# Filter HTTP traffic
tcpdump port 80

# Write capture to file
tcpdump -w capture.pcap

---

netstat – Network Statistics

Displays network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.

# Active connections
netstat -an

# Listening ports
netstat -an | grep LISTEN

# Interface statistics
netstat -i

---

🧠 Memory Debugging

valgrind – Memory Error Detector¹

Available via Ports or Packages, detects memory leaks and invalid accesses.

# Check for memory errors
valgrind --tool=memcheck ./my_program

# Generate suppressions
valgrind --gen-suppressions=all ./my_program

¹ Note: Must be installed separately using pkg install valgrind.

---

/proc/*/status – Process Memory Info

Access process memory information directly from procfs (if mounted).

# Check process memory usage
cat /proc/1234/status | grep VmRSS

---

🕵️‍♂️ Logs and System Messages

dmesg – Kernel Ring Buffer

Views boot-time messages and runtime kernel diagnostics.

# Recent kernel messages
dmesg | tail

# Highlight errors/warnings
dmesg -T | grep -E "(error|warning)"

---

syslogd + logger – System Logging

Standard Unix logging facilities supported in FreeBSD.

# Send custom log message
logger "Test log entry"

# View recent logs
tail /var/log/messages

---

🧰 Advanced Diagnostic Tools

DTrace – Dynamic Tracing Framework²

Powerful instrumentation framework for examining live systems.

Example probe:

syscall::open:entry
{
    printf("Opening file: %s\n", copyinstr(arg0));
}

Run with:

sudo dtrace -s script.d

² Available on most architectures except some ARM variants.

---

kgdb – Kernel Debugger³

Used for debugging the running kernel or crash dumps.

# Analyze core dump
kgdb /boot/kernel/kernel /var/crash/vmcore.0

³ Requires kernel debugging enabled and proper setup.

---

🧾 Summary Table

Tool	Purpose
truss	System call tracing
ktrace	Kernel-level process tracing
gdb	Source-level debugging
procstat	Detailed process information
fstat	Open file descriptor tracking
sockstat	Socket and connection monitoring
tcpdump	Network packet capture
valgrind	Memory error detection
dmesg	Kernel diagnostic messages
DTrace	Dynamic system-wide tracing

---

🧠 Best Practices

✅ Use truss for quick syscall troubleshooting ✅ Leverage procstat for deep process introspection ✅ Enable DTrace for production-safe observability ✅ Regularly review dmesg for hardware or driver issues ✅ Combine tools for layered diagnostics ✅ Store logs centrally for audit trails

---

🧾 See Also

• FreeBSD BSD Userland Differences
• FreeBSD RC and Jails
• FreeBSD Shell Basics
• Reference: BSD vs GNU Tools
• Patterns: Logging and Telemetry

---