How to collect VM exit numbers in KVM

VM Exit Monitoring in KVM

VM exits occur when a guest VM transitions to hypervisor mode to handle privileged operations, I/O requests, or other events. Tracking exit counts and types helps identify performance bottlenecks, inefficient guest behavior, and hypervisor overhead.

Quick Exit Statistics

The debugfs interface provides a simple view of exit counts:

cat /sys/kernel/debug/kvm/exits

This shows aggregate exit counts across all VMs on the system. The output format lists exit reason codes and their frequencies. Note that this requires CONFIG_KVM_DEBUG_STATS enabled in the kernel and debugfs mounted at /sys/kernel/debug.

If you’re running with SELinux or AppArmor, ensure your user has appropriate access to debugfs. On hardened systems, you may need to disable enforcing mode temporarily:

semanage fcontext -a -t debugfs_t "/sys/kernel/debug(/.*)?"
restorecon -Rv /sys/kernel/debug

Per-VM Exit Tracking

For per-VM statistics, check the VM-specific stats:

cat /sys/kernel/debug/kvm/*/exits

This gives you visibility into which VMs are generating the most exits. High exit rates from specific guests indicate either workload characteristics (expected) or tuning opportunities.

Detailed Tracing with perf

For performance analysis beyond raw counts, use perf to trace exit reasons with full context:

perf record -e kvm:kvm_exit -a sleep 30
perf script | grep kvm_exit | head -20

This captures exit events including the specific exit reason (e.g., EXIT_REASON_HLT, EXIT_REASON_IO_INSTRUCTION, EXIT_REASON_CPUID, EXIT_REASON_EPT_VIOLATION). Exit reasons vary by CPU architecture—x86_64 and ARM64 have different sets.

For more granular analysis, combine multiple KVM tracepoints:

perf record -e kvm:kvm_exit,kvm:kvm_entry,kvm:kvm_page_fault -a -p <pid>
perf report --stdio

This traces entries, exits, and page faults for a specific QEMU process, useful when debugging single VM performance issues.

ftrace Alternative

If perf isn’t available or you prefer kernel tracing directly:

echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_exit/enable
cat /sys/kernel/debug/tracing/trace_pipe

Capture to a file for later analysis:

echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_exit/enable
timeout 10 cat /sys/kernel/debug/tracing/trace_pipe > /tmp/kvm_exits.txt

Interpreting Exit Reasons

Common high-volume exits typically include:

• HLT: Guest CPU idle loop—usually normal and low-cost
• IO_INSTRUCTION: I/O port access requiring emulation—investigate if excessive
• EPT_VIOLATION: Memory mapping faults—high rates suggest memory pressure or suboptimal guest memory configuration
• CPUID: CPUID instruction execution—usually infrequent unless workload heavily queries CPU features
• PAUSE: Guest pause instruction—normal in spinlock scenarios, but can indicate contention

Use perf stat -e kvm:* to get a quick breakdown across your running VMs, then drill into specific reasons with perf record and perf script filtering.

Kernel Requirements

Exit statistics require:

• CONFIG_KVM_DEBUG_STATS=y in kernel config
• debugfs mounted (usually automatic on modern systems)
• Sufficient permissions (typically requires root or kvm group membership)

Check your kernel config:

grep KVM_DEBUG_STATS /boot/config-$(uname -r)

For production workloads, reserve detailed tracing for specific troubleshooting windows—continuous high-frequency KVM event tracing adds measurable overhead.