QEMU/KVM Network Mechanisms
Introduction
As we know, network subsystems are important in computer systems since they are I/O systems and need to be optimized with many algorithms and skills. This article will introduce how QEMU/KVM [2] network part works. In order to put everything simple and easy to understand, we will begin with several examples and then understand how it works internally.
Examples
In this example, we will use TAP device [1] as QEMU/KVM host network device driver and VirtIO driver will be used to send/receive network packets/data between Host OS and Guest OS. See following “Start QEMU/KVM virtual machine” commands in details.
sudo x86_64-softmmu/qemu-system-x86_64 /home/hkucs/colo_ubuntu_single.img -machine pc-i440fx-2.3,accel=kvm,usb=off -netdev tap,id=hn0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown -device virtio-net-pci,id=net-pci0,netdev=hn0 -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor stdio
Actually, according to my understanding, TAP device and driver are used in Host OS to send/receive packets/data to/from Gust OS since TAP driver can connect Gust OS with Internet. TAP driver will send the packets to Virtio in Host OS part and Virtio puts these packets into buffers, and at last, it notifies Virtio driver in Guest OS part to receive these packets. It has the same theory for how TAP devices receive packets from Guest OS. See following function stack for this part.
tap_send -----------> HOST TAP device driver
-> qemu_send_packet_async
-> qemu_send_packet_async_with_flags
->qemu_net_queue_send
->qemu_net_queue_deliver
->qemu_deliver_packet
->nc->info->receive
->virtio_net_receive -------------> since we use virtio-net-pci driver
->virtqueue_fill, virtqueue_flush, virtio_notify
->virtio_notify_vector
->k->notify
->virtio_pci_notify
->msix_notify
-> msi_send_message
->address_space_stl_le
->address_space_stl_internal
Actually, I think there is another example in [1]’s “Connecting VLANs To TAP Devices” part, and the function call stack should be like following.
tap_send -----------> HOST TAP device driver
-> qemu_send_packet_async
-> qemu_send_packet_async_with_flags
->qemu_net_queue_send
->qemu_net_queue_deliver
->qemu_deliver_packet
->nc->info->receive
->net_hub_port_receive ---> Start VLAN
->net_hub_receive
->qemu_send_packet
-> qemu_send_packet_async
->qemu_sendv_packet_async
...
->e1000_receive ---->NIC
Mechanisms
We need to look into mechanisms from above examples. So firstly, we will find that, in order to improve system performance, network part in QEMU/KVM will handle these packets in asynchronous way (it is also the way to be used in critical section). This is also due to these packets in memory cannot be operated by two threads simultaneously. On the other hand, these layers (TAP layer, Virtio layer, etc) are decoupled so they system design/implementation will be more flexibility and scalability. At last, Virtio is a para-virtualized technology to improve the performance for I/O systems in virtualization environments.
References
[1] https://people.gnome.org/~markmc/qemu-networking.html
[2] https://lxr.missinglinkelectronics.com/qemu+v2.4.0/