I find there are too many TCP segments retransmited in a Xen virtual machine.
sudo netstat -s | grep segments 537559 segments received 558908 segments send out 3533 segments retransmited 2677 bad segments received.
There are 4 VMs on a host which has 4 cores. The segment retransmision rate is too high.
It finally turns out that there are too many VMs on the same host. Adjusting the number of VMs on one single host to 3 makes the tcp segments retransmision rate much lower. My guess of the reason is that the CPU turns to the bottleneck (not confirmed).