Using Open vSwitch with libvirt and QEMU/KVM

[UPDATE 7/21/2012]

I have written an updated blog post on Libvirt/Open vSwitch intergration now that Libvirt supports Open vSwitch directly since 0.9.11. If your libvirt isn't version 0.9.11 or higher you will still need to use bridge compatibility mode as described below. Otherwise check out the new post http://binarybitme.blogspot.com/2012/07/libvirt-0911-has-support-for-open.html

----

This is a quick and dirty write-up of using Open vSwitch(OVS) with libvirt and QEMU/KVM. I assume you already have libvirt(http://libvirt.org) up and running with QEMU/KVM. Other hypervisors may work but I haven't tried them at this time.

Grab the master of Open vSwitch(http://openvswitch.org/) with git because in previous versions there is an issue with removing ports from a fake bridge when the TAP is just deleted without first removing it from the bridge. This is what happens with libvirt and KVM when a VM is shutdown. This is fixed in the OVS master. The fix will also be in versions 1.2.2, 1.3.0,  and later when released.

1) Compile and install OVS by following INSTALL.Linux

2) Compile and install OVS bridge compatibility mode by following INSTALL.bridge

Now you should have a working OVS install with bridge compatibility mode.

3) Create a fake bridge for each VLAN you want available. So say you have br0 as your real bridge and you want a VLAN 101.

ovs-vsctl add-br br101 br0 101

This will create a fake bridge tagged with VLAN 101. Do this for every VLAN you want.

4) In libvirt use a bridge interface setting the source to the appropriate fake bridge.

For example if you want a virtual machine on VLAN 101 assign it to br101

You can add the network interface by editing domain's XML directly with virsh.

<interface type='bridge'>
  <source bridge='br101'/>
</interface>

Add this in the <devices> section. Libvirt will fill in the <mac/> and <address/>.

You can also add the network interface with virt-manager.

Now when you boot the VM libvirt will create the TAP interface and add it to br101. Since bridge compatibility mode is running OVS will add the port tagged with the appropriate VLAN when libvirt adds the interface to the bridge.

After starting the VM you should see the associated TAP interface when you list the ports in OVS.

ovs-vsctl list port

It should have the appropriate tag. In this case the tag should be 101.

Notice the port will not show up under br0 if you list-ports. You will have to list-ports on the fake bridge

ovs-vsctl list-ports br101

When you shutdown the VM the TAP interface will be destroyed and the port will be removed from OVS. If the TAP interface is destroyed, but the port still exists in OVS then you are not using a new enough version of OVS.

This will not help with mirrors or other advanced features but will give you the basic ability to automatically add and remove ports on the virtual switch with the appropriate VLAN tags for your virtual machines' interfaces.

I am working on a helper service that will enable the ability to automatically handle mirrors and other advanced functionality. It will also allow you to avoid using bridge compatibility mode if you desire. Eventually I hope to see OVS support built into libvirt but this will help until then.

pfSense in a Virtualized Environment

Just a quick note on pfSense in a virtualized environment. For whatever reason pfSense hangs during installation in a VM with default VM settings on all hypervisors I have tried. I have seen this on 1.2.3 and 2.0-RC1 on VirtualBox, KVM, and Xen. I have not tested with pfSense 2.0-RC3. The way I found to get around this is to put the cdrom and the harddisk on separate virtual disk controllers. For example sticking one on a SCSI and the other IDE seems to work around the issue. I have tested this in both VBox and KVM.

If you haven't checked out pfSense it complimants Open VSwitch quite well in a virtualized lab environment as a LAN router and firewall. My favorite firewall/router distribution I have used hands down.

VirtualBox 4 Networking Notes

I recently setup a lab environment on older hardware that didn't have VT-x. For this setup I used VirtualBox 4 headless, phpVirtualBox, and Open VSwitch. I have a few notes worth documenting in one place.

1. If you are trunking to a VBox VM use the vnic type of AMD PCNet PCI II (Am79C970A). This is not the default vnic type. I have been able to trunk to Linux and FreeBSD guests including pfSense using the PCNet II vnic with a trunk from Open VSwitch with no issues. Every other vnic type I tried didn't work for me. I haven't tried the virtio drivers on VBox they may work as well. I do know the virtio drivers work when trunking to a Linux VM on KVM.
2. If you are setting the destination of a mirror in Open VSwitch to a VM vnic on VBox you will need to set extra data on the vnic to allow promisc mode to be enabled.
   
   VBoxManage setextradata VM_NAME \
   "VBoxInternal/Devices/pcnet/0/LUN#0/Config/IfPolicyPromisc" "allow-all"
   
   Replace pcnet with the appropriate vnic type you are using.
3. VBox comes with VBoxTunctl so you don't need to install tunctl seperatly to create tap interfaces for use with Open VSwitch.

A lot of people don't think of VBox except for desktop use but it works very well in a headless server environment. Especially when combined with phpVirtualBox for quick and easy managment and VM console access.

As a side note you don't get 64 bit guests or hardware breakpoints while debugging in a guest without VT-x on VirtualBox :-( but at least you get full virtualization with good performance :-).

Not entirely related but I would love to see enhanced protection such as sVirt like functionally developed for libvirt/KVM on a system with SELinux or Apparmor available. That would go a long why to help protect the host and other guests if an escape from a guest was achieved.