KVM is based on virtualization instruction set – either Intel vmx or AMD svm, which provides the ability to run the VM directly without emulation or translation. In other words, KVM could only be enabled if it sees vmx/svm in /proc/cpuinfo. By default, there is not way to run a nested KVM within a KVM guest, as the CPU of KVM guest does not have vmx/svm. This post introduces a general way to have nested KVM on Fedora 20. Have fun!

0. Host (Fedora 20, x86_64)

Make sure your CPU has vmx/svm for KVM – ‘cat /proc/cpuinfo | grep vmx‘. If lucky, move on installing KVM – ‘yum install kvm‘. Then we have to enable the nested KVM feature, which is disabled by default (Run ‘cat /sys/module/kvm_intel/parameters/nested‘ – ‘N’). To turn it on (ref a):
echo “options kvm-intel nested=1” | sudo tee /etc/modprobe.d/kvm-intel.conf
(Run ‘cat /sys/module/kvm_intel/parameters/nested’ – ‘Y’)

1. KVM Guest (fedora20kvm, x86_64)

Within the host, use ‘virt-install’ (CLI) or ‘virtual machine manager’ (GUI) to install a Fedora 20 VM named fedora20kvm. It does not matter what is the hard configuration. Just be aware that we will create a nested KVM guest within it. So make sure we will have enough memory and disk space. Once created, we can login our KVM Guest and ‘cat /proc/cpuinfo | grep vmx’ and won’t find it. Installing KVM on the guest currently is not possible. Now it is time to expose the vmx instruction set to the KVM Guest. BTW, I am using the Intel CPU and will talk about vmx in the following. If AMD is the target, replace it with svm.

Run ‘virsh edit fedora20kvm’ to edit the XML configuration for our KVM guest fedora20kvm. Find the CPU tag and add the line in green, which will expose the vmx to the KVM guest (ref a). Note that this XML is usually under /etc/libvirt/qemu/. Manually editing also works.

<cpu mode=’custom’ match=’exact’>
<model fallback=’allow’>SandyBridge</model>
<feature policy=’require’ name=’vmx’/>
</cpu>

After fedora20 is booted, ‘cat /proc/cpuinfo | grep vmx’ will find it – Yes!

2. nested KVM Guest (fedora20kvmkvm, x86_64)

Within the KVM guest (fedora20kvm), we can install KVM now. Once KVM is there, we can go ahead to create a nested KVM guest named fedora20kvmkvm, like what we have done for the KVM guest. However, we may find that we are not able to create/start this fedora20kvmkvm and KVM complains “Network is already in use by interface by eth0“. You may also find the error in /var/log/libvirt/libvirtd.log if you are using Ubuntu. Before we fix this, we needs few words to make clear.

The networking among KVM guests, the host and the Internet is implemented via a bridge called ‘virbr0‘ (or ‘br0’ in older releases). This ‘virbr0’ is a bridge usually to a ‘real’ interface, like ‘eth0’, providing the NAT functionality between the host IP and KVM guest IP. The IP of ‘virbr0’ by default is 192.168.122.1 and all the KVM guests created based on ‘virbr0’ has the IP format ‘192.168.122.X’, where X is in range [2, 254].

Once KVM is installed, libvirtd would try to create this bridge/interface after the system is booted. The bridge would NOT be created if it collides with its host routing (ref b). Running ‘ifconfig’, indeed, we are not able to find ‘virbr0’ within the KVM guest. The reason is simple, within the KVM guest, ‘eth0’ IP is eventually based on ‘virbr0’ of the host, which is something like ‘192.168.122.Y’. Within the route table of the KVM guest (‘cat /proc/net/route‘), there is a routing entry with destination IP ‘192.168.122.0’ and netmask ‘255.255.255.0’ for ‘eth0’. Unfortunately, libvirtd of the KVM guest was trying to create a ‘virbr0’, which would have the same routing entry but with different interface name! It failed to do that.

Now we will fix that. Let us find the default configuration file for ‘virbr0’ used by libvirtd – /etc/libvirt/qemu/networks/default.xml. By default, it looks like below:

<network>
<name>default</name>
<uuid>dc0ed217-7e39-42c4-b1e0-f2112b9bae58</uuid>
<bridge name=”virbr0″ />
<forward/>
<ip address=”192.168.122.1” netmask=”255.255.255.0″>
<dhcp>
<range start=”192.168.122.2” end=”192.168.122.254” />
</dhcp>
</ip>
</network>

Now let’s change it using some other IP block, like 10.0.0.0:

<network>
<name>default</name>
<uuid>dc0ed217-7e39-42c4-b1e0-f2112b9bae58</uuid>
<bridge name=”virbr0″/>
<forward/>
<ip address=”10.0.0.1” netmask=”255.255.255.0″>
<dhcp>
<range start=”10.0.0.2” end=”10.0.0.254“/>
</dhcp>
</ip>
</network>

Reboot your KVM guest to make sure the changes work. After rebooting, you will find the missing ‘virbr0’ and you should be able to create/start a nested KVM – fedora20kvmkvm – Yes!

3. ifconfig

3.1 Host (Fedora 20)

[root@daveti daveti]# ifconfig
em1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
inet 184.171.61.83  netmask 255.255.254.0  broadcast 184.171.61.255
inet6 fe80::1a03:73ff:fe25:3571  prefixlen 64  scopeid 0x20<link>
inet6 2001:468:d01:c6:1a03:73ff:fe25:3571  prefixlen 64  scopeid 0x0<global>
ether 18:03:73:25:35:71  txqueuelen 1000  (Ethernet)
RX packets 1636026  bytes 2213024469 (2.0 GiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 696327  bytes 52861586 (50.4 MiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
device interrupt 20  memory 0xf3500000-f3520000

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
inet 127.0.0.1  netmask 255.0.0.0
inet6 ::1  prefixlen 128  scopeid 0x10<host>
loop  txqueuelen 0  (Local Loopback)
RX packets 731874  bytes 2290638009 (2.1 GiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 731874  bytes 2290638009 (2.1 GiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
ether 52:54:00:a7:cd:75  txqueuelen 0  (Ethernet)
RX packets 459232  bytes 32469028 (30.9 MiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 634892  bytes 2123001894 (1.9 GiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vmnet1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
inet 172.16.39.1  netmask 255.255.255.0  broadcast 172.16.39.255
inet6 fe80::250:56ff:fec0:1  prefixlen 64  scopeid 0x20<link>
ether 00:50:56:c0:00:01  txqueuelen 1000  (Ethernet)
RX packets 115  bytes 0 (0.0 B)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 6059  bytes 0 (0.0 B)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vmnet8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
inet 192.168.232.1  netmask 255.255.255.0  broadcast 192.168.232.255
inet6 fe80::250:56ff:fec0:8  prefixlen 64  scopeid 0x20<link>
ether 00:50:56:c0:00:08  txqueuelen 1000  (Ethernet)
RX packets 115  bytes 0 (0.0 B)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 6054  bytes 0 (0.0 B)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
inet6 fe80::fc54:ff:fed9:177a  prefixlen 64  scopeid 0x20<link>
ether fe:54:00:d9:17:7a  txqueuelen 500  (Ethernet)
RX packets 229738  bytes 16326182 (15.5 MiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 364690  bytes 1533862479 (1.4 GiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@daveti daveti]#

3.2 KVM Guest (fedora20kvm)

[root@localhost networks]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
inet 192.168.122.172  netmask 255.255.255.0  broadcast 0.0.0.0
inet6 fe80::5054:ff:fed9:177a  prefixlen 64  scopeid 0x20<link>
ether 52:54:00:d9:17:7a  txqueuelen 1000  (Ethernet)
RX packets 151421  bytes 322443953 (307.5 MiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 104436  bytes 7136384 (6.8 MiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
inet 127.0.0.1  netmask 255.0.0.0
inet6 ::1  prefixlen 128  scopeid 0x10<host>
loop  txqueuelen 0  (Local Loopback)
RX packets 87268  bytes 289788554 (276.3 MiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 87268  bytes 289788554 (276.3 MiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
inet 10.0.0.1  netmask 255.255.255.0  broadcast 10.0.0.255
ether fe:54:00:ff:01:0c  txqueuelen 0  (Ethernet)
RX packets 104648  bytes 5653017 (5.3 MiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 140706  bytes 321786351 (306.8 MiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
inet6 fe80::fc54:ff:feff:10c  prefixlen 64  scopeid 0x20<link>
ether fe:54:00:ff:01:0c  txqueuelen 500  (Ethernet)
RX packets 104393  bytes 7094414 (6.7 MiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 149982  bytes 322261530 (307.3 MiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@localhost networks]#

3.3 nested KVM Guest (fedora20kvmkvm)

[daveti@localhost ~]$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
inet 10.0.0.34  netmask 255.255.255.0  broadcast 0.0.0.0
inet6 fe80::5054:ff:feff:10c  prefixlen 64  scopeid 0x20<link>
ether 52:54:00:ff:01:0c  txqueuelen 1000  (Ethernet)
RX packets 150045  bytes 322268521 (307.3 MiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 104432  bytes 7099768 (6.7 MiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
inet 127.0.0.1  netmask 255.0.0.0
inet6 ::1  prefixlen 128  scopeid 0x10<host>
loop  txqueuelen 0  (Local Loopback)
RX packets 4  bytes 340 (340.0 B)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 4  bytes 340 (340.0 B)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[daveti@localhost ~]$

4. Something about Ubuntu…

If you are using Ubuntu, be aware. I was trying to set up a nested KVM following the similar procedure using Ubuntu 12.04.4 x86_64 but failed finally. The reason is changing default.xml for ‘virbr0’ did not work. I assume Ubuntu was using some buggy libvirt.

5. Reference

a. http://www.rdoxenham.com/?p=275
b. http://www.redhat.com/archives/libvir-list/2010-May/msg01088.html
c. http://kashyapc.com/2012/01/14/nested-virtualization-with-kvm-intel/
d. http://lxr.linux.no/linux+v3.13.5/Documentation/virtual/kvm/nested-vmx.txt