Libvirt xml rpc error

A.19.1. libvirtd failed to start

Symptom

The libvirt daemon does not start automatically. Starting the libvirt daemon manually fails as well:

# systemctl start libvirtd.service
* Caching service dependencies ...                                                                                             [ ok ]
* Starting libvirtd ...
/usr/sbin/libvirtd: error: Unable to initialize network sockets. Check /var/log/messages or run without --daemon for more info.
* start-stop-daemon: failed to start `/usr/sbin/libvirtd'                                                                      [ !! ]
* ERROR: libvirtd failed to start

Moreover, there is not 'more info' about this error in /var/log/messages.

Investigation

Change libvirt’s logging in /etc/libvirt/libvirtd.conf by enabling the line below. To enable the setting the line, open the /etc/libvirt/libvirtd.conf file in a text editor, remove the hash (or #) symbol from the beginning of the following line, and save the change:

log_outputs="3:syslog:libvirtd"

This line is commented out by default to prevent libvirt from producing excessive log messages. After diagnosing the problem, it is recommended to comment this line again in the /etc/libvirt/libvirtd.conf file.

Restart libvirt to determine if this has solved the problem.

If libvirtd still does not start successfully, an error similar to the following will be printed:

# systemctl restart libvirtd
Job for libvirtd.service failed because the control process exited with error code. See "systemctl status libvirtd.service" and "journalctl -xe" for details.

Sep 19 16:06:02 jsrh libvirtd[30708]: 2017-09-19 14:06:02.097+0000: 30708: info : libvirt version: 3.7.0, package: 1.el7 (Unknown, 2017-09-06-09:01:55, js
Sep 19 16:06:02 jsrh libvirtd[30708]: 2017-09-19 14:06:02.097+0000: 30708: info : hostname: jsrh
Sep 19 16:06:02 jsrh libvirtd[30708]: 2017-09-19 14:06:02.097+0000: 30708: error : daemonSetupNetworking:502 : unsupported configuration: No server certif
Sep 19 16:06:02 jsrh systemd[1]: libvirtd.service: main process exited, code=exited, status=6/NOTCONFIGURED
Sep 19 16:06:02 jsrh systemd[1]: Failed to start Virtualization daemon.

-- Subject: Unit libvirtd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit libvirtd.service has failed.
--
-- The result is failed.

The libvirtd man page shows that the missing cacert.pem file is used as TLS authority when libvirt is run in Listen for TCP/IP connections mode. This means the --listen parameter is being passed.

Solution

Configure the libvirt daemon’s settings with one of the following methods:

• Install a CA certificate.

• Do not use TLS; use bare TCP instead. In /etc/libvirt/libvirtd.conf set listen_tls = 0 and listen_tcp = 1. The default values are listen_tls = 1 and listen_tcp = 0.

• Do not pass the --listen parameter. In /etc/sysconfig/libvirtd.conf change the LIBVIRTD_ARGS variable.

A.19.2. The URI Failed to Connect to the Hypervisor

A.19.3. PXE Boot (or DHCP) on Guest Failed

Symptom

A guest virtual machine starts successfully, but is then either unable to acquire an IP address from DHCP or boot using the PXE protocol, or both. There are two common causes of this error: having a long forward delay time set for the bridge, and when the iptables package and kernel do not support checksum mangling rules.

Long forward delay time on bridge

Investigation

This is the most common cause of this error. If the guest network interface is connecting to a bridge device that has STP (Spanning Tree Protocol) enabled, as well as a long forward delay set, the bridge will not forward network packets from the guest virtual machine onto the bridge until at least that number of forward delay seconds have elapsed since the guest connected to the bridge. This delay allows the bridge time to watch traffic from the interface and determine the MAC addresses behind it, and prevent forwarding loops in the network topology.

If the forward delay is longer than the timeout of the guest’s PXE or DHCP client, the client’s operation will fail, and the guest will either fail to boot (in the case of PXE) or fail to acquire an IP address (in the case of DHCP).

Solution

If this is the case, change the forward delay on the bridge to 0, disable STP on the bridge, or both.

This solution applies only if the bridge is not used to connect multiple networks, but just to connect multiple endpoints to a single network (the most common use case for bridges used by libvirt).

If the guest has interfaces connecting to a libvirt-managed virtual network, edit the definition for the network, and restart it. For example, edit the default network with the following command:

# virsh net-edit default

Add the following attributes to the <bridge> element:

<name_of_bridge='virbr0' delay='0' stp='on'/>

delay='0' and stp='on' are the default settings for virtual networks, so this step is only necessary if the configuration has been modified from the default.

If the guest interface is connected to a host bridge that was configured outside of libvirt, change the delay setting.

Add or edit the following lines in the /etc/sysconfig/network-scripts/ifcfg-name_of_bridge file to turn STP on with a 0 second delay:

STP=on DELAY=0

After changing the configuration file, restart the bridge device:

/usr/sbin/ifdown name_of_bridge
/usr/sbin/ifup name_of_bridge

If name_of_bridge is not the root bridge in the network, that bridge’s delay will be eventually reset to the delay time configured for the root bridge. To prevent this from occurring, disable STP on name_of_bridge.

The iptables package and kernel do not support checksum mangling rules

Investigation

This message is only a problem if all four of the following conditions are true:

• The guest is using virtio network devices.

  If so, the configuration file will contain model type='virtio'

• The host has the vhost-net module loaded.

  This is true if ls /dev/vhost-net does not return an empty result.

• The guest is attempting to get an IP address from a DHCP server that is running directly on the host.

• The iptables version on the host is older than 1.4.10.

  iptables 1.4.10 was the first version to add the libxt_CHECKSUM extension. This is the case if the following message appears in the libvirtd logs:

  warning: Could not add rule to fixup DHCP response checksums on network default
  warning: May need to update iptables package and kernel to support CHECKSUM rule.

  Unless all of the other three conditions in this list are also true, the above warning message can be disregarded, and is not an indicator of any other problems.

When these conditions occur, UDP packets sent from the host to the guest have uncomputed checksums. This makes the host’s UDP packets seem invalid to the guest’s network stack.

Solution

To solve this problem, invalidate any of the four points above. The best solution is to update the host iptables and kernel to iptables-1.4.10 or newer where possible. Otherwise, the most specific fix is to disable the vhost-net driver for this particular guest. To do this, edit the guest configuration with this command:

virsh edit name_of_guest

Change or add a <driver> line to the <interface> section:

<interface type='network'>
  <model type='virtio'/>
  <driver name='qemu'/>
  ...
</interface>

Save the changes, shut down the guest, and then restart it.

If this problem is still not resolved, the issue may be due to a conflict between firewalld and the default libvirt network.

To fix this, stop firewalld with the service firewalld stop command, then restart libvirt with the service libvirtd restart command.

In addition, if the /etc/sysconfig/network-scripts/ifcfg-network_name file is configured correctly, you can ensure that the guest acquires an IP address by using the dhclient command as root on the guest.

A.19.4. Guest Can Reach Outside Network, but Cannot Reach Host When Using macvtap interface

Symptom

A guest virtual machine can communicate with other guests, but cannot connect to the host machine after being configured to use a macvtap (also known as type='direct') network interface.

Investigation

Even when not connecting to a Virtual Ethernet Port Aggregator (VEPA) or VN-Link capable switch, macvtap interfaces can be useful. Setting the mode of such an interface to bridge allows the guest to be directly connected to the physical network in a very simple manner without the setup issues (or NetworkManager incompatibility) that can accompany the use of a traditional host bridge device.

However, when a guest virtual machine is configured to use a type='direct' network interface such as macvtap, despite having the ability to communicate with other guests and other external hosts on the network, the guest cannot communicate with its own host.

This situation is actually not an error — it is the defined behavior of macvtap. Due to the way in which the host’s physical Ethernet is attached to the macvtap bridge, traffic into that bridge from the guests that is forwarded to the physical interface cannot be bounced back up to the host’s IP stack. Additionally, traffic from the host’s IP stack that is sent to the physical interface cannot be bounced back up to the macvtap bridge for forwarding to the guests.

Solution

Use libvirt to create an isolated network, and create a second interface for each guest virtual machine that is connected to this network. The host and guests can then directly communicate over this isolated network, while also maintaining compatibility with NetworkManager.

Procedure A.8. Creating an isolated network with libvirt

1. Add and save the following XML in the /tmp/isolated.xml file. If the 192.168.254.0/24 network is already in use elsewhere on your network, you can choose a different network.

   ...
   <network>
     <name>isolated</name>
     <ip address='192.168.254.1' netmask='255.255.255.0'>
       <dhcp>
         <range start='192.168.254.2' end='192.168.254.254'/>
       </dhcp>
     </ip>
   </network>
   ...
   

   Figure A.3. Isolated Network XML

2. Create the network with this command: virsh net-define /tmp/isolated.xml

3. Set the network to autostart with the virsh net-autostart isolated command.

4. Start the network with the virsh net-start isolated command.

5. Using virsh edit name_of_guest, edit the configuration of each guest that uses macvtap for its network connection and add a new <interface> in the <devices> section similar to the following (note the <model type='virtio'/> line is optional to include):

   ...
   <interface type='network' trustGuestRxFilters='yes'>
     <source network='isolated'/>
     <model type='virtio'/>
   </interface>
   

   Figure A.4. Interface Device XML

6. Shut down, then restart each of these guests.

The guests are now able to reach the host at the address 192.168.254.1, and the host will be able to reach the guests at the IP address they acquired from DHCP (alternatively, you can manually configure the IP addresses for the guests). Since this new network is isolated to only the host and guests, all other communication from the guests will use the macvtap interface. For more information, see Section 23.17.8, “Network Interfaces”.

A.19.5. Could not add rule to fixup DHCP response checksums on network ‘default’

Symptom

This message appears:

Could not add rule to fixup DHCP response checksums on network 'default'

Investigation

Although this message appears to be evidence of an error, it is almost always harmless.

Solution

Unless the problem you are experiencing is that the guest virtual machines are unable to acquire IP addresses through DHCP, this message can be ignored.

A.19.6. Unable to add bridge br0 port vnet0: No such device

Symptom

The following error message appears:

Unable to add bridge name_of_bridge port vnet0: No such device

For example, if the bridge name is br0, the error message appears as:

Unable to add bridge br0 port vnet0: No such device

In libvirt versions 0.9.6 and earlier, the same error appears as:

Failed to add tap interface to bridge name_of_bridge: No such device

Or for example, if the bridge is named br0:

Failed to add tap interface to bridge 'br0': No such device

Investigation

Both error messages reveal that the bridge device specified in the guest’s (or domain’s) <interface> definition does not exist.

To verify the bridge device listed in the error message does not exist, use the shell command ip addr show br0.

A message similar to this confirms the host has no bridge by that name:

br0: error fetching interface information: Device not found

If this is the case, continue to the solution.

However, if the resulting message is similar to the following, the issue exists elsewhere:

br0        Link encap:Ethernet  HWaddr 00:00:5A:11:70:48
           inet addr:10.22.1.5  Bcast:10.255.255.255  Mask:255.0.0.0
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:249841 errors:0 dropped:0 overruns:0 frame:0
           TX packets:281948 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
	   RX bytes:106327234 (101.4 MiB)  TX bytes:21182634 (20.2 MiB)

Solution

Edit the existing bridge or create a new bridge with virsh

Use virsh to either edit the settings of an existing bridge or network, or to add the bridge device to the host system configuration.

Edit the existing bridge settings using virsh

Use virsh edit name_of_guest to change the <interface> definition to use a bridge or network that already exists.

For example, change type='bridge' to type='network', and <source bridge='br0'/> to <source network='default'/>.

Create a host bridge using virsh

For libvirt version 0.9.8 and later, a bridge device can be created with the virsh iface-bridge command. This creates a bridge device br0 with eth0, the physical network interface that is set as part of a bridge, attached:

virsh iface-bridge eth0 br0

Optional: If needed, remove this bridge and restore the original eth0 configuration with this command:

virsh iface-unbridge br0

Create a host bridge manually

A.19.7. Migration Fails with error: unable to resolve address

Symptom

QEMU guest migration fails and this error message appears:

# virsh migrate qemu qemu+tcp://192.168.122.12/system
  error: Unable to resolve address name_of_host service '49155': Name or service not known

For example, if the destination host name is newyork, the error message appears as:

# virsh migrate qemu qemu+tcp://192.168.122.12/system
error: Unable to resolve address 'newyork' service '49155': Name or service not known

However, this error looks strange as we did not use newyork host name anywhere.

Investigation

During migration, libvirtd running on the destination host creates a URI from an address and port where it expects to receive migration data and sends it back to libvirtd running on the source host.

In this case, the destination host (192.168.122.12) has its name set to ‘newyork’. For some reason, libvirtd running on that host is unable to resolve the name to an IP address that could be sent back and still be useful. For this reason, it returned the ‘newyork’ host name hoping the source libvirtd would be more successful with resolving the name. This can happen if DNS is not properly configured or /etc/hosts has the host name associated with local loopback address (127.0.0.1).

Note that the address used for migration data cannot be automatically determined from the address used for connecting to destination libvirtd (for example, from qemu+tcp://192.168.122.12/system). This is because to communicate with the destination libvirtd, the source libvirtd may need to use network infrastructure different from the type that virsh (possibly running on a separate machine) requires.

Solution

The best solution is to configure DNS correctly so that all hosts involved in migration are able to resolve all host names.

If DNS cannot be configured to do this, a list of every host used for migration can be added manually to the /etc/hosts file on each of the hosts. However, it is difficult to keep such lists consistent in a dynamic environment.

If the host names cannot be made resolvable by any means, virsh migrate supports specifying the migration host:

# virsh migrate qemu qemu+tcp://192.168.122.12/system tcp://192.168.122.12

Destination libvirtd will take the tcp://192.168.122.12 URI and append an automatically generated port number. If this is not desirable (because of firewall configuration, for example), the port number can be specified in this command:

# virsh migrate qemu qemu+tcp://192.168.122.12/system tcp://192.168.122.12:12345

Another option is to use tunneled migration. Tunneled migration does not create a separate connection for migration data, but instead tunnels the data through the connection used for communication with destination libvirtd (for example, qemu+tcp://192.168.122.12/system):

# virsh migrate qemu qemu+tcp://192.168.122.12/system --p2p --tunnelled

A.19.8. Migration Fails with Unable to allow access for disk path: No such file or directory

Symptom

A guest virtual machine (or domain) cannot be migrated because libvirt cannot access the disk image(s):

# virsh migrate qemu qemu+tcp://name_of_host/system
error: Unable to allow access for disk path /var/lib/libvirt/images/qemu.img: No such file or directory

For example, if the destination host name is newyork, the error message appears as:

# virsh migrate qemu qemu+tcp://newyork/system
error: Unable to allow access for disk path /var/lib/libvirt/images/qemu.img: No such file or directory

Investigation

By default, migration only transfers the in-memory state of a running guest (such as memory or CPU state). Although disk images are not transferred during migration, they need to remain accessible at the same path by both hosts.

Solution

Set up and mount shared storage at the same location on both hosts. The simplest way to do this is to use NFS:

Procedure A.9. Setting up shared storage

1. Set up an NFS server on a host serving as shared storage. The NFS server can be one of the hosts involved in the migration, as long as all hosts involved are accessing the shared storage through NFS.

   # mkdir -p /exports/images
   # cat >>/etc/exports <<EOF
   /exports/images    192.168.122.0/24(rw,no_root_squash)
   EOF

2. Mount the exported directory at a common location on all hosts running libvirt. For example, if the IP address of the NFS server is 192.168.122.1, mount the directory with the following commands:

   # cat >>/etc/fstab <<EOF
   192.168.122.1:/exports/images  /var/lib/libvirt/images  nfs  auto  0 0
   EOF
   # mount /var/lib/libvirt/images

It is not possible to export a local directory from one host using NFS and mount it at the same path on another host — the directory used for storing disk images must be mounted from shared storage on both hosts. If this is not configured correctly, the guest virtual machine may lose access to its disk images during migration, because the source host’s libvirt daemon may change the owner, permissions, and SELinux labels on the disk images after it successfully migrates the guest to its destination.

If libvirt detects that the disk images are mounted from a shared storage location, it will not make these changes.

A.19.9. No Guest Virtual Machines are Present when libvirtd is Started

Symptom

The libvirt daemon is successfully started, but no guest virtual machines appear to be present.

# virsh list --all
 Id    Name                           State
----------------------------------------------------

Investigation

There are various possible causes of this problem. Performing these tests will help to determine the cause of this situation:

Verify KVM kernel modules

Verify that KVM kernel modules are inserted in the kernel:

# lsmod | grep kvm
kvm_intel             121346  0
kvm                   328927  1 kvm_intel

If you are using an AMD machine, verify the kvm_amd kernel modules are inserted in the kernel instead, using the similar command lsmod | grep kvm_amd in the root shell.

If the modules are not present, insert them using the modprobe <modulename> command.

Although it is uncommon, KVM virtualization support may be compiled into the kernel. In this case, modules are not needed.

Verify virtualization extensions

Verify that virtualization extensions are supported and enabled on the host:

# egrep "(vmx|svm)" /proc/cpuinfo
flags		: fpu vme de pse tsc ... svm ... skinit wdt npt lbrv svm_lock nrip_save
flags		: fpu vme de pse tsc ... svm ... skinit wdt npt lbrv svm_lock nrip_save

Enable virtualization extensions in your hardware’s firmware configuration within the BIOS setup. See your hardware documentation for further details on this.

Verify client URI configuration

Verify that the URI of the client is configured as intended:

# virsh uri
vbox:///system

For example, this message shows the URI is connected to the VirtualBox hypervisor, not QEMU, and reveals a configuration error for a URI that is otherwise set to connect to a QEMU hypervisor. If the URI was correctly connecting to QEMU, the same message would appear instead as:

# virsh uri
qemu:///system

This situation occurs when there are other hypervisors present, which libvirt may speak to by default.

Solution

After performing these tests, use the following command to view a list of guest virtual machines:

# virsh list --all

A.19.10. Common XML Errors

Although it is not recommended, it is sometimes necessary to edit a guest virtual machine’s (or a domain’s) XML file manually. To access the guest’s XML for editing, use the following command:

# virsh edit name_of_guest.xml

This command opens the file in a text editor with the current definition of the guest virtual machine. After finishing the edits and saving the changes, the XML is reloaded and parsed by libvirt. If the XML is correct, the following message is displayed:

# virsh edit name_of_guest.xml

Domain name_of_guest.xml XML configuration edited.

After saving the XML file, use the xmllint command to validate that the XML is well-formed, or the virt-xml-validate command to check for usage problems:

# xmllint --noout config.xml

# virt-xml-validate config.xml

If no errors are returned, the XML description is well-formed and matches the libvirt schema. While the schema does not catch all constraints, fixing any reported errors will further troubleshooting.

Syntax errors are caught by the XML parser. The error message contains information for identifying the problem.

This example error message from the XML parser consists of three lines — the first line denotes the error message, and the two following lines contain the context and location of the XML code containing the error. The third line contains an indicator showing approximately where the error lies on the line above it:

error: (name_of_guest.xml):6: StartTag: invalid element name
<vcpu>2</vcpu><
-----------------^

A well-formatted XML document can contain errors that are correct in syntax but libvirt cannot parse. Many of these errors exist, with two of the most common cases outlined below.