I had the pleasure of meeting Russell Pavlicek, who shares Xen community management responsibilities with Lars Kurth, at SUSECon last November and he, along with Dario Faggioli of the Xen community, poked me about writing a blog post describing the state of Xen support in libvirt. I found this to be an excellent idea and good reason to get off my butt and write! Far too much time passes between my musings.
Xen has had a long history in libvirt. In fact, it was the first hypervisor supported by libvirt. I’ve witnessed an incredible evolution of libvirt over the years and now not only does it support managing many hypervisors such as Xen, KVM/QEMU, LXC, VirtualBox, hyper-v, ESX, etc., but it also supports managing a wide range of host subsystems used in a virtualized environment such as storage pools and volumes, networks, network interfaces, etc. It has really become the swiss army knife of virtualization management on Linux, and Xen has been along for the entire ride.
libvirt supports multiple hypervisors via a hypervisor driver interface, which is defined in $libvirt_root/src/drvier.h – see struct _virDriver. libvirt’s virDomain* APIs map to functions in the hypervisor driver interface, which are implemented by the various hypervisor drivers. The drivers are located under $libvirt_root/src/<hypervisor-name>. Typically, each driver has a $libvirt_root/src/<hypervisor-name>/<hypervisor-name>_driver.c file which defines a static instance of virDriver and fills in the functions it implements. As an example, see the definition of libxlDriver in $libvirt_root/src/libxl/libxl_driver.c, the firsh few lines of which are
static virDriver libxlDriver = {
.no = VIR_DRV_LIBXL,
.name = “xenlight”,
.connectOpen = libxlConnectOpen, /* 0.9.0 */
.connectClose = libxlConnectClose, /* 0.9.0 */
.connectGetType = libxlConnectGetType, /* 0.9.0 */
….
}
The original Xen hypervisor driver is implemented using a variety of Xen tools: xend, xm, xenstore, and the hypervisor domctrl and sysctrl interfaces. All of these “sub-drivers” are controlled by an “uber driver” known simply as the “xen driver”, which resides in $libvirt_root/src/xen/. When an API in the hypervisor driver is called on a Xen system, e.g. virDomainCreateXML, it makes its way to the xen driver, which funnels the request to the most appropriate sub-driver. In most cases, this is the xend sub-driver, although the other sub-drivers are used for some APIs. And IIRC, there are a few APIs for which the xen driver will iterate over the sub-drivers until the function succeeds. I like to refer to this xen driver, and its collection of sub-drivers, as the “legacy Xen driver”. Due to its heavy reliance on xend, and xend’s deprecation in the Xen community, the legacy driver became just that – legacy. With the introduction of libxenlight (aka libxl), libvirt needed a new driver for Xen.
In 2011 I had a bit of free time to work on a hypervisor driver for libxl, committing the initial driver in 2b84e445. As mentioned above, this driver resides in $libvirt_root/src/libxl/. Subsequent work by SUSE, Univention, Redhat, Citrix, Ubuntu, and other community contributors has resulted in a quite functional libvirt driver for the libxl toolstack.
The libxl driver only supports Xen >= 4.2. The legacy Xen driver should be used on earlier versions of Xen, or installations where the xend toolstack is used. In fact, if xend is running, the libxl driver won’t even load. So if you want to use the libxl driver but have xend running, xend must be shutdown followed by a restart of libvirtd to load the libxl driver. Note that if xend is not running, the legacy Xen driver will not load.
Currently, there are a few differences between the libxl driver and the legacy Xen driver. First, the libxl driver is clueless about domains created by other libxl applications such as xl. ‘virsh list’ will not show domains created with ‘xl create …’. This is not the case with the legacy Xen driver, which is just a broker to xend. Any domains managed by xend are also manageable with the legacy Xen driver. Users of the legacy Xen driver in libvirt are probably well aware that ‘virsh list’ will show domains defined with ‘xm new …’ or created with ‘xm create …’, and might be a bit surprised to find this in not the case with the libxl driver. But this could be addressed by implementing functionality similar to the ‘qemu-attach’ capability supported by the QEMU driver, which allows “importing” a QEMU instance created directly with e.g. ‘qemu -m 1024 -smp …’. Contributions are warmly welcomed if this functionality is important to you :-).
A second difference between the libxl and legacy Xen drivers is related to the first one. xend is the stateful service in the legacy stack, maintaining state of defined and running domains. As a result, the legacy libvirt Xen driver is stateless, generally forwarding requests to xend and allowing xend to maintain state. In the new stack, however, libxl is stateless. Thererfore, the libvirt libxl driver itself must now maintain the state of all domains. An interesting side affect of this is losing all your domains when upgrading from libvirt+xend to libvirt+libxl. For a smooth upgrade, all running domains should be shutdown and their libvirt domXML configuration exported for post-upgrade import into the libvirt libxl driver. For example, in psuedo-code
for each domain
virsh shutdown domain
virsh dumpxml > domain-name.xml
perform xend -> libxl upgrade
restart libvirtd
for each domain
virsh define domain-name.xml
It may also be possible to import xend managed domains after upgrading to libxl. On most installations, the configuration of xend managed domains is stored in /var/lib/xend/domains/<dom-uuid>/config.sxp. Since the legacy Xen driver already supports parsing SXP, this code could be used read any existing xend managed domains and import those into libvirt. I will need to investigate the feasibility of this approach, and report any findings in a future blog post.
The last (known) difference between the drivers is the handling of domain0. The legacy xen driver handles domain0 as any other domain. The libxl driver currently treats domain0 as part of the host, thus e.g. it is not shown in ‘virsh list’. This behavior is similar to the QEMU driver, but is not necessarily correct. Afterall, domain0 is just another domain in Xen, which can have devices attached and detached, memory ballooned, etc., and should probably be handled as such by the libvirt libxl driver. Contributions welcomed!
Otherwise, the libxl driver should behave the same as the legacy Xen driver, making xend to libxl upgrades quite painless, outside of the statefullness issue discussed above. Any other differences between the legacy Xen driver and the libxl driver are bugs – or missing features. Afterall, the goal of libvirt is to insulate users from underlying churn in hypervisor-specific tools.
At the time of this writing, the important missing features in the libxl driver relative to the legacy Xen driver are PCI passthrough and migration. Chunyan Liu has provided patches for both of these features, the first of which is close to committing upstream IMO
https://www.redhat.com/archives/libvir-list/2014-January/msg00400.html
https://www.redhat.com/archives/libvir-list/2013-September/msg00667.html
The libxl driver is also in need of improved parallelization. Currently, long running operations such as create, save, restore, core dump, etc. lock the driver, blocking other operations, even those that simply get state. I have some initial patches that introduce job support in the libxl driver, similar to the QEMU driver. These patches allow classifying driver operations into jobs that modify state, and thus block any other operations on the domain, and jobs that can run concurrently. Bamvor Jian Zhang is working on a patch series to make use of libxl’s asynchronous variants of these long running operations. Together, these patch sets will greatly improve parallelism in the libxl driver, which is certainly important in for example cloud environments where many virtual machine instances can be started in parallel.
Beyond these sorely needed features and improvements, there is quite a bit of work required to reach feature parity with the QEMU driver, where it makes sense. The hypervisor driver interface currently supports 193 functions, 186 of which are implemented in the QEMU driver. By contrast, only 86 functions are implemented in the the libxl driver. To be fair, quite a few of the unimplemented functions don’t apply to Xen and will never be implemented. Nonetheless, for any enthusiastic volunteers, there is quite a bit of work to be done in the libvirt libxl driver.
Although I thoroughly enjoy working on libvirt and have healthy respect for the upstream community, my available time to work on upstream libvirt is limited. Currently, I’m the primary maintainer of the Xen drivers, so my limited availability is a bottleneck. Other libvirt maintainers review and commit Xen stuff, but their primary focus is on the rapid development of other hypervisor drivers and host subsystems. I’m always looking for help in not only implementation of new features, but also reviewing and testing patches from other contributors. If you are part of the greater Xen ecosystem, consider lending a hand with improving Xen support in libvirt!
Jim Fehlig 
Pingback: Links 16/1/2014: Instructionals | Techrights
Hi Jim,
These days we noticed that Xen 4.4 released with Libvirt integration, and you are the important maintainer for this project ! However, we cannot find any libvirt config guide for xl through the internet.
We encountered such kinds of problems since Xen 4.3 and cannot make xl config with upstream qemu work with Libvirt.
In our environment with Xen 4.3 and libvirt 1.2.1, libvirt cannot start a predefined xm config file,complaining cannot find qemu-xen, nor qemu-xen-traditional.
Could you please give us some guide or example on Xen 4.4 Libvirt confg, especially on upstream qemu new features and corresponding Libvirt version information?
Thank you very much!
Hi Kevin,
Have you converted the xl config to libvirt domain XML? libvirt domain configuration format is XML (http://libvirt.org/formatdomain.html), it does not directly consume the various Xen config formats. You can however convert xm/xl config to libvirt domain XML with ‘virsh domxml-from-native xen-xm /path/to/xm-or-xl.config’ and then define or create the domain with the resulting XML, e.g. ‘virsh define /path/to/converted.xml’. Also, new to libvirt 1.2.3, is the libxl driver’s ability to parse Xen’s s-expression (sxpr) config format, e.g. ‘virsh domxml-from-native xen-sxpr /path/to/sxpr-config’.
Are you using distro-provided Xen and libvirt packages? If so, I’d suggest using the distro’s bug tool or appropriate mailing list for additional help.
Hi jfehlig,
Thank you for your help. I’ve also got some hints from xen-users, and made libvirt work with xen 4.4 now. Everything goes well except that I cannot make spice related configuration work. It seems that virsh domxml-from-native command
does not have parser for spice parameters. What’s more, if I copy spice related xml config from KVM, libxenlight driver still cannot recognize them.
Whether I need the specific newest libvirt version or any patch?
Good to hear that you have libvirt working with the Xen 4.4 libxl stack! Unfortunately, there are still quite a few missing features in the libvirt libxl driver, spice support being one of them. Patches are always welcome, if contributing to libvirt is something you can do. Cheers!
Though I am not familiar with libvirt development, I will continue to test xen 4.4+ and libvirt integration then! Thank you for you advice again, and I will also post my questions here if you would like!
Hm, to be honest I think the current driver implementation is a huge mistake. It pretends that there’s adequate Xen support as it was with the xend driver, but actually it is (for my usage) far from production ready.
It’s a misconception to assume that xend is stateful. It may be in some way, but it always reflects the correct state of running domains. While some stuff is cached in /var/lib/xend, the runstate is extracted from the hypervisor. Try it: if /var/lib/xend is emptied and xend is restarted, all domains show up correctly. Mixing xl and xm on Xen 4.1 isn’t recommended, but possible.
In contrast, the libvirt XL implementation just caches some values in /var/lib which might reflect what’s going on, or not. That’s simply unusable for real-world use.
I got an app that uses libvirt to retrieve the current domain states, but I’ll have to migrate it away from libvirt now to obtain reliable information. Too sad, this renders libvirt’s claim to cover all hypervisors moot.
My apologies, this rant might sound quite rude, offending somebody who stepped up to implement a missing driver, but sometimes implementing the wrong stuff prevents somebody else doing it right.
libvirt is open source software. There is nothing preventing you from contributing patches for missing functionality, or to change whatever is wrong by “doing it right”.
I’m afraid patching or extending won’t be satisfying. Having libvirt-local state info about VMs appears plain wrong to me (all information is retrievable using xenstore and xc api), so a complete rewrite would be more feasible.
I used libvirt as kind of remote api, so instead of trying to extend a broken driver (I wasn’t too happy about some libvirt-on-xm behaviour) I decided to write my own daemon that makes xc, xl and xenstore available via https, which works fine for me. I might be publishing this xld daemon some day open sourced.