Xen live migration in OpenStack Grizzly

I recently experimented with live migration in an OpenStack Grizzly cluster of Xen compute nodes and thought it useful to write a short blog post for the benefit of others interested in OpenStack Xen deployments.

My OpenStack Grizzly cluster consisted of three nodes: One controller node hosting most of the OpenStack services (rabbitmq, mysql, cinder, keystone, nova-api, nova-scheduler, etc.) and two Xen compute nodes running nova-compute, nova-network, and nova-novncproxy.  All nodes were running fully patched SLES11 SP2.  devstack was used to deploy the OpenStack services.

For the most part, I used the OpenStack documentation for configuring live migration to setup the environment.  The main configuration tasks include

  1. Configuring shared storage on the compute nodes involved in live migration.  I took the simple approach and used NFS for shared storage between the compute nodes, mounting an NFS share at /opt/stack/data/nova/instances in each compute node.
  2. Ensure that the UID and GID of your nova (or stack) and libvirt users are identical between each of your servers. This ensures that the permissions on the NFS mount will work correctly.
  3. Ensure the shared directory has ‘execute/search’ bit set, e.g. chmod o+x /opt/stack/data/nova/instances
  4. Ensure firewall is properly configured to allow migrations.  For Xen, port 8002 needs to be open.  This is the port xend listens on for migration requests.

In addition to the steps described in the OpenStack documentation, the following configuration needs to be performed on the Xen compute nodes

  1. Enable migration (aka relocation) in /etc/xen/xend-config.sxp: (xend-relocation-server yes)
  2. Define a list of hosts allowed to connect to the migration port in /etc/xen/xend-config.sxp.  To allow all hosts, leave the list empty:  (xend-relocation-hosts-allow ”)
  3. Set the ‘live_migration_flag’ option in /etc/nova/nova.conf.  In the legacy xm/xend toolstack, xend implements all of the migration logic.  Unlike the libvirt qemu driver, the libvirt legacy xen driver can only pass the migration request to the Xen toolstack, so the only migration flags needed are VIR_MIGRATE_LIVE and VIR_MIGRATE_UNDEFINE_SOURCE:  live_migration_flag=VIR_MIGRATE_LIVE,VIR_MIGRATE_UNDEFINE_SOURCE
  4. Set the live_migration_uri option in /etc/nova/nova.conf.  The default for this option is ‘qemu+tcp://%s/system’.  For Xen, this needs to be ‘xenmigr://%s/system’:  live_migration_uri = xenmigr://%s/system

After these configuration steps, restart xend and nova-compute on the Xen compute nodes to reload the new configuration.  Your OpenStack Xen cluster should now be able to perform live migration as per the OpenStack Using Migration documentation.

On my small cluster, xen71 is the controller node and xen76 and xen77 are the Xen compute nodes.  I booted an instance of a SLES11 SP2 Xen HVM image that was provisioned on xen76.

stack@xen71:~> nova list
+--------------------------------------+------------------------+--------+--------------------+
|ID                                    | Name                   | Status | Networks           |
+--------------------------------------+------------------------+--------+--------------------+
| 6b45baa2-3dc2-420c-a7ab-aad25fc1aa2a | sles11sp2-xen-hvm-test | ACTIVE | private=10.4.128.2 |
+--------------------------------------+------------------------+--------+--------------------+
stack@xen71:~> nova show 6b45baa2-3dc2-420c-a7ab-aad25fc1aa2a
+-------------------------------------+----------------------------------------------------------+
| Property                            | Value                                                    |
+-------------------------------------+----------------------------------------------------------+
| status                              | ACTIVE                                                   |
| updated                             | 2013-04-05T17:27:16Z                                     |
| OS-EXT-STS:task_state               | None                                                     |
| OS-EXT-SRV-ATTR:host                | xen76                                                    |
| key_name                            | None                                                     |
| image                               | SLES11SP2-xen-hvm (5b39e6b3-bc3f-4fb0-81a0-b115cb8ada80) |
| private network                     | 10.4.128.2                                               |
| hostId                              | cca619a77da34c0c26001fb2438d7cce6a5da6408ae8ec111401f627 |
| OS-EXT-STS:vm_state                 | active                                                   |
| OS-EXT-SRV-ATTR:instance_name       | instance-0000000a                                        |
| OS-EXT-SRV-ATTR:hypervisor_hostname | xen76.virt.lab.novell.com                                |
| flavor                              | m1.tiny (1)                                              |
| id                                  | 6b45baa2-3dc2-420c-a7ab-aad25fc1aa2a                     |
| security_groups                     | [{u'name': u'default'}]                                  |
| user_id                             | 86b77dee688e4eff957865205d27464a                         |
| name                                | sles11sp2-xen-hvm-test                                   |
| created                             | 2013-04-05T17:10:04Z                                     |
| tenant_id                           | 0833047bb70d4b38874328aad83b7140                         |
| OS-DCF:diskConfig                   | MANUAL                                                   |
| metadata                            | {}                                                       |
| accessIPv4                          |                                                          |
| accessIPv6                          |                                                          |
| progress                            | 0                                                        |
| OS-EXT-STS:power_state              | 1                                                        |
| OS-EXT-AZ:availability_zone         | nova                                                     |
| config_drive                        |                                                          |
+-------------------------------------+----------------------------------------------------------+

Now let’s migrate the instance to the xen77 compute node

stack@xen71:~> nova live-migration 6b45baa2-3dc2-420c-a7ab-aad25fc1aa2a xen77

While the migration is in progess, we can see the status and task state as migrating

stack@xen71:~> nova show 6b45baa2-3dc2-420c-a7ab-aad25fc1aa2a
+-------------------------------------+----------------------------------------------------------+
| Property                            | Value                                                    |
+-------------------------------------+----------------------------------------------------------+
| status                              | MIGRATING                                                |
| updated                             | 2013-04-05T20:16:27Z                                     |
| OS-EXT-STS:task_state               | migrating                                                |
| OS-EXT-SRV-ATTR:host                | xen76                                                    |
| key_name                            | None                                                     |
| image                               | SLES11SP2-xen-hvm (5b39e6b3-bc3f-4fb0-81a0-b115cb8ada80) |
| private network                     | 10.4.128.2                                               |
| hostId                              | cca619a77da34c0c26001fb2438d7cce6a5da6408ae8ec111401f627 |
| OS-EXT-STS:vm_state                 | active                                                   |
| OS-EXT-SRV-ATTR:instance_name       | instance-0000000a                                        |
| OS-EXT-SRV-ATTR:hypervisor_hostname | xen76.virt.lab.novell.com                                |
| flavor                              | m1.tiny (1)                                              |
| id                                  | 6b45baa2-3dc2-420c-a7ab-aad25fc1aa2a                     |
| security_groups                     | [{u'name': u'default'}]                                  |
| user_id                             | 86b77dee688e4eff957865205d27464a                         |
| name                                | sles11sp2-xen-hvm-test                                   |
| created                             | 2013-04-05T17:10:04Z                                     |
| tenant_id                           | 0833047bb70d4b38874328aad83b7140                         |
| OS-DCF:diskConfig                   | MANUAL                                                   |
| metadata                            | {}                                                       |
| accessIPv4                          |                                                          |
| accessIPv6                          |                                                          |
| OS-EXT-STS:power_state              | 1                                                        |
| OS-EXT-AZ:availability_zone         | nova                                                     |
| config_drive                        |                                                          |
+-------------------------------------+----------------------------------------------------------+

Once the migration completes, we can see that the instance is now running on xen77

stack@xen71:~> nova show 6b45baa2-3dc2-420c-a7ab-aad25fc1aa2a
+-------------------------------------+----------------------------------------------------------+
| Property                            | Value                                                    |
+-------------------------------------+----------------------------------------------------------+
| status                              | ACTIVE                                                   |
| updated                             | 2013-04-05T20:11:37Z                                     |
| OS-EXT-STS:task_state               | None                                                     |
| OS-EXT-SRV-ATTR:host                | xen77                                                    |
| key_name                            | None                                                     |
| image                               | SLES11SP2-xen-hvm (5b39e6b3-bc3f-4fb0-81a0-b115cb8ada80) |
| private network                     | 10.4.128.2                                               |
| hostId                              | cabdc8468130edd0f85440f1b2922419b359b3da36a40de98713dbda |
| OS-EXT-STS:vm_state                 | active                                                   |
| OS-EXT-SRV-ATTR:instance_name       | instance-0000000a                                        |
| OS-EXT-SRV-ATTR:hypervisor_hostname | xen76.virt.lab.novell.com                                |
| flavor                              | m1.tiny (1)                                              |
| id                                  | 6b45baa2-3dc2-420c-a7ab-aad25fc1aa2a                     |
| security_groups                     | [{u'name': u'default'}]                                  |
| user_id                             | 86b77dee688e4eff957865205d27464a                         |
| name                                | sles11sp2-xen-hvm-test                                   |
| created                             | 2013-04-05T17:10:04Z                                     |
| tenant_id                           | 0833047bb70d4b38874328aad83b7140                         |
| OS-DCF:diskConfig                   | MANUAL                                                   |
| metadata                            | {}                                                       |
| accessIPv4                          |                                                          |
| accessIPv6                          |                                                          |
| progress                            | 0                                                        |
| OS-EXT-STS:power_state              | 1                                                        |
| OS-EXT-AZ:availability_zone         | nova                                                     |
| config_drive                        |                                                          |
+-------------------------------------+----------------------------------------------------------+

You might notice a small bug here in that OS-EXT-SRV-ATTR:hypervisor_hostname is not updated with the xen77 host now running the instance.  A minor issue that I will add to my list of bugs needing investigation.

About these ads

2 thoughts on “Xen live migration in OpenStack Grizzly

    • It depends on the size of the instance, and the rate at which the working set is dirtied I suppose. I was testing with m1.tiny flavors (512M of memory) with instances that were basically idle, and migration was taking about 8 seconds.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s