14c38ac0f2
This was noticed in a downstream bug when a Nova instance with Cinder
volume (in this case, both the Nova instance storage _and_ Cinder volume
are located on Ceph) is migrated to a target Compute node, the disk
cache value for the Cinder volume gets changed. I.e. the QEMU
command-line for the Cinder volume stored on Ceph turns into the
following:
Pre-migration, QEMU command-line for the Nova instance:
[...] -drive file=rbd:volumes/volume-[...],cache=writeback
Post-migration, QEMU command-line for the Nova instance:
[...] -drive file=rbd:volumes/volume-[...],cache=none
Furthermore, Jason Dillaman from Ceph confirms RBD cache being enabled
pre-migration:
$ ceph --admin-daemon /var/run/qemu/ceph-client.openstack.[...] \
config get rbd_cache
{
"rbd_cache": "true"
}
And disabled, post-migration:
$ ceph --admin-daemon /var/run/qemu/ceph-client.openstack.[...] \
config get rbd_cache
{
"rbd_cache": "false"
}
This change in cache value post-migration causes I/O latency on the
Cinder volume.
From a chat with Daniel Berrangé on IRC: Prior to live migration, Nova
rewrites all the <disk> elements, and passes this updated guest XML
across to target libvirt. And it is never calling _set_cache_mode()
when doing this. So `nova.conf`'s `writeback` setting is getting lost,
leaving the default `cache=none` setting. And this mistake (of leaving
the default cache value to 'none') will of course be correct when you
reboot the guest on the target later.
So:
- Call _set_cache_mode() in _get_volume_config() method -- because it
is a callback function to _update_volume_xml() in
nova/virt/libvirt/migration.py.
- And remove duplicate calls to _set_cache_mode() in
_get_guest_storage_config() and attach_volume().
- Fix broken unit tests; adjust test_get_volume_config() to reflect
the disk cache mode.
Thanks: Jason Dillaman of Ceph for observing the change in cache modes
in a downstream bug analysis, Daniel Berrangé for help in
analysis from a Nova libvirt driver POV, and Stefan Hajnoczi
from QEMU for help on I/O latency instrumentation with `perf`.
Closes-bug: 1706083
Change-Id: I4184382b49dd2193d6a21bfe02ea973d02d8b09f
|
||
|---|---|---|
| api-guide/source | ||
| api-ref/source | ||
| contrib | ||
| devstack | ||
| doc | ||
| etc/nova | ||
| gate | ||
| nova | ||
| placement-api-ref/source | ||
| plugins/xenserver | ||
| releasenotes | ||
| tools | ||
| .coveragerc | ||
| .gitignore | ||
| .gitreview | ||
| .mailmap | ||
| .testr.conf | ||
| CONTRIBUTING.rst | ||
| HACKING.rst | ||
| LICENSE | ||
| MAINTAINERS | ||
| README.rst | ||
| babel.cfg | ||
| bindep.txt | ||
| requirements.txt | ||
| setup.cfg | ||
| setup.py | ||
| test-requirements.txt | ||
| tests-functional-py3.txt | ||
| tests-py3.txt | ||
| tox.ini | ||
README.rst
Team and repository tags
OpenStack Nova
OpenStack Nova provides a cloud computing fabric controller, supporting a wide variety of compute technologies, including: libvirt (KVM, Xen, LXC and more), Hyper-V, VMware, XenServer and OpenStack Ironic.
OpenStack Nova is distributed under the terms of the Apache License, Version 2.0. The full terms and conditions of this license are detailed in the LICENSE file.
API
To learn how to use Nova's API, consult the documentation available online at:
https://developer.openstack.org/api-guide/compute/ https://developer.openstack.org/api-ref/compute/
For more information on OpenStack APIs, SDKs and CLIs, please see:
https://www.openstack.org/appdev/ https://developer.openstack.org/
Operators
To learn how to deploy and configure OpenStack Nova, consult the documentation available online at:
For information about the different compute (hypervisor) drivers supported by Nova, please read:
https://docs.openstack.org/developer/nova/feature_classification.html
In the unfortunate event that bugs are discovered, they should be reported to the appropriate bug tracker. If you obtained the software from a 3rd party operating system vendor, it is often wise to use their own bug tracker for reporting problems. In all other cases use the master OpenStack bug tracker, available at:
Developers
For information on how to contribute to Nova, please see the contents of the CONTRIBUTING.rst.
Any new code must follow the development guidelines detailed in the HACKING.rst file, and pass all unit tests.
Further developer focused documentation is available at: