This adds interfaces to the glance plugin for uploading / downloading
vhd files by glance url instead of with a host / port / use_ssl
tuple. This makes it possible to have glance hosted on a non root url
and still work with xenserver.
This bumps the plugin version to 1.3 as these are new functions exposed.
Convert xenapi driver to using up/down vhd2
All unit test converted to use this new functionality.
Change-Id: I73f60de0b1644827238961e1659b0119b80bf82f
This makes internal changes to the xen glance plugins so that they are
internally passing information about the glance servers by urls. It
does not change the surface of the plugin (that will come later), so
at a top level we are still thinking of glance as a host, port, and
use_ssl. However that's immediately converted to an endpoint url which
is then used for all the rest of the effort.
Change-Id: Ie1dce099846c725ff1ad7bb32f74aa6ae701ee9a
Currently tgz images are supported only when the names of VHD files
within follow the format of "0.vhd, 1.vhd, ... (n-1).vhd" called VDI
chain.
To maximize the range of support, the arbitrary named VHD file could be
renamed to 0.vhd when the tgz image only has a single VHD file.
Change-Id: I01cadf26262b20f1ad5a19e4441b42a0155f5a52
Currently the tox py34 target uses testtools.run to run
a subset of our test harness. We need to be able to use
pretty_tox.sh just like py27 as we make progress with py34
support.
The first step is to make sure we can discover all the
tests using:
python -m subunit.run discover -t . ./nova/tests/ --list
So, we need to fix a bunch of things for the discovery
to work including updating to a new version of websockify.
In the xen code, we should keep the original import and
add except for py34 as xen uses an older python that does
not work with six.moves.
Depends-On: Ib4ef2e79b28b7180e564b3d6dc2188456c66c08a
Change-Id: I88b6746da6136a7386a173f6cacd42f0b470deee
If we send the terminator on an error, it looks like we have actually
completed the upload, when that was not actually the case, we have
failed, and ideally we want glance to notice and mark the snapshot as
failed.
This partly reverts the following commit:
e039b036b5e9dbaff8b37f7ab22c209b71bdc182
It is not a full revert of the above commit, and some of the refactoring
and error handling changes are still useful.
Closes-Bug: #1398826
Change-Id: Ib608a32556135a5b7e666d02c4ee77722ea6e07c
Currently, the Xen API plugin sends the data payload to Glance
without checking the image status. In consequence, if the image is
not in queued status Glance will not let the image upload succeed.
This is by design, please refer [0].
The issue here being; buffering of the data will happen on the Glance
server. Consequently, the bound to be failed upload operation will not
get a 409 until after a large chunk of data has already been streamed
to the Glance server. This may result into client side wastage of
bandwidth and wait time.
This patch proposes a HEAD call be made from the glance plugin to
Glance API; thus in turn checking the valid image state for the
data transfer.
[0] http://docs.openstack.org/developer/glance/statuses.html
Change-Id: I24c1222a3b1ac809353724500117e58d50615ac8
This commit is a small refactor to the XenAPI glance plugin removing
the use of locals() to improve the readability.
Change-Id: I5de095ac1f7f38fd37cc9d6010a876995f0f8841
Add a comment documenting the restriction that our xenapi plugins must
target Python 2.4 features because of restrictions with the XenServer
dom0 userspace.
Change-Id: Ide130923b16c0555c3190cbd63ce97e18d18aae0
The xen plugin does not catch all cases where upload can throw
an exception due to connection failures and timeouts. This
change extends the try/execpt window to catch these cases.
Also added original connect which in part keeps us from lazy-loading
the connection and logging of exception information so we can
figure out later what happened.
Change-Id: If643a7da9f602ab7bb5d255031a841c674fa6c6f
Fixes-bug: 1378903
HTTP errors can be split into a few categories: client ephemeral, server
ephemeral, server permanent, and globally permanent. You could argue
there is even more permutations. However, for simplicity, these errors
can be viewed as ephemeral and permanent.
The Glance XenAPI plugin has been updated to raise a RetryableError for
ephemeral and unexpected (i.e. errors that are categorized as neither
permanent or ephemeral) errors.
Additionally, an instance fault will be logged every time an error
occurs. This will serve as transaction history where every attempt is a
transaction on the state. If an ephemeral error occurs, there is a
retry, then a permanent error, the history of each error will be in the
instance_faults.
Deployers should configure the num_retries relative to the number of
api_servers. Right now, servers are put in random order and then cycled.
Trying the same server multiple times could cause unnecessary load and
delays. However, trying multiple servers, is ideal when a single server
is behaving badly or cannot reach another server it needs to communicate
with to fulfill a request.
Closes-Bug: 1380776
Change-Id: I267a5b524c3ff8a28edf1a2285b77bb09049773c
The latest version of the agent has started changing the hostname
through the official windows APIs, when resetnetwork is called. This
means a reboot might happen after resetnetwork.
A reboot happens when polling for the version. Until now we wait for the
call timeout, before fetching the new dom id. This change ensures that
if we spot a reboot, the plugin exits early rather than keeping polling
the wrong dom id.
Turns out its best to wait for the dom_id to change, before trying
to poll the agent again. Once the dom_id in xenapi has been updated,
the xenstore keys are always in place.
Trying too early leads to lots of reboot detections because we are
retrying with the old dom_id. XenServer continues to return the old
dom_id for a little while after the reboot.
Closes-Bug: #1370999
Change-Id: Id0bf5b64f2b271d162db5bbce50167ab1f665c87
We can hit issues with corrupted VHDs if we copy a VHD while XenServer
is performing other operations. This happens because there are times
when we copy the VHD chains while XenServer is still performing a
coalesce of the VHD chain.
In most cases, vhd-util should be able to safely repair any metadata
corruption. It can copy the copy of the VHD footer at the front of the
VHD file and add it at the bottom on the VHD file. There is no VM data
loss, due to the way the coalesce happens, but the chain will be bigger
than it would be both before and after the coalesce.
This does not, however, ensure that snapshots are valid before uploading
them to glance. But should you launch a corrupted snapshot, this change
would fix up the snapshot, and allow it to boot correctly.
Closes-Bug: #1362595
Change-Id: I88b737d7e97964a9db5ccf2c39dea7fd0701ead4
During chunk upload if a subprocess exception is thrown, a malformed
chunk upload occurs.
This patch adds logging for subprocess exceptions and sends the
terminator even in the event of a subprocess exception. The incomplete
file upload will then be cleaned up by the client.
Change-Id: I559f201430bcd4c36d669c7772a10bfe3cd54d41
Closes-Bug: 1365637
If Glance returns a 500 response on an initial attempt to upload
a snapshot image, it will set the image status to killed. Any retry
attempts will fail with a 409 response.
Hence, Nova should not attempt to retry the upload in this case, which
will eliminate many pointless retries.
Closes-Bug: 1349933
Change-Id: I1a8f2416923a368b02cf8963c747ebbb24d749a1
pep8 E265 makes sure block comment start with '# '. Fix and gate on this
new rule as it helps improve comment readability.
In the few cases where the comment was just commented out code, remove
the comment.
Change-Id: Iea1c445df8ddc2b6c17a4ab697ad756eef2f91fa
Both H305 and H307 are part of the OpenStack style guide.
Ensure that there are 3 groups of imports, stdlib,
third-party and project specific. Within each category
ensure that imports are in alphabetical order.
Change-Id: Id4994551c135c3c1a4982153f0c6cacba6176b95
This commit addresses some optimizations when using libtorrent to
download. The settings are documented here:
http://www.rasterbar.com/products/libtorrent/reference-Settings.html
I'm proposing to add these with dom0 environment overrides in case
any other adjustments are needed. The changes effectively double the
download rate from the default settings. It also allows unbalanced
download/upload which is preferred in openstack environments.
No unit tests were added as plugins are not currently unit tested.
Closes-bug: #1303993
Change-Id: I503a6a521c196e4ad58e0a16aba2988f66a9cdca
Using ssh depends on the host being set up to be able to SSH into
itself which is not a common scenario. While this is unavoidable for
the current implementation of resize across multiple hosts, if there
is a single host (i.e. a test scenario) or the resize is restricted
to the same host then we can succeed without SSH access
Dependency on Ia310e31d31aaf5c979e41c64af8223202a18e03a is because the
tests will always fail without Ia310 therefore this fix cannot be tested
without taking Ia310 as well.
Closes-bug: 1308064
Change-Id: I15802a1d97d380b1c5b74fc9f92ece2494fe789a
Currently, when the network connection to glance is interrupted, there
is different behaviour on upload and download.
This change ensures the behaviour between the two code paths is more
consistent.
Uploads generally need to be given more time before they timeout, so
to keep a single timeout between upload and download, the timeout is
increased to 90 seconds.
At the same time, it ensures the tar process gets killed when any
issues occur with the communication between the hypervisor and
glance.
Change-Id: Id5396e5d3c1052dc2979476a886412da65e08670
Closes-Bug: #1284596
Fixes bug 1286187
Neither execute_get_output() nor execute() would wait for the process to
fully finish executing. This could potentially create a race condition
where commands execute in a different order than intended.
Also, for execute() it could cause a process to never finish executing if
it generates enough output to block writing to the pipe.
Change-Id: I3404f4b3ca1cddeec2f3e7b393817a6ccc42bec7
This check indicates on comments in which multi line docstring should
start without a leading new line. This change fixed all violators of
said check.
Change-Id: Ic7357b8c7420767dba611f6fcee07b7700f3aea8
We added a function into the xenhost plugin to get information
about PCI devices. Roughly we run the lspci command on dom0. This
information will be used to get the list of pci devices that are
passed on the pciback.hide dom0 command line.
The hide option is used to hide the devices from the normal guest
drivers and assign them to the pciback kernel driver at boot on dom0
instead of their normal driver. We will parse the output of the lspci
command to find which device is using the pciback kernel driver and
thus to know if it has been passed to the pciback.hide option.
This information will be used to perform the match with the
list of pci devices provided in pci_whitelist into /etc/nova/nova.conf.
Implements: blueprint pci-passthrough-xenapi
Change-Id: I465fc5d29f3c47ab0079adcfcc2d7d6501bd4b20
Add a new host plugin to monitor whether the garbage collector is running.
This is then used by the wait_for_vhd_coalesce function to know whether
waiting any more is likely to produce the expected results.
Increased the number of times we loop waiting for coalesce as the loop
now has knowledge that something is still happening.
DocImpact
Change to parameter default.
Change-Id: Idcc0738945bd1aee8cdb52c9cfabd798f85c31db
Closes-bug: 1258169
We don't need to have the vi modelines in each source file,
it can be set in a user's vimrc if required.
Also a check is added to hacking to detect if they are re-added.
Change-Id: I347307a5145b2760c69085b6ca850d6a9137ffc6
Closes-Bug: #1229324
It seems there are cases when the server gets stuck in a building state
because the download from glance fails, but the nova-compute doesn't
find out.
After some code inspect, a likely cause seems to related to having no
socket timeout inside urllib2 when we download the image from glance. A
workaround for this issue is to configure the default socket timeout as
described here:
http://docs.python.org/3.1/howto/urllib2.html#sockets-and-layers
Fixes bug 1251650
Change-Id: I8b34d95d3524f3825b28e61cbbb59290653d3711
Fix the agent update message send so Windows guests will work
Message must be formatted: "url,md5sum"
Change-Id: I675cc5a7ca522696eab1820c14c48361ff7e0f74
Closes-Bug: 1250165
Nova shouldn't be building any types of packages. And if we are we should
be testing this code in gate somewhere.
Change-Id: I70d82a6173b5eda183d4e49d330ae97dd14f9078
Because the plugins live on a host seperate to Nova we need an interface
to test whether they are the expected version. We can't use pip or other
requirement systems as they are cross-machine.
Closes bug 1226622
Change-Id: I58ab669061f51bd87071e2cf0d93d33021001309
Due to some issues in the gate, several flake8 errors got merged
recently. This patch fixes the most recent issues found.
For the H304 relative import errors, we have to skip those because they
are put into /etc/xensource/scripts which is the current working
directory when running the plugin.
For the H231 incompatible python 3 'except x,y' construct, we have to
skip those because this code is written to run on python 2.4 and 'except
x as y' does not work with python 2.4.
Note that in cleaning up some of the H304 failures for relative imports,
I also re-arranged the imports to follow the hacking guide of doing
standard library packages first, then third party packages, and finally
nova-specific packages.
Closes-Bug: #1229753
Change-Id: I1c2211fd6a10d43d7e65cdb4e18530397788cf2c
* Removes dead code
* Prefixes internal functions with underscore
* Removes a few one-off functions in favor of inlining
Change-Id: Ic2f8c2e512df270f3b1b0e35ded98832a5152c72
Due to a previous commit, a client of make_subprocess is making a call
with an env parameter. As make_subprocess does not have such a
parameter, image upload fails. This patch adds the env keyword argument
to make_subprocess, and use that in the Popen call.
Fixes bug 1216267
Change-Id: I8098b484067b54c15fe95f2bf84c5890db7ebcb3
This commit adds the instance uuid to the rsync debug output in
the debug logs. The issue we were seeing is that when multiple
migrations are ongoing at the same time, it's not easy to
track the rsync progress of each migration.
Fixes: bug #1215629
Change-Id: Iec662bdc0aed1c6f54ecaa062335dc5504ac030f
Currently only used by XenAPI.
Sets GZIP environment variable before spawning tar -cz to force the
compression level. Defaults to none, which means the GZIP environment
variable is not set and the default (usually -6) is used.
Implements: blueprint image-compression-mode
Change-Id: I02e136b7215a4f2cefc259f40bde9c1b205737ed
This patch adds support for a new kind of Glance image, a specially crafted ISO
which supports iPXE booting, giving customers a means to roll their own image.
Two virt-layer modifications were needed. The first was adding configurations
for the iPXE ISO feature (network to use, boot menu, mkisofs_cmd). The second
was the ability to inject networking info into the ISO after it was
downloaded.
To use this feature, operators should enable the `ipxe_boot` image-property.
DocImpact
Implements blueprint xenapi-ipxe-iso-boot-support
Change-Id: I33acf9dfdff0a5ed9797723a142bc451348e8549
Send identity headers while uploading/downloading images from glance. Related
to:
1) Add identity headers while calling glanceclient from nova
Ife02059abbdce7920f1f408b71e9745d777fa770
2) Pass identity headers received in glanceclient to nova
Ifbef582aa4e64a2e7a46db43a9cc6cf8c3531dbd
3) Pass identity headers received by glance api to glance registry
Ie5f07ed6dfeaa8428de4f79c4d40d182328e6ab4
NOTE: since this is a change to a Dom0 plugin API, it requires the lock-step
upgrade of both the compute-manager and the dom0 plugin.
DocImpact
Change-Id: I6d5e3448d2c0acc392d18e0b88cec25cf313da5b
Recently a change was made to how/where the '_' method is imported. This
seems to have broken most xenserver plugins. These plugins shouldn't be
doing * imports anyway so I've gone in and cleaned up all places where
"from pluginlib_nova import *" was used and replaced with "import
pluginlib_nova".
Fixes bug 1207107
Change-Id: I3b2a299cf60543e0f766becfeb4740af791b3576
There are a few upload errors that glance plugin should
not retry upload on, as it will fail again. This patch changes
upload to not retry on the following conditions in addition
to the 401 error that it already handles:
- CONFLICT
- REQUEST_ENTITY_TOO_LARGE
- PRECONDITION_FAILED
- FORBIDDEN
Fixes bug 1203726
Change-Id: I538a4f9b133169b2e398e7dfcb3af58fef204f03
Previous _ was monkey patched into builtins whenever
certain modules were imported. This removes that and
simply imports it when it is needed.
Change-Id: I0af2c6d8a230e94440d655d13cab9107ac20d13c