389 Commits

Author SHA1 Message Date
Sean Dague
89de4ed5e9 Add upload/download vhd2 interfaces
This adds interfaces to the glance plugin for uploading / downloading
vhd files by glance url instead of with a host / port / use_ssl
tuple. This makes it possible to have glance hosted on a non root url
and still work with xenserver.

This bumps the plugin version to 1.3 as these are new functions exposed.

Convert xenapi driver to using up/down vhd2

All unit test converted to use this new functionality.

Change-Id: I73f60de0b1644827238961e1659b0119b80bf82f
2015-12-18 12:37:35 +00:00
Sean Dague
6ebd61b28b retool xen glance plugin to work with urls
This makes internal changes to the xen glance plugins so that they are
internally passing information about the glance servers by urls. It
does not change the surface of the plugin (that will come later), so
at a top level we are still thinking of glance as a host, port, and
use_ssl. However that's immediately converted to an endpoint url which
is then used for all the rest of the effort.

Change-Id: Ie1dce099846c725ff1ad7bb32f74aa6ae701ee9a
2015-12-11 08:28:58 -05:00
Jenkins
1a86864e8b Merge "Fix a few typos" 2015-09-15 11:24:36 +00:00
Takashi NATSUME
e5f457d66f Fix a few typos
adress => address
curent => current
concurent => concurrent
a instance => an instance
optinally => optionally

Change-Id: I08e38d8418c2133475e4631bfaef17e34dd1c5bc
Closes-Bug: #1481123
2015-09-08 10:36:14 +09:00
John Hua
b97e409dfd xenapi: Support extra tgz images that with only a single VHD
Currently tgz images are supported only when the names of VHD files
within follow the format of "0.vhd, 1.vhd, ... (n-1).vhd" called VDI
chain.

To maximize the range of support, the arbitrary named VHD file could be
renamed to 0.vhd when the tgz image only has a single VHD file.

Change-Id: I01cadf26262b20f1ad5a19e4441b42a0155f5a52
2015-09-07 12:39:01 +00:00
Davanum Srinivas
ed0196ebb6 Get py34 subunit.run test discovery to work
Currently the tox py34 target uses testtools.run to run
a subset of our test harness. We need to be able to use
pretty_tox.sh just like py27 as we make progress with py34
support.

The first step is to make sure we can discover all the
tests using:
python -m subunit.run discover -t . ./nova/tests/ --list

So, we need to fix a bunch of things for the discovery
to work including updating to a new version of websockify.

In the xen code, we should keep the original import and
add except for py34 as xen uses an older python that does
not work with six.moves.

Depends-On: Ib4ef2e79b28b7180e564b3d6dc2188456c66c08a
Change-Id: I88b6746da6136a7386a173f6cacd42f0b470deee
2015-07-28 10:40:56 +00:00
John Garbutt
979d1116fe xenapi: don't send terminating chunk on errors
If we send the terminator on an error, it looks like we have actually
completed the upload, when that was not actually the case, we have
failed, and ideally we want glance to notice and mark the snapshot as
failed.

This partly reverts the following commit:
e039b036b5e9dbaff8b37f7ab22c209b71bdc182

It is not a full revert of the above commit, and some of the refactoring
and error handling changes are still useful.

Closes-Bug: #1398826
Change-Id: Ib608a32556135a5b7e666d02c4ee77722ea6e07c
2015-01-14 13:58:18 +00:00
Nikhil Komawar
9211822923 XenAPI: Check image status before uploading data
Currently, the Xen API plugin sends the data payload to Glance
without checking the image status. In consequence, if the image is
not in queued status Glance will not let the image upload succeed.
This is by design, please refer [0].

The issue here being; buffering of the data will happen on the Glance
server. Consequently, the bound to be failed upload operation will not
get a 409 until after a large chunk of data has already been streamed
to the Glance server. This may result into client side wastage of
bandwidth and wait time.

This patch proposes a HEAD call be made from the glance plugin to
Glance API; thus in turn checking the valid image state for the
data transfer.

[0] http://docs.openstack.org/developer/glance/statuses.html

Change-Id: I24c1222a3b1ac809353724500117e58d50615ac8
2015-01-05 16:23:27 -05:00
Nikhil Komawar
43e89abeb2 XenAPI: Refactor message strings to remove locals
This commit is a small refactor to the XenAPI glance plugin removing
the use of locals() to improve the readability.

Change-Id: I5de095ac1f7f38fd37cc9d6010a876995f0f8841
2015-01-05 16:20:22 -05:00
Johannes Erdfelt
3664d8e466 xenapi plugins must target only Python 2.4 features
Add a comment documenting the restriction that our xenapi plugins must
target Python 2.4 features because of restrictions with the XenServer
dom0 userspace.

Change-Id: Ide130923b16c0555c3190cbd63ce97e18d18aae0
2014-11-10 07:29:03 -08:00
Christopher Lefelhocz
e03525bf1a Fix xen plugin to retry on upload failure
The xen plugin does not catch all cases where upload can throw
an exception due to connection failures and timeouts.  This
change extends the try/execpt window to catch these cases.

Also added original connect which in part keeps us from lazy-loading
the connection and logging of exception information so we can
figure out later what happened.

Change-Id: If643a7da9f602ab7bb5d255031a841c674fa6c6f
Fixes-bug: 1378903
2014-11-06 12:58:21 -06:00
Jesse J. Cook
cc2a34ce05 update retryable errors & instance fault on retry
HTTP errors can be split into a few categories: client ephemeral, server
ephemeral, server permanent, and globally permanent. You could argue
there is even more permutations. However, for simplicity, these errors
can be viewed as ephemeral and permanent.

The Glance XenAPI plugin has been updated to raise a RetryableError for
ephemeral and unexpected (i.e. errors that are categorized as neither
permanent or ephemeral) errors.

Additionally, an instance fault will be logged every time an error
occurs. This will serve as transaction history where every attempt is a
transaction on the state. If an ephemeral error occurs, there is a
retry, then a permanent error, the history of each error will be in the
instance_faults.

Deployers should configure the num_retries relative to the number of
api_servers. Right now, servers are put in random order and then cycled.
Trying the same server multiple times could cause unnecessary load and
delays. However, trying multiple servers, is ideal when a single server
is behaving badly or cannot reach another server it needs to communicate
with to fulfill a request.

Closes-Bug: 1380776

Change-Id: I267a5b524c3ff8a28edf1a2285b77bb09049773c
2014-10-27 12:25:40 -05:00
John Garbutt
d65ca49990 xenapi: deal with reboots while talking to agent
The latest version of the agent has started changing the hostname
through the official windows APIs, when resetnetwork is called. This
means a reboot might happen after resetnetwork.

A reboot happens when polling for the version. Until now we wait for the
call timeout, before fetching the new dom id. This change ensures that
if we spot a reboot, the plugin exits early rather than keeping polling
the wrong dom id.

Turns out its best to wait for the dom_id to change, before trying
to poll the agent again. Once the dom_id in xenapi has been updated,
the xenstore keys are always in place.

Trying too early leads to lots of reboot detections because we are
retrying with the old dom_id. XenServer continues to return the old
dom_id for a little while after the reboot.

Closes-Bug: #1370999

Change-Id: Id0bf5b64f2b271d162db5bbce50167ab1f665c87
2014-09-26 14:57:05 +01:00
Jenkins
daed1ebdd5 Merge "XenAPI: run vhd-util repair if VHD check fails" 2014-09-21 01:01:14 +00:00
John Garbutt
bfdae32efb XenAPI: run vhd-util repair if VHD check fails
We can hit issues with corrupted VHDs if we copy a VHD while XenServer
is performing other operations. This happens because there are times
when we copy the VHD chains while XenServer is still performing a
coalesce of the VHD chain.

In most cases, vhd-util should be able to safely repair any metadata
corruption. It can copy the copy of the VHD footer at the front of the
VHD file and add it at the bottom on the VHD file. There is no VM data
loss, due to the way the coalesce happens, but the chain will be bigger
than it would be both before and after the coalesce.

This does not, however, ensure that snapshots are valid before uploading
them to glance. But should you launch a corrupted snapshot, this change
would fix up the snapshot, and allow it to boot correctly.

Closes-Bug: #1362595

Change-Id: I88b737d7e97964a9db5ccf2c39dea7fd0701ead4
2014-09-17 13:46:39 +01:00
Jesse J. Cook
e039b036b5 xenapi: send chunk terminator on subprocess exc
During chunk upload if a subprocess exception is thrown, a malformed
chunk upload occurs.

This patch adds logging for subprocess exceptions and sends the
terminator even in the event of a subprocess exception. The incomplete
file upload will then be cleaned up by the client.

Change-Id: I559f201430bcd4c36d669c7772a10bfe3cd54d41
Closes-Bug: 1365637
2014-09-04 14:48:51 -05:00
Brian Elliott
77aea0e6a3 xenapi: Do not retry snapshot upload on 500
If Glance returns a 500 response on an initial attempt to upload
a snapshot image, it will set the image status to killed.  Any retry
attempts will fail with a 409 response.

Hence, Nova should not attempt to retry the upload in this case, which
will eliminate many pointless retries.

Closes-Bug: 1349933

Change-Id: I1a8f2416923a368b02cf8963c747ebbb24d749a1
2014-06-10 17:10:03 +00:00
Joe Gordon
855fe98ef4 Fix and Gate on E265
pep8 E265 makes sure block comment start with '# '. Fix and gate on this
new rule as it helps improve comment readability.

In the few cases where the comment was just commented out code, remove
the comment.

Change-Id: Iea1c445df8ddc2b6c17a4ab697ad756eef2f91fa
2014-07-24 08:11:00 -04:00
Davanum Srinivas
f3dc6eefe7 Fix and gate on H305 and H307
Both H305 and H307 are part of the OpenStack style guide.

Ensure that there are 3 groups of imports, stdlib,
third-party and project specific. Within each category
ensure that imports are in alphabetical order.

Change-Id: Id4994551c135c3c1a4982153f0c6cacba6176b95
2014-07-15 01:13:26 +00:00
Christopher Lefelhocz
5255364933 XenAPI: adjust bittorrent settings
This commit addresses some optimizations when using libtorrent to
download.  The settings are documented here:
http://www.rasterbar.com/products/libtorrent/reference-Settings.html

I'm proposing to add these with dom0 environment overrides in case
any other adjustments are needed.  The changes effectively double the
download rate from the default settings.  It also allows unbalanced
download/upload which is preferred in openstack environments.

No unit tests were added as plugins are not currently unit tested.

Closes-bug: #1303993

Change-Id: I503a6a521c196e4ad58e0a16aba2988f66a9cdca
2014-05-07 18:46:45 +00:00
Alex Gaynor
b055e16846 Fixed many typos.
These were found using: https://github.com/intgr/topy

Change-Id: Ia4e14508c285d95ab4eaeabbde032ecc5e7c9e4b
2014-04-25 21:47:41 -07:00
Bob Ball
b634bf3a0b XenAPI: Use local rsync rather than remote if possible
Using ssh depends on the host being set up to be able to SSH into
itself which is not a common scenario.  While this is unavoidable for
the current implementation of resize across multiple hosts, if there
is a single host (i.e. a test scenario) or the resize is restricted
to the same host then we can succeed without SSH access

Dependency on Ia310e31d31aaf5c979e41c64af8223202a18e03a is because the
tests will always fail without Ia310 therefore this fix cannot be tested
without taking Ia310 as well.

Closes-bug: 1308064

Change-Id: I15802a1d97d380b1c5b74fc9f92ece2494fe789a
2014-04-22 18:06:59 +01:00
John Garbutt
6b578665d4 xenapi: Cleanup tar process on glance error
Currently, when the network connection to glance is interrupted, there
is different behaviour on upload and download.

This change ensures the behaviour between the two code paths is more
consistent.

Uploads generally need to be given more time before they timeout, so
to keep a single timeout between upload and download, the timeout is
increased to 90 seconds.

At the same time, it ensures the tar process gets killed when any
issues occur with the communication between the hypervisor and
glance.

Change-Id: Id5396e5d3c1052dc2979476a886412da65e08670
Closes-Bug: #1284596
2014-04-08 15:48:00 +01:00
Jenkins
416edbe2c3 Merge "Rename Openstack to OpenStack" 2014-03-18 01:17:47 +00:00
Johannes Erdfelt
27c8086028 xenapi plugins: Make sure subprocesses finish executing
Fixes bug 1286187

Neither execute_get_output() nor execute() would wait for the process to
fully finish executing. This could potentially create a race condition
where commands execute in a different order than intended.

Also, for execute() it could cause a process to never finish executing if
it generates enough output to block writing to the pipe.

Change-Id: I3404f4b3ca1cddeec2f3e7b393817a6ccc42bec7
2014-02-28 16:25:21 +00:00
Alexander Bochkarev
dd4032e9fb Enable flake8 H404 checking
This check indicates on comments in which multi line docstring should
start without a leading new line. This change fixed all violators of
said check.

Change-Id: Ic7357b8c7420767dba611f6fcee07b7700f3aea8
2014-02-27 11:15:55 +04:00
guillaume-thouvenin
d5b8d5b18d XenAPI: Add the support for updating the status of the host.
We added a function into the xenhost plugin to get information
about PCI devices. Roughly we run the lspci command on dom0. This
information will be used to get the list of pci devices  that are
passed on the pciback.hide dom0 command line.

  The hide option is used to hide the devices from the normal guest
drivers and assign them to the pciback kernel driver at boot on dom0
instead of their normal driver. We will parse the output of the lspci
command to find which device is using the pciback kernel driver and
thus to know if it has been passed to the pciback.hide option.

  This information will be used to perform the match with the
list of pci devices provided in pci_whitelist into /etc/nova/nova.conf.

Implements: blueprint pci-passthrough-xenapi

Change-Id: I465fc5d29f3c47ab0079adcfcc2d7d6501bd4b20
2014-02-17 11:56:18 +01:00
tanlin
186faf363f Rename Openstack to OpenStack
Change-Id: Iacc51d8ad071c3b22b7bb6b21cff972599b59a7b
2014-02-13 16:20:50 +08:00
Bob Ball
270d4f1d6b XenAPI: Monitor the GC when coalescing
Add a new host plugin to monitor whether the garbage collector is running.
This is then used by the wait_for_vhd_coalesce function to know whether
waiting any more is likely to produce the expected results.

Increased the number of times we loop waiting for coalesce as the loop
now has knowledge that something is still happening.

DocImpact
Change to parameter default.

Change-Id: Idcc0738945bd1aee8cdb52c9cfabd798f85c31db
Closes-bug: 1258169
2014-02-05 23:54:37 +00:00
liu-sheng
74f953a1d7 Remove vi modelines
We don't need to have the vi modelines in each source file,
it can be set in a user's vimrc if required.

Also a check is added to hacking to detect if they are re-added.

Change-Id: I347307a5145b2760c69085b6ca850d6a9137ffc6
Closes-Bug: #1229324
2014-02-03 14:19:44 +00:00
John Garbutt
89b0d540d5 xenapi: stop hang during glance download
It seems there are cases when the server gets stuck in a building state
because the download from glance fails, but the nova-compute doesn't
find out.

After some code inspect, a likely cause seems to related to having no
socket timeout inside urllib2 when we download the image from glance. A
workaround for this issue is to configure the default socket timeout as
described here:
http://docs.python.org/3.1/howto/urllib2.html#sockets-and-layers

Fixes bug 1251650
Change-Id: I8b34d95d3524f3825b28e61cbbb59290653d3711
2013-12-03 18:42:24 +00:00
Brian Elliott
f20523aeca xenapi: Fix agent update message format
Fix the agent update message send so Windows guests will work

Message must be formatted: "url,md5sum"

Change-Id: I675cc5a7ca522696eab1820c14c48361ff7e0f74
Closes-Bug: 1250165
2013-11-12 20:19:25 +00:00
Joe Gordon
df4eb86234 Remove xenapi rpm building code
Nova shouldn't be building any types of packages. And if we are we should
be testing this code in gate somewhere.

Change-Id: I70d82a6173b5eda183d4e49d330ae97dd14f9078
2013-10-15 14:38:40 -07:00
Bob Ball
24fd331b8c XenAPI: Add versioning for plugins
Because the plugins live on a host seperate to Nova we need an interface
to test whether they are the expected version.  We can't use pip or other
requirement systems as they are cross-machine.

Closes bug 1226622

Change-Id: I58ab669061f51bd87071e2cf0d93d33021001309
2013-10-03 13:08:56 +00:00
Matt Riedemann
016e39734e Fix several flake8 issues in the plugins/xenserver code
Due to some issues in the gate, several flake8 errors got merged
recently. This patch fixes the most recent issues found.

For the H304 relative import errors, we have to skip those because they
are put into /etc/xensource/scripts which is the current working
directory when running the plugin.

For the H231 incompatible python 3 'except x,y' construct, we have to
skip those because this code is written to run on python 2.4 and 'except
x as y' does not work with python 2.4.

Note that in cleaning up some of the H304 failures for relative imports,
I also re-arranged the imports to follow the hacking guide of doing
standard library packages first, then third party packages, and finally
nova-specific packages.

Closes-Bug: #1229753

Change-Id: I1c2211fd6a10d43d7e65cdb4e18530397788cf2c
2013-09-24 09:04:21 -07:00
Jenkins
fc07ac93ba Merge "xenapi: Cleanup pluginlib_nova" 2013-09-24 00:27:12 +00:00
Rick Harris
097f9a34a2 xenapi: Cleanup pluginlib_nova
* Removes dead code
* Prefixes internal functions with underscore
* Removes a few one-off functions in favor of inlining

Change-Id: Ic2f8c2e512df270f3b1b0e35ded98832a5152c72
2013-09-23 18:49:03 +00:00
jmeridth
2266c04458 xenapi: fix pep8 violations in nova plugins
Removed plugins/xenserver from tox.ini excludes list, ran tox -e pep8, and
fixed violations.

Fixes: bug 1197893
Change-Id: I8e0116304b07d54c49a01202b3ce6d3426097bb4
2013-09-17 16:44:59 +00:00
Jenkins
7e8f27e1eb Merge "Adds Instance UUID to rsync debug logging" 2013-08-30 22:52:24 +00:00
Mate Lakat
36be3145c7 Add env to make_subprocess
Due to a previous commit, a client of make_subprocess is making a call
with an env parameter. As make_subprocess does not have such a
parameter, image upload fails. This patch adds the env keyword argument
to make_subprocess, and use that in the Popen call.

Fixes bug 1216267

Change-Id: I8098b484067b54c15fe95f2bf84c5890db7ebcb3
2013-08-27 10:43:32 +01:00
Jenkins
c1742adfe1 Merge "xenapi: Added iPXE ISO boot support" 2013-08-26 17:00:23 +00:00
Zaina Afoulki
1df56a5c4a Adds Instance UUID to rsync debug logging
This commit adds the instance uuid to the rsync debug output in
the debug logs. The issue we were seeing is that when multiple
migrations are ongoing at the same time, it's not easy to
track the rsync progress of each migration.
Fixes: bug #1215629

Change-Id: Iec662bdc0aed1c6f54ecaa062335dc5504ac030f
2013-08-22 16:09:09 -05:00
Teran McKinney
9a24ae88da nova.conf configurable gzip compression level
Currently only used by XenAPI.

Sets GZIP environment variable before spawning tar -cz to force the
compression level. Defaults to none, which means the GZIP environment
variable is not set and the default (usually -6) is used.

Implements: blueprint image-compression-mode

Change-Id: I02e136b7215a4f2cefc259f40bde9c1b205737ed
2013-08-22 17:22:36 +00:00
Josh Kearney
614c4a2359 xenapi: Added iPXE ISO boot support
This patch adds support for a new kind of Glance image, a specially crafted ISO
which supports iPXE booting, giving customers a means to roll their own image.

Two virt-layer modifications were needed. The first was adding configurations
for the iPXE ISO feature (network to use, boot menu, mkisofs_cmd). The second
was the ability to inject networking info into the ISO after it was
downloaded.

To use this feature, operators should enable the `ipxe_boot` image-property.

DocImpact

Implements blueprint xenapi-ipxe-iso-boot-support

Change-Id: I33acf9dfdff0a5ed9797723a142bc451348e8549
2013-08-21 22:17:33 +00:00
Jared Culp
47a20a9f44 xenapi: send identity headers from glance plugin
Send identity headers while uploading/downloading images from glance. Related
to:

1) Add identity headers while calling glanceclient from nova
Ife02059abbdce7920f1f408b71e9745d777fa770

2) Pass identity headers received in glanceclient to nova
Ifbef582aa4e64a2e7a46db43a9cc6cf8c3531dbd

3) Pass identity headers received by glance api to glance registry
Ie5f07ed6dfeaa8428de4f79c4d40d182328e6ab4

NOTE: since this is a change to a Dom0 plugin API, it requires the lock-step
upgrade of both the compute-manager and the dom0 plugin.

DocImpact

Change-Id: I6d5e3448d2c0acc392d18e0b88cec25cf313da5b
2013-08-08 17:51:22 +00:00
Jenkins
3475662e02 Merge "xenapi: no image upload retry on certain errors" 2013-08-08 16:02:56 +00:00
Brian Lamar
6d3d641de5 Remove * import from xenserver plugins
Recently a change was made to how/where the '_' method is imported. This
seems to have broken most xenserver plugins. These plugins shouldn't be
doing * imports anyway so I've gone in and cleaned up all places where
"from pluginlib_nova import *" was used and replaced with "import
pluginlib_nova".

Fixes bug 1207107

Change-Id: I3b2a299cf60543e0f766becfeb4740af791b3576
2013-07-31 18:17:45 -04:00
Sulochan Acharya
7aa6f8465c xenapi: no image upload retry on certain errors
There are a few upload errors that glance plugin should
not retry upload on, as it will fail again. This patch changes
upload to not retry on the following conditions in addition
to the 401 error that it already handles:

- CONFLICT
- REQUEST_ENTITY_TOO_LARGE
- PRECONDITION_FAILED
- FORBIDDEN

Fixes bug 1203726

Change-Id: I538a4f9b133169b2e398e7dfcb3af58fef204f03
2013-07-31 17:03:41 -05:00
Alex Gaynor
cd9d8e45c6 Remove the monkey patching of _ into the builtins
Previous _ was monkey patched into builtins whenever
certain modules were imported. This removes that and
simply imports it when it is needed.

Change-Id: I0af2c6d8a230e94440d655d13cab9107ac20d13c
2013-07-26 07:31:17 -07:00
Jenkins
e6812fc4e6 Merge "xenapi: Pass string arguments to popen" 2013-07-25 23:08:41 +00:00