XenServer has released os-xenapi lib on pypi, I have made a patch
https://review.openstack.org/#/c/406059/ use os-xenapi in nova
project. In this patch, we just delete those useless files and
folders.
Change-Id: I7cd83a492bae38fd9249f25b95663fa4385b262c
Partially-Implements: blueprint add-os-xenapi-library
String interpolation should be delayed to be handled
by the logging code, rather than being done
at the point of the logging call.
So add the following hacking rule for it.
- [N354] String interpolation should be delayed at logging calls.
See the oslo i18n guideline.
* http://docs.openstack.org/developer/oslo.i18n/guidelines.html
Change-Id: Ief6d3ee3539c0857098fffdb7acfeec3e0fed6eb
Closes-Bug: #1596829
All Python files, except executables, should have a '.py' extension.
This wasn't being done for files in the 'plugins/xenserver' folder,
likely because it seems unnatural to call a "plugin" with a file
extension.
Begin working towards a resolution for this by renaming all files to
include a '.py' extension. To avoid forcing operators to upgrade the
Nova service and per-host plugins at the same time, continue to
reference the old, non-'.py' filenames, using symlinks to ensure the
references continue to work. Once Ocata has been released, these
symlinks can be removed, the API version updated and the reference
updated to include the '.py' extension.
Change-Id: Icf4cf2424c15276a6c01778fb92f76bbdc9b2068
Closes-bug: #1302831
When introducing interim bridge for XenAPI, it introduced a interface
which is connected to the interim bridge and it has the name as
vifxxxxxxxx-xx. In the xenapi plugin of bandwidth, it uses
the pattern of "^vif*" to filter out VM's interfaces. Due to the
existing of vifxxxxxxxx-xx, it will get exception when parsing the
interface's name basing on "vif<dom#>.<dev>".
So let's give more restrict pattern to filter VM's vifs e.g. change from
"^vif*" to "^vif<dom#>.<dev#>". May consider to use another interface
name for the interface on interim bridge; but need cover both name
patterns when delete interfaces to avoid upgrade issues which may make
things complicated.
Change-Id: I36b96bbe2bb764f3d60cbeb43aa3dbd9a0a43f61
Closes-Bug: #1597551
Add unit test for plugin bandwidth. And there is a bug in
the plugin which does not close the opened file. This change
will try to fix it. It also splits _read_proc_net() into two
functions, so that it's easier to make the UT.
Change-Id: Ie871b1d16cf180e9709e46bb0d67573053d14376
Move some disk operations from DomU to Dom0 to allow Nova
to run independently from the hypervisor.
Some disk types can be created in Dom0 (e.g. swap, ext3) but
others cannot. Add support here for creating those that can
in Dom0. This will be documented in the independent_compute
config flag in the last patch in this series.
Implements: blueprint xenapi-independent-nova
Change-Id: I634e783a5e19f9710544920e12300b299fc06b36
This implementation is to give support on neutron security group with
XenServer as compute driver. When using neutron+openvswitch, the ovs
agent on compute node cannot run correctly due to lack of qbr linux
bridge on compute node. This change will add qbr linux bridge when
xenserver as hypervisor
Xenserver driver now doesn't have linux bridge, the connection is:
compute node: vm-vif -> br-int -> br-eth
network node: br-eth -> br-int -> br-ex
With this implemented, linux bridge(qbr) will be added in compute
node. Thus the security group rules can be applied on qbr bridge.
The connection will look like:
compute node: vm-vif -> qbr(linux bridge) -> br-int -> br-eth
network node: br-eth -> br-int -> br-ex
Closes-Bug: #1526138
Implements: blueprint support-neutron-security-group
DocImpact: /etc/modprobe.d/blacklist-bridge file in dom0 should be
deleted since it prevent loading linux bridge module in dom0
Depends-On: I377f8ad51e1d2725c3e0153e64322055fcce7b54
Change-Id: Id9b39aa86558a9f7099caedabd2d517bf8ad3d68
With xen hypervisor, it will create two sets of port/interface
for each VM vif: One is named as tapx.y which is qemu-emulated
NIC and is used when no PV drivers installed with the VM; The
other one is named as vifx.0 which is the xen network frontend
NIC and is used when VM has PV drivers installed. But neutron
only handle one port. If the finally used port is not the one
handled by neutron, the guest VM will have no network access.
The solution is to change nova xenapi to add a bridge, which
we call the 'interim bridge', between the VM and the OVS
integration bridge. In this way it will only expose to neutron
with the integration bridge's port which connected to the
interim bridge. So it will fix the issue mentioned above.
Closes-Bug: #1268955
Change-Id: I0cfc0284e1fcd1a6169d31a7ad410716037e5cc2
This code makes xenplugin version agnostic
and allows to work with both apis, depending
on what version is considered as 'current'
Plugin version is bumped to 1.4.
partially implements bp use-glance-v2-api
Co-Authored-By: Sudipta Biswas <sbiswas7@in.ibm.com>
Change-Id: I84f6971afaeaeed69a2d3e93a34af8df70d1fb00
This patchset gets rid of all the old download_vhd/upload_vhd code.
The old _vhd calls won't work anymore since we won't send them the
right parameters.
Change-Id: I8c0b47a7e8a94ee4b20c888bd86c1a77920103b8
Some files have regressed since '798058' ensured PEP8 compliance.
Correct these regressions in preparation for enabled PEP8 checks.
Change-Id: I2d0bae469673e4b5e2238978a7ba63f6bfe0358a
Partial-Bug: #1302831
When downloading a streamed chunk from swift through glance gives tar
invalid input, tar will exit, but leave the plugin still trying to write
data to the dead process.
To stop this we can spot when the tar process exits early, and be sure
to stop trying to write more data to the dead process.
Change-Id: Ic2bc89fa6d08db505b044a9498c1bfa5b884a056
Closes-Bug: 1552293
Before enabling PEP8 checks for these files, the actual PEP8 issues
need to be resolved. There are a number of "types" of issues that need
to be fixed:
* E112 expected an indented block
* E241 multiple spaces after ','
* E265 block comment should start with '# '
* E302 expected 2 blank lines, found 1
* H102 Apache 2.0 license header not found
* H237 module module is removed in Python 3
* H303 No wildcard (*) import.
* H404 multi line docstring should start without a leading new line
* H231 Python 3.x incompatible 'except x,y:' construct
* F403 from module import *' used; unable to detect undefined names
* F841 local variable 'name' is assigned to but never used
* N344 Use six.iteritems() instead of dict.iteritems().
The fixes for the above are all pretty self-explanatory, but there is
one exception. The 'N344' issues are not ignorable due to how the
"hacking" check is implemented. Rather than ignoring them, skip the
issue altogether by replacing 'iteritems' with 'items'. The difference
in memory usage that this incurs should not be noticable.
Change-Id: Ie26e3daca3b408c9ea43613bcb9996618b9abc69
Partial-Bug: #1302831
This adds interfaces to the glance plugin for uploading / downloading
vhd files by glance url instead of with a host / port / use_ssl
tuple. This makes it possible to have glance hosted on a non root url
and still work with xenserver.
This bumps the plugin version to 1.3 as these are new functions exposed.
Convert xenapi driver to using up/down vhd2
All unit test converted to use this new functionality.
Change-Id: I73f60de0b1644827238961e1659b0119b80bf82f
This makes internal changes to the xen glance plugins so that they are
internally passing information about the glance servers by urls. It
does not change the surface of the plugin (that will come later), so
at a top level we are still thinking of glance as a host, port, and
use_ssl. However that's immediately converted to an endpoint url which
is then used for all the rest of the effort.
Change-Id: Ie1dce099846c725ff1ad7bb32f74aa6ae701ee9a
Currently tgz images are supported only when the names of VHD files
within follow the format of "0.vhd, 1.vhd, ... (n-1).vhd" called VDI
chain.
To maximize the range of support, the arbitrary named VHD file could be
renamed to 0.vhd when the tgz image only has a single VHD file.
Change-Id: I01cadf26262b20f1ad5a19e4441b42a0155f5a52
Currently the tox py34 target uses testtools.run to run
a subset of our test harness. We need to be able to use
pretty_tox.sh just like py27 as we make progress with py34
support.
The first step is to make sure we can discover all the
tests using:
python -m subunit.run discover -t . ./nova/tests/ --list
So, we need to fix a bunch of things for the discovery
to work including updating to a new version of websockify.
In the xen code, we should keep the original import and
add except for py34 as xen uses an older python that does
not work with six.moves.
Depends-On: Ib4ef2e79b28b7180e564b3d6dc2188456c66c08a
Change-Id: I88b6746da6136a7386a173f6cacd42f0b470deee
If we send the terminator on an error, it looks like we have actually
completed the upload, when that was not actually the case, we have
failed, and ideally we want glance to notice and mark the snapshot as
failed.
This partly reverts the following commit:
e039b036b5e9dbaff8b37f7ab22c209b71bdc182
It is not a full revert of the above commit, and some of the refactoring
and error handling changes are still useful.
Closes-Bug: #1398826
Change-Id: Ib608a32556135a5b7e666d02c4ee77722ea6e07c
Currently, the Xen API plugin sends the data payload to Glance
without checking the image status. In consequence, if the image is
not in queued status Glance will not let the image upload succeed.
This is by design, please refer [0].
The issue here being; buffering of the data will happen on the Glance
server. Consequently, the bound to be failed upload operation will not
get a 409 until after a large chunk of data has already been streamed
to the Glance server. This may result into client side wastage of
bandwidth and wait time.
This patch proposes a HEAD call be made from the glance plugin to
Glance API; thus in turn checking the valid image state for the
data transfer.
[0] http://docs.openstack.org/developer/glance/statuses.html
Change-Id: I24c1222a3b1ac809353724500117e58d50615ac8
This commit is a small refactor to the XenAPI glance plugin removing
the use of locals() to improve the readability.
Change-Id: I5de095ac1f7f38fd37cc9d6010a876995f0f8841
Add a comment documenting the restriction that our xenapi plugins must
target Python 2.4 features because of restrictions with the XenServer
dom0 userspace.
Change-Id: Ide130923b16c0555c3190cbd63ce97e18d18aae0
The xen plugin does not catch all cases where upload can throw
an exception due to connection failures and timeouts. This
change extends the try/execpt window to catch these cases.
Also added original connect which in part keeps us from lazy-loading
the connection and logging of exception information so we can
figure out later what happened.
Change-Id: If643a7da9f602ab7bb5d255031a841c674fa6c6f
Fixes-bug: 1378903
HTTP errors can be split into a few categories: client ephemeral, server
ephemeral, server permanent, and globally permanent. You could argue
there is even more permutations. However, for simplicity, these errors
can be viewed as ephemeral and permanent.
The Glance XenAPI plugin has been updated to raise a RetryableError for
ephemeral and unexpected (i.e. errors that are categorized as neither
permanent or ephemeral) errors.
Additionally, an instance fault will be logged every time an error
occurs. This will serve as transaction history where every attempt is a
transaction on the state. If an ephemeral error occurs, there is a
retry, then a permanent error, the history of each error will be in the
instance_faults.
Deployers should configure the num_retries relative to the number of
api_servers. Right now, servers are put in random order and then cycled.
Trying the same server multiple times could cause unnecessary load and
delays. However, trying multiple servers, is ideal when a single server
is behaving badly or cannot reach another server it needs to communicate
with to fulfill a request.
Closes-Bug: 1380776
Change-Id: I267a5b524c3ff8a28edf1a2285b77bb09049773c
The latest version of the agent has started changing the hostname
through the official windows APIs, when resetnetwork is called. This
means a reboot might happen after resetnetwork.
A reboot happens when polling for the version. Until now we wait for the
call timeout, before fetching the new dom id. This change ensures that
if we spot a reboot, the plugin exits early rather than keeping polling
the wrong dom id.
Turns out its best to wait for the dom_id to change, before trying
to poll the agent again. Once the dom_id in xenapi has been updated,
the xenstore keys are always in place.
Trying too early leads to lots of reboot detections because we are
retrying with the old dom_id. XenServer continues to return the old
dom_id for a little while after the reboot.
Closes-Bug: #1370999
Change-Id: Id0bf5b64f2b271d162db5bbce50167ab1f665c87
We can hit issues with corrupted VHDs if we copy a VHD while XenServer
is performing other operations. This happens because there are times
when we copy the VHD chains while XenServer is still performing a
coalesce of the VHD chain.
In most cases, vhd-util should be able to safely repair any metadata
corruption. It can copy the copy of the VHD footer at the front of the
VHD file and add it at the bottom on the VHD file. There is no VM data
loss, due to the way the coalesce happens, but the chain will be bigger
than it would be both before and after the coalesce.
This does not, however, ensure that snapshots are valid before uploading
them to glance. But should you launch a corrupted snapshot, this change
would fix up the snapshot, and allow it to boot correctly.
Closes-Bug: #1362595
Change-Id: I88b737d7e97964a9db5ccf2c39dea7fd0701ead4
During chunk upload if a subprocess exception is thrown, a malformed
chunk upload occurs.
This patch adds logging for subprocess exceptions and sends the
terminator even in the event of a subprocess exception. The incomplete
file upload will then be cleaned up by the client.
Change-Id: I559f201430bcd4c36d669c7772a10bfe3cd54d41
Closes-Bug: 1365637
If Glance returns a 500 response on an initial attempt to upload
a snapshot image, it will set the image status to killed. Any retry
attempts will fail with a 409 response.
Hence, Nova should not attempt to retry the upload in this case, which
will eliminate many pointless retries.
Closes-Bug: 1349933
Change-Id: I1a8f2416923a368b02cf8963c747ebbb24d749a1
pep8 E265 makes sure block comment start with '# '. Fix and gate on this
new rule as it helps improve comment readability.
In the few cases where the comment was just commented out code, remove
the comment.
Change-Id: Iea1c445df8ddc2b6c17a4ab697ad756eef2f91fa
Both H305 and H307 are part of the OpenStack style guide.
Ensure that there are 3 groups of imports, stdlib,
third-party and project specific. Within each category
ensure that imports are in alphabetical order.
Change-Id: Id4994551c135c3c1a4982153f0c6cacba6176b95
This commit addresses some optimizations when using libtorrent to
download. The settings are documented here:
http://www.rasterbar.com/products/libtorrent/reference-Settings.html
I'm proposing to add these with dom0 environment overrides in case
any other adjustments are needed. The changes effectively double the
download rate from the default settings. It also allows unbalanced
download/upload which is preferred in openstack environments.
No unit tests were added as plugins are not currently unit tested.
Closes-bug: #1303993
Change-Id: I503a6a521c196e4ad58e0a16aba2988f66a9cdca
Using ssh depends on the host being set up to be able to SSH into
itself which is not a common scenario. While this is unavoidable for
the current implementation of resize across multiple hosts, if there
is a single host (i.e. a test scenario) or the resize is restricted
to the same host then we can succeed without SSH access
Dependency on Ia310e31d31aaf5c979e41c64af8223202a18e03a is because the
tests will always fail without Ia310 therefore this fix cannot be tested
without taking Ia310 as well.
Closes-bug: 1308064
Change-Id: I15802a1d97d380b1c5b74fc9f92ece2494fe789a
Currently, when the network connection to glance is interrupted, there
is different behaviour on upload and download.
This change ensures the behaviour between the two code paths is more
consistent.
Uploads generally need to be given more time before they timeout, so
to keep a single timeout between upload and download, the timeout is
increased to 90 seconds.
At the same time, it ensures the tar process gets killed when any
issues occur with the communication between the hypervisor and
glance.
Change-Id: Id5396e5d3c1052dc2979476a886412da65e08670
Closes-Bug: #1284596
Fixes bug 1286187
Neither execute_get_output() nor execute() would wait for the process to
fully finish executing. This could potentially create a race condition
where commands execute in a different order than intended.
Also, for execute() it could cause a process to never finish executing if
it generates enough output to block writing to the pipe.
Change-Id: I3404f4b3ca1cddeec2f3e7b393817a6ccc42bec7
This check indicates on comments in which multi line docstring should
start without a leading new line. This change fixed all violators of
said check.
Change-Id: Ic7357b8c7420767dba611f6fcee07b7700f3aea8