When using the tasks API, sdk was unable to delete an image from
the provider. This is fixed in 0.32.0.
Change-Id: I97ebc2bc4ad458926b400c1ccf12e5853fc18761
It looks like 0.9.0 has released, and it seems like it requires
creating a dynamic client now. We're seeing jobs using nodepool fail
with
File "... /nodepool/driver/openshift/provider.py", line 21, in <module>
from openshift import client as os_client
ImportError: cannot import name 'client' from 'openshift'
This does break things like the project-config gate. Pin to 0.8.9 for
now until it is fixed.
Change-Id: Ie9ccb56f987b4d05c6fe9d4938ca74a110f55653
Glance uploads images in two major ways. There is the direct way and the
task based async way. openstacksdk 0.28.0-0.31.0 do not have functional
task based image uploads so we exclude them.
Change-Id: I1ed56fa199a6ace6c1292f9edc14ea6b497d4c89
Related-Story: https://storyboard.openstack.org/#!/story/2005851
Support for concurrency and rate limiting has been added to keystoneauth,
which is the library openstacksdk uses to talk to OpenStack. Instead
of managing concurrency in nodepool using the TaskManager and pool of
worker threads, let keystoneauth take over. This also means we no longer
have a hook into the request process, so we defer statsd reporting to
the openstacksdk layer as well.
Change-Id: If21a10c56f43a121d30aa802f2c89d31df97f121
The next patch removes the use of TaskManager from nodepool. This
patch is mostly to cover any momentary weirdness.
Change-Id: I95df226792b716ba0f60e70eb9e4072e097c85af
This change adds an experimental AWS driver. It lacks some of the deeper
features of the openstack driver, such as quota management and image
building, but is highly functional for running tests on a static AMI.
Note that the test base had to be refactored to allow fixtures to be
customized in a more flexible way.
Change-Id: I313f9da435dfeb35591e37ad0bec921c8b5bc2b5
Co-Authored-By: Tristan Cacqueray <tdecacqu@redhat.com>
Co-Authored-By: David Moreau-Simard <dmsimard@redhat.com>
Co-AUthored-By: Clint Byrum <clint@fewbar.com>
This reverts commit 7e1b8a7261.
openstacksdk >=0.19.0 fixes the filtering problems leading to all
ports being deleted. However openstacksdk <0.21.0 has problems with
dogpile.cache so use 0.21.0 as a minimum.
Change-Id: Id642d074cbb645ced5342dda4a1c89987c91a8fc
There is a bug in Rackspace Public Cloud that causes keystoneauth to
fail doing discovery which 0.20.0 of openstacksdk exposes because it
starts using keystoneauth discovery directly. Until the keystoneauth fix
lands and is released, running nodepool with 0.20.0 of openstacksdk will
fail when attempting to use Rackspace Public Cloud.
Just for the record, this is due to the fact that Rackspace Public
Cloud:
- has invalid integer project ids
- still senselessly keeps them in the compute service URL
- blocks access to the compute discovery document
The keystone team are kindly accepting a workaround fix to keystoneauth
even though it is a workaround for what is a completely invalid setup.
Change-Id: I72ec16ecb7770d97aa5703bdcfd3e8b188c89f19
This changes implements a Kubernetes resource provider.
The driver supports namespace request and pod request to enable both
containers as machine and native containers workflow.
Depends-On: https://review.openstack.org/605823
Change-Id: I81b5dc5abe92b71cc98b0d71c8a2863cddff6027
devstack will happily install openstacksdk from source here just from
required-projects, but we install nodepool into a virtualenv, and that
is just doing a pip install.
Add an install_openstacksdk function that installs openstacksdk into the
virtualenv so that the devstack magic for installing from source works.
Change-Id: I68c16d8601f6fd96863f205094d27c0da09ef73c
os-client-config is now just a wrapper around openstacksdk. The shade
code has been imported into openstacksdk. To reduce complexity, just use
openstacksdk directly.
openstacksdk's TaskManager has had to grow some features to deal with
SwiftService. Making nodepool's TaskManager a subclass of openstacksdk's
TaskManager ensures that we get the thread pool set up properly.
Change-Id: I3a01eb18ae31cc3b61509984f3817378db832b47
This small match had a surprising number of issues...
Even though we're testing returning 'text/plain' when 'text/html' is
specified, it turns out the accept headers real browsers send actually
specify "*/*;q=0.8" which means that unmatched types will be given the
same, lower weight.
This means in the extant code, the best_match() for a normal browser
request (without 'text/plain') will think that 'application/json' and
'text/plain' are the same preference, and will take the first in the
list, which happens to be 'application/json'. The result is that
*real* webrowsers are getting back json, when we wanted them to get a
human readable result.
Also, in the mean time webob has a new accept handling model, and
best_match() is deprecated anyway. Update the requirements to the
latest WebOb (new handling was added in 1.8.0, 1.8.1 is a bugfix)
So now we use the new API, and list 'text/html' as an acceptable
offer, which ensures it will be chosen if a browser is requesting
things. It still returns a text/plain table.
A check for this is added to the test suite. Also a bug with setting
the headers after the request is fixed, which went unnoticed because
of the prior default behaviour.
Change-Id: I84094ca67b16ce9246507aa0010646ffc3e9dbff
nodepool is python3 only now, so there is no need to continue to use the
six library. Remove its use and shift to just using python3 versions of
things.
Change-Id: Ic1d1b4b736deba1b8325adf0b446e2121eaa6b20
It seems like recent version fixes deadlock issue and there is security
concerns regarding PyCrypto: http://www.paramiko.org/installing-1.x.html
Change-Id: Ia464931228ebdec15354cfd1176c48f6ee48eb0e
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
For example, a cloud may get better preformance from a cinder volume
then the local compute drive. As a result, give nodepool to option to
choose if the server should boot from volume or not.
Change-Id: I3faefe99096fef1fe28816ac0a4b28c05ff7f0ec
Depends-On: If58cd96b0b9ce4569120d60fbceb2c23b2f7641d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Recent version of diskimage-builder now include the virtualenv logic,
so we can delete our copy.
Change-Id: Ic2d99258b3bd3487e8f10c50bfa10f8215d66b14
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Nova has an API call that can fetch the list of available AZs. Use it to
provide a default list so that we can provide sane choices to the
scheduler related to multi-node requests rather than just letting nova
pick on a per-request basis.
Change-Id: I1418ab8a513280318bc1fe6e59301fda5cf7b890
Even though there is nothing to read from secure.conf anymore, it
is kept around intact since we may want to use this for ZooKeeper
credentials at some point.
Change-Id: Ieb3a93b09c889f74da3463494957335aaaa9f40f
Remove files and fakes related to Jenkins. Since the 'targets'
config section was for mapping to Jenkins, this isn't needed either.
Change-Id: Ib5c615a95fcdce5234b3c63957171d77b8fbc65d
Sets up a shim for the new nodepool.NodePool.run() method that does
not require any service except ZooKeeper and removes all references
to Gearman/ZMQ.
Change-Id: I452c24d631592f47eb3f4cbffb56f3252f36c298
We have been using DIB_CHECKSUM for a few weeks now, and things
appear to be working well. Now switch to passing --checksum via
disk-iamge-create and properly clean up the checksum files when we
delete our images.
Change-Id: Ia849aa179750648be13455d1f38a6a01423bd8c0
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
For unknown reasons nodepoold will currently deadlock using
paramiko==2.0.0. I think the deadlock is related to when we bring a
new node online and SSH into it for the first time to run the
ready-script.
We should be able to reproduce it using a devstack functional test.
Change-Id: If4cbed55b23aff508d153164d7d7f90f9f3d5dc5
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
It should not happen in a neutron setup that we have leaked floating
ips. However, sometimes it seems that it happens around startup. It's
also safe in a neutron context to just clean the unattached ones. So
assume that sometimes clouds get into weird states and just clean them.
Change-Id: I1a30efb3b7994381592c2391881711d6b1f32dff
Depends-On: I93b0c7d0b0eefdfe0fb1cd4a66cdbba9baabeb09
Adding the config value "nat_destination" increased the surface area of
the nodepool.yaml networks list being used for two reasons. The primary
purpose of nodepool.yaml is to tell nodepool what to create - and the
networks list says "please boot the vms for this provider on all of the
networks in this list". However, there are, for some clouds, additional
information that needs to be configured about the cloud that should not
also imply that nodes should boot on the associated networks. A good
example is internap, where the networks do not follow patterns that
allow them to be discovered. One needs, in config, to tell shade "this
network is public and this network is private" but nodepool really only
wants to boot with a nic connected to the public network.
Long story short - we should not aim to duplicate all of the features of
clouds.yaml in nodepool.yaml. They are two different files with two
different purposes in life. One should use clouds.yaml to configure
qualities of the cloud, and nodepool.yaml to configure how you want to
use the cloud.
This reverts commit 2a30810b2e.
Change-Id: Ic7801d2ad49574f2b24c28c5a8f747ba73c6bf24
We need to know which networks are public/private, which we already have
in nodepool, but were not passing in to the OCC constructor. We also
need to be able to indicate which network should be the target of NAT in
the case of multiple private networks, which can be done via
nat_destination and the new networks list argument support in OCC.
Finally, 'use_neutron' is purely the purview of shade now, so remove it.
Depends-On: I0d469339ba00486683fcd3ce2995002fa0a576d1
Change-Id: I70e6191d60e322a93127abf4105ca087b785130e
We wrote shade as an extraction of the logic we had in nodepool, and
have since expanded it to support more clouds. It's time to start
using it in nodepool, since that will allow us to add more clouds
and also to handle a wider variety of them.
Making a patch series was too tricky because of the way fakes and
threading work, so this is everything in one stab.
Depends-On: I557694b3931d81a3524c781ab5dabfb5995557f5
Change-Id: I423716d619aafb2eca5c1748bc65b38603a97b6a
Co-Authored-By: James E. Blair <jeblair@linux.vnet.ibm.com>
Co-Authored-By: David Shrewsbury <shrewsbury.dave@gmail.com>
Co-Authored-By: Yolanda Robla <yolanda.robla-mota@hpe.com>
This adds a per-test zookeeper server fixture and demonstrates
its use in preparation for actual zookeeper use.
Change-Id: I99ac519057cdd9d9c829928e0e12ece65b0a5c43
Depends-On: I66d2bc90eb350424c051f480b4102209054ce2eb
This patch upgrades nodepool to support APScheduler 3.0. For the most
part, 3.0 was a rewrite but our changes seem to be limited.
This is part of the effort to packaged our CI tools for fedora
rawhide.
Change-Id: Ia39dca8327cbc89c6bfe707a82a41afe03f52ffa
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This is mostly needed for the fedora packaging effort and generally a
good ideal to keep in sync with other OpenStack projects.
Change-Id: I04f04c007bd85e4fa1627a8dd954dfd720bed270
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
As discussed in the original bug [1] one of the reasons to not
daemonize after import of paramiko is that it creates a global
Crypto.Random() object which is used around the place, however the
close of fd's in the daemonize breaks that object's connection to
/dev/urandom.
However, it seems that this issue is no longer as paramiko now
directly uses urandom and has no global object [2]. Requirements is
updated to the first release after this change.
We have also removed the use of the global statsd object in
I6d339a8c631f8508a60e9ef890173780157adefd.
Thus we should be able to return this to it's simpler form
[1] https://github.com/paramiko/paramiko/issues/59
[2] 6f211115f4
Change-Id: I6cf3ba0df857f4213b0ec658ebd0511a24618175
This is part of the effort to package our CI tools for Fedora rawhide.
Change-Id: I8c1ddd6778d121c8629cd612667281056ff276d9
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2.1.0 throws the exception:
daemon.daemon.DaemonOSEnvironmentError: Unable to change process owner ([Errno 1] Operation not permitted)
Change-Id: I1fde7751d0a480982e14c77e672cb38e3485314b
Using shade earlier than 0.12.0 causes our shade integration test to
fail with a missing argument.
Change-Id: I5f60225283ebd60cf8359ca3d962b04c6ddc8b8b
This is to fix a downstream issue between nodepool and
diskimage-builder pip dependencies. Since nodepool depends on
diskimage-builder, it would be nice to have common depends.
We also bump hacking to deal with pbr version issues.
Change-Id: I0efa395bc66c9e50bfe4a13e971232ad9184ad48
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Antoine moved the node management APIs to python-jenkins in
change I29423d8bab5fd So now nodepool can just use python-jenkins
directly. No need to maintain the same code in this project.
Change-Id: I3852106ae67ad3da4f14423c19f39af03694cb22
David Shrewsbury