2497 Commits

Author SHA1 Message Date
James E. Blair
1845788a95 Add more detail to TESTING.rst
Add additional sections to the testing instructions; these are based
on the same sections in the Zuul repo.  This brings the nodepool repo
to parity.

Change-Id: I7a1b2c62963a815a1ab864e3685be1653e5734fc
2020-06-09 08:48:43 -07:00
Zuul
fa2a850cb9 Merge "openshiftpods: define ca_crt parameter if available" 2020-05-18 12:14:53 +00:00
Zuul
2456820f89 Merge "Update dib to 2.36.0" 2020-05-14 15:11:02 +00:00
Zuul
9d2d6cf942 Merge "container functional test : allow to specify elements-dir" 2020-05-13 19:09:48 +00:00
Zuul
ee72fd8586 Merge "Functional container tests: update to CentOS 8" 2020-05-12 21:09:44 +00:00
Zuul
47193a46b9 Merge "Allow disabling build-log-retention" 2020-05-12 20:59:37 +00:00
Zuul
108e5c9644 Merge "Use fqdn for builder hostname info" 2020-05-12 20:45:06 +00:00
Zuul
dd4a993e38 Merge "Logs stats for nodepool automated cleanup" 2020-05-07 23:52:39 +00:00
Zuul
5cd0f371c5 Merge "Dockerfile: use local openstack-ci-core PPA key" 2020-05-07 21:24:17 +00:00
Pierre-Louis Bonicoli
48524da2ca
openshiftpods: define ca_crt parameter if available
Reuse the CA certificate if one is available. CA certificate can be
defined in the kube/config file used by Nodepool service.

Fix the following error:

    DEBUG zuul.AnsibleJob.output: [...]
    Ansible output: b'fatal: [molecule]: FAILED! => {
        [...]
        "failed_modules": {
            "setup": {
                "failed": true,
                "module_stderr": "Unable to connect to the server: x509: certificate signed by unknown authority",
                "module_stdout": "",
                "msg": "MODULE FAILURE See stdout/stderr for the exact error",
                "rc": 1,
            }
        },
        "msg": "The following modules failed to execute: setup"
    }

Change-Id: Ic2b764e88d966a5c501e72ba3dfb46436979072c
2020-05-07 19:16:46 +02:00
Ian Wienand
e8bb64a52a Use fqdn for builder hostname info
After Ia94f20b15ab9cf2ae4969891eedccec8d5291d36 the hostname fields
are just used for informational purposes.  Use the FQDN so something
like 'nb01.opendev.org' and 'nb01.openstack.org' are differentiated in
the output of something like 'nodepool dib-image-list'.

Change-Id: I1f061958ff271f707fddbe9ef74fd2e2a228e4ca
2020-05-07 12:47:25 +10:00
Zuul
30c1ccb882 Merge "Build nodepool with python3.8" 2020-05-06 23:09:33 +00:00
Zuul
d4651fe89d Merge "Build multi-arch images for x86 and arm" 2020-05-06 23:07:45 +00:00
Monty Taylor
ad625b8d8c Build nodepool with python3.8
We updated the zuul images to py3.8, let's update the nodepool ones
too.

Change-Id: Id4b9c00d14c5fb0dd192760084c6563a8bae3636
2020-05-05 09:23:02 -05:00
Ian Wienand
ae784c57a2 Handle ipv6 literal zookeeper addresses
If you use a ipv6 literal as the host: parameter in zookeeper-services
section, it gets incorretly concatentated with the port when building
the host list.  Detect ipv6 literals and quote them with rfc2732 []'s
when building the host string.

Change-Id: I49a749142e8031be37a08960bf645faca4e97149
2020-05-04 06:52:07 -07:00
Monty Taylor
5530478860 Build multi-arch images for x86 and arm
So that we can run nodepool-builder in containers on arm
hosts to better build arm images, update our zuul config
to build multi-arch docker images.

Change-Id: I98ec4cef2ff35c707ff43d0b8e554b969d720250
2020-04-30 14:22:06 -05:00
Ian Wienand
043a4258fc Dockerfile: use local openstack-ci-core PPA key
We see timeouts trying to get this key fairly frequently in the gate.
Store it locally and use that in the container build.

Change-Id: Ifd706849f1fad88c8ec4afc79090df4afb88abb4
2020-04-30 17:32:02 +10:00
Zuul
fd5915a070 Merge "Switch to fedora-30 for the openshift integration job" 2020-04-30 03:46:16 +00:00
Ian Wienand
c22fe0518d container functional test : allow to specify elements-dir
This will allow testing of external elements with this playbook

Change-Id: I6e6aeaa60ca30f10734ed19cb7bd949d621c0636
2020-04-30 11:45:38 +10:00
Ian Wienand
b9f6f6bf62 Allow disabling build-log-retention
This allows setting build-log-retention to -1 to disable automatic
collection of logs.  This would facilitate managing these logs with an
external tool like logrotate.  Another case is where you have the
builds failing very quickly -- say, one of the builds has destroyed
the container and so builds fail to even exec dib correctly.  In this
case it's difficult to get to the root-cause of the problem because
the first build's logs (the one that destroyed the container) have
been repead just seconds after the failure.

Change-Id: I259c78e6a0e30b4c0a8d2f4c12a6941a2d227c38
2020-04-29 13:07:07 +10:00
Ian Wienand
e998607471 Update dib to 2.36.0
The most important change here is fixes for building SuSE with
pip-and-virtualenv; something that has been failing for some time.

Change-Id: I90328693f3ad45f44bec72fb9e72f45ac3be6790
2020-04-28 10:31:38 +10:00
Ian Wienand
ec5e0909de Add nodepool as required project to siblings image build
When used outside the nodepool repository, we need to ensure the
nodepool project is included for sibling image build.

Change-Id: Ie696abe98620ff1036fe12069f0df89eaab47ef7
2020-04-24 11:23:43 +10:00
Zuul
ca31fcaa68 Merge "func-tests: drop debuntu specific env vars" 2020-04-23 23:50:21 +00:00
Zuul
0b5f2b29d2 Merge "Siblings container build: set work dir to nodepool" 2020-04-23 23:29:20 +00:00
Zuul
3f082077c8 Merge "Parallelize initial static node synchronization" 2020-04-23 21:58:18 +00:00
Monty Taylor
ed75d0536b Actually install extras from nodepool_base
We added two packages to extras so that they'd end up in the
container images, but we never told anything to install them.
It became clear that that's confusing, so we added an api
to python-builder to allow specifying a list of extras to
install.

Depends-On: https://review.opendev.org/722125
Change-Id: I27e10822744863560febcdad8bab9a4f3cf8fc8e
2020-04-22 15:01:30 -05:00
Zuul
cbdf37450c Merge "Dockerfile: incorporate workaround deboostrap" 2020-04-22 13:57:31 +00:00
Ian Wienand
10ff9dcf1c Functional container tests: update to CentOS 8
Update the functional tests to CentOS 8

Depends-On: https://review.opendev.org/721726
Change-Id: I08b7173767fd4dedd81f932bbfbf8761fd38b653
2020-04-22 10:40:38 +10:00
Ian Wienand
140068b514 func-tests: drop debuntu specific env vars
These tests are used generically to build a range of images, not just
debuntu based ones.  Remove these platform specific flags.

In the jobs based on this for the nodepool gate we are building
centos images, so these don't apply there.

Change-Id: Ia4dde1fb01da284a5e681237ab88c68fb9afcbef
2020-04-22 10:40:38 +10:00
Ian Wienand
8ab4c48d24 Siblings container build: set work dir to nodepool
Set the working directory to the nodepool checkout so that when we use
this job from other repos, it finds and builds the right Dockerfile.

Change-Id: I8578dd612d58387ad20c43404444df43f41a6723
2020-04-22 10:40:38 +10:00
Ian Wienand
edba92e412 Dockerfile: incorporate workaround deboostrap
As described in the updated comment section, this debootstrap from the
openstack-ci PPA works around some issues building inside a container.

Change-Id: I0887a801bb6dd4ce992c39d9e332a18f8194a7b9
2020-04-22 10:40:38 +10:00
Zuul
5d27775bc5 Merge "Set pool info on leaked instances" 2020-04-21 19:13:24 +00:00
Zuul
47cbae1985 Merge "Log openstack console at info" 2020-04-21 18:13:14 +00:00
Clark Boylan
257e26b0a4 Set pool info on leaked instances
We need to set pool info on leaked instances so that they are properly
accounted for against quota. When a znode has provider details but not
pool details we can't count it against quota used but also don't count
it as unmanaged quota so end up in limbo.

To fix this we set pool info metadata so that when an instance leaks we
can recover the pool info and set it on the phony instance znode records
used to delete those instances.

Change-Id: Iba51655f7bf86987f9f88bb45059464f9f211ee9
2020-04-21 10:41:40 -07:00
James E. Blair
5ff79aa67d Log openstack console at info
This only happens if the user has enabled console-log on the label,
so log it at info level so it appears in the standard logs.

Change-Id: I74dbc82bbbd310ba788250d864869681603babd2
2020-04-21 10:21:54 -07:00
Tobias Henkel
de6eaacc8f
Parallelize initial static node synchronization
We currently register static nodes serially on startup of the static
node provider. This is not a problem as long as all static nodes are
reachable. However if there are multiple static nodes unreachable and
fail with timeout the startup of the provider can take very long since
it must wait for many timeouts one after another. In order to handle
this better parallelize the initial sync of static nodes.

Change-Id: I4e9b12de277ef0c3140815fb61fed612be2d9396
2020-04-21 09:44:40 +02:00
Monty Taylor
d63c833815 Add yappi and objgraph to an extra
So that they wind up in the nodepool images, we need to list
them somewhere.

Change-Id: I47b43407c86c5846ca658963c298b05f280c3b42
2020-04-16 10:02:01 -05:00
Zuul
4c521fd208 Merge "config_validator: refactor the schema to a static method" 2020-04-16 03:15:28 +00:00
Zuul
003bd04e38 Merge "Test alternative username in unit tests" 2020-04-15 19:44:09 +00:00
Zuul
06aacd17be Merge "diskimage.username setting was not read from configuration file" 2020-04-15 19:35:14 +00:00
Zuul
a5924345d5 Merge "Mark dib as deleting before erasing files from disk" 2020-04-15 18:26:58 +00:00
James E. Blair
fe5685237c Mark dib as deleting before erasing files from disk
To avoid a situation where nodepool deletes an image build but
continues to indicate that it is "ready" in the ui, make sure that
we set the state to "deleting" before we start deleting.

Change-Id: I0f87dd93262ba46f42931d83d123244dfe85cd2f
2020-04-15 10:29:27 -07:00
Zuul
3c2d8ef64b Merge "Add requires to zuul-quick-start job" 2020-04-15 14:36:58 +00:00
Zuul
f29f1646e1 Merge "k8s/OKD Provider: Don't Set ca_cert if TLS verification is skipped" 2020-04-15 13:15:21 +00:00
Ian Wienand
ce00f347a4
Logs stats for nodepool automated cleanup
As a follow-on to I81b57d6f6142e64dd0ebf31531ca6489d6c46583, bring
consistency to the resource leakage cleanup statistics provided by
nodepool.

New stats for cleanup of leaked instances and floating ips are added
and documented.  For consistency, the downPorts stat is renamed to
leaked.ports.

The documenation is re-organised slightly to group common stats
together.  The nodepool.task.<provider>.<task> stat is removed because
it is covered by the section on API stats below.

Change-Id: I9773181a81db245c5d1819fc7621b5182fbe5f59
2020-04-15 14:48:36 +02:00
Zuul
28b1f67dd0 Merge "Add debug info on json parse error of image upload" 2020-04-15 12:39:18 +00:00
Ian Wienand
bff3e7f554 Test alternative username in unit tests
As a follow-on to I615e530decbee6a46167a40748342d2193851c02, switch
one of the usernames in testing away from the default to test setting
the username from config.

Change-Id: Id05561b942ed96bc3cc011df6906d706f12d80bf
2020-04-15 15:31:45 +10:00
Zuul
775cd32028 Merge "Add ZooKeeper TLS support" 2020-04-15 01:41:47 +00:00
Zuul
f0532631a6 Merge "Stop checking user_data in func test" 2020-04-15 01:27:24 +00:00
Michał Suszko
314a6ac48a diskimage.username setting was not read from configuration file
Change-Id: I615e530decbee6a46167a40748342d2193851c02
2020-04-15 09:48:45 +10:00