Commit Graph

413 Commits

Author SHA1 Message Date
Clark Boylan
a6d4fae070 Tune sshd connections settings on test nodes
Update the sshd_config on our test nodes to accomodate what appears to
be an increase in ssh scanner traffic. In particular LoginGraceTime
defaults to 120 seconds. We reduce that to 30 seconds to cycle
connections more quickly. Then we also increase the maximum number of
connection startups to 30 from the default of 10. We also reduce the
random fail rate from 30% to 10% between 31 and 100 connections.

I'm not entirely certain this will fix things, but based on what we've
seen from logs it may be what we need to make ssh to test nodes more
reliable.

Change-Id: Ifacf7d00de157ab2fb60cde990f0b49f03f71415
2022-08-17 12:40:46 -07:00
Ian Wienand
4fbc3e1b30 infra-package-needs: blank out coreutils for Rocky 9
Rocky 9 has coreutils-single package installed, so trying to install
coreutils package conflicts.  Just blank this out for this platform
like 8.

Change-Id: I48933a61a065cee9402cb803b0da214eafe2cd8a
2022-08-11 13:37:55 +10:00
Ian Wienand
c043b9c2b6 nodepool: update package maps for Rocky 9
Update the package maps.  It also seems like matching just "9" will
cover 9-stream and Rocky, which just uses DIB_RELEASE "9".  Also fixup
the 80-enable-haveged to skip on rocky.

Change-Id: Ia352d217d00e10068a463b62f7d9aca72cb88a8c
2022-08-11 10:17:10 +10:00
wangxiyuan
e37ee42a41 Update openeEuler pkg mapping
openEuler 22.03 supports python3 only. There is no
python-devel package.

This patch update the pkg mapping to fix the package
install problem in nodepool.

Change-Id: I11750048841ec49c893b4c9332a6029b329b54cb
2022-08-05 09:17:42 +08:00
Ian Wienand
efbb9b8961 nodepool elements: fix pip upgrade venv
Change I316e9587b6e290cd421b47f506c91dbebe0975c0 had a rather
embarrasing oversight in that it copied the /usr/bindep-env/bin/pip
invocation for upgrading pip to all the other venv's.

i.e. we were upgrading hte bindep-env pip over and over, and not
actually the pip in the working venv.  The os-testr install on older
platforms has now broken because it still tries to install with the
ancient inbuilt pip -- local testing has confirmed that it works with
the updated pip.

Change-Id: I22c549b5f9b9e3882fcd2340946d2850b0b2f86b
2022-07-06 13:27:58 +10:00
Clark Boylan
3e57097f50 Install setfacl on test nodes
Ansible v5 appears to rely on setfacl more than ansible 2.9 did when
running tasks as a different unprivileged user than the one currently
running ansible. Without setfacl installed we get errors like:

  Failed to set permissions on the temporary files Ansible needs to
  create when becoming an unprivileged user (rc: 1, err: chmod:
  invalid mode: ‘A+user:stack:rx:allow’ Try 'chmod --help' for more
  information.}). For information on working around this, see
  https://docs.ansible.com/ansible-core/2.12/user_guide/become.html#risks-of-becoming-an-unprivileged-user

Installing setfacl makes the error go away as ansible get use setfacl
instead of chown/chmod.

Ubuntu, Debian, Fedora, CentOS, and OpenSUSE all appear to call the
package 'acl'. We assume that openeuler and rocky inherit this package
name. That means we only need to override the package name for Gentoo.

Change-Id: I71736578dbd5e0683b18023e73ab44255eb6eb18
2022-05-24 13:02:25 -07:00
Michael Johnson
4ceb4adc83 Fix selinux context for unbound.log
The previous patch[1] applied a partial context to the unbound.log file.
This patch applies a full context to resolve the "partial context" error, using semanage to make the file label persistent.

[1] https://review.opendev.org/c/openstack/project-config/+/841546

Change-Id: Ic15957fa4ef58355efd2e96f143386f393b0a59d
2022-05-12 19:25:06 +00:00
Ian Wienand
4be8c66630 Set context for unbound.log on selinux systems
I4f3265c16320613d4ba74a02df1361c5d9cf2fb1 moved this file to
/var/lib/unbound on selinux systems, as it was getting permissions
errors trying to write into /var/log.

This turns out to make it harder to collect the logs from projects
like devstack.  It's simpler if we just have the log file in a
consistent place.  On selinux systems, set the context, and revert
things to just writing into /usr/log/unbound.log

Change-Id: I6bb58ef0d6bf4cbbb7fd4066e01b7a01d05009c3
2022-05-12 15:16:08 +10:00
Dr. Jens Harbott
a694b41c74 Fix apt-phased-updates
Missing then

Change-Id: Ifb446e44a788583c6ab7b93747927b921932aba6
2022-05-04 17:56:39 +02:00
Dr. Jens Harbott
37a934aacb Always include phased updates for Jammy
Our images builds always install the newest updates,
so we need this in order to avoid conflicts.

Change-Id: I7e14968f2aa7beed88e806ac1eef9370a5388bbb
2022-05-04 15:16:14 +00:00
Dr. Jens Harbott
f5b01d71b7 Start bulding ubuntu-jammy images
Jammy was released yesterday, we can start building images for it.

Change-Id: I5fd16df8bf0e3b74711875ad0573be69db12791b
2022-04-24 15:23:28 +02:00
Ian Wienand
39d2332299 infra-package-needs: don't require coreutils for Rocky Linux 8
coreutils comes in two variants now, 'coreutils-single' which is a
busybox-like single binary called through symlinks and the regular
coreutils.  Both satisfy the dependency for coreutils for any other
packages, but if you explicitly ask to install coreutils over
coretuils-single you get an error.

Since coreutils-single is already in the base-image, just skip
installing it on Rocky 8.

Change-Id: I89f8cb49b0cd373e454dd37439bf6efd971233e5
2022-02-21 08:13:11 +11:00
Neil Hanlon
cc6b6de7a0
Add Rocky Linux to nodepool elements tooling
Change-Id: I2a94a34b1518c90ef5f0d91c9131482520c38c62
2022-02-15 17:31:34 -05:00
Dr. Jens Harbott
eebe545182
Add py36 variant of get-pip.py
The common version of get-pip.py no longer supports python3.6 or older.
Devstack has amended it's use of this file to use a stable version of
the installer for python3.6 [0]. Pre-cache this version in our images,
too, so that it can be consumed in the CI.

[0] https://review.opendev.org/#/q/Iab2c391d5388461fe9e9037cee81884ce8032e72

Change-Id: Ied14dc7188e27e2a4b144d597b696ee85f25721d
2022-02-02 16:42:04 +01:00
wangxiyuan
c28cce5d75 Add openEuler disto support for elements
This patch add openEuler disto support for the diskimage-builder
elements in project-config

Change-Id: I7d7804376c4579b550c865a3ad5719fea1ec703b
2021-12-16 09:37:19 +08:00
Ian Wienand
d54c2b46a7 cache-devstack: use dstat_graph from opendev
We have incoporated outstanding changes into a new opendev.org hosted
repository.

Change-Id: Ie4d31fa380573eddf6a5c31eeb6403d7c3b00159
2021-12-07 13:07:35 +11:00
Alfredo Moralejo
42fa5a7e73 Fix haveged installation in CentOS7
When adding support for CentOS Stream 9 [1], I made dib to install haveged
in centos8 or centos8s only. This broke centos7 images.

This patch should get haveged installed in all centos releases != 9-stream and fix
centos7 one.

[1] https://review.opendev.org/c/openstack/project-config/+/811442

Change-Id: I5a33160c6272ee4e452b83599ca3ed552422c6d2
2021-11-05 11:06:08 +01:00
Ian Wienand
7df74a82fa infra-package-needs: skip haveged start on 9-stream
This package is not installed (see
I9b88baf422d947d5209d036766a86b09dca0c21a) so we can't enable this
service on 9-stream.

Change-Id: Ie42d73e7cd12c80b076429a643d95778ff5665b8
2021-11-05 17:40:46 +11:00
Ian Wienand
04b3eb3d43 nodepool elements: use yaml.safe_load
The recent pyyaml 6 release, incorporated in the latest
nodepool-builder/dib images, requires a specific loader now.  Use
safe_load as there's nothing crazy going on here.

Change-Id: I26e5e1cf6778cb51872d6a65b2ac683335201bb4
2021-11-05 11:25:17 +11:00
Alfredo Moralejo
4e88e748bb Add support for CentOS Stream 9 in nodepool elements
This patch is adding support for CentOS Stream 9 in elements
infra-package-needs and nodepool-base which are used in nodepool images.

- Remove installation of ntpdate (it has been removed in CS9).
- It skips installation of haveged until it's available in EPEL9.
- It maps package iptables to iptables-service.

Note that this patch has been tested together with dib patch in Depends-On.

Depends-On: https://review.opendev.org/c/openstack/diskimage-builder/+/811392
Change-Id: I25d9bb7272edc3215840a53e5d79efe5d1fd7210
2021-11-03 12:47:06 +01:00
Ian Wienand
30fd4b4549 dstat graph: update to version with fixes
This updates the on-disk cache version to the same as the default
version in zuul-jobs from I0d5e1b567c364a9e6c7aa0b95de17abffaef0434

Note we have a pull request open on this original project to
incorporate these changes, but the project has been dormat for a long
time.  If we don't have a response in the medium-term future, we can
bring this project into opendev.org git and maintain our enhancments
there.

Change-Id: I26b1a7d0dde12be7c493d44d754b9ea8f72e6e66
2021-10-29 09:36:33 +11:00
Ian Wienand
00d29d829b infra-package-needs: install latest pip
The pip installed in the venv with "python3 -m venv" on Xenial is 8.X
-- this does not understand python_requires metadata on packages and
can thus pull in requirements that won't actually run inside the
virtualenv.

Avoid this by upgrading pip in the venv before installing.

While this is the immediate need; do the same on the other venv's we
create for general sanity.

Change-Id: I316e9587b6e290cd421b47f506c91dbebe0975c0
2021-10-20 09:56:06 +11:00
Ian Wienand
c554a6b3b3 infra-package-needs: don't start ntp for Fedora
See I361059c6b62ea240b6fef5a61d254959622199d7 where we modified Fedora
to not install the deprecated ntp package.

Change-Id: I9147f16a4e67b15ac7cc0bc4684ad8390718525f
2021-06-16 14:08:14 +10:00
Ian Wienand
20d69256ff infra-package-needs: stub ntp for Fedora
ntp/ntpdate isn't a package on Fedora any more [1].  Make this like
centos 8 above and install chrony for time services.

[1] https://fedoraproject.org/wiki/Changes/NtpReplacement

Change-Id: I361059c6b62ea240b6fef5a61d254959622199d7
2021-06-15 19:43:31 +10:00
Ian Wienand
f7a8f02462 nodepool elements: create suse boot rc directory
As noted inline, the /etc/init.d directory appears to have been
somehow remove/no longer created with a recent update.  I've added
this manually and the image builds, and the rc-local.service still
runs.  Do this for now to unblock other builds.

Change-Id: I0b0b2e38951bad656bcfdb47b6470e033564db59
2021-03-17 13:37:12 +11:00
Lee Yarwood
388d227704 Add Cirros 0.5.2 to cache
Recently released [1] and including a fix [2] required by the
Iad1adbc23b31dd54a96299e7a8a4b622c15eed8d change introducing q35 testing
in the nova-next job this image needs to cached before we update
devstack.

[1] https://github.com/cirros-dev/cirros/releases/tag/0.5.2
[2] https://github.com/cirros-dev/cirros/pull/65

Change-Id: I064f1e4a8ba0bab5b759fb419fa9f9c52e122baa
2021-03-08 08:46:15 +00:00
Lee Yarwood
cf2ba9c2e9 Revert "Add custom cirros image with ahci module enabled to cache"
This reverts commit 5ee0780486.

0.5.2 [1] was cut after another colleague asked for a release. I guess their release build issues have been resolved since I asked a few weeks ago. As a result this build is no longer required once we've bumped to 0.5.2.

[1] https://github.com/cirros-dev/cirros/releases/tag/0.5.2

Change-Id: I5332d0e47ad863ca9795a8b0b86b73156621622d
2021-03-08 08:43:53 +00:00
Lee Yarwood
5ee0780486 Add custom cirros image with ahci module enabled to cache
As discussed on the ML [1] the nova-next job is looking to start testing
the q35 machine type. In order to do this *before* the next Cirros
release a custom dev build of the Cirros image has been built with the
ahci module included, as is now required to allow for SATA based config
drives to work.

[1] http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020823.html

Change-Id: I67912064487598c0e5b4ce3001276f42e0ebcad1
2021-03-03 22:28:05 +00:00
Zuul
e707d93f2b Merge "zuul-worker: remove additional install of apt-transport-https" 2021-02-04 18:20:14 +00:00
Zuul
fb00c509f8 Merge "add gnupg2 to package map for gentoo" 2021-01-17 14:47:18 +00:00
Matthew Thode
a76ab5745b
add gnupg2 to package map for gentoo
Change-Id: Ia0967f8669a6cdd4606f0b10aae3a224b2cf4cee
2021-01-17 02:41:06 -06:00
Matthew Thode
d29d9313d7
allow create-repo-list to be run without six
Some distros (gentoo) do not come with six in the base image (stage3),
it is installed later in the image build.  Use the native
urlopen/URLError if the six version is not available.

Change-Id: Id1c52aa17e565c16369645508e43e4f485defa66
2021-01-17 02:38:37 -06:00
Tristan Cacqueray
66cb2317fd infra-package-needs: add support for centos-8-stream
This change fix the centos version check to work with both '8' and '8-stream'
value for DIB_RELEASE. The current test is failing with:

  $ export DIB_RELEASE=8-stream
  $ set -u
  $ [[ $DIB_RELEASE -gt 8 ]]
  bash: stream: unbound variable

The fix is lifted from
https://review.opendev.org/#/c/734083/14/diskimage_builder/elements/simple-init/environment.d/15-simple-init-networkmanager

Change-Id: I23dca12eef1c3cc2aacf6ac50029e2bc9fde72dc
2020-10-02 18:19:07 +00:00
Clark Boylan
c6660ecfcb Revert "Pin setuptools<50 in our image venvs"
This reverts commit 6f992efbc5.

Setuptools 50.1.0 has been released which reverts the breaking behavior.
https://review.opendev.org/#/c/749766/ tests that these fixes work in
other venvs that exhibited the same problems. That change looks happy so
I think this revert is ready to go.

Change-Id: I31b62be4f85f40f4d99e463cd961dec0a3542f47
2020-09-03 09:19:52 -07:00
Clark Boylan
6f992efbc5 Pin setuptools<50 in our image venvs
setuptools==50 doesn't work with python older than python3.8. There are
a number of issues [0][1][2] the first of which we are tripping over. Be
conservative here and install older setuptools until these issues on
python3.5, 3.6, and 3.7 are sorted out.

[0] https://github.com/pypa/setuptools/issues/2352
[1] https://github.com/pypa/setuptools/issues/2363
[2] https://github.com/pypa/setuptools/issues/2357

Change-Id: I1ae7251c8a4c214544524871fdfe51e947b638f8
2020-08-31 14:02:24 -07:00
Hervé Beraud
95c01abf38 Stop to use the __future__ module.
The __future__ module [1] was used in this context to ensure compatibility
between python 2 and python 3.

We previously dropped the support of python 2.7 [2] and now we only support
python 3 so we don't need to continue to use this module and the imports
listed below.

Imports commonly used and their related PEPs:
- `division` is related to PEP 238 [3]
- `print_function` is related to PEP 3105 [4]
- `unicode_literals` is related to PEP 3112 [5]
- `with_statement` is related to PEP 343 [6]
- `absolute_import` is related to PEP 328 [7]

[1] https://docs.python.org/3/library/__future__.html
[2] https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html
[3] https://www.python.org/dev/peps/pep-0238
[4] https://www.python.org/dev/peps/pep-3105
[5] https://www.python.org/dev/peps/pep-3112
[6] https://www.python.org/dev/peps/pep-0343
[7] https://www.python.org/dev/peps/pep-0328

Change-Id: Icc115b2271a6fd84f3952b03406250367babd273
2020-06-02 20:44:50 +02:00
Monty Taylor
0a4e2aab9a Don't install python-dev on focal
Also, install yamllint in the dib env, as it's a requirement
of dib-lint now but is only in test-requirements for dib.

Change-Id: I083bca901ca51438099d1d3bbbd0076ac3d7da07
2020-05-27 10:24:51 -05:00
Ian Wienand
9485aaaca3 infra-package-needs: drop ntp for Focal
systemd timesyncd is the default mechanism for timesync on Focal;
let's reduce our modification footprint by not overinstalling ntp or
trying to enable it.

Change-Id: I60e15b9101511e9008159b7a0b63f1b4b3febb96
2020-04-30 11:07:11 +10:00
Dmitry Tantsur
d30035b433 Cache cirros UEC images
Ironic uses them in its gate jobs, downloading every time. With
github broken all these jobs are failing now.

Change-Id: I8649d2cd530bdedcbd333991f7376fe9cd9bf267
2020-04-23 16:43:43 +02:00
Clark Boylan
a8a41f162f Install gpg tooling on dib images
This is particularly important for debuntu where we need working gpg for
apt and a missing gpg-agent is fatal. We install it globally so that
consistent tooling is available across systems.

Note everyone but suse seems to have a gnupg2 package. Suse calls it
gpg2.

Change-Id: I6c56e85db501f2c9d7c648e614f1efbaadc213a2
2020-04-03 13:30:26 -07:00
Ian Wienand
2ef9b25101 zuul-worker: remove additional install of apt-transport-https
This is installed for the base images in the dependent change

Depends-On: https://review.opendev.org/716788
Change-Id: Ic6ae3c5406fc0efd7fff1875459dfab85b4f702c
2020-04-03 09:20:21 -05:00
Ian Wienand
8a1b382301 zuul-worker: remove python-apt & libselinux deps
These have been here since the initial commit
I0e76931fdb4ca0c7445b1e72dc348f0cf03eaee5 in 2016.

These have now bifurcated into python3 packages and are brought in
elsewhere; we don't need to additionally bring it in here.  At best we
duplicate the install, at worse we get it wrong
(I563896f1caa1a68781f10b27cd7d9cf2606aaa17).

libselinux is installed for all platforms by the redhat-common package
[1].  This element probably pre-dates that.

python[3]-apt is brought in by software-properties-common [2] for all
platforms, which will bring in the correct python[3] version.

[1] https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/redhat-common/package-installs.yaml#L14
[2] https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/base/pre-install.d/03-baseline-tools

Change-Id: Ie1ca9bda749e17cdd5269850c03feca100395f1f
2020-04-03 09:20:05 -05:00
Ian Wienand
539b1a3b6c zuul-worker element: use python3-libselinux for fedora
It seems Fedora 31 has completely remvoed python-libselinux, so this
now causes a build failure.

Change-Id: I563896f1caa1a68781f10b27cd7d9cf2606aaa17
2020-04-02 09:34:09 +11:00
Radosław Piliszek
624eec932b Cache CirrOS 0.5.1 for AArch64 too
OpenStack is getting tested on AArch64 CI nodes.
Let's cache CirrOS for it as well.

Change-Id: I3138bda523533bbbdc354d50c57179dca60d6d9c
2020-03-23 16:49:36 +01:00
Radosław Piliszek
dc68586431 Cache CirrOS 0.5.1
We are moving towards CirrOS 0.5.1 usage in CI.
DevStack's patch is pending [1] and Kolla Ansible merged [2].

[1] https://review.opendev.org/711182
[2] https://review.opendev.org/711492

Change-Id: I4e82a955b59dfdcebbcd1f1687466e546781d28b
2020-03-23 16:27:36 +01:00
Clark Boylan
9e1c44d9a4 Statically cache devstack images and packages
We had been running a script to generate a list of things to cache for
devstack. Unfortunately, we've discovered that this attempts to perform
unsafe actions which illustrates that this is unsafe (and creates
unnecessary relationship between opendev images and openstack/devstack).

Address this by providing a static list of things to cache.

Note this does not do anything for arm64 images (that will need to be
addressed in a follow on but they are largely not running devstack there
yet).

On a Bionic node this is what we have in /opt/cache/files/:

cirros-0.3.2-i386-disk.vmdk
cirros-0.3.4-x86_64-disk.img
cirros-0.3.4-x86_64-disk.vhd.tgz
cirros-0.3.4-x86_64-uec.tar.gz
cirros-0.3.5-x86_64-disk.img
cirros-0.3.5-x86_64-disk.vhd.tgz
cirros-0.3.5-x86_64-uec.tar.gz
cirros-0.4.0-x86_64-disk.img
cirros-0.4.0-x86_64-uec.tar.gz
etcd-v3.1.10-linux-amd64.tar.gz
etcd-v3.2.17-linux-amd64.tar.gz
etcd-v3.3.12-linux-amd64.tar.gz
get-pip.py
stackviz-latest.tar.gz
zanata-cli-4.3.3-dist.tar.gz

I've trimmed out the vmdk, vhd, and tarball based images as we should
all be using qcow2s. Everything under etcd is provided by preexisting
static lists.

Change-Id: Iff741e8ed4c517ccabae6e6d6ba730f0aa37a272
2020-03-13 16:52:21 -07:00
Clark Boylan
8dffaef5c1 Install tox into a virtualenv on our images
This will install tox into a virtualenv on our images. On our older
images with globally installed tox this can be ignored but as we move to
"plain" images this can be used as an opt in tox executable by jobs.
Jobs can set the tox_executable path for the ensure-tox role.

We don't install it globally to avoid polluting package manager managed
paths.

Change-Id: If5397d731e9fb04431482529aed23cd9fdaecc1d
2020-03-13 09:27:30 -07:00
Ian Wienand
c142764ec4 nodepool-elements: skip tox install with no pip-and-virtualenv
This is a follow-on to I85438baf5bb31790a56fe5b38327361f0a2398e9.

Skip over this install of tox, which no longer works without the
"pip-and-virtualenv" element define of $DIB_PYTHON_PIP.  We want to
not install globally in the image, but move things like this to
ansible roles in base jobs if required.

Change-Id: Id1571210f0778019c78aec9f38e9f1254c1d68f9
2020-03-11 13:25:55 +11:00
Ian Wienand
7602343b6a nodepool-elements: Use venv for utilities
Since all platforms have Python 3, use the new ensure-venv element
from the dependent change to install bindep and os-testr.

Since we are no longer using pip to install anything during the
builder, this drops the dependency on pip-and-virtualenv from
nodepool-base.  Avoiding this element is our long-term goal, as it's
modification to system state are problematic in a number of ways.  To
maintain the status-quo, the pip-and-virtualenv element is added
explicitly to each build's element list, with a note on it's future.

The current plan for backwards compatability is to replicate the
environment pip-and-virtualenv provides in a base role/job that can be
optionally included.  To test this, provide a new node type
"ubuntu-bionic-plain" that will not include the pip-and-virtualenv
element.  This is put on just one provider (rax) to minimise impact.

The dependent-change (and a dib release) is required before merge so
the ensure-venv element is available.

Depends-On: https://review.opendev.org/707513
Change-Id: I85438baf5bb31790a56fe5b38327361f0a2398e9
2020-03-11 10:12:00 +11:00
Ian Wienand
7801ad35a6 Fix new dib-lint errors
The dib 2.34.0 release uncapped hacking and has found some new minor
issues.  Add missing readmes and fix whitespace.

Change-Id: Ia05e54c26988774bf03b0764a6df5e60e8ddaca8
2020-03-11 10:10:57 +11:00