765 Commits

Author SHA1 Message Date
Zuul
3a278e46a9 Merge "CI: Move NFV reqs installation to where it belongs" into stable/train 2020-06-19 20:58:16 +00:00
Michal Nasiadka
eba42fa8c3 Run tox in venv in case of building images
tox depends on virtualenv, when it's installed using pip - it tries to upgrade
virtualenv, and then fails.

Change-Id: Ic9566778f87f8b81563ed012e651cef50d1baa38
2020-06-19 06:19:39 +00:00
Radosław Piliszek
31fb5cc672 CI: Move NFV reqs installation to where it belongs
This ought to fix #1864238 for the stable branches.

Change-Id: I218905247a4a4003ecfc2c9ab3e47767bb5ab33e
Related-bug: #1864238
2020-06-12 21:37:37 +02:00
Radosław Piliszek
cfc1ba2c33 Check that used Ansible can see Kolla Ansible
Fix-feature following up on the original check [1] to make it
test the correct interpreter.

Additionally, this change removes last, unneeded call to
random python - getting script directory is perfectly
doable in bash.

All checks are done from Python, not Ansible, due to its
performance. Python version feels snappy (0.2 s to check),
compared to sluggish Ansible (2.0 s to check).
What is more, relying on Ansible would require hacky solutions
to e.g. prevent custom config from interfering with it.
We might be willing to steer Ansible from Python in the future
anyhow.

[1] Icf0399d21b3fde8d530d73e6e7ee4a57665da276

Change-Id: Ib8f2e6b6672e7c06aa94bc226c4d72640d25d8c2
Closes-Bug: #1856346
(cherry picked from commit 32fc2599a665e01b47c74588cdc2b547dc208874)
(cherry picked from commit 1f16c04a50e0c6e9614ede128736c2c97d4efe1d)
2020-05-25 14:33:25 +00:00
xiaojueguan
afc5c99747 fix can not generate ovs-dpdk.conf
Change-Id: I0ecc10f8035f421d7dc1132c2b29db0d9cc84cdf
Closes-Bug: #1879125
(cherry picked from commit 593b7b4ce4467c7687638bd0e1eced62f638d653)
2020-05-18 15:15:42 +00:00
Mark Goddard
57440edf80 CI: Reduce unnecessary gate image builds
Backport note: added CentOS 8 skip for tgtd present in master - somehow
this was missed in backport.

Due to regex matching of images, we are currently building quite a few
unnecessary images in CI. Images that can unexpectedly match:

* barbican-keystone-listener
* ironic-neutron-agent
* prometheus-haproxy-exporter
* prometheus-memcached-exporter

This change improves the situation by anchoring matches to the start of
the name. A full solution would be to specify every image required for
each job. This would be quite verbose and require maintenance.

Change-Id: I7387ccd0cb394c54c79bfba8b1c09fce7ee2973e
(cherry picked from commit 8b86c8194b2726dcecbc1487a8f567043828dcde)
2020-03-11 13:33:56 +00:00
Zuul
f5ba6754d7 Merge "CentOS 8: Add deploy jobs in CI" into stable/train 2020-03-02 20:26:14 +00:00
Mark Goddard
6cd2077cce CI: build swift images for swift scenario
Change-Id: I957fcea4ab180dd82d1faaa9a012f83d3647f043
(cherry picked from commit f30320285055467ec30d6179a7c50f261c38f5d7)
2020-02-28 13:36:58 +00:00
Radosław Piliszek
21fa358ceb CentOS 8: Add deploy jobs in CI
Backport note: Using latest Ansible for CentOS 8 rather than 2.6 which
is used for CentOS 7.

Adds new CI job definitions for CentOS 8:

- kolla-ansible-centos8-source
- kolla-ansible-centos8-binary
- kolla-ansible-centos8-source-cinder-lvm
- kolla-ansible-centos8-source-mariadb
- kolla-ansible-centos8-source-bifrost
- kolla-ansible-centos8-source-zun
- kolla-ansible-centos8-source-swift
- kolla-ansible-centos8-source-scenario-nfv
- kolla-ansible-centos8-source-ironic
- kolla-ansible-centos8-binary-ironic
- kolla-ansible-centos8-source-masakari
- kolla-ansible-centos8-source-cells

The following jobs are added to the check pipeline:

- kolla-ansible-centos8-source
- kolla-ansible-centos8-binary
- kolla-ansible-centos8-source-cinder-lvm
- kolla-ansible-centos8-source-mariadb
- kolla-ansible-centos8-source-zun
- kolla-ansible-centos8-source-swift
- kolla-ansible-centos8-source-scenario-nfv
- kolla-ansible-centos8-source-ironic
- kolla-ansible-centos8-binary-ironic
- kolla-ansible-centos8-source-cells

The following jobs are not yet passing so are not added to the check
pipeline:

- kolla-ansible-centos8-source-bifrost
- kolla-ansible-centos8-source-masakari

The kolla-ansible-centos8-source job is added to the gate.

Depends-On: https://review.opendev.org/709204
Depends-On: https://review.opendev.org/709205
Depends-On: https://review.opendev.org/709207
Depends-On: https://review.opendev.org/710326

Co-Authored-By: Mark Goddard <mark@stackhpc.com>

Change-Id: Ibd806feee71721b122b77d7eff33228ca1cc2853
Partially-Implements: blueprint centos-rhel-8
(cherry picked from commit 287adab05ebcae4bef16759795ad0907ffb2489c)
2020-02-28 09:18:37 +00:00
Zuul
b05d9f94b6 Merge "CI: Use upper constraints when installing clients" into stable/train 2020-02-26 17:17:14 +00:00
Zuul
aeb4396ce2 Merge "CI: Move ansible installation & configuration to Ansible" into stable/train 2020-02-26 17:06:30 +00:00
Zuul
79d142f4a8 Merge "Support python 3 in kolla-ansible script" into stable/train 2020-02-26 12:36:04 +00:00
Mark Goddard
6401c3b40c CI: Use upper constraints when installing clients
Clients are starting to release versions that don't support Python 2.
The ironic scenario is currently failing on stable branches for this
reason.

Use upper constraints to avoid installing these new versions on stable
branches.

Change-Id: I4f91b53cbf2297d70da4b54d6c402c1427aacdd9
2020-02-26 10:34:17 +00:00
Mark Goddard
08039b02d5 CI: Move ansible installation & configuration to Ansible
Continue to reduce the scope of setup_gate.sh. Allows us to more easily
select python 2 or 3.

Change-Id: If2eeeacbbbdf58afb765b4a39772b5a1af7b952b
Partially Implements: blueprint python-3
(cherry picked from commit c320077f08f39be6b0083e053da02e6ca45e4c0a)
2020-02-25 12:43:50 +00:00
Mark Goddard
a86b465390 Support python 3 in kolla-ansible script
The kolla-ansible script uses a python interpreter to detect the
location of playbooks and to check that the kolla_ansible python module
is importable. This change updates the script to support a python
interpreter named python or python3.

Partially Implements: blueprint python-3

Change-Id: Id5dcc53cc5dd9780632c04b6b73c56ea5da484a2
(cherry picked from commit 3a6577badacd8f2d03ec2eaaf1336c27a8c1479d)
2020-02-25 12:24:08 +00:00
Radosław Piliszek
ee19e8f062 Allow to override external network params in init-runonce
I'm not marking it CI-only as ppl seem to be using it.
OTOH, not adding a release note as we are not promoting its usage.

This is to allow us to customize for CI.

Backport to Train for hassle-free upgrades testing. :-)

Change-Id: I8100a6cb63b1e54078629bd6ca8475dc5896784a
(cherry picked from commit effbb209976f3c74592c2a69bb4086541b567f06)
2020-02-25 09:17:18 +00:00
Radosław Piliszek
64051bbbdb CI: Replace cinder-lvm scenario with zun scenario
As discussed during the meeting. [1]

[1] http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-02-05-15.00.html

Change-Id: I339fc7f01f640a32ff35d19ded54b6f8e2fdec15
(cherry picked from commit 5b0894299ef8fca62f9d8eab7f0e229548da26cb)
2020-02-07 18:38:48 +01:00
Hongbin Lu
241aaac4b2 CI: Test Zun with Cinder LVM backend (iSCSI)
Co-authored-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
Depends-on: https://review.opendev.org/694476
Change-Id: I6e7f2f4229c8b579dcc17dacffeb74160875ae29
(cherry picked from commit c6481469e66311681dcd61a15db5e8d8c0e7be6a)
2020-02-07 18:35:30 +01:00
Radosław Piliszek
4beb760d8d CI: Refactor a lot
Separate upgrade logic to is_upgrade job var and rename
scenarios to match.

Rename "ACTION" to "SCENARIO" (as it is a scenario).

Separate testing of dashboard (aka Horizon) and increase
its timeout to 5 minutes (CentOS 7 slow as always).

Separate initialization of core OpenStack.

Use gate setup script from ./tests/

Remove useless tox setupenv.

Do not deploy Heat when not really necessary.

Change-Id: I4fca319ccc3de7188f8b7b44c9c71321e3899467
(cherry picked from commit a2fc684164cb5169492d6e0c007ad006f814492c)
2019-11-29 08:57:01 +01:00
Zuul
f51565d877 Merge "CI/Debian: Push images after building" into stable/train 2019-11-28 18:15:02 +00:00
Mark Goddard
7ecba94dd5 Fix kolla_ansible python module check
The kolla-ansible script now (Train+) checks whether the kolla_ansible
python module is installed, and emits the following message if not:

ERROR: kolla_ansible has to be available in the PYTHONPATH (e.g.
installed)

It is supposed to exit 1 if this check fails, but does not.

This change fixes this by removing a subshell from the kolla-ansible
script.

Change-Id: I2c1a7398f2b8f876abcf2508874835154cb4ce57
Closes-Bug: #1854049
(cherry picked from commit 692d176cd5e716ad24e6f4fe57f2c345417c0793)
2019-11-27 11:06:24 +00:00
Radosław Piliszek
de9e0392c7 CI/Debian: Push images after building
This fixes Debian job failures during image building.

See https://github.com/docker/for-linux/issues/711
for upstream details.

Change-Id: Icf3ffb261605ffe5d8f2618c2ed4cb97db97dd49
(cherry picked from commit 6ab144aae405db6d3ab672c58b7675b49b86d009)
2019-11-26 11:34:08 +00:00
Zuul
44302eaa46 Merge "CI: Remove playbook-setup-nodes.yml" into stable/train 2019-11-25 09:52:02 +00:00
Zuul
ca2edcff21 Merge "CI: Add mariadb test" into stable/train 2019-11-25 09:17:50 +00:00
Mark Goddard
56a2bedbed CI: Remove playbook-setup-nodes.yml
This playbook is neither kolla-ansible nor Zuul. Much of what it does is
stale and no longer required. Remove it.

Change-Id: I6fed9d8f2439a53c6ac35c670b56b43579236ee9
(cherry picked from commit 43bb5ae6fbbab2b365cc51a24a26567092a19b91)
2019-11-21 17:28:20 +00:00
Mark Goddard
93b89194a7 CI: Add mariadb test
Tests the following operations for MariaDB:

* Stop
* Recovery

Backup and restore will be added in a separate change.

Depends-On: https://review.opendev.org/693329
Change-Id: I836d91554715cce0e82c1bbebb7430c457418b2d
(cherry picked from commit ed996ef90d35fa1a75eea3b1c9ec4dcdc6193bd3)
2019-11-21 17:22:59 +00:00
Radosław Piliszek
d1693f1723 CI: Add more service list checks to OS "smoke testing"
Change-Id: Ie7ee70e19536c2d8f4300df55c9c6ca38abb7ae4
(cherry picked from commit f59f6a6104ecf928a21d8a2c01ceb5d442db6743)
2019-11-21 17:20:30 +00:00
Radosław Piliszek
af7305f2aa Fail kolla-ansible early when not available in the PYTHONPATH
Starting with the Train release, kolla_ansible (the package) must be
available in the PYTHONPATH to run most playbooks.
It is documented.
This change makes the error message friendlier.

Change-Id: Icf0399d21b3fde8d530d73e6e7ee4a57665da276
Closes-bug: #1849943
2019-10-26 15:09:15 +02:00
Radosław Piliszek
bc053c09c1 Implement IPv6 support in the control plane
Introduce kolla_address filter.
Introduce put_address_in_context filter.

Add AF config to vars.

Address contexts:
- raw (default): <ADDR>
- memcache: inet6:[<ADDR>]
- url: [<ADDR>]

Other changes:

globals.yml - mention just IP in comment

prechecks/port_checks (api_intf) - kolla_address handles validation

3x interface conditional (swift configs: replication/storage)

2x interface variable definition with hostname
(haproxy listens; api intf)

1x interface variable definition with hostname with bifrost exclusion
(baremetal pre-install /etc/hosts; api intf)

neutron's ml2 'overlay_ip_version' set to 6 for IPv6 on tunnel network

basic multinode source CI job for IPv6

prechecks for rabbitmq and qdrouterd use proper NSS database now

MariaDB Galera Cluster WSREP SST mariabackup workaround
(socat and IPv6)

Ceph naming workaround in CI
TODO: probably needs documenting

RabbitMQ IPv6-only proto_dist

Ceph ms switch to IPv6 mode

Remove neutron-server ml2_type_vxlan/vxlan_group setting
as it is not used (let's avoid any confusion)
and could break setups without proper multicast routing
if it started working (also IPv4-only)

haproxy upgrade checks for slaves based on ipv6 addresses

TODO:

ovs-dpdk grabs ipv4 network address (w/ prefix len / submask)
not supported, invalid by default because neutron_external has no address
No idea whether ovs-dpdk works at all atm.

ml2 for xenapi
Xen is not supported too well.
This would require working with XenAPI facts.

rp_filter setting
This would require meddling with ip6tables (there is no sysctl param).
By default nothing is dropped.
Unlikely we really need it.

ironic dnsmasq is configured IPv4-only
dnsmasq needs DHCPv6 options and testing in vivo.

KNOWN ISSUES (beyond us):

One cannot use IPv6 address to reference the image for docker like we
currently do, see: https://github.com/moby/moby/issues/39033
(docker_registry; docker API 400 - invalid reference format)
workaround: use hostname/FQDN

RabbitMQ may fail to bind to IPv6 if hostname resolves also to IPv4.
This is due to old RabbitMQ versions available in images.
IPv4 is preferred by default and may fail in the IPv6-only scenario.
This should be no problem in real life as IPv6-only is indeed IPv6-only.
Also, when new RabbitMQ (3.7.16/3.8+) makes it into images, this will
no longer be relevant as we supply all the necessary config.
See: https://github.com/rabbitmq/rabbitmq-server/pull/1982

For reliable runs, at least Ansible 2.8 is required (2.8.5 confirmed
to work well). Older Ansible versions are known to miss IPv6 addresses
in interface facts. This may affect redeploys, reconfigures and
upgrades which run after VIP address is assigned.
See: https://github.com/ansible/ansible/issues/63227

Bifrost Train does not support IPv6 deployments.
See: https://storyboard.openstack.org/#!/story/2006689

Change-Id: Ia34e6916ea4f99e9522cd2ddde03a0a4776f7e2c
Implements: blueprint ipv6-control-plane
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-10-16 10:24:35 +02:00
Marcin Juszkiewicz
b9d6cc2a7f Add Debian/source CI job
Change-Id: I0628b16e3ebdb3fa8196acdc1bd9c63e75bcfb09
2019-10-04 08:57:55 +00:00
Kris Lindgren
2fe0d98ebb Add a job that *only* deploys updated containers
Sometimes as cloud admins, we want to only update code that is running
in a cloud.  But we dont need to do anything else.  Make an action in
kolla-ansible that allows us to do that.

Change-Id: I904f595c69f7276e71692696471e32fd1f88e6e8
Implements: blueprint deploy-containers-action
2019-09-26 17:51:14 +01:00
Mark Goddard
8c8adb0e45 Update minimum version of Ansible to 2.6
This is required for the dict2items filter.

Change-Id: I60a04e839bf06506ff36c2631a286130d5fde972
2019-09-11 17:09:32 +01:00
Kien Nguyen
fbac54c5f5 Testing Masakari role in gate
Add Masakari testing into the Gate.

Change-Id: I52df33f963e7d2ae4059887df3d24d9e6642134e
Depends-On: https://review.opendev.org/#/c/615469/
Depends-On: https://review.opendev.org/#/c/615715
Implements: blueprint ansible-masakari
Co-Authored-By: Gaëtan Trellu <gaetan.trellu@incloudus.com>
2019-08-14 12:32:51 -04:00
Radosław Piliszek
8a543098d6 CI: clean up requirements installation
We install kolla-ansible requirements in Zuul's Ansible playbooks.
This patch cleans up the installation in scripts so that they are
only concerned with auxiliary requirements:
- ansible (since we do not track it in requirements)
- ara (for log summaries)
- openstack clients (for first init and tests after deployment)

Additionally this patch installs openstack clients in a separate
virtualenv.
Note that all kolla-ansible requirements, ansible and ara are still
installed system-wide.

Change-Id: Iac04082ad39a9d823c515ba11c5db9af50ed225f
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-07-16 20:43:22 +02:00
Zuul
c2244260c6 Merge "Test minimum supported and latest versions of Ansible" 2019-07-15 09:54:39 +00:00
Zuul
103e0e43e7 Merge "init-runonce: make public1 network optional" 2019-07-11 09:30:53 +00:00
Mark Goddard
3026fd9129 init-runonce: make public1 network optional
Skip creation by setting ENABLE_EXT_NET to 0.

Since adding errexit we are failing in kayobe CI, since we have a
conflicting flat network on physnet1.

Change-Id: I88429f30eb81a286f4b8104d5e7a176eefaad667
2019-07-10 17:48:28 +01:00
Mark Goddard
b9da05be55 Test minimum supported and latest versions of Ansible
We have a minimum supported version of Ansible, currently 2.5. We should
test this in addition to the latest version. This change tests latest on
Ubuntu, and minimum on other distros.

Change-Id: I45a7173139f057177a71e919ad3e718a99d9f87b
2019-07-09 08:04:26 +00:00
Zuul
887938bbcb Merge "Exit on failure in init-runonce" 2019-07-09 07:33:46 +00:00
Zuul
ec78645928 Merge "Bump minimum Ansible version to 2.5" 2019-07-08 09:21:53 +00:00
Zuul
fb964ce41b Merge "CI - remove unused setup scripts" 2019-07-05 15:42:42 +00:00
Zuul
dfa1a3844d Merge "Add upgrade-bifrost command" 2019-07-05 09:17:16 +00:00
Zuul
56c3603586 Merge "CI: Keep stderr in ansible logs" 2019-07-04 07:45:54 +00:00
Radosław Piliszek
b9aa8b38f4 CI: Keep stderr in ansible logs
Otherwise ara had only the stderr part and logs only the
stdout part which made ordered analysis harder.

Additionally add -vvv for the bootstrap-servers run.

Change-Id: Ia42ac9b90a17245e9df277c40bda24308ebcd11d
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-07-02 20:44:33 +02:00
Mark Goddard
9cac1137d0 Add upgrade-bifrost command
This performs the same as a deploy-bifrost, but first stops the
bifrost services and container if they are running.

This can help where a docker stop may lead to an ungraceful shutdown,
possibly due to running multiple services in one container.

Change-Id: I131ab3c0e850a1d7f5c814ab65385e3a03dfcc74
Implements: blueprint bifrost-upgrade
Closes-Bug: #1834332
2019-07-02 14:30:14 +01:00
Radosław Piliszek
20ab480ca5 CI: Use template-overrides.j2 from kolla
Some kolla-ansible jobs failed due to using external mirrors
instead of local ones.
This was due to not using the template override provided by kolla.
This patch fixes that.

Depends-On: https://review.opendev.org/668226
Change-Id: I27f714fdf05e521aa8ce25c5683a452ceb35eeb8
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-07-01 17:00:53 +00:00
Mark Goddard
0a769dc30b Bump minimum Ansible version to 2.5
This is necessary for some Ansible tests which were renamed in 2.5 -
including 'version' and 'successful'.

Change-Id: Iacf88ef5589c7571fcf56ba8b99d3dbe76975195
2019-07-01 09:38:01 +01:00
Mark Goddard
bc08b44fd1 Exit on failure in init-runonce
Previously we sourced this script in tests/deploy.sh, but this was
recently changed. Following that change we lost the errexit setting,
meaning we ignore errors in init-runonce.

Adding errexit in the script itself means that all callers get error
handling.

Also log init-runonce output.

TrivialFix

Change-Id: I9b35bd5f0f76eec26ddd968d093a3a5fd55a7ce2
2019-06-28 14:31:24 +00:00
Radosław Piliszek
480e575000 CI - remove unused setup scripts
This patch removes old, unused setup scripts.
They belong to kolla anyway.

Change-Id: Ia8dcd39991177153af61c638c7ae05123da08e82
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-06-21 15:10:34 +00:00
Mark Goddard
845040ad3f Add CI job for ironic
Adds four new CI jobs for testing centos/ubuntu binary/source deploys
with ironic enabled. These are run only when there are changes to the
ironic role.

Performs some simple testing by creating a node using the fake-hardware
hardware type and creating a server.

Change-Id: Ie669e57ce2af53257b4ca05f45193cb73f48827a
Depends-On: https://review.opendev.org/664011
2019-06-11 10:22:04 +01:00