70 Commits

Author SHA1 Message Date
Stanislav Egorov
2605cd9159 Gate scripts refactoring
Added flexible boot mode configuration for VMs based on BareMetalHost definition.

Change-Id: I1332ff99e7ec539ac280efda515f736b7c18f459
2020-08-19 14:49:03 -07:00
Alexander Noskov
582894d389 Switch to Ubuntu focal (20.04) based image
Since we have switched from debian to ubuntu based image and changed the
name (in https://review.opendev.org/733078) we need to align it with
airshipctl repository.
Renaming the following names:
1) debian-custom.iso -> ubuntu-focal.iso
2) quay.io/airshipit/isogen:latest-debian_stable -> quay.io/airshipit/isogen:latest-ubuntu_focal

Updating the following packages and resources:
1) k8s 1.17.3 -> 1.18.6
2) docker 19.03.9 -> 19.03.12
3) Calico 3.9 -> 3.15

Change-Id: I7eaf382acb79016a511db6e0955fa932c02963c4
2020-08-13 13:11:25 -05:00
Nikolay Fedorov
bd6f2f8e32 Add usage MAC for Ephemeral VM deploy
We should use interface naming independent on OS, type, speed, slot etc.
This CR takes an approach that is already used for the target node.
To define VMs, their script uses MAC addresses from manifests.

Relates-To: #285

Change-Id: Iafffd23bc584d2daf3fed5ee301491f447498193
2020-08-05 12:29:39 +04:00
Stanislav Egorov
e20c8ae1fa Preparation for workers deployment
Added air-worker VM
Import kubeconfig and check cluster
Initinfra and capi for target cluster

Change-Id: Ib05f805b428a1eab20d62df67ac97670714eebd0
2020-07-28 18:35:19 +00:00
Matt McEuen
0324993d60 Add an m3 host generator function
This PS has a function which constructs a collection of Metal3 BareMetalHost
resources, along with associated configuration Secrets.
It solves for a couple of things:

1. pulling the nitty gritty details for generating BMH into one reusable place,
2. allowing the site-specific details to be filled in via catalogues of values

This function leverages a couple of different plugins in sequence:
The airshipctl Replacement plugin, which pulls the site-specific data from
the catalogue documents into a Templater plugin configuration; and then
the airshipctl Templater plugin, which generates a variable number of
BMHs in a data-driven fashion.

More details can be found in the README.md in this patchset.

Closes: #245

Change-Id: I3ddbd36dc53ea6afbd633098c985f4b28bcbb793
2020-07-17 15:30:26 -05:00
Stanislav Egorov
f1725f3123 Cleaning for gate scripts
Removed orphaned local scripts because some roles were removed.
Fixed env vars for proper configuration.
Updated documentation accordingly.

Change-Id: I37f0c8d038fb51ddaa57664a65d347056df6f007
2020-07-09 13:16:05 -07:00
Stanislav Egorov
7e5ba1a195 Reverting back vars for build gate
Change-Id: Ia880686a9398ce0ce8b4d3b062d94601a53ca66f
2020-07-08 22:58:14 -07:00
Roman Gorshunov
eddd09f0a0 CI: remove redundant Ansiblle-based CI jobs
CI jobs have been rewritten in Bash scripts and seem to be working fine for the
past few weeks. Removing now redundant Ansiblle-based CI jobs.

This would reduce workload we impose in CI infrastructure.

Change-Id: I690405f5401da0beeb4b176ab22cffdfda24a09f
2020-07-08 16:19:56 +02:00
Ruslan Aliev
876c6043ee Replace apt/yum ansible modules, use package instead
apt/yum use is not convenient, package module automatically uses
the underlying OS package manager. Also, some ansible roles currently
use only apt module without yum, therefore patch fixes this bug too.

Change-Id: I5dd49d513d1a791ab51ca6ce6eb1c079542c5624
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
2020-06-29 20:28:49 +00:00
Sirajudeen
1fd55a7abd install python3-pip explicitly.
* zuul gate failing due to pip3 missing, so had to install it explicitly

Change-Id: I8d24cc506d58312f338e3632e97a567b23dc71a1
2020-06-24 16:54:34 -05:00
Sirajudeen
6aaceb1e7f Use sudo to install python3-setuptools
Change-Id: I4119486e8388ca69fce8a135d785f66cc9c4e880
2020-06-23 17:47:38 -05:00
Sirajudeen
08c5d6acbb Add python3-setuptools to required packages
* This commit adds python3-setuptools to required packages, so that
  the job airship-airshipctl-update-github-issues will not fail

Change-Id: I7b5a93e83175bb19dd6145f1af6bf7f6fd7a4d61
2020-06-23 09:30:26 -05:00
Sreejith Punnapuzha
c3002cabda Add bash scripts to run airshipctl
* Add playbook to run the scripts
 * Add script to install kubectl
 * Add script to create airshipctl executable
 * Add script to generate test configuration
 * Add script to pull site documents
 * Add script to build ephemeral iso
 * Add script to deploy ephemeral node
 * Add script to deploy control plane

Closes: #223
Closes: #224
Closes: #225
Closes: #226
Closes: #227

Change-Id: Ied63e2a733c3abcc6b1a822624c5ffce92e05a44
Signed-off-by: Sreejith Punnapuzha <sreejith.punnapuzha@outlook.com>
2020-06-19 15:44:06 +00:00
Ruslan Aliev
19c37fef7f Fix contol plane deployment ansible task
There is need to ensure that appropriate directory does not exist
before clonining source code into it, otherwise git throws an error.
The destination folder name for cloning was fixed. Variable name which
uses for task fail verification was corrected.

Relates-To: #271
Closes: #271
Change-Id: I9c2bd03f68e6c34fc28b900f11dbaadd190087e8
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
2020-06-15 02:17:22 -05:00
Yasin, Siraj (SY495P)
458fb2579b Fix gate scripts for local setup
* when remote_work_dir is set, document pull happens from the
  remote_work_dir (/tmp/airship) with a folder created within
  remote_work_dir with repo name (/tmp/airship/airshipctl).
  However the manifest yamls of kind "Clusterctl"
  refers to a relative path from work_dir and it fails
  finding expected objects.
   example: manifests/function/capm3/v0.3.1
* So trying to avoid document pull, and use current dir for workspace.
  similar to Zuul gate implementation

Change-Id: I63fd5476247f957745e15cbdfceb5fb483758e83
2020-06-12 03:00:32 +00:00
Dmitry Ukov
df2fff0acf Add control plane deployment
K8s control plane is deployed by KubeadmControlPlane controller. This
controller creates CAPI machines and infrastructure objects
(Metal3Machine). Metal3Machine objects are created based on a template
which contins host selector label. Control plane label is assigned to a
particular BareMetalHots object defined inside of the shared
kustomization.

Relates-To: #149
Closes: #221
Change-Id: I3be1750aacf9736ece2944045c036f405e404561
2020-06-11 13:44:42 +00:00
Matt McEuen
958d783919 Add site doc validation gate
This adds a gate which loops over all phases in all sites,
and performs an airshipctl apply --dry-run on them to ensure YAML
validity and schema adherence.  Aside from installation tasks,
the gate is run via a makefile entrypoint so that it can be
easily consumed by developers or by non-zuul CICD platforms.

Change-Id: Ie4ab246848a580ab20c3153af1e3749a27e3f770
2020-06-04 18:04:47 -05:00
Ian Howell
7243176e44 Add golint as a non-voting gate
Change-Id: I9850528a720ff517a3f400553fa8ab66bf4b1b3b
Closes: #253
2020-05-29 11:40:45 -05:00
Yasin, Siraj (SY495P)
9cde7fc8f5 Fixed the script path issue
* created a files dir within playbooks and moved
  update_github_issues.py

Change-Id: Icb621ec00bab0b431c09405370968c0b0ac975b7
Relates-To: #196
2020-05-27 14:34:03 +00:00
Zuul
57b47c9640 Merge "Ansible syntax fix and added message decrypt." 2020-05-22 15:55:10 +00:00
Yasin, Siraj (SY495P)
b248f6e4a4 Ansible syntax fix and added message decrypt.
* Decrypt message which is base64 encoded in secrets
* read change_url from zuul dictionary instead of its
  subelement items

Change-Id: I2137be52d4ad8c0a158b6192b6e0a08527c60328
Relates-To: #196
2020-05-21 12:25:34 -05:00
Dmitry Ukov
a301de202b Add CAPI initialization steps to gating
This change enables execute 'airshipctl cluster init' command
as a part of zuul pipeline

Co-Authored-By: Kostyantyn Kalynovskyi <kkalynovskyi@mirantis.com>
Change-Id: I9d499ae66f8d847759c2e61e5d4415f2c9f3c860
2020-05-20 15:17:20 +04:00
Zuul
66980ed432 Merge "Replace become: true notation" 2020-05-20 04:27:57 +00:00
Ruslan Aliev
76d8716968 Replace become: true notation
Ansible documentation recommends to use yes/no as boolean value,
the last mention of possible using true/yes related to version 2.4 [1].
Starting from version 2.5 only become: yes specified in docs [2][3].
Since we have both variations, this patch aligns become statement to
use only yes value.

[1] https://docs.ansible.com/ansible/2.4/become.html
[2] https://docs.ansible.com/ansible/2.5/user_guide/become.html
[3] https://docs.ansible.com/ansible/latest/user_guide/become.html

Change-Id: I706e6e1b9d706d6bfcc38c1a3d0d7304575946ce
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
2020-05-16 01:48:42 -05:00
Ruslan Aliev
917d6896b6 Remove "cluster initinfra" command
Command "cluster initinfra" should be removed since
"phase apply" was introduced instead.

Change-Id: I263f8988a4456fc03314093a55ee1c1da8048404
Relates-To: #162
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
2020-05-16 01:45:34 -05:00
Zuul
528407edc7 Merge "Add pod log collecting role" 2020-05-12 16:10:18 +00:00
Dmitry Ukov
ea79cd6a71 Add pod log collecting role
Closes: #220
Change-Id: I5a4b682f8f4b94fa6827e6d304d07d825811916b
2020-05-12 18:08:03 +04:00
Zuul
3fb7c774e7 Merge "Reusing apache-server instead of httpd" 2020-05-11 15:39:30 +00:00
Zuul
4e0af74f46 Merge "Fixed document pull issue in gate script 21" 2020-05-08 20:47:48 +00:00
James Gu
301de51467 Fixed document pull issue in gate script 21
Changed systemwide-executable to run in local_src_dir because the script needs
to build airshipctl binary first, before doc pull command can be invoked and populate
the remote_work_dir. Changed the order of the roles to allow systemwide-executable
to run before test-config and pull-docs.

Change-Id: Ia47423d91cdfb2dbbf0259344daab1c7b8cd47f5
Signed-off-by: James Gu <james.gu@att.com>
2020-05-08 18:30:51 +00:00
Albin Vass
ee2b91efed Use in repo docker-install instead of install-docker from zuul-jobs
Change-Id: Id94ef7f5a0327aff5c184a85d11f457e0436630f
2020-05-08 18:59:00 +02:00
Drew Walters
f729fdd1a1 Replace venv with tox for documentation builds
OpenStack infra has created Zuul project templates that will enable us
to push our documentation to Read the Docs and the Airship website.
While adding a tox file to this project is not ideal, it appears to be
the only way to leverage these jobs at the moment.

This change adds a tox.ini file to airshipctl to build our documentation
and removes the venv solution. The Airship Working Committee will work
with the OpenStack infra team, if possible, to make the publish jobs
more flexible in the future and remove the tox.ini file.

Change-Id: I274cf69a7c79e0aad1d4c9b1decfaf5630f0b18f
Signed-off-by: Drew Walters <andrew.walters@att.com>
2020-05-08 15:47:41 +00:00
Stanislav Egorov
b599664bb7 Fixed local gate runner
Role airshipctl-test-configs should be run after building airshipctl
because it's running binary that is not built yet

Change-Id: I7d352bfbd80f9db50913fae3620170bea9460325
2020-05-06 11:14:51 -07:00
Alexey Odinokov
16d2c8d8c1 Reusing apache-server instead of httpd
Since we already introduced apache server for redfish,
it will be a good idea to use it for sharing iso also.
These changes add file-exchanger config that can be
used to share iso. In addition apache allows the
specified users (see rbac config) to upload files.
This may help in future to better emulate deployment
environment, where it will be necessary to do this
step after airshipctl generated iso and before
airshipctl invokes redfish to run that iso.

Change-Id: Ice47fe977fb71cf9df14d39c5ac26e365e2006f2
2020-05-01 03:17:02 +00:00
James Gu
d26df73b4b [#112] Add doc pull in deployment scripts
Added the optional use of doc pull in gate deployment scripts. When the
remote_work_dir is defined, the new pull doc role will invoke airshipctl
document pull command to download the manifests to the remote_work_dir.
This role is not included in zuul job so the zuul job will continue to use
the already checked out manifests in the zuul.project.src directory.

Fixes #112

Change-Id: I754c08661ad0b635819cb3730e05a6986d90f5ca
Signed-off-by: James Gu <james.gu@att.com>
2020-04-30 15:11:44 -07:00
James Gu
afcfdb5c6e Proxy support enhancement
Added gate scripts and playbook enhancement to run airshipctl behind proxy.

Change-Id: Icc3cdcd6fd150420103f907cd1871feba1ccd223
Signed-off-by: James Gu <james.gu@att.com>
2020-04-28 18:04:55 +00:00
Vamsi Savaram
e40b5ffec5 Add initinfra gating tests
Relates-To: #180

Change-Id: Ie7a8daa532596f2d1d0fb2059ef4630607511711
2020-04-23 14:41:45 +00:00
Alexey Odinokov
bb7bd1c58e Substituting redfish-emulator and reverse-proxy with Apache
The current implementation of airship-libvirt-gate is using
sushy-emulator binary to emulate redfish. Sushy-emulator works
only for http and also can’t authenticate users out-of-box if
ran by itself. In order to check https and authentication the
reverse-proxy was introduced. This approach had several
drawbacks:
1) http still doesn’t check auth
2) to use apache for https only is too heavy solution for https
This change converts reverse proxy to apache running
sushy-emulator as wsgi backend, that gives an ability to check
authentication for both http and https.
We’re also getting rid of ad-hoc sushy-emulator service and
using out-of-box apache service implementation.
The code also introduces gathering of apache resulting configs
and logs for quicker debug if needed.
Right now authentication is disabled, since manifests are
written in a way so they don’t use them. If it’s necessary to
enable it, just set username here[1]

PS
There is ability to use apache for http-server [2], but it’s
better to do as a separate PR

[1]
roles/airship-libvirt-gate/defaults/main.yaml
[2]
roles/http-fileserver

Change-Id: I43b5bca41519c88b01535c156b2db0e9edaa81bb
2020-04-17 15:55:48 +00:00
Yasin, Siraj (SY495P)
c25d223c7b Add copyright for missing files
* added license templates for go, bash & yaml in tools dir
* added a script that will add license information for all
    missing files. Type:  go, yaml, yml, sh
* skip adding license for all files within testdata
* Syntax:
   > ./tools/add_license.sh

* Skip license for manifests folder
* Added one extra line after licene for yaml files
* Added License after Hashbang for bash.
* Add an extra line after hashbang and before license
* Updated the go template to use multiline comments

New Files:
  1. tools/add_license.sh
  2. tools/license_go.txt
  3. tools/license_yaml.txt
  4. tools/license_bash.txt

Change-Id: Ia4da5b261e7cd518d446896b72c810421877472a
Realtes-To:#147
2020-04-09 08:35:59 -05:00
Stanislav Egorov
b4a4be2a7f Switch gate to official isogen image
Change-Id: I65e02251b2f6c747c72a38bb20caf2dd3978ea7c
2020-04-02 14:17:15 -07:00
Sreejith Punnapuzha
d39e490f06 [#106] Add docker install role
This PS adds role which installs docker

Depends-On: https://review.opendev.org/#/c/714109/

Change-Id: I2d114fbb8bd93f88bd9526529e985f92125dc428
Signed-off-by: Sreejith Punnapuzha <sreejith.punnapuzha@outlook.com>
2020-03-23 15:25:15 -05:00
Zuul
b2a602fa07 Merge "Revert "[#89] Gather container logs in check pipeline"" 2020-03-19 21:36:30 +00:00
Roman Gorshunov
a8c91a7c40 Revert "[#89] Gather container logs in check pipeline"
This reverts commit 5c5ff8b46f254c5004bbf20b9b7da4933d476b58.

Logs are not being collected under artifacts section of Zuul.
Container logs collector patch set 712555 needs further work.

Change-Id: If8703812cf101506de2808365dfee6c032a0f35f
2020-03-19 15:26:58 +00:00
Zuul
906c2b2ec2 Merge "[#89] Gather container logs in check pipeline" 2020-03-19 11:54:05 +00:00
Vamsi Savaram
5c5ff8b46f [#89] Gather container logs in check pipeline
Change-Id: I043e7bf1cbb4cf67e3b811df1c0e84ecfc137988
Signed-off-by: Vamsi Savaram <vamsi.savaram@gmail.com>
2020-03-17 16:27:17 +01:00
Zuul
8785d15836 Merge "[#106] Add Gather logs role" 2020-03-17 13:50:55 +00:00
Sreejith Punnapuzha
7f3a2991dd [#106] Add Gather logs role
This PS adds role which does the log collection

Change-Id: Id249d87be4e95568659bdd5c6b1a20d279bc5e9f
Signed-off-by: Sreejith Punnapuzha <sreejith.punnapuzha@outlook.com>
2020-03-12 14:44:58 -05:00
Kostiantyn Kalynovskyi
147b97048b Split document model, add entrypoints for repos
Add NewBundleByPath function, that would return bundle built from
the specified path argument

Add CurrentContextEntryPoint method of the config
object, that would allow easily get kustomize root path based on
clusterType and phase. You can also leave phase arg empty string,
which would try to return bundle for all phases

Introduce changes to config pakage objects:

- Manifest:
  SubPath: this is relative path to the root of the repository that
contains directories with sites (SiteNames)
    PrimaryRepositoryName: this is a string that must correspond to a key
of the Repositories map of manifest object, which is used to derive
primary repository
    Repositories object is a map, map keys correspond to names of the
directories where `document pull` command will download repositories
defined in manifest prepended by manifest.TargetPath.

Introduce new config method CurrentContextEntryPoint(), method takes
TargetPath, PrimaryRepo.URL, SubPath, and clusterType and phase
constructs a path to the entry point out of which the DocumentBundle
should be build, and returns it to the caller. After that caller can
build a bundle out of it, the bundle will contain documents relevant to
particular cluster-type and phase.

All objects that depend on bundle interface are updated to use the
CurrentContextEntryPoint() method of the config object

Relates-To: #99

Closes: #99

Change-Id: I99320c4cb626841d46f4c298b583e9af90b1dce4
2020-03-12 09:55:05 -05:00
Alexander Hughes
ea0dd64935 [#70] Resolve kubectl gate errors
In [0] the kubectl subcommand was removed from airshipctl, which is
now causing gates [1] and [2] to fail. This change addresses these
failures by ensuring kubectl is installed via curl, and that kubectl
commands are run in place of the original airshipctl kubectl commands.

[0] https://review.opendev.org/#/c/710094/
[1] https://zuul.opendev.org/t/openstack/job/airship-airshipctl-functional-existing-k8s
[2] https://zuul.opendev.org/t/openstack/job/airship-airshipctl-gate-test

Relates-To: #70

Change-Id: I03b3420588a4abb5cd24fdc62a07019783a40557
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
2020-03-10 19:15:07 +00:00
Alexander Hughes
abb28716c8 [#81] makefile to check for unused test data
This change adds logic to the docker-image-unit-tests to ensure that
code is checked within the container for coverage, valid test data,
and reduces the number of containers created in the gate - reducing
the runtime of each gate.

Relates-To: #81

Change-Id: Iab3d9bd74266d32404e6eb06ecebc234df04fd5c
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
2020-03-08 00:05:48 +00:00