101 Commits

Author SHA1 Message Date
Zuul
4e4d0f6568 Merge "Disable the redundant github issue update job" 2021-04-21 01:31:37 +00:00
Shturm, Svetlana
220f0673cb Remove publish job dependency from build job
With latest changes build images jobs uses zuul.change variable
which is not defined for merge event.
Also using build job as a parent for publish creates an unnessesarly
execution of make command which we repeat later under publish playbook.

That change removes that dependency.

Change-Id: I7e02c2f3ef2ccd8b9db9872b123222020f80664a
2021-04-19 14:39:06 -05:00
siraj.yasin
d4bae30fb3 Disable the redundant github issue update job
* airship-airshipctl-update-github-issues job adds comments
  to related github issue on PS merge based on the tag(Realtes-To/Closes)
  in commit message.
* airshipbot is already doing the job of adding comments to issues on new PSs
  and updating the status (closed, reopen) as applicable.
* So the job airship-airshipctl-update-github-issues is redundant when airshipbot
  is running.

Change-Id: I2c6fa748d3334384bf5b31c87d9501a006a153c1
Relates-To: #502
2021-04-19 13:26:40 +00:00
James Gu
06609c3180 Added script for the missing controlplane_target phase
controlplane_target phase is needed for target cluster more than a
single node

Signed-off-by: James Gu <james.gu@att.com>
Change-Id: I6e77d4268cdee0ebcc65e1f9172ef645ced53337
2021-04-13 10:47:47 -07:00
Alexey Odinokov
a7e332f9ec SOPS improvements
Added 2 phases:
1. secret-reencrypt - This phase can be used to
reecnrypt the existing secrets with new key.
To do so SOPS_IMPORT_PGP must contain
old public key and new private key (but
may also include other data).
SOPS_PGP_FP must contain fingerprint of
new private key.

2. secret-show - This phase may be useful for some users
that need to see what generated by secret-generate phase.

Disabled SOPS debug by default.
To enable it back run commands with
env variable DEBUG_SOPS_GPG=true

Change-Id: Id7fe13d6943d386577df25dba4aaa83e62e58980
2021-03-19 21:31:25 +00:00
Andrii Ostapenko
3f559c3c1e
Implements custom QCOW delivery
With this commit QCOW images are pulled as a docker image to new
qcow-bundle container within ironic pod and copy them to shared ironic
volume to be served to hosts.

Also squashed with [0], manifests are adjusted to consume new QCOWs and
old QCOW related funtionality is removed.

[0] https://review.opendev.org/c/airship/airshipctl/+/776270

Co-Authored-By: Alexey Odinokov <aodinokov@mirantis.com>
Co-Authored-By: Craig Anderson <craig.anderson@att.com>

Change-Id: I958184b34ae94206bc5e87993e9287587b6a11d9
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2021-02-23 15:43:45 -06:00
Alexey Odinokov
b51e7559b6 Adding encryption of k8s secrets and iso users passwords
This patchset introduces a generated with template [1] and encrypted
VariableCatalogue generated-secrets that contains steps to
generate: ephemeral and target CA+admin key/cert and passwords for
users in ephemeral bootstrap iso.

It also introduces the way how these secrets are used in manifests:
They're decrypted by kustomize and incorporated into the folders
`catalogues` in the site, so they can be used by replacement plugin.

This patchset contains modifications in replacement plugin
configurations to put the decrypted values from VariableCatalogue
in place.

Since k8s secrets were substituted with generated values
this patchset removes pre-generated k8s secrets.

[1]
manifests/type/gating/target/generator/secret-template.yaml

Change-Id: I0898c74012833f0e171d36bb8145acf358510b69
2021-02-12 04:07:36 +00:00
Alexey Odinokov
a098fb908d Preparing zuul tests for secrets generation
zuul takes playbooks from master
and if it's necessary to add one more step to
that playbooks, it's necessary to add that first
to playbook, merge and only after that merge the step
itself.

Change-Id: I9569e1e7e4b8be216563ee66b6e6adea170f5aa1
2021-02-10 20:34:39 +00:00
Ruslan Aliev
54256dbc70 Repair validate-site job
This job doesn't work properly since a lot of changes in the
airshipctl logic were applied. All the issues were addressed.

Change-Id: Iec6fa7e6a3aa1ab46d496a8fd63822df1f8124cc
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Relates-To: #19
2021-02-08 23:54:57 -06:00
Sirajudeen
00130a9b53 Cleanup CAPI deployment script
* Removed the duplicate script for CAPI ephemeral node deployment
* Updated the generic script to support all providers

Change-Id: Icc1bed5c1b62662109b43ec94ee2fdb5de6de09b
2021-01-28 19:24:20 +00:00
Sreejith Punnapuzha
0b2c9925bf Add encryption to gates
This commit fixes validate site by introducing the encryption mechanism

Signed-off-by: Sreejith Punnapuzha <Sreejith.Punnapuzha@outlook.com>
Change-Id: I01f6aa7ddfee16b06b40f6bfa06b2192a7931cd3
2021-01-18 16:16:49 +00:00
Vladislav Kuzmin
8dba799c18 Add secrets generator phase
This phase builded on top of generic executor container.
It uses kustomize generator to generate secrets
and SOPS function to encrypt secrets.

Usage:
    1. `curl -fsSL -o key.asc https://raw.githubusercontent.com/mozilla/sops/master/pgp/sops_functional_tests_key.asc`
       Copy existing key from sops project
    2. `export SOPS_IMPORT_PGP="$(cat key.asc)" && export SOPS_PGP_FP="FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4"`
    3. `airshipctl phase run secret-generate`
        It will generate and encrypt secret in
        manifests/site/test-site/target/generator/results/generated/
    4. `KUSTOMIZE_PLUGIN_HOME=$(pwd)/manifests SOPS_IMPORT_PGP=$(cat key.asc) kustomize build --enable_alpha_plugins
        manifests/site/test-site/target/catalogues/ > output.txt`
	It will decrypt encrypted secret

Co-authored-by: Alexey Odinokov <aodinokov@mirantis.com>
Change-Id: I1682d71b7805eb36c407e712dcb747de799bc8bb
Relates-To: #379
2021-01-14 18:57:15 +00:00
Anderson, Craig (ca846m)
2daacf5f2a image-builder integration for ISO builds
This introduces airshipctl integration with image-builder [0], which
replaces the existing isogen tool for ephemeral ISO generation.

The airshipctl isogen executor has been updated for building ephemeral
ISOs using the image-builder container. The ability for user-declared
filenames for cloud-init user data and network data was removed, since
the user's only interest is in supplying the relevant overrides, not in
transparent naming coordination with the image-builder container. A new
object is added to the document package to identify the document kind,
label, and key to retrieve data from since this is pattern we will
reuse elsewhere.

Progress flag removed as requsted. Progress is reported directly by the
image-builder container.

Isogen debug flag removed in favor of using log.DebugEnabled()

[0] https://review.opendev.org/#/c/730777/

Depends-On: https://review.opendev.org/c/airship/images/+/730777/
Change-Id: I545004feaf2116f8ffb29faf6f7f7f5fcfe24fff
2020-12-17 14:47:53 -08:00
Battina, Sai (sb464f)
f49b509534 Update IPA packages in ironic
This PS updates ironic deployment with IPA init container
which contains python agent agent packages which are ubuntu based
packages. The docker image for this can found here [0]

[0]
https://opendev.org/airship/images/src/branch/master/ipa-downloader-image

Change-Id: Idaaafb2ddb562cf22a62df36100dd1e6c76211fc
2020-12-10 15:07:04 +00:00
Zuul
df3764b7ac Merge "Remove document plugin subcommand" 2020-11-12 03:03:33 +00:00
Dmitry Ukov
9c69138990 Remove document plugin subcommand
This change finalises shift to new version of kustomize plugins

Change-Id: Ie47f42a9e2c4dc03fc4dca437020ad71a564d894
2020-11-11 14:59:14 +04:00
Rajat Sharma
854e25355b Incorporate HWCC in deployment gate.
Added sample HCC profiles and deployment script to apply profiles.

Change-Id: If8152a0afda142c40a53ddf9a4bd6866b435b0e5
Co-Authored-By: Rajat Sharma <rajatcu143@gmail.com>
Relates-To-Issue: #326
Depends-On: #738410
2020-11-10 16:12:28 +05:30
Sirajudeen
a360ebf9da Check if commit is linked to github issue
* Non-Voting gate to check if commit is linked to github issue

Change-Id: I8a4ea350382f8f2ed58ce49d52f1386d9c0d9c92
Closes: #387
2020-11-06 18:45:34 +00:00
Sirajudeen
c0f116a68c Separate lint job from the test
* Split lint and test jobs
* Add lint job to be run against all files

Change-Id: I74da7999324283984d61f38ae9fb394ee5b37ce6
Closes: #386
2020-11-05 22:35:32 +00:00
Zuul
340bed7ac5 Merge "Use token for github Authentication" 2020-10-26 18:32:56 +00:00
Sirajudeen
c33f5f5104 Use token for github Authentication
* Move from user/passwd to token based auth for
  github api access
* Github password based authentication will be deprecated soon
  https://developer.github.com/changes/2020-02-14-deprecating-password-auth/

Change-Id: Ic5af5bcf6ceb9828f2df7cca97e73c713dc79c1e
Closes: #373
2020-10-26 13:44:27 +00:00
Andrii Ostapenko
81efbceea6
Collect description of k8s object to job logs
Change-Id: I15226b9700acf169c100431d5dc7fafe41fbcb87
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-10-23 11:58:29 -05:00
Dmitry Ukov
d783d71f05 Introduce Replacement Transformer plugin container
Relates-To: #341
Change-Id: I65a5b8cf3bcf94321fc02f240b1d8075bca45c0c
2020-10-16 16:00:56 +04:00
Zuul
f92e8bd042 Merge "Replace PrimaryRepositoryName with PhaseRepositoryName" 2020-10-13 20:50:53 +00:00
Sirajudeen
8307148482 Remove clusterctl install from gate script
* clusterctl commands are driven through phase run.
  So removing install clusterctl CLI utility from gate
  script

Change-Id: I1c57aa07a9e19495c94c3080d0ce40dfc0e5dd47
2020-10-13 15:11:59 +00:00
Vladimir Kozhukalov
0dc4ab7491 Replace PrimaryRepositoryName with PhaseRepositoryName
Change-Id: I2276a3b3ea50b8e81dd240815cd00a0a2a2af96e
Relates-To: #356
2020-10-12 15:28:27 +03:00
Sidney Shiba
d0683139fe Integration of Azure provider to Airship 2.0
This commit integrates the Azure provider to the Airship 2.0 project.

It adds the following folders:
- manifest/function/capz: This folder contains all manifests required
for the integration of Azure provider.
- manifest/function/k8scontrol-capz: This folder contains the base
manifest for the Azure Workload cluster.
- manifest/site/az-test-site: This folder contains the manifests used
for initializing the CAPI and CAPZ components on the management cluster
invoking "airshipctl cluster init" and manifests used for deploying a
Workload cluster on the Azure Cloud by invoking the command
"airshipctl phase apply azure".
- tools/deployment/azure: provides the script shell that are used in the
zuul gates and local test.

Updated files:
- zuul.d/project.yaml and zuul.d/jobs.yaml have been updated to include
gates for validating the Azure provider integration.

Change-Id: Icbdc7f6f42c159f48dd11e35626da3bc016f5487
2020-10-05 19:27:31 +00:00
Sean Eagan
eb40a5700f Add gating for helm release management
This adds gating for helm release management by including a minimal
example workload phase which consists of a HelmRelease for the nginx
ingress controller and a corresponding deployment script including
supporting validation logic.

Change-Id: Ia21a799030289c7e40a0e61292578987ea0f6c63
Relates-To: #351
2020-09-23 14:59:24 +00:00
Sreejith Punnapuzha
a6528dc564 Enhance log collection zuul jobs
* Change name for gate script runner job
  * Make log collection job parameterize

Change-Id: I9f7c711556996af2da350d0160e572e1a26efd0f
Signed-off-by: Sreejith Punnapuzha <sreejith.punnapuzha@outlook.com>
2020-09-11 14:46:50 -05:00
Stanislav Egorov
393b75bc08 Worker node deployment
Change-Id: I1a1c06ee9fa16e2c56374e98f8aaea5da80898cd
2020-09-10 12:11:10 -07:00
Sreejith Punnapuzha
f95da5a56c Enhance target node deployment on gates
* Add script to install clusterctl
  * Add controlplane yamls for target node
  * Add script to perform cluster move
  * Add script to zuul jobs

Closes: #288
Change-Id: Ia6891df9c9b1da333396e76f11332deeb17ab807
Signed-off-by: Sreejith Punnapuzha <sreejith.punnapuzha@outlook.com>
2020-08-27 11:12:05 -05:00
Stanislav Egorov
2605cd9159 Gate scripts refactoring
Added flexible boot mode configuration for VMs based on BareMetalHost definition.

Change-Id: I1332ff99e7ec539ac280efda515f736b7c18f459
2020-08-19 14:49:03 -07:00
Alexander Noskov
582894d389 Switch to Ubuntu focal (20.04) based image
Since we have switched from debian to ubuntu based image and changed the
name (in https://review.opendev.org/733078) we need to align it with
airshipctl repository.
Renaming the following names:
1) debian-custom.iso -> ubuntu-focal.iso
2) quay.io/airshipit/isogen:latest-debian_stable -> quay.io/airshipit/isogen:latest-ubuntu_focal

Updating the following packages and resources:
1) k8s 1.17.3 -> 1.18.6
2) docker 19.03.9 -> 19.03.12
3) Calico 3.9 -> 3.15

Change-Id: I7eaf382acb79016a511db6e0955fa932c02963c4
2020-08-13 13:11:25 -05:00
Nikolay Fedorov
bd6f2f8e32 Add usage MAC for Ephemeral VM deploy
We should use interface naming independent on OS, type, speed, slot etc.
This CR takes an approach that is already used for the target node.
To define VMs, their script uses MAC addresses from manifests.

Relates-To: #285

Change-Id: Iafffd23bc584d2daf3fed5ee301491f447498193
2020-08-05 12:29:39 +04:00
Stanislav Egorov
e20c8ae1fa Preparation for workers deployment
Added air-worker VM
Import kubeconfig and check cluster
Initinfra and capi for target cluster

Change-Id: Ib05f805b428a1eab20d62df67ac97670714eebd0
2020-07-28 18:35:19 +00:00
Matt McEuen
0324993d60 Add an m3 host generator function
This PS has a function which constructs a collection of Metal3 BareMetalHost
resources, along with associated configuration Secrets.
It solves for a couple of things:

1. pulling the nitty gritty details for generating BMH into one reusable place,
2. allowing the site-specific details to be filled in via catalogues of values

This function leverages a couple of different plugins in sequence:
The airshipctl Replacement plugin, which pulls the site-specific data from
the catalogue documents into a Templater plugin configuration; and then
the airshipctl Templater plugin, which generates a variable number of
BMHs in a data-driven fashion.

More details can be found in the README.md in this patchset.

Closes: #245

Change-Id: I3ddbd36dc53ea6afbd633098c985f4b28bcbb793
2020-07-17 15:30:26 -05:00
Stanislav Egorov
f1725f3123 Cleaning for gate scripts
Removed orphaned local scripts because some roles were removed.
Fixed env vars for proper configuration.
Updated documentation accordingly.

Change-Id: I37f0c8d038fb51ddaa57664a65d347056df6f007
2020-07-09 13:16:05 -07:00
Stanislav Egorov
7e5ba1a195 Reverting back vars for build gate
Change-Id: Ia880686a9398ce0ce8b4d3b062d94601a53ca66f
2020-07-08 22:58:14 -07:00
Roman Gorshunov
eddd09f0a0 CI: remove redundant Ansiblle-based CI jobs
CI jobs have been rewritten in Bash scripts and seem to be working fine for the
past few weeks. Removing now redundant Ansiblle-based CI jobs.

This would reduce workload we impose in CI infrastructure.

Change-Id: I690405f5401da0beeb4b176ab22cffdfda24a09f
2020-07-08 16:19:56 +02:00
Ruslan Aliev
876c6043ee Replace apt/yum ansible modules, use package instead
apt/yum use is not convenient, package module automatically uses
the underlying OS package manager. Also, some ansible roles currently
use only apt module without yum, therefore patch fixes this bug too.

Change-Id: I5dd49d513d1a791ab51ca6ce6eb1c079542c5624
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
2020-06-29 20:28:49 +00:00
Sirajudeen
1fd55a7abd install python3-pip explicitly.
* zuul gate failing due to pip3 missing, so had to install it explicitly

Change-Id: I8d24cc506d58312f338e3632e97a567b23dc71a1
2020-06-24 16:54:34 -05:00
Sirajudeen
6aaceb1e7f Use sudo to install python3-setuptools
Change-Id: I4119486e8388ca69fce8a135d785f66cc9c4e880
2020-06-23 17:47:38 -05:00
Sirajudeen
08c5d6acbb Add python3-setuptools to required packages
* This commit adds python3-setuptools to required packages, so that
  the job airship-airshipctl-update-github-issues will not fail

Change-Id: I7b5a93e83175bb19dd6145f1af6bf7f6fd7a4d61
2020-06-23 09:30:26 -05:00
Sreejith Punnapuzha
c3002cabda Add bash scripts to run airshipctl
* Add playbook to run the scripts
 * Add script to install kubectl
 * Add script to create airshipctl executable
 * Add script to generate test configuration
 * Add script to pull site documents
 * Add script to build ephemeral iso
 * Add script to deploy ephemeral node
 * Add script to deploy control plane

Closes: #223
Closes: #224
Closes: #225
Closes: #226
Closes: #227

Change-Id: Ied63e2a733c3abcc6b1a822624c5ffce92e05a44
Signed-off-by: Sreejith Punnapuzha <sreejith.punnapuzha@outlook.com>
2020-06-19 15:44:06 +00:00
Ruslan Aliev
19c37fef7f Fix contol plane deployment ansible task
There is need to ensure that appropriate directory does not exist
before clonining source code into it, otherwise git throws an error.
The destination folder name for cloning was fixed. Variable name which
uses for task fail verification was corrected.

Relates-To: #271
Closes: #271
Change-Id: I9c2bd03f68e6c34fc28b900f11dbaadd190087e8
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
2020-06-15 02:17:22 -05:00
Yasin, Siraj (SY495P)
458fb2579b Fix gate scripts for local setup
* when remote_work_dir is set, document pull happens from the
  remote_work_dir (/tmp/airship) with a folder created within
  remote_work_dir with repo name (/tmp/airship/airshipctl).
  However the manifest yamls of kind "Clusterctl"
  refers to a relative path from work_dir and it fails
  finding expected objects.
   example: manifests/function/capm3/v0.3.1
* So trying to avoid document pull, and use current dir for workspace.
  similar to Zuul gate implementation

Change-Id: I63fd5476247f957745e15cbdfceb5fb483758e83
2020-06-12 03:00:32 +00:00
Dmitry Ukov
df2fff0acf Add control plane deployment
K8s control plane is deployed by KubeadmControlPlane controller. This
controller creates CAPI machines and infrastructure objects
(Metal3Machine). Metal3Machine objects are created based on a template
which contins host selector label. Control plane label is assigned to a
particular BareMetalHots object defined inside of the shared
kustomization.

Relates-To: #149
Closes: #221
Change-Id: I3be1750aacf9736ece2944045c036f405e404561
2020-06-11 13:44:42 +00:00
Matt McEuen
958d783919 Add site doc validation gate
This adds a gate which loops over all phases in all sites,
and performs an airshipctl apply --dry-run on them to ensure YAML
validity and schema adherence.  Aside from installation tasks,
the gate is run via a makefile entrypoint so that it can be
easily consumed by developers or by non-zuul CICD platforms.

Change-Id: Ie4ab246848a580ab20c3153af1e3749a27e3f770
2020-06-04 18:04:47 -05:00
Ian Howell
7243176e44 Add golint as a non-voting gate
Change-Id: I9850528a720ff517a3f400553fa8ab66bf4b1b3b
Closes: #253
2020-05-29 11:40:45 -05:00
Yasin, Siraj (SY495P)
9cde7fc8f5 Fixed the script path issue
* created a files dir within playbooks and moved
  update_github_issues.py

Change-Id: Icb621ec00bab0b431c09405370968c0b0ac975b7
Relates-To: #196
2020-05-27 14:34:03 +00:00