102 Commits

Author SHA1 Message Date
Ruslan Aliev
d632a5de5f Enable voting for golint job
Since golint job was fixed and it's essential to maintain
code standards this job should have voting status.

Change-Id: I623171ce89b3906eac30e11fa3d7ebeb2725372b
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
2021-01-20 21:47:35 -06:00
Sirajudeen
d9a7200698 Removing the non-voting jobs from dependency
* Removing the non-voting job from dependency. So that
  for any reason if the dependent non-voting job fails,
  the gate runner job will not be skipped.

Change-Id: If2f208d41a6df1ce7286654fe6e203d309042874
2021-01-18 21:04:24 +00:00
Zuul
42e7dea246 Merge "Fix container image publishing" 2021-01-08 20:00:09 +00:00
Zuul
6397810826 Merge "Integrate Sonobuoy Conformance Test Scripts" 2021-01-07 17:08:07 +00:00
Rishabh Kumar Jain
b0217a8ba1 Integrate Sonobuoy Conformance Test Scripts
Add jobs in experimental pipeline to do the following:

  - install Sonobuoy
  - run CNCF Conformace Tests
  - run CIS Benchmarks Tests

Conformance tests include:

  - CNCF Compliance: uses sonobuoy end-to-end (e2e) and systemd-logs
                     plugins

  - CIS Benchmarks: utilizes the kube-bench implementation
                    of the CIS security benchmarks plugin

Pipeline can be triggered by comment
   - "check experimental"
Change-Id: I7d08ae42512dc4c83e2f550c4809ce1f8ddccc7b

Change-Id: I2e6469f5b8e229828532ce5499498da639d23fe6
2021-01-06 16:28:06 -05:00
Sirisha Gopigiri
52151ef9fd Zuul gate to check rotate-sa-token and certificate expiry commands
This Patch Set uses docker zuul gate and
checks the certificate expiration
and certiticate rotate-sa-token commands

Change-Id: I76f902d4fcacdbfe168abd58ec707282e46a3f91
2021-01-06 18:16:45 +05:30
Anderson, Craig (ca846m)
2daacf5f2a image-builder integration for ISO builds
This introduces airshipctl integration with image-builder [0], which
replaces the existing isogen tool for ephemeral ISO generation.

The airshipctl isogen executor has been updated for building ephemeral
ISOs using the image-builder container. The ability for user-declared
filenames for cloud-init user data and network data was removed, since
the user's only interest is in supplying the relevant overrides, not in
transparent naming coordination with the image-builder container. A new
object is added to the document package to identify the document kind,
label, and key to retrieve data from since this is pattern we will
reuse elsewhere.

Progress flag removed as requsted. Progress is reported directly by the
image-builder container.

Isogen debug flag removed in favor of using log.DebugEnabled()

[0] https://review.opendev.org/#/c/730777/

Depends-On: https://review.opendev.org/c/airship/images/+/730777/
Change-Id: I545004feaf2116f8ffb29faf6f7f7f5fcfe24fff
2020-12-17 14:47:53 -08:00
Rishabh Kumar Jain
ae6ce6931b Integrate common infra provider zuul scripts
* add common scripts for providers capd/capg/capz/capo
    to do the following:
    - install kind (reuses  tools/document/get_kind.sh)
    - improve tools/document/start_kind.sh to use
      custom kind config file
    - created a script similar to 26_deploy_metal3_capi_ephemeral_node.sh,
      called tools/deployment/26_deploy_capi_ephemeral_node.sh and
      made it flexible for all providers (metal3/capd/capz/capg/capo)
    - deploy ephemeral control plane
    - initialize target node
    - move crds from ephemeral to target
    - deploy target workers

Closes: #413
Change-Id: If3520d000a03e5ed26d54e1ceedb2febfceccb3d
2020-12-11 16:45:07 -05:00
Sean Eagan
3e694a7349 Fix container image publishing
The existing `image_repo` value was leading to invalid image tag name
errors e.g. "https://quay.io/api/v1//airshipit/airshipctl:latest". This
updates the value to match the default DOCKER_REGISTRY value in the Makefile.

Change-Id: Ifa9112c8b91475593300ff04fd069dff703762af
Closes: #437
2020-12-09 11:15:56 -06:00
Zuul
a5c96cb430 Merge "Removing Azure Zuul Gate" 2020-12-03 23:18:39 +00:00
Dinesh Garg
396a160271 Allow proxy and container repository customization
Modify following roles to accept the proxy and env
variables from zuul jobs and project manifest so they
can be overwritten to run them on downstream zuul

- airshipctl-build-images
- airshipctl-publish-images

Change-Id: Ied06e60507f125c60ca7b9d47331e967fd69bf14
2020-11-18 15:29:41 +00:00
Sidney Shiba
2de2016f9a Removing Azure Zuul Gate
This patchset removes the Zuul gate for the Azure integration, which is
not relevant anymore.

Change-Id: I4e2751965f6b3fbc8c2b062f2acfa03c1575521a
2020-11-16 14:59:49 -06:00
Zuul
7a404c6fa6 Merge "Incorporate HWCC in deployment gate." 2020-11-11 12:08:22 +00:00
Roman Gorshunov
5b862d81c3 CI: run gate runner only on request and on merge
The airshipctl gate runner consumes 16GB RAM VM node. Currently we have
insufficient compute capacity to run it on every patch set change.

This patch set makes gate runner run only on request and on Zuul gate
(pre-merge) to reduce workload imposed onto community CI.

Request to run airshipctl gate runner is a comment
"check experimental"
left to a patch set in Gerrit.

Change-Id: I5cb2a80b5cdc9dd025249e313d29eab62cbb3446
2020-11-10 14:07:00 +01:00
Rajat Sharma
854e25355b Incorporate HWCC in deployment gate.
Added sample HCC profiles and deployment script to apply profiles.

Change-Id: If8152a0afda142c40a53ddf9a4bd6866b435b0e5
Co-Authored-By: Rajat Sharma <rajatcu143@gmail.com>
Relates-To-Issue: #326
Depends-On: #738410
2020-11-10 16:12:28 +05:30
Sirajudeen
a360ebf9da Check if commit is linked to github issue
* Non-Voting gate to check if commit is linked to github issue

Change-Id: I8a4ea350382f8f2ed58ce49d52f1386d9c0d9c92
Closes: #387
2020-11-06 18:45:34 +00:00
Sirajudeen
f259bc8a62 Fix the lint job name in azure gate runner
* Recently the lint and test jobs are separated
* update the reference to lint and test job names
  marked as dependency for azure gate runner

Change-Id: I9bf495143b14d248c54c20e4814ac236b167eba1
2020-11-06 18:44:25 +00:00
Zuul
a4d7410830 Merge "Separate lint job from the test" 2020-11-06 18:16:39 +00:00
Sirajudeen
c0f116a68c Separate lint job from the test
* Split lint and test jobs
* Add lint job to be run against all files

Change-Id: I74da7999324283984d61f38ae9fb394ee5b37ce6
Closes: #386
2020-11-05 22:35:32 +00:00
Alexander Hughes
2e92614239 Skip azure gate runner on irrelevant files
Change-Id: I693d78a2f400b45c6095c039c3a91e7c3d9fb4c8
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
2020-11-05 14:26:23 +00:00
Kostiantyn Kalynovskyi
68542a8c0a Enable gate runner job during gating
This would will make sure that issues like #382 would not arise
in the future.

Change-Id: I3f65c760fc2d893a2e93ae8d606b2282bb1008b3
Relates-To: #383
Closes: #383
2020-10-29 02:46:41 +00:00
Zuul
340bed7ac5 Merge "Use token for github Authentication" 2020-10-26 18:32:56 +00:00
Sirajudeen
c33f5f5104 Use token for github Authentication
* Move from user/passwd to token based auth for
  github api access
* Github password based authentication will be deprecated soon
  https://developer.github.com/changes/2020-02-14-deprecating-password-auth/

Change-Id: Ic5af5bcf6ceb9828f2df7cca97e73c713dc79c1e
Closes: #373
2020-10-26 13:44:27 +00:00
Andrii Ostapenko
81efbceea6
Collect description of k8s object to job logs
Change-Id: I15226b9700acf169c100431d5dc7fafe41fbcb87
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-10-23 11:58:29 -05:00
Matt McEuen
f164c97e6a Increase doc validation job timeout
This increases the timeout of the document validation job from
30min (default) to 60min.  This is required because of the recent
addition of provider-specific site definitions; since a kind cluster
is set up / torn down for each site definition, it can take longer
than a half hour when running on slower open infra VMs.

Change-Id: I23e7201365058d96ccdc07e310c3c921daf952a7
2020-10-21 09:29:46 -05:00
Kostiantyn Kalynovskyi
81de5ee252 Enable voting of gate tests
Opendev infra 16G flavors are much more stable now and we can
enable this test again.

Change-Id: Ie6b5a97030b89e3235814e9cae8bb238b02087ce
2020-10-13 16:32:03 -05:00
Sirajudeen
8307148482 Remove clusterctl install from gate script
* clusterctl commands are driven through phase run.
  So removing install clusterctl CLI utility from gate
  script

Change-Id: I1c57aa07a9e19495c94c3080d0ce40dfc0e5dd47
2020-10-13 15:11:59 +00:00
Zuul
ce3d3b3242 Merge "Make document validatation job non voting" 2020-10-09 14:48:24 +00:00
Zuul
c6a9082b2c Merge "Making gate runner job dependent of small jobs." 2020-10-05 22:37:29 +00:00
Sidney Shiba
d0683139fe Integration of Azure provider to Airship 2.0
This commit integrates the Azure provider to the Airship 2.0 project.

It adds the following folders:
- manifest/function/capz: This folder contains all manifests required
for the integration of Azure provider.
- manifest/function/k8scontrol-capz: This folder contains the base
manifest for the Azure Workload cluster.
- manifest/site/az-test-site: This folder contains the manifests used
for initializing the CAPI and CAPZ components on the management cluster
invoking "airshipctl cluster init" and manifests used for deploying a
Workload cluster on the Azure Cloud by invoking the command
"airshipctl phase apply azure".
- tools/deployment/azure: provides the script shell that are used in the
zuul gates and local test.

Updated files:
- zuul.d/project.yaml and zuul.d/jobs.yaml have been updated to include
gates for validating the Azure provider integration.

Change-Id: Icbdc7f6f42c159f48dd11e35626da3bc016f5487
2020-10-05 19:27:31 +00:00
Zuul
e884972839 Merge "Increase gate test timeout to 2 hours" 2020-10-05 15:01:52 +00:00
Sirajudeen
c643f7c3f4 Making gate runner job dependent of small jobs.
* The idea here is to execute all the applicable small jobs first.
* If all the applicable small jobs succeeds, then proceed with the
  long running gate runner job.

Change-Id: I807ad3bf725db35945e83feff0601b011c771d5a
2020-10-02 21:00:57 +00:00
Roman Gorshunov
1ad48c0983 Reduce Zuul CI pool workload
Added patterns of files which should and should not trigger jobs on Zuul CI,
to reduce workload imposed onto the CI infrastructure.

Change-Id: I7081c7a2a4902a29e571dce1133465f2821d71a3
2020-09-30 19:38:55 +02:00
Roman Gorshunov
92ffcb6e66 Reduce Zuul CI pool workload
Added patterns of files which should not trigger the most resource-consuming
job (on 16GB VM) on Zuul CI.

Change-Id: I143a15f531f5179cebf2741eacd67818976fda94
2020-09-29 15:09:40 +02:00
Ruslan Aliev
d8a6139e81 Add document pull step to gate script runner
Currently we don't test document pull step by rewriting
manifest_directory variable as current source path, which
makes a mess in target_path, manifest_path and related paths
(concatenation of target_path+manifest_path won't return
an existing and proper location). This patch organizes
mentioned variables and enables document pull step which
starts working properly. The latest repo state used as
git source for this command.

Change-Id: I5abce73877441c7529f2f77add79cf410e2226d8
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
2020-09-25 00:39:28 -05:00
Sean Eagan
eb40a5700f Add gating for helm release management
This adds gating for helm release management by including a minimal
example workload phase which consists of a HelmRelease for the nginx
ingress controller and a corresponding deployment script including
supporting validation logic.

Change-Id: Ia21a799030289c7e40a0e61292578987ea0f6c63
Relates-To: #351
2020-09-23 14:59:24 +00:00
Kostiantyn Kalynovskyi
f56f34d731 Increase gate test timeout to 2 hours
When the successful gate test runs for approximately 1:20, current
timeout does not allow jobs to pass if there is bad latency at the
DC in which vm was spawned. This will reduce the amount of rechecks
that patch sets owners have to perform.

Change-Id: Icd7410126c6bcdf01ff6763255023539e1950c45
2020-09-22 13:07:45 -05:00
Kostiantyn Kalynovskyi
7836e73773 Make document validatation job non voting
The job needs to be redesigned in general, and it doesn't belong
to airshipctl repository. Document validation should be performed
in treasure map repository.

Next step would be to move validation job to treasure map repo

Change-Id: I627eec5c5d7efddd22c6aaa672091210451e89ae
2020-09-22 10:18:37 -05:00
Sreejith Punnapuzha
a6528dc564 Enhance log collection zuul jobs
* Change name for gate script runner job
  * Make log collection job parameterize

Change-Id: I9f7c711556996af2da350d0160e572e1a26efd0f
Signed-off-by: Sreejith Punnapuzha <sreejith.punnapuzha@outlook.com>
2020-09-11 14:46:50 -05:00
Stanislav Egorov
393b75bc08 Worker node deployment
Change-Id: I1a1c06ee9fa16e2c56374e98f8aaea5da80898cd
2020-09-10 12:11:10 -07:00
Sreejith Punnapuzha
f95da5a56c Enhance target node deployment on gates
* Add script to install clusterctl
  * Add controlplane yamls for target node
  * Add script to perform cluster move
  * Add script to zuul jobs

Closes: #288
Change-Id: Ia6891df9c9b1da333396e76f11332deeb17ab807
Signed-off-by: Sreejith Punnapuzha <sreejith.punnapuzha@outlook.com>
2020-08-27 11:12:05 -05:00
Sirajudeen
4f19a506a3 Increase Job timeout due to recent infra changes
Change-Id: I41120dfd1e0fe0715eff0003a93e6dfa38b336cd
2020-08-05 16:40:53 +00:00
Stanislav Egorov
e20c8ae1fa Preparation for workers deployment
Added air-worker VM
Import kubeconfig and check cluster
Initinfra and capi for target cluster

Change-Id: Ib05f805b428a1eab20d62df67ac97670714eebd0
2020-07-28 18:35:19 +00:00
Matt McEuen
2dbd13c644 Re-enable doc validation gate
This sets the validation gate to be voting again; it had been turned
off while some work was in flight.  However, the check is passing again
and re-enabling it will help ensure document goodness.

A couple of tweaks were made based on issues seen during multi-cluster
testing:

1. The kind cluster is now torn down between each cluster (eph and target).
   We'd been reusing the cluster for the sake of speed, but deleting
   everything out of it still left some unexplained cruft behind,
   which broke validation of the second (target) cluster.

2. This reverts back to a simple approach to phase selection and sequencing,
   away from a fancier technique which was failing the only phase
   was "initinfra".

Note: once `airshipctl phase validate` and a queryable phase map are
in place, the gate should be reworked to leverage them instead.

Change-Id: I37c51e859e2df880981e2e138a67c23e17eabd93
2020-07-23 18:55:09 -05:00
Matt McEuen
0324993d60 Add an m3 host generator function
This PS has a function which constructs a collection of Metal3 BareMetalHost
resources, along with associated configuration Secrets.
It solves for a couple of things:

1. pulling the nitty gritty details for generating BMH into one reusable place,
2. allowing the site-specific details to be filled in via catalogues of values

This function leverages a couple of different plugins in sequence:
The airshipctl Replacement plugin, which pulls the site-specific data from
the catalogue documents into a Templater plugin configuration; and then
the airshipctl Templater plugin, which generates a variable number of
BMHs in a data-driven fashion.

More details can be found in the README.md in this patchset.

Closes: #245

Change-Id: I3ddbd36dc53ea6afbd633098c985f4b28bcbb793
2020-07-17 15:30:26 -05:00
Roman Gorshunov
041dcdb82c CI: Remove job running on 32GB instances
CI fits well on 16GB RAM instance just fine and only runs a few minutes longer.
Overall 32GB instances have shown to be stable. In currenct situation of lack
of resources this patch removes job which runs on 32GB instances. Identical job
continues to run on 16GB instances.

Change-Id: I0681894a6d8cee22048febe21152a03c47ac7ede
2020-07-08 18:10:27 +02:00
Roman Gorshunov
eddd09f0a0 CI: remove redundant Ansiblle-based CI jobs
CI jobs have been rewritten in Bash scripts and seem to be working fine for the
past few weeks. Removing now redundant Ansiblle-based CI jobs.

This would reduce workload we impose in CI infrastructure.

Change-Id: I690405f5401da0beeb4b176ab22cffdfda24a09f
2020-07-08 16:19:56 +02:00
Roman Gorshunov
dda36462e2 CI: Temporarily disable failing gate
CI jobs using 16GB and 32GN Zuul nodes are failing on provider side.
Temporarily disabling voting on those gates, letting the code to be merged if
necessary.

Change-Id: I99ec2ccacf732844b7e3e56b41372e5623d4e9ee
2020-07-08 14:50:59 +02:00
Sirajudeen
c37b27b8d0 Zuul Job with Bash oriented gate tests
* Added bash scripts in place of ansible playbooks/roles
* zuul will run bash scripts, and this means that they are actually tested
  which will make development experience easier

Change-Id: I60d1dac589ae8ad835d3f1dda5e2a45523367496
Relates-To: #222
2020-06-19 15:44:27 +00:00
Dmitry Ukov
df2fff0acf Add control plane deployment
K8s control plane is deployed by KubeadmControlPlane controller. This
controller creates CAPI machines and infrastructure objects
(Metal3Machine). Metal3Machine objects are created based on a template
which contins host selector label. Control plane label is assigned to a
particular BareMetalHots object defined inside of the shared
kustomization.

Relates-To: #149
Closes: #221
Change-Id: I3be1750aacf9736ece2944045c036f405e404561
2020-06-11 13:44:42 +00:00