114 Commits

Author SHA1 Message Date
Sean Eagan
f5f4f31063 Trigger Github mirroring on tag push
Currently tags aren't mirrored until the next patchset is merged,
which delays the Github release workflow.

Change-Id: Ib50b6369d4f8ddeaf832f1de2c8f6b133b9b40c5
Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
2021-04-01 13:46:53 -05:00
Ruslan Aliev
fa6b217297 Bump timeout for airshipctl-gate-script-runner job
Currently, about 15% of the last jobs were failed [1]
due to exceeding timeout error, however in most cases the job
was running just fine, but it was interrupted due to global
timeout. This patch increases timeout for this job by 20 minutes.

[1] https://zuul.opendev.org/t/openstack/builds?job_name=airship-airshipctl-gate-script-runner
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>

Change-Id: I5d8058aa5edd56052d51b67fbf23c0665ee1e6a9
2021-03-25 19:46:34 -05:00
Kostiantyn Kalynovskyi
3fbf865048 Fix CAPD deployment
Commit fixes CAPD deployment and removes redundant scripts
that check expiration for CAPD site.

They must be tested separately outside CAPD pipeline

Related-To: #482
Closes: #482
Change-Id: I60ffd76a4f3f08bd7bd198a0c2b15483dfbdd6a6
2021-03-19 11:31:21 +00:00
Zuul
4c1d1f87c0 Merge "Increase site doc validation job timeout" 2021-02-22 21:27:38 +00:00
Alexey Odinokov
b51e7559b6 Adding encryption of k8s secrets and iso users passwords
This patchset introduces a generated with template [1] and encrypted
VariableCatalogue generated-secrets that contains steps to
generate: ephemeral and target CA+admin key/cert and passwords for
users in ephemeral bootstrap iso.

It also introduces the way how these secrets are used in manifests:
They're decrypted by kustomize and incorporated into the folders
`catalogues` in the site, so they can be used by replacement plugin.

This patchset contains modifications in replacement plugin
configurations to put the decrypted values from VariableCatalogue
in place.

Since k8s secrets were substituted with generated values
this patchset removes pre-generated k8s secrets.

[1]
manifests/type/gating/target/generator/secret-template.yaml

Change-Id: I0898c74012833f0e171d36bb8145acf358510b69
2021-02-12 04:07:36 +00:00
Kostiantyn Kalynovskyi
1055f5c82a Increase site doc validation job timeout
the job takes around 45-60 minutes to complete on most of the nodes
however some nodes are slower and it make take a little bit more.

As future enhancement we should pararelize validation and refactor
scripts to run two cluster validation at the same time.

Change-Id: I01ac2ed15727931126058c343dbbb826790e940d
2021-02-11 03:50:15 +00:00
Ruslan Aliev
54256dbc70 Repair validate-site job
This job doesn't work properly since a lot of changes in the
airshipctl logic were applied. All the issues were addressed.

Change-Id: Iec6fa7e6a3aa1ab46d496a8fd63822df1f8124cc
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Relates-To: #19
2021-02-08 23:54:57 -06:00
Zuul
5632c0b169 Merge "Revert "Disabling airship-airshipctl-gate-script-runner"" 2021-02-04 19:17:04 +00:00
Roman Gorshunov
a28c710528 Revert "Disabling airship-airshipctl-gate-script-runner"
This reverts commit 9bd01de3dad7304b4eaedf09dba081b92470d736.

Reason for revert: cloud instances we use for CI are more stable now.

Change-Id: Id14f84e2a5b40d537161f1b4ea35fc97e9b1467d
2021-02-04 12:27:11 +00:00
Zuul
08681115d5 Merge "Cleanup CAPI deployment script" 2021-02-02 16:38:40 +00:00
Alexey Odinokov
9bd01de3da Disabling airship-airshipctl-gate-script-runner
Change-Id: I98c102258f63813a43299048e618549ffaca3e74
2021-02-01 16:12:15 +00:00
Sirajudeen
00130a9b53 Cleanup CAPI deployment script
* Removed the duplicate script for CAPI ephemeral node deployment
* Updated the generic script to support all providers

Change-Id: Icc1bed5c1b62662109b43ec94ee2fdb5de6de09b
2021-01-28 19:24:20 +00:00
Ruslan Aliev
d632a5de5f Enable voting for golint job
Since golint job was fixed and it's essential to maintain
code standards this job should have voting status.

Change-Id: I623171ce89b3906eac30e11fa3d7ebeb2725372b
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
2021-01-20 21:47:35 -06:00
Sirajudeen
d9a7200698 Removing the non-voting jobs from dependency
* Removing the non-voting job from dependency. So that
  for any reason if the dependent non-voting job fails,
  the gate runner job will not be skipped.

Change-Id: If2f208d41a6df1ce7286654fe6e203d309042874
2021-01-18 21:04:24 +00:00
Zuul
42e7dea246 Merge "Fix container image publishing" 2021-01-08 20:00:09 +00:00
Zuul
6397810826 Merge "Integrate Sonobuoy Conformance Test Scripts" 2021-01-07 17:08:07 +00:00
Rishabh Kumar Jain
b0217a8ba1 Integrate Sonobuoy Conformance Test Scripts
Add jobs in experimental pipeline to do the following:

  - install Sonobuoy
  - run CNCF Conformace Tests
  - run CIS Benchmarks Tests

Conformance tests include:

  - CNCF Compliance: uses sonobuoy end-to-end (e2e) and systemd-logs
                     plugins

  - CIS Benchmarks: utilizes the kube-bench implementation
                    of the CIS security benchmarks plugin

Pipeline can be triggered by comment
   - "check experimental"
Change-Id: I7d08ae42512dc4c83e2f550c4809ce1f8ddccc7b

Change-Id: I2e6469f5b8e229828532ce5499498da639d23fe6
2021-01-06 16:28:06 -05:00
Sirisha Gopigiri
52151ef9fd Zuul gate to check rotate-sa-token and certificate expiry commands
This Patch Set uses docker zuul gate and
checks the certificate expiration
and certiticate rotate-sa-token commands

Change-Id: I76f902d4fcacdbfe168abd58ec707282e46a3f91
2021-01-06 18:16:45 +05:30
Anderson, Craig (ca846m)
2daacf5f2a image-builder integration for ISO builds
This introduces airshipctl integration with image-builder [0], which
replaces the existing isogen tool for ephemeral ISO generation.

The airshipctl isogen executor has been updated for building ephemeral
ISOs using the image-builder container. The ability for user-declared
filenames for cloud-init user data and network data was removed, since
the user's only interest is in supplying the relevant overrides, not in
transparent naming coordination with the image-builder container. A new
object is added to the document package to identify the document kind,
label, and key to retrieve data from since this is pattern we will
reuse elsewhere.

Progress flag removed as requsted. Progress is reported directly by the
image-builder container.

Isogen debug flag removed in favor of using log.DebugEnabled()

[0] https://review.opendev.org/#/c/730777/

Depends-On: https://review.opendev.org/c/airship/images/+/730777/
Change-Id: I545004feaf2116f8ffb29faf6f7f7f5fcfe24fff
2020-12-17 14:47:53 -08:00
Rishabh Kumar Jain
ae6ce6931b Integrate common infra provider zuul scripts
* add common scripts for providers capd/capg/capz/capo
    to do the following:
    - install kind (reuses  tools/document/get_kind.sh)
    - improve tools/document/start_kind.sh to use
      custom kind config file
    - created a script similar to 26_deploy_metal3_capi_ephemeral_node.sh,
      called tools/deployment/26_deploy_capi_ephemeral_node.sh and
      made it flexible for all providers (metal3/capd/capz/capg/capo)
    - deploy ephemeral control plane
    - initialize target node
    - move crds from ephemeral to target
    - deploy target workers

Closes: #413
Change-Id: If3520d000a03e5ed26d54e1ceedb2febfceccb3d
2020-12-11 16:45:07 -05:00
Sean Eagan
3e694a7349 Fix container image publishing
The existing `image_repo` value was leading to invalid image tag name
errors e.g. "https://quay.io/api/v1//airshipit/airshipctl:latest". This
updates the value to match the default DOCKER_REGISTRY value in the Makefile.

Change-Id: Ifa9112c8b91475593300ff04fd069dff703762af
Closes: #437
2020-12-09 11:15:56 -06:00
Zuul
a5c96cb430 Merge "Removing Azure Zuul Gate" 2020-12-03 23:18:39 +00:00
Dinesh Garg
396a160271 Allow proxy and container repository customization
Modify following roles to accept the proxy and env
variables from zuul jobs and project manifest so they
can be overwritten to run them on downstream zuul

- airshipctl-build-images
- airshipctl-publish-images

Change-Id: Ied06e60507f125c60ca7b9d47331e967fd69bf14
2020-11-18 15:29:41 +00:00
Sidney Shiba
2de2016f9a Removing Azure Zuul Gate
This patchset removes the Zuul gate for the Azure integration, which is
not relevant anymore.

Change-Id: I4e2751965f6b3fbc8c2b062f2acfa03c1575521a
2020-11-16 14:59:49 -06:00
Zuul
7a404c6fa6 Merge "Incorporate HWCC in deployment gate." 2020-11-11 12:08:22 +00:00
Roman Gorshunov
5b862d81c3 CI: run gate runner only on request and on merge
The airshipctl gate runner consumes 16GB RAM VM node. Currently we have
insufficient compute capacity to run it on every patch set change.

This patch set makes gate runner run only on request and on Zuul gate
(pre-merge) to reduce workload imposed onto community CI.

Request to run airshipctl gate runner is a comment
"check experimental"
left to a patch set in Gerrit.

Change-Id: I5cb2a80b5cdc9dd025249e313d29eab62cbb3446
2020-11-10 14:07:00 +01:00
Rajat Sharma
854e25355b Incorporate HWCC in deployment gate.
Added sample HCC profiles and deployment script to apply profiles.

Change-Id: If8152a0afda142c40a53ddf9a4bd6866b435b0e5
Co-Authored-By: Rajat Sharma <rajatcu143@gmail.com>
Relates-To-Issue: #326
Depends-On: #738410
2020-11-10 16:12:28 +05:30
Sirajudeen
a360ebf9da Check if commit is linked to github issue
* Non-Voting gate to check if commit is linked to github issue

Change-Id: I8a4ea350382f8f2ed58ce49d52f1386d9c0d9c92
Closes: #387
2020-11-06 18:45:34 +00:00
Sirajudeen
f259bc8a62 Fix the lint job name in azure gate runner
* Recently the lint and test jobs are separated
* update the reference to lint and test job names
  marked as dependency for azure gate runner

Change-Id: I9bf495143b14d248c54c20e4814ac236b167eba1
2020-11-06 18:44:25 +00:00
Zuul
a4d7410830 Merge "Separate lint job from the test" 2020-11-06 18:16:39 +00:00
Sirajudeen
c0f116a68c Separate lint job from the test
* Split lint and test jobs
* Add lint job to be run against all files

Change-Id: I74da7999324283984d61f38ae9fb394ee5b37ce6
Closes: #386
2020-11-05 22:35:32 +00:00
Alexander Hughes
2e92614239 Skip azure gate runner on irrelevant files
Change-Id: I693d78a2f400b45c6095c039c3a91e7c3d9fb4c8
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
2020-11-05 14:26:23 +00:00
Kostiantyn Kalynovskyi
68542a8c0a Enable gate runner job during gating
This would will make sure that issues like #382 would not arise
in the future.

Change-Id: I3f65c760fc2d893a2e93ae8d606b2282bb1008b3
Relates-To: #383
Closes: #383
2020-10-29 02:46:41 +00:00
Zuul
340bed7ac5 Merge "Use token for github Authentication" 2020-10-26 18:32:56 +00:00
Sirajudeen
c33f5f5104 Use token for github Authentication
* Move from user/passwd to token based auth for
  github api access
* Github password based authentication will be deprecated soon
  https://developer.github.com/changes/2020-02-14-deprecating-password-auth/

Change-Id: Ic5af5bcf6ceb9828f2df7cca97e73c713dc79c1e
Closes: #373
2020-10-26 13:44:27 +00:00
Andrii Ostapenko
81efbceea6
Collect description of k8s object to job logs
Change-Id: I15226b9700acf169c100431d5dc7fafe41fbcb87
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-10-23 11:58:29 -05:00
Matt McEuen
f164c97e6a Increase doc validation job timeout
This increases the timeout of the document validation job from
30min (default) to 60min.  This is required because of the recent
addition of provider-specific site definitions; since a kind cluster
is set up / torn down for each site definition, it can take longer
than a half hour when running on slower open infra VMs.

Change-Id: I23e7201365058d96ccdc07e310c3c921daf952a7
2020-10-21 09:29:46 -05:00
Kostiantyn Kalynovskyi
81de5ee252 Enable voting of gate tests
Opendev infra 16G flavors are much more stable now and we can
enable this test again.

Change-Id: Ie6b5a97030b89e3235814e9cae8bb238b02087ce
2020-10-13 16:32:03 -05:00
Sirajudeen
8307148482 Remove clusterctl install from gate script
* clusterctl commands are driven through phase run.
  So removing install clusterctl CLI utility from gate
  script

Change-Id: I1c57aa07a9e19495c94c3080d0ce40dfc0e5dd47
2020-10-13 15:11:59 +00:00
Zuul
ce3d3b3242 Merge "Make document validatation job non voting" 2020-10-09 14:48:24 +00:00
Zuul
c6a9082b2c Merge "Making gate runner job dependent of small jobs." 2020-10-05 22:37:29 +00:00
Sidney Shiba
d0683139fe Integration of Azure provider to Airship 2.0
This commit integrates the Azure provider to the Airship 2.0 project.

It adds the following folders:
- manifest/function/capz: This folder contains all manifests required
for the integration of Azure provider.
- manifest/function/k8scontrol-capz: This folder contains the base
manifest for the Azure Workload cluster.
- manifest/site/az-test-site: This folder contains the manifests used
for initializing the CAPI and CAPZ components on the management cluster
invoking "airshipctl cluster init" and manifests used for deploying a
Workload cluster on the Azure Cloud by invoking the command
"airshipctl phase apply azure".
- tools/deployment/azure: provides the script shell that are used in the
zuul gates and local test.

Updated files:
- zuul.d/project.yaml and zuul.d/jobs.yaml have been updated to include
gates for validating the Azure provider integration.

Change-Id: Icbdc7f6f42c159f48dd11e35626da3bc016f5487
2020-10-05 19:27:31 +00:00
Zuul
e884972839 Merge "Increase gate test timeout to 2 hours" 2020-10-05 15:01:52 +00:00
Sirajudeen
c643f7c3f4 Making gate runner job dependent of small jobs.
* The idea here is to execute all the applicable small jobs first.
* If all the applicable small jobs succeeds, then proceed with the
  long running gate runner job.

Change-Id: I807ad3bf725db35945e83feff0601b011c771d5a
2020-10-02 21:00:57 +00:00
Roman Gorshunov
1ad48c0983 Reduce Zuul CI pool workload
Added patterns of files which should and should not trigger jobs on Zuul CI,
to reduce workload imposed onto the CI infrastructure.

Change-Id: I7081c7a2a4902a29e571dce1133465f2821d71a3
2020-09-30 19:38:55 +02:00
Roman Gorshunov
92ffcb6e66 Reduce Zuul CI pool workload
Added patterns of files which should not trigger the most resource-consuming
job (on 16GB VM) on Zuul CI.

Change-Id: I143a15f531f5179cebf2741eacd67818976fda94
2020-09-29 15:09:40 +02:00
Ruslan Aliev
d8a6139e81 Add document pull step to gate script runner
Currently we don't test document pull step by rewriting
manifest_directory variable as current source path, which
makes a mess in target_path, manifest_path and related paths
(concatenation of target_path+manifest_path won't return
an existing and proper location). This patch organizes
mentioned variables and enables document pull step which
starts working properly. The latest repo state used as
git source for this command.

Change-Id: I5abce73877441c7529f2f77add79cf410e2226d8
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
2020-09-25 00:39:28 -05:00
Sean Eagan
eb40a5700f Add gating for helm release management
This adds gating for helm release management by including a minimal
example workload phase which consists of a HelmRelease for the nginx
ingress controller and a corresponding deployment script including
supporting validation logic.

Change-Id: Ia21a799030289c7e40a0e61292578987ea0f6c63
Relates-To: #351
2020-09-23 14:59:24 +00:00
Kostiantyn Kalynovskyi
f56f34d731 Increase gate test timeout to 2 hours
When the successful gate test runs for approximately 1:20, current
timeout does not allow jobs to pass if there is bad latency at the
DC in which vm was spawned. This will reduce the amount of rechecks
that patch sets owners have to perform.

Change-Id: Icd7410126c6bcdf01ff6763255023539e1950c45
2020-09-22 13:07:45 -05:00
Kostiantyn Kalynovskyi
7836e73773 Make document validatation job non voting
The job needs to be redesigned in general, and it doesn't belong
to airshipctl repository. Document validation should be performed
in treasure map repository.

Next step would be to move validation job to treasure map repo

Change-Id: I627eec5c5d7efddd22c6aaa672091210451e89ae
2020-09-22 10:18:37 -05:00