334 Commits

Author SHA1 Message Date
Zuul
8a3dd8a466 Merge "AIAP: Fix manifest caching" 2021-10-04 21:42:09 +00:00
Ruslan Aliev
1faeb29f43 Disable capd job from voting
* slightly increase timeout for gate script runner job
* disable capd job from voting due to instability
* fix validate_docs script:
  we should either filter site to validate from command line or
  from sites_to_skip variable
* enable document validation for docker test site

Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: If38e6e24f7187d59072bc7435c700b829e2dafba
Closes: #649
2021-10-01 13:37:44 -05:00
Ian Howell
6cd4d6ed17 AIAP: Fix manifest caching
This prevents the runner container from using the cached manifests from
a prior execution of AIAP (as this should only happen when a developer
specifically tells AIAP to).

Change-Id: I79afefd859776654e455265cf69e201039e8ce9e
2021-09-29 14:10:37 -05:00
Siraj Yasin
23ccec608c make SITES_TO_SKIP configurable from env
* Read SITES_TO_SKIP from env, so that treasuremap can
  set its own list of sites to be skipped in the environment
  variable

Change-Id: I14507b77cfcb348d310156fac22a1ec39dba0731
2021-09-29 16:23:03 +00:00
Zuul
f2363dafd1 Merge "Prevent 22_test_configs.sh from deleting too much" 2021-09-28 19:45:00 +00:00
Zuul
b1bbb820b7 Merge "Allow pulling from a private airshipctl repo" 2021-09-28 19:44:56 +00:00
Manoj Alva(ma257n)
faa13c51ad Support for MAC address when VM count > 1 in AIAP
Current AIAP implementation has support for multiple VMs for
target/worker with an attribute "count".
But this is not supported for corresponding new MAC addresses.
The current implementation has mac address fixed for one
target node/one worker node.
This PS provide change in ansible playlist to handle
MAC addresses as list.

Closes: #640
Relates-To: #640
Change-Id: I14b6e2e4d9e81a4f1cf114c26f94328a1461134f
2021-09-24 08:04:13 +00:00
Siraj Yasin
907070454f Allow pulling from a private airshipctl repo
* Added the logic to support pulling airshipctl from a
  private github repo with basic http authentication.

* Added Username, Password in secrets

Relates-To: #632
Relates-To: #636
Change-Id: If1a1bbfaf9ce543cf35377d367b70676f9e3e9ef
2021-09-23 18:40:17 +00:00
Zuul
75044b33b1 Merge "Adjust conformance scripts (capd)" 2021-09-23 16:48:11 +00:00
Zuul
eb9ac97ce2 Merge "AIAP - check if docker server is ready" 2021-09-22 01:18:25 +00:00
Zuul
e7a84be28d Merge "AIAP: Update host file location" 2021-09-22 01:18:22 +00:00
Rishabh Kumar Jain
6ff6f8120b Adjust conformance scripts (capd)
* adjust kubebench conformance scripts
      to use ~/.airship/kubeconfig
    * adjust sonobuoy conformance scripts
      to use ~/.airship/kubeconfig
    * update sonobuoy version to 0.51

Relates-To: #557
Change-Id: I5c300e6740f0d050c0619473934af6f4d3b48816
2021-09-20 20:35:26 +00:00
Siraj Yasin
76849d28ae AIAP - check if docker server is ready
* docker version returns both client and server version
  So even if server is not running, the client version will be returned
  and docker build will fail.
* So checking for containerd in docker version output. This will be true
  only if the server is also ready

Change-Id: Idd4355b4f8d3617a339df67ac9820e33311b7d79
2021-09-20 19:56:25 +00:00
Ian Howell
0a09ba491b AIAP: Fix artifact-setup bug
This fixes the artifact-setup container to use the new `signal_status`
script created in b77f9e2910f81e3716a7011baa5581239d523c74

Change-Id: Ia0958f9d1e7050d8d8f37b9d66f471fd35156e13
2021-09-20 14:04:10 -05:00
Ian Howell
f00f40f0bd AIAP: Update host file location
The location of the host config file was moved in
e2c56108eef38dd83df52fcfd1fa6844e5376a56, which resulted in breakages in
AIAP. This updates the file's location in the runner container to
circumvent the breakages.

Change-Id: I3c4b9c92cabcf7549e75da1c77b1775b4cdbc6bb
2021-09-20 12:21:23 -05:00
Zuul
6cdf28fb40 Merge "Clear bad defaults in AIAP/airshipctl replacements" 2021-09-20 16:44:45 +00:00
Ian Howell
5cfdfc823f Prevent 22_test_configs.sh from deleting too much
This states explicitly which files should be removed by
22_test_configs.sh as opposed to wiping out anything containing the
string "config". This prevents the script from deleting files it
shouldn't, e.g. config.bak

Change-Id: I1d7b3563c3b5088b496b79f6416fd1d50ce6a600
2021-09-20 16:33:12 +00:00
Zuul
dad459f5cb Merge "Nextgen secrets implementation with separation per cluster" 2021-09-18 06:24:57 +00:00
Ian Howell
b77f9e2910 AIAP: Cleanup completion statuses
This adds the `status-checker` container, which aggregates the statuses
of the task containers. This is useful for quickly checking whether AIAP
is in a failed or successful state, allowing for early exit during
gating or testing. This also prevents the containers from stopping for
any reason, allowing for easy debugging.

Change-Id: I1571d006fb3c856e4d2bedee0befdccae6082a66
2021-09-16 16:06:52 -05:00
Zuul
d58ddbdb05 Merge "Fix CAPD deployment using phase plan" 2021-09-16 20:05:28 +00:00
Zuul
4f42338019 Merge "Removed the hardcoded wait logic" 2021-09-15 23:28:17 +00:00
Zuul
ecccdd7641 Merge "Cleanup logs" 2021-09-15 18:00:30 +00:00
Ian Howell
bedca67420 Clear bad defaults in AIAP/airshipctl replacements
Change-Id: I17dd940448c1263f00e865239b6e69201e61d55b
2021-09-15 10:20:05 -05:00
Siraj Yasin
c7be426de3 Removed the hardcoded wait logic
* Removed the hardcoded wait condition ( 30s delay )
  for docker and libvirtd service
* Added logic to wait until docker is available for artifact-setup.
* Added logic to wait until libvritd is available for infra-builder.
* Added "imagePullPolicy: IfNotPresent" to make use of local changes.

Closes: #629
Change-Id: I8956cf5927be6142fcc0a3b1a112ea82511fd9bb
2021-09-15 15:12:53 +00:00
Zuul
d5690e20fe Merge "AIAP: Update docs" 2021-09-14 15:26:19 +00:00
Ian Howell
fd04e9cd00 AIAP: Update docs
This change aims to clarify the differentiation between deploying AIAP
to Azure and deploying to a native Linux environment.

Change-Id: I3e014589dc02cce1bac5ff32a69c29ef6fc5b9f3
2021-09-13 09:17:30 -05:00
Ian Howell
fc20c79e47 AIAP: Keep containers alive in successful runs
Currently the task containers will reach a conclusive state (success or
failure), and will promptly stop running. This creates the following
problems:
  1. In the case of failure, a user doesn't have the ability to exec into
     the container for debugging purposes.
  2. In the case a success, the readiness probe for each task container
     will begin to fail. This happens because readiness is defined by
     the existence of a file; if the container stops running, the file
     no longer exists.

This change provides a solution to the 2nd bullet, keeping the container
alive in case of success.

Change-Id: Iae432caada3d2a338493b1534da72957808185d2
2021-09-10 11:21:42 -05:00
Zuul
885187f37c Merge "AIAP: Fix filepath name for manifests" 2021-09-09 15:30:30 +00:00
Ian Howell
e241430712 AIAP: Update documentation
This updates the documentation for working with AIAP. This includes
changes to the README as well as adding all configurable values to the
examples (having their default values).

Change-Id: I7c1b92d6a324b0fbd07f27ec0607fc62765c4fa2
2021-09-09 00:10:17 +00:00
Ian Howell
b921364ece AIAP: Fix filepath name for manifests
AIAP was unable to modify certain manifest files when the manifest url
ended in `.git`. This fixes that by stripping off the file extension
from the target path directory.

Change-Id: I3915deeb109fb1df726b27e1dd1deb1c987e39c5
2021-09-08 16:38:29 -05:00
Ian Howell
cde9648718 AIAP: Cause AIAP to fail faster
Currently, when a container in AIAP fails, the container continues to
run in the backgroud. This causes the container to be marked as failed
so that the deployment will fail faster.

Closes: #626

Change-Id: I34de366390b6ddc0ef434f7e642e73797a790edf
2021-09-07 17:02:52 -05:00
Ian Howell
b32e9b669b AIAP: Utilize document pull
Closes: #623

Change-Id: I559c160eb33710c3e3105e5454d035a833ce4abf
2021-09-07 16:54:14 -05:00
Zuul
3a537cc859 Merge "AIAP: Export KUBECONFIG var to environment" 2021-09-07 21:44:54 +00:00
Zuul
7dab6429d6 Merge "AIAP: Cleanup kubeconfig after running aks script" 2021-09-07 19:53:54 +00:00
Zuul
5ccc79ab18 Merge "Configure AIAP pod anti-affinity" 2021-09-07 19:33:26 +00:00
Zuul
ccc5bbb5e4 Merge "AIAP: Add environment variables for auth settings" 2021-09-07 16:41:21 +00:00
Zuul
e24d828f73 Merge "Update 22_test_configs.sh to use new document pull" 2021-09-07 16:41:20 +00:00
Zuul
be518102df Merge "Add auth to template for 22_test_configs.sh" 2021-09-07 16:00:34 +00:00
Ratnopam Chakrabarti
441ef0e3da Fix CAPD deployment using phase plan
This PS introduces phase plan for CAPD provider
 * Adds phase to merge kubeconfig
 * Patches type/gating plan to use capd phase plan
 * Patches cluster_map and executor to work with "default" namespace
 * Fixes CAPD zuul failures

Relates-To: #564
Relates-To: #580
Relates-To: #587
Change-Id: I5007970c907bc87dccf6dd9fcb052afc1b5c13f7
2021-09-07 15:00:25 +00:00
Ian Howell
f6f2372a8b AIAP: Cleanup kubeconfig after running aks script
The `az aks get-credentials` command adds a context, user, and cluster
to the kubeconfig. This cleans that up when CLEANUP_GROUP is true.

Change-Id: I6563d278e4700363d4d33c2ab746ac18dd0e51dc
2021-09-04 19:04:38 +00:00
Alexey Odinokov
e2c56108ee Nextgen secrets implementation with separation per cluster
1. Extending templater with kyaml functions and creating combined catalogue
to be able to request/update the existing resources.
This is based on 'everything is transformer' concept introduced in kustomize 4.x
That includes gathering all secrets into 1 variable catalogue and
special mechanism to regenerate/merge with manual secrets.

2. Implementing 'catalogue per cluster' approach for secrets.

3. Rearranging secrets so it's possible to use:
pgp (each person may have his own key), age, Hachicorp Vault and etc
and the list of people who can decrypt documents is set in a special file.
Since in some cases there should be a separate list of people who can decrypt
data - this list is set for each cluster (ephemeral and target) separatelly.

Closes: #586
Change-Id: I038f84dd138d5ad4a35f4862c61ff2124c2fd530
2021-09-03 20:46:15 +00:00
Ian Howell
0cb8e0f5f4 AIAP: Parameterize base image
This parameterizes the base image location used for the libvirt,
artifact-setup, infra-builder, and runner images.

Closes: #628

Change-Id: Idca33b5571017700fead0eb758af1900b4a4fcf4
2021-09-03 12:32:38 -05:00
Zuul
362cc4b880 Merge "Add a script to run AIAP in AKS" 2021-09-03 14:35:52 +00:00
Ian Howell
4dffb330ec AIAP: Add environment variables for auth settings
This adds the required auth variables to the runner environment so that
a user can configure passwords/keys needed to pull manifests from a
private repository.

Closes: #627

Change-Id: I185a86c0c82be38a0d41bb69f425a8fac8b5b913
2021-09-03 09:20:29 -05:00
Ian Howell
1d09fa090c Add auth to template for 22_test_configs.sh
This adds the following vairables to be used with the 22_test_configs.sh
script, so that it can generate configs which have access to private
repos.

* AIRSHIP_CONFIG_MANIFEST_REPO_AUTH_TYPE
* AIRSHIP_CONFIG_MANIFEST_REPO_AUTH_KEY_PASSWORD
* AIRSHIP_CONFIG_MANIFEST_REPO_AUTH_KEY_PATH
* AIRSHIP_CONFIG_MANIFEST_REPO_AUTH_HTTP_PASSWORD
* AIRSHIP_CONFIG_MANIFEST_REPO_AUTH_SSH_PASSWORD
* AIRSHIP_CONFIG_MANIFEST_REPO_AUTH_USERNAME

Relates-To: #627

Change-Id: I791e45eb6747e76f0e69a7cdcfde90db570fa1d1
2021-09-03 09:16:36 -05:00
Matt McEuen
a67286151f Configure AIAP pod anti-affinity
This adds an anti-affinity rule to airship-in-a-pod, so that only
a single AIAP can run on a given node at a time.  This is currently
required because of the way AIAP configures host mounts; future work
will allow multiple AIAPs to co-exist on a node, at which point this
anti-affinity rule can be taken out.

Change-Id: Ib8f5062a3bd2b4ea6d0a95c42cce08ac9789c362
2021-09-02 19:59:16 -05:00
Matt McEuen
47de62a11c Add a script to run AIAP in AKS
This adds a script that a developer can use to run airship in a pod
in a dynamically created AKS cluster.  By default a resource group is
created and then cleaned up following the test.

Further work needs to be done to surface some indication of a failed
test out of the pod, and watch for that condition in this script.

Change-Id: I29f462b9becd6d1de0a9e0e1d7877c6019cbd6ee
2021-09-02 18:11:10 -05:00
Ian Howell
e8d8b1e135 AIAP: Remove tty setting from libvirt container
The `tty` setting may be required in some instances dependent upon the
environment in which AIAP is deployed. However, these instances are much
less frequent than successful deployments when the setting is set to
false. Further investigation will be needed to determine when it is
appropriate to use this flag.

Change-Id: Ibbd44cbf0697a71f8081cd170dbdf5f7b4e4f106
2021-09-02 13:28:59 -05:00
Ian Howell
95a43b48ae Update 22_test_configs.sh to use new document pull
This causes the 22_test_configs.sh to create an airship config file
which supports the new document pull feature introduced in
cb080a2066

Change-Id: I057bd2420f488a9f91ed69d09f73b1336827e7a2

Relates-To: #623
Change-Id: I7bbf052877e4918e168fc4005ea899aa700bae41
2021-09-02 09:09:12 -05:00
Zuul
bd3b7f8436 Merge "Add proxy variable to AIAP" 2021-09-01 07:39:57 +00:00