48 Commits

Author SHA1 Message Date
Dmitry Ukov
cad8177fee Make API compatible with k8s apimachinary
Each object must implement runtime.Object interface. Therefore we
need DeepCopy methods implemented for each API struct.
Implementation is generated by controller-get utility.

Relates-To: #259
Change-Id: I535bf9ec3ff61597a778f73ca449313b7455b989
2020-06-23 13:01:16 +04:00
Zuul
f1471d1189 Merge "Zuul Job with Bash oriented gate tests" 2020-06-22 18:36:31 +00:00
Sreejith Punnapuzha
c64ad2db4f Update kubectl version
This commit updates kubectl version from 1.16.2 to 1.17.4 as part of
gate setup

Change-Id: I88728be65474ee7ce57f243e1356b84c7609a45f
Signed-off-by: Sreejith Punnapuzha <sreejith.punnapuzha@outlook.com>
2020-06-19 14:31:31 -05:00
Sirajudeen
c37b27b8d0 Zuul Job with Bash oriented gate tests
* Added bash scripts in place of ansible playbooks/roles
* zuul will run bash scripts, and this means that they are actually tested
  which will make development experience easier

Change-Id: I60d1dac589ae8ad835d3f1dda5e2a45523367496
Relates-To: #222
2020-06-19 15:44:27 +00:00
Sreejith Punnapuzha
c3002cabda Add bash scripts to run airshipctl
* Add playbook to run the scripts
 * Add script to install kubectl
 * Add script to create airshipctl executable
 * Add script to generate test configuration
 * Add script to pull site documents
 * Add script to build ephemeral iso
 * Add script to deploy ephemeral node
 * Add script to deploy control plane

Closes: #223
Closes: #224
Closes: #225
Closes: #226
Closes: #227

Change-Id: Ied63e2a733c3abcc6b1a822624c5ffce92e05a44
Signed-off-by: Sreejith Punnapuzha <sreejith.punnapuzha@outlook.com>
2020-06-19 15:44:06 +00:00
Yasin, Siraj (SY495P)
458fb2579b Fix gate scripts for local setup
* when remote_work_dir is set, document pull happens from the
  remote_work_dir (/tmp/airship) with a folder created within
  remote_work_dir with repo name (/tmp/airship/airshipctl).
  However the manifest yamls of kind "Clusterctl"
  refers to a relative path from work_dir and it fails
  finding expected objects.
   example: manifests/function/capm3/v0.3.1
* So trying to avoid document pull, and use current dir for workspace.
  similar to Zuul gate implementation

Change-Id: I63fd5476247f957745e15cbdfceb5fb483758e83
2020-06-12 03:00:32 +00:00
Dmitry Ukov
df2fff0acf Add control plane deployment
K8s control plane is deployed by KubeadmControlPlane controller. This
controller creates CAPI machines and infrastructure objects
(Metal3Machine). Metal3Machine objects are created based on a template
which contins host selector label. Control plane label is assigned to a
particular BareMetalHots object defined inside of the shared
kustomization.

Relates-To: #149
Closes: #221
Change-Id: I3be1750aacf9736ece2944045c036f405e404561
2020-06-11 13:44:42 +00:00
Matt McEuen
958d783919 Add site doc validation gate
This adds a gate which loops over all phases in all sites,
and performs an airshipctl apply --dry-run on them to ensure YAML
validity and schema adherence.  Aside from installation tasks,
the gate is run via a makefile entrypoint so that it can be
easily consumed by developers or by non-zuul CICD platforms.

Change-Id: Ie4ab246848a580ab20c3153af1e3749a27e3f770
2020-06-04 18:04:47 -05:00
Yasin, Siraj (SY495P)
90e27b542e Add check_copyright as part of gate script
* Added check_copyright to validate if all source files has
  expected copyright header

Change-Id: I8d9f3510db9cb5cbc510748a2f71ad95aa8d366b
2020-06-01 18:52:28 +00:00
Zuul
1bc62bb55f Merge "Add golint as a non-voting gate" 2020-06-01 16:48:25 +00:00
Ruslan Aliev
6577d3fc64 Install netaddr as dependency of ansible ipaddr/ipwrap filter
Currently there is no installation procedure for netaddr package
on ansible controller. The absence of this package causes
apache-wsgi-sushy-emulator or apache-file-exchanger tasks to fail.
There is need to manually install it according to ansible docs [1].

[1] https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters_ipaddr.html

Change-Id: I29fa7086ea637859655e0021a3fcbe4282fe09ae
Relates-To: #239
Closes: #239
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
2020-05-30 03:03:35 +00:00
Ian Howell
7243176e44 Add golint as a non-voting gate
Change-Id: I9850528a720ff517a3f400553fa8ab66bf4b1b3b
Closes: #253
2020-05-29 11:40:45 -05:00
Stanislav Egorov
fbd8c89180 Fixed local scripts
Added remote_work_dir for template to use the same config options
that we have for regular gate.

Without this change when we run local srcipts we have a conflict
when 'airshipctl cluster init' tried to use manifests from '/tmp/airship'
which is targetPath from airconfig. But 'airshipctl document pull'
downloded manifests to '/tmp/airship/airshipctl'

Change-Id: I1eaa2f7525471c756f79a0bc1dd566124f40d531
2020-05-28 15:30:44 -07:00
Dmitry Ukov
a301de202b Add CAPI initialization steps to gating
This change enables execute 'airshipctl cluster init' command
as a part of zuul pipeline

Co-Authored-By: Kostyantyn Kalynovskyi <kkalynovskyi@mirantis.com>
Change-Id: I9d499ae66f8d847759c2e61e5d4415f2c9f3c860
2020-05-20 15:17:20 +04:00
Ruslan Aliev
917d6896b6 Remove "cluster initinfra" command
Command "cluster initinfra" should be removed since
"phase apply" was introduced instead.

Change-Id: I263f8988a4456fc03314093a55ee1c1da8048404
Relates-To: #162
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
2020-05-16 01:45:34 -05:00
James Gu
d26df73b4b [#112] Add doc pull in deployment scripts
Added the optional use of doc pull in gate deployment scripts. When the
remote_work_dir is defined, the new pull doc role will invoke airshipctl
document pull command to download the manifests to the remote_work_dir.
This role is not included in zuul job so the zuul job will continue to use
the already checked out manifests in the zuul.project.src directory.

Fixes #112

Change-Id: I754c08661ad0b635819cb3730e05a6986d90f5ca
Signed-off-by: James Gu <james.gu@att.com>
2020-04-30 15:11:44 -07:00
Dmitry Ukov
25e7d92d48 Rename gating test site
* New name for gating site is 'test-site'
* Renamed 'test-bootstrap' type. New name is 'gating'.
* Moved BareMetalHost and its settings from ephemeral function to
  shared directory within the site

Change-Id: I510e74c1a553bb2b1f84e66f8e4a7796640b59cc
2020-04-29 17:32:40 +04:00
James Gu
afcfdb5c6e Proxy support enhancement
Added gate scripts and playbook enhancement to run airshipctl behind proxy.

Change-Id: Icc3cdcd6fd150420103f907cd1871feba1ccd223
Signed-off-by: James Gu <james.gu@att.com>
2020-04-28 18:04:55 +00:00
Zuul
585f885409 Merge "Add initinfra gating tests" 2020-04-23 18:28:27 +00:00
Sreejith Punnapuzha
53c0be6776 Add proxy support for pip3 package installation
This commit adds proxy support for pip installation. if environment
variable http_proxy is set, pip uses it for package installation else
it will continue without proxy

Change-Id: Iffa0f592853f3e05824263734410c64b5774ab33
Signed-off-by: Sreejith Punnapuzha <sreejith.punnapuzha@outlook.com>
2020-04-23 11:31:16 -05:00
Vamsi Savaram
e40b5ffec5 Add initinfra gating tests
Relates-To: #180

Change-Id: Ie7a8daa532596f2d1d0fb2059ef4630607511711
2020-04-23 14:41:45 +00:00
Alexey Odinokov
1505353333 Switching local gating scripts to Python3 to comply with Zuul
The initial version of gating playbook contained ability
to run them either on Python2 or Python3 ansible.
But it causes a lot of issues e.g. a need to install[1] both
versions of libs to support both possible ways to run,
discrepancy between local and Zuul run[2]
Zuul testing is done with ansible running on Python3 [3].
Meanwhile ppa repo for ansible contains ansible pachage
that depends only from python2. There is no package for
Python3 and we used only Python2-ansyble.
The fact that Zuul runs it with different configuration
means that we'll keep facing the issues with dependencies
in future, since Gating doesn't test it.
This patch changes makes local scripts run as Zuul.

PS
The script partially takes care of reverting intallation
from its previous runs. To cleanup completely it's possible
to run [4] in case these pacakges are not used by anyone else

[1]
https://opendev.org/airship/airshipctl/src/branch/master/roles/libvirt-install/tasks/main.yaml#L38
[2]
https://review.opendev.org/#/c/717841/
https://review.opendev.org/#/c/718590/
[3]
https://zuul.opendev.org/t/openstack/build/4096513604a84cd2b2f2578fc3f98660/log/zuul-info/host-info.primary.yaml#498
[4]
if dpkg -l | grep "ansible.*ppa~bionic"; then
      sudo apt -y remove ansible
      sudo apt -y autoremove
      sudo add-apt-repository -r -y ppa:ansible/ansible
      sudo apt-get -y update
fi
+ possibly
sudo apt remove virtinst python-libvirt python-docker python-requests python-lxml python-libvirt

Change-Id: Ia1ac3aa4128f6c4797dea1709ccb8516ae342a66
2020-04-11 06:53:40 +00:00
Zuul
788dc35b0a Merge "Add copyright for missing files" 2020-04-10 17:51:51 +00:00
Yasin, Siraj (SY495P)
c25d223c7b Add copyright for missing files
* added license templates for go, bash & yaml in tools dir
* added a script that will add license information for all
    missing files. Type:  go, yaml, yml, sh
* skip adding license for all files within testdata
* Syntax:
   > ./tools/add_license.sh

* Skip license for manifests folder
* Added one extra line after licene for yaml files
* Added License after Hashbang for bash.
* Add an extra line after hashbang and before license
* Updated the go template to use multiline comments

New Files:
  1. tools/add_license.sh
  2. tools/license_go.txt
  3. tools/license_yaml.txt
  4. tools/license_bash.txt

Change-Id: Ia4da5b261e7cd518d446896b72c810421877472a
Realtes-To:#147
2020-04-09 08:35:59 -05:00
Stanislav Egorov
9e0b8b2a96 Fixed local gate scripts
Change-Id: I3fc0c597f1b4f050b40905b5415069c5d0b8101f
2020-04-08 18:31:30 -07:00
Stanislav Egorov
2933dfdc51 Moved last role to airshipctl
All roles should be developed in airshipctl roles dir

Change-Id: Ide949c95889746595d41c4db913619f7140daf35
2020-03-25 13:16:38 -07:00
Zuul
891995a646 Merge "tools: Use local connection for ansible runner" 2020-03-11 23:02:19 +00:00
Ian H. Pittwood
388dcc0517 Target test files only in git diff check
Change-Id: I758636bc1669c1618b41db9e672664fedfc7b65a
Co-Authored-By: Francis Yi <francisyi19@gmail.com>
Co-Authored-By: Ian Pittwood <pittwoodian@gmail.com>
2020-03-10 15:55:22 -04:00
Alexander Hughes
abb28716c8 [#81] makefile to check for unused test data
This change adds logic to the docker-image-unit-tests to ensure that
code is checked within the container for coverage, valid test data,
and reduces the number of containers created in the gate - reducing
the runtime of each gate.

Relates-To: #81

Change-Id: Iab3d9bd74266d32404e6eb06ecebc234df04fd5c
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
2020-03-08 00:05:48 +00:00
Zuul
cadd8ad608 Merge "Uplift linter" 2020-03-06 20:25:54 +00:00
Drew Walters
2a91ed613c tools: Use local connection for ansible runner
The scripts in the tools/gate directory do not run ansible with a local
conenction; the ansible playbooks attempt to SSH to a VM on localhost
and fail. This change adds the local connection flag in order to run
directly on the host.

Change-Id: I3d7c7b6d21fc4972bd8a084ea49c945af4bfc518
Signed-off-by: Drew Walters <andrew.walters@att.com>
2020-03-06 19:42:56 +00:00
Ian Howell
b669efb95b Uplift linter
The golangci-lint linter runs into issue [0] when linting airshipctl
with go1.14. This uplifts the linter to a version in which the issue is
fixed.

[0] https://github.com/golangci/golangci-lint/issues/893

Change-Id: I50d3c4ab2533c1d14d01effed21906f0ef79aafc
2020-03-04 09:19:19 -06:00
Stanislav Egorov
d0e1c0258f [#32]: scripts for local run playbooks
Change-Id: Ia3f0e1461149f4ff887f965dd5b5a00fe2f8f04b
2020-02-28 17:24:32 -08:00
Ian Howell
091fa09a23 [#50] Clean up temp files from unit tests
This commit adds the utility testing function TempDir, which provides a
tester with a temporary directory as well as a means of cleaning up that
directory. The new function is implemented everywhere that makes sense
throughout the code base.

This also cleans up the directories left behind by go-git's testing
fixtures.

Some light refactoring was also performed in this change.

Change-Id: I754484934660487140f57671bacb5463cf669e3e
2020-02-20 11:48:11 -06:00
Ian Howell
1b8a965105 Add a simple whitespace linter
The goimports tool does not check for trailing whitespaces in multiline
strings. This commit adds a simple linter to address this deficiency.

This change will also have the benefit of providing light linting of any
non-go files.

Change-Id: I253dd7c4895f476fa3a6f0c4fccf7c9cada16366
2020-01-15 10:57:44 -06:00
Ian Howell
7ecae507e0 Add make tidy to check for up-to-date go.mod
This commit adds the `tidy` target. This target will fail if the go.mod
or go.sum files are not up to date with the code. It will also be a
prerequisite to the `lint` command, implying that it will also be caught
by the Zuul gates.

Change-Id: Ie33de2d793beca100435670346eb0e2ea7b3b0dd
2019-11-15 09:41:41 -06:00
Ian Howell
5b25f0bd00 This removes golangci-lint as a go module dependency.
golangci-lint is GPL-licensed, and its presence in go.mod might lead
people to believe that airshipctl links against a GPL-licensed
dependency. In reality airshipctl does not link against golangci-lint
(thanks to the // +build tools comment), but this change helps to avoid
that misconception from being made.

Change-Id: I95c223a7f6597e1aa07a025ed2cebbbd6bd7cbd8
2019-10-30 09:26:42 -05:00
Ian Howell
dc9c78b210 Fix the coverage tests
This fixes a bug where `make cover` was missing packages. The target now
covers all code except for code placed in the top level package (such as
main.go) and anything placed in the testutils directory.

This also fixes minor issues with the Dockerfile and the coverage_check
script

Note that this commit also strives to increase code coverage beyond the
80% margin

Change-Id: I9e1cbcf841cc869345a00f05e39774cb3da10065
2019-10-02 15:18:56 -05:00
Ian Howell
1c999e2095 Add coverage checks
This commit adds a makefile target for generating a report for unit test
coverage. It also adds the coverage_check tool to assert that the actual
test coverage meets the specified requirement.

This also improves various aspects of the testing utilities:
* The "test" package has been renamed to "testutil"
* The "Objs" member has been removed from the CmdTest object
* The "Cmd" member has been added to the CmdTest object. This allows
  testing of multiple variants of a root airshipctl command

Finally, this commit includes additional tests for root.go. These are
required in order to meet the required coverage threshold.

Change-Id: Id48343166c0488c543a405ec3143e4a75355ba43
2019-08-15 11:11:51 -05:00
Ian Howell
3cda375947 Remove dependency on k8s-code-generator
The deletions from the go.mod in this change were a relic of an earlier
change, where the k8s-code-generator was being used to generated client
code for Argo workflows. Now that Argo has been properly integrated into
the project, this is no longer needed.

These deletions were caught by a `go mod tidy`

Change-Id: I3e9f115dc31ef30dee59ddeeee513282bc555fcf
2019-07-01 12:57:29 -05:00
Ian Howell
e4b45002c9 Enable the linter, and fix uncovered linting issues
This uses the current master commit of golangci-lint. We will want to
update to use a tagged version as soon as the golangci-lint team pushes
a new tag
2019-05-30 10:02:54 -05:00
Ian Howell
615fe6c15f Add the first test for workflow init 2019-05-28 16:47:51 -05:00
Ian Howell
55a2dcb84e Add a more complex test to workflow list 2019-05-24 14:35:34 -05:00
Ian Howell
3ea070bcd5 Refrain from linting until golangci-lint issues are fixed
See the following issues:
https://github.com/golangci/golangci-lint/issues/538
https://github.com/golangci/golangci-lint/issues/539
2019-05-21 12:31:16 -05:00
Ian Howell
2624fda7f9 Remove old dependency on go-bindata 2019-05-21 12:02:39 -05:00
Ian Howell
34d0ff7e4d Remove dead code related to old plugin method 2019-05-17 14:37:30 -05:00
Ian Howell
f76eb8e027 Add a bin packer 2019-05-15 16:10:50 -05:00
Ian Howell
1d624fb05f Add a linter 2019-05-01 12:26:51 -05:00