29 Commits

Author SHA1 Message Date
Ruslan Aliev
ca71de3951 Move document filesystem to a separate package
In order to widely use filesystem interface in airshipctl
there is need to move it to separate package to avoid
importing unnecessary dependencies from document
package and, as a consequence, possible cyclic dependencies.
 * filesystem moved from document/fs to pkg/fs

Change-Id: I3b6298462f03db43594a9fa26bf23ab7687c5589
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Relates-To: #415
2020-11-30 19:51:30 -06:00
Dmitry Ukov
9c69138990 Remove document plugin subcommand
This change finalises shift to new version of kustomize plugins

Change-Id: Ie47f42a9e2c4dc03fc4dca437020ad71a564d894
2020-11-11 14:59:14 +04:00
Ruslan Aliev
0c870e5244 Remove clustertype-related functions
We don't use clustertype anymore, so all the related functions must
be deleted or adjusted as well as appropriate config fields.

Change-Id: I3931fdc71d4318e916f8bbc2d94e062c9df5f641
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Relates-To: #349
2020-10-16 11:31:58 -05:00
Kostiantyn Kalynovskyi
e6e56ffa2e Use BundleFactory instead of bundle in executors
This will allow to avoid using nil bundle and performing
additional checks inside the executors

Change-Id: Ia061cccf7e0a7948bbc5c8fce24e0e6f8b3256e1
2020-09-28 19:39:05 -05:00
Vladislav Kuzmin
d7101bcb2c Refactor environment module
Move code from environment module to util and document
modules to prevent circular dependencies.

Relates-To: #264

Change-Id: Ifec9ab3f1ada01cc662e655ff4a6f2cfebe4150a
2020-07-31 14:03:39 +04:00
Kostiantyn Kalynovskyi
edf09b0fd5 Add Append method to bundle interface
This method would allow us to append bundle with new documents

Change-Id: I201a2a0f2e7f707556a53495538c0f075ce6fe17
Relates-To: #238
2020-07-07 14:45:26 +00:00
Matt McEuen
4e64e74b6f Add a setting for kustomize plugins
This adds an env var-driven configuration for the filesystem path
in which kustomize should expect its plugins (including airshipctl)
to be.  The value defaults to a subfolder off the `.airshipit` folder.

The config is persisted as a singleton rather than a member of
AirshipCTLSettings (or the Config) because
1. the settings object would have had to have been passed around
   a couple dozen additional files/functions,
2. it's reasonable to expect the plugin location to be consistent
   across threads in a multi-threaded, airshipctl-as-library context.

Settings_test.go was moved in to an environment_test package to avoid
a circular import dependency.

Change-Id: Icdd21bd3687ef42492e388af982d7b490af3eff3
2020-05-15 15:06:32 -05:00
Dmitry Ukov
39ee048451 Introduce document plugin subcommand
airship document plugin is intended to be executed as an exec plugin
for kustomize document model.
Environment variable is used to gather plugin configuration. Plugin to
execute is determined based on group-version-kind specified in plugin
configuration. Each airship plugin must implement plugin interface.

Relates-To: #173
Change-Id: I4f6c3b5be140c0d8fd7519f1cedd33de1cef662c
2020-04-21 15:00:22 +04:00
Matt McEuen
e52ed80363 Update Kustomize integration to api/v0.3.1
This updates the Kustomize dependency for airshipctl to
api/v0.3.1, which is the oldest version which will support the
Replacement Transformer plugin.

Some changes were needed to accomodate the fact that various apis
that airshipctl was relying on were moved to kustomize-insternal
packages, namely:
- Integrated with the krusty.Kustomizer to drive kustomization
- Removed the custom plugin loader which leveraged the Unknown type
- Worked around NoFieldError becoming private, inc. removing a test

As a follow on we'll need to re-integrate plugin functionality somehow.

Also, in this release Kustomize has implemented support for the
"config.kubernetes.io/local-config" annotation, which we'd planned
to use to to avoid deploying some documents to the Kubernetes API.
It turns out the semantics are different than we anticipated;
Kustomize also fails to return these docs via document *selection*.
Therefore, this change reverts to an earlier approach which uses
a custom airshipit.org/deploy-k8s label.

Change-Id: I7022e12464ea7b6a3ca8609f99f3699bf8da0edd
2020-04-13 10:06:41 -05:00
Zuul
121adcddc8 Merge "Fixed TODO about error definitions" 2020-04-13 13:34:51 +00:00
Yasin, Siraj (SY495P)
c25d223c7b Add copyright for missing files
* added license templates for go, bash & yaml in tools dir
* added a script that will add license information for all
    missing files. Type:  go, yaml, yml, sh
* skip adding license for all files within testdata
* Syntax:
   > ./tools/add_license.sh

* Skip license for manifests folder
* Added one extra line after licene for yaml files
* Added License after Hashbang for bash.
* Add an extra line after hashbang and before license
* Updated the go template to use multiline comments

New Files:
  1. tools/add_license.sh
  2. tools/license_go.txt
  3. tools/license_yaml.txt
  4. tools/license_bash.txt

Change-Id: Ia4da5b261e7cd518d446896b72c810421877472a
Realtes-To:#147
2020-04-09 08:35:59 -05:00
Stanislav Egorov
f7a2e33b8b Fixed TODO about error definitions
Use SelectOne to replace Select and additional checks.

Change-Id: I40b4780d62fa4b3a0e01cc54ef7dbe4b32a4763d
2020-04-06 11:23:42 -07:00
Alan Meadows
e2cca32748 Add Redfish Authentication Support
This commit also introduces a dochelper concept.  This provides
some convenience methods to the document pkg that help extract data
from known document types as well as walk document relationships to
discover related information such as BMC credentials for baremetal
hosts.

Once merged, a follow up patchset will leverage these within the
cloud-init code to deduplicate some of these lookups.

Change-Id: Ie6a770ce4b34adbea30281917f0cb2fdc460b4fb
2020-03-27 15:29:07 -07:00
Kostiantyn Kalynovskyi
7a2d01789e Add ordering of the documents
By default kustomize cli orders k8s resources, in a way, that they
can easily by applied to cluster, namespaces and crds first. In this
patch set we implementing same behavior

Relates-To: #131
Closes: #131

Change-Id: I4fc75366627ed361ac1da48e89a35949bcb79801
2020-03-26 09:54:42 -05:00
Dmitry Ukov
09ef460e31 Add filtering method for document bundle
SelectByFieldValue method returns new bundle with documents which
values referenced by JSON path comply with function passed as input
argument to the method

Related: #14
Change-Id: I31bb6beee96b0d65719b6976caac63e54c65dcd2
2020-03-24 23:23:58 +04:00
Stanislav Egorov
3eacfc6c71 [#102]: removed methods for bundle interface
Removed two unused methods from bundle interface.

Two other methods are used internally and there is no need to keep it in the interface.

Change-Id: I5d7999e4b2c83dd43025442e5549ac7826009a87
2020-03-16 15:08:19 -07:00
Kostiantyn Kalynovskyi
147b97048b Split document model, add entrypoints for repos
Add NewBundleByPath function, that would return bundle built from
the specified path argument

Add CurrentContextEntryPoint method of the config
object, that would allow easily get kustomize root path based on
clusterType and phase. You can also leave phase arg empty string,
which would try to return bundle for all phases

Introduce changes to config pakage objects:

- Manifest:
  SubPath: this is relative path to the root of the repository that
contains directories with sites (SiteNames)
    PrimaryRepositoryName: this is a string that must correspond to a key
of the Repositories map of manifest object, which is used to derive
primary repository
    Repositories object is a map, map keys correspond to names of the
directories where `document pull` command will download repositories
defined in manifest prepended by manifest.TargetPath.

Introduce new config method CurrentContextEntryPoint(), method takes
TargetPath, PrimaryRepo.URL, SubPath, and clusterType and phase
constructs a path to the entry point out of which the DocumentBundle
should be build, and returns it to the caller. After that caller can
build a bundle out of it, the bundle will contain documents relevant to
particular cluster-type and phase.

All objects that depend on bundle interface are updated to use the
CurrentContextEntryPoint() method of the config object

Relates-To: #99

Closes: #99

Change-Id: I99320c4cb626841d46f4c298b583e9af90b1dce4
2020-03-12 09:55:05 -05:00
Alan Meadows
28db50e6d6 [#45] iso generation pulls network data from ephemeral host
This commit updates the iso generation process to pull the
network data from the ephemeral host network data from the
baremetalhost spec of the host with the right label, namely:

airshipit.org/ephemeral-node=true

It will back into the secret name and namespace specified
for the given host and extract the network data from this
secret which should be identical to the network configuration
the host would receive during normal provisioning.

It also pulls the user-data for the iso generation process
which is specific to the iso generation process from a secret
with a similar special label:

airshipit.org/ephemeral-user-data=true

Change-Id: Iae6edeb231d9dbae0b316aa73009f145d57ac316
2020-03-11 19:07:14 +04:00
Kostiantyn Kalynovskyi
76a280ead0 Add Kustomize document filesystem layer
Add document filesystem object to avoid direct kustomize imports by
other packages in airshipctl, document.filesystem also extends
kustomize filesystem with temporary file methods.

Relates-To: #11
Closes: #11

Change-Id: Ia8034048d80d79d5996dce0e283828644fbef906
2020-03-04 20:04:35 +00:00
Zuul
7f06e556a4 Merge "[#11] Implement selector kustomize primitive" 2020-03-04 19:24:58 +00:00
Kostiantyn Kalynovskyi
746141f6ac [#11] Implement selector kustomize primitive
Introduce Selector object that would serve as adaptor to kustomize,
this provides a layer between kustomize in a way that there is no
more need to import types package from kustomize by other modules.

Change-Id: Ib9e892e8b71d1808e427dd5240e4f313cc366225
2020-03-03 22:31:26 +00:00
Alexander Hughes
2bf3117197 [#58] Update types for golint
This patch addresses guidance from effective go [0] and golint
failures such as:

pkg/document/document.go:8:6: type name will be used as
document.DocumentFactory by other packages, and that stutters; consider
calling this Factory
pkg/remote/errors.go:9:6: type name will be used as
remote.RemoteDirectError by other packages, and that stutters; consider
calling this DirectError
pkg/remote/remote_direct.go:27:6: type name will be used as
remote.RemoteDirectClient by other packages, and that stutters; consider
calling this DirectClient
pkg/remote/redfish/errors.go:10:6: type name will be used as
redfish.RedfishClientError by other packages, and that stutters;
consider calling this ClientError
pkg/remote/redfish/redfish.go:14:6: type name will be used as
redfish.RedfishRemoteDirect by other packages, and that stutters;
consider calling this RemoteDirect

[0] https://golang.org/doc/effective_go.html#package-names

Relates-To: #58

Change-Id: I6d94da7755c8719bbcc4a77917e283074281309a
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
2020-03-03 01:41:43 +00:00
Alexander Hughes
f66f94dc80 [#58] Update error strings to be golint compliant
Error strings by convention should not end in punctuation, or contain
uppercase letters.

Relates-To: #58

Change-Id: I027fb21a20f08fdd24cf654f3ae1cbceb3e5a6c6
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
2020-02-28 16:31:53 -05:00
Ian H. Pittwood
c7c1011a5c Fix various code style issues
Fixes possible name collisions between variable names and package names

Remove redundant import naming

Use nil slices for slice declarations instead of empty slices

Use make for slices of fixed lengths

Remove redundant parentheses

Replace deprecated `SetOutput` method with `SetOut`

Fix swapped actual/expected arguments on assertEqualGolden

Change-Id: Ia39ef44372c3e44948e5440575125bdb470898df
2020-02-07 09:28:18 -06:00
Dmitry Ukov
b1361e05f8 Implement document plugin loader
Loader associated with Unknown type of a transformer kustomize plugin
and considered 'builtin'. Kustomize plugin system executes Config and
Transform method of builtin plugins. Therefore appropriate methods
of the document plugin loaders are executed as well. Main goal for
airship document plugin loaded is to determine desired aitship plugin
based on Kind field and execute its Config or Transform methods

Change-Id: Ic26a880570491ac3a59f2357ed455a2a7362387b
2020-01-21 10:25:57 +00:00
Dmitry Ukov
34cca34796 Fix NewBundle error return
Defer is executed at the very end of a functions. If we have named
return paramteters Cleanp() function will overwrite function result.
So if 'err' is not 'nil' Clenup() will overwrite 'err' with 'nil'

Change-Id: I97bbce53be26281515a287b513a3727aa199260d
2020-01-21 14:23:31 +04:00
Ian Howell
49027f4151 Tighten the restrictions of the linter
This change causes the linter to be a bit more complainy. The hope is
that this will cut down on some of the more pedantic issues being caught
in code reviews, and thus reduce the overall time a change spends in the
review process.

This change includes various changes to the codebase to bring it up to
the new standards.

Change-Id: I570d304bca5554404354f972d8a2743279a0171b
2020-01-10 14:54:34 -06:00
Kostiantyn Kalynovskyi
f8f6f8be27 [AIR-97] Moving WriteOut function from bundle pkg
This commit moves Write function to separate yaml package, and makes
it available to anyone for dumping documents in single file, with
correct yaml separators '---', '...'.

This will allow, for example dumping of filtered from bundle documents
to disk or any writer, like Stdout, for debugging purposes. Specifically
it will be used as part of `airshipctl cluster initinfra` command as a
buffer for delivering resources to kubernetes cluster.

Change-Id: I780e3f2d2ff446b8787153f500d04d10487ed71b
2019-10-11 17:13:59 -05:00
Alan Meadows
7eeb529053 Initial commit for document pkg library
This introduces two new interfaces to airshipctl:

* Bundle: represents a Kustomize rendered
collection of your documents that allows you to
iterate over or filter the documents in the bundle.

* Document: a thin wrapper around a rendered
document.

This commit also includes a proposal for a way to
represent YAML test fixtures as testdatafs.go
and a new testdata subfolder within the document
pkg.

Change-Id: I4282b27977617552c25c5ae2c3b7da44e67dd014
2019-10-04 14:22:26 -07:00