130 Commits

Author SHA1 Message Date
Zuul
d6ddb502c3 Merge "Isolate document pull unit test" 2020-05-21 03:23:56 +00:00
Yasin, Siraj (SY495P)
fffc42ad8a Enable golint as part of golangci-lint.
Issues raised after enabling golint as part of golangci-lint

  * error var ToDiscoveryError should have name of the
	form ErrFoo (golint)
  * error strings should not be capitalized or end with punctuation
	or a newline (golint)

Issues not raised by golint embedded within golangci-lint

  * comment on exported function <funcname> should be
      of the form "<funcname> ..."

So kept the golint standalone tool also in Makefile

Change-Id: I7f2ce66d1e757bc24c042fc212226cecea297f27
2020-05-20 13:29:11 -05:00
Ian Howell
f5846161c2 Isolate document pull unit test
This causes the document pull unit tests to use a git repo stored
locally on the machine, rather than reaching out to github

Change-Id: I0add3cdf60d7fe584c0194148e539a7a290f2bc4
2020-05-19 16:43:23 -05:00
Yasin, Siraj (SY495P)
8d16fffb7c Fix Lint warnings: Missing comments for functions
* Added comments wherever missing for exported functions/constants

Change-Id: I7d0b79b5f56fc6c3edb60a3b58c0962ebd76314c
Relates-To: #148
2020-05-19 16:54:09 +00:00
Zuul
7276dd68d8 Merge "RT: add error handling for missing target docs" 2020-05-18 15:54:18 +00:00
Zuul
fde808c068 Merge "Validate config before document pull" 2020-05-16 07:14:07 +00:00
Matt McEuen
4e64e74b6f Add a setting for kustomize plugins
This adds an env var-driven configuration for the filesystem path
in which kustomize should expect its plugins (including airshipctl)
to be.  The value defaults to a subfolder off the `.airshipit` folder.

The config is persisted as a singleton rather than a member of
AirshipCTLSettings (or the Config) because
1. the settings object would have had to have been passed around
   a couple dozen additional files/functions,
2. it's reasonable to expect the plugin location to be consistent
   across threads in a multi-threaded, airshipctl-as-library context.

Settings_test.go was moved in to an environment_test package to avoid
a circular import dependency.

Change-Id: Icdd21bd3687ef42492e388af982d7b490af3eff3
2020-05-15 15:06:32 -05:00
Yasin, Siraj (SY495P)
007e36281e Validate config before document pull
* invoke EnsureComplete before document pull to make sure
  all expected sections (cluster, context, authinfo, currentcontext)
  of config are present

Change-Id: If2ef4611cddc6ae96797dc26675551ef81ca8768
Closes: #215
2020-05-14 14:56:00 -05:00
Matt McEuen
2e7224a7bf RT: add error handling for missing target docs
The ReplacementTransformer already checks for the error condition where
a replacement source doesn't match any of the input documents.
This change adds similar error handling for missing replacement
target documents, and tweaks the naming/messages to make it clear
which situation is encountered.

This will make it easier for manifest engineers to catch typos
in their RT rules, or documents that have failed to be kustomized in
or authored.

Change-Id: I0efbcfbc2952afae05e48b9ff8e0b3c67df7e7df
2020-05-11 17:45:34 -05:00
Alexander Noskov
753d8a456c Fix pull documents for non-master branch
By specifying any other branch rather than master in airshipctl config,
`airshipctl document pull` command failing with `reference not found`
error.
The resulted target dir, after execution, still pointing to master.
The idea of this change is to use checkout options during the clone
stage, so we will be clonning the specific branch.
Added debug log about cloning default(master) branch in case of
missing checkout options.
Enabled validation for incompatible parameters in checkout options.
Aligned unit tests with code changes.

Closes: #197
Change-Id: I50ac18289a8f02997d5b90c82f1688083cff8bf3
2020-05-09 01:22:54 -05:00
Zuul
9d7c8e5c83 Merge "Add phase commandline option for interacting with phases" 2020-04-30 22:42:53 +00:00
Ruslan Aliev
0bbc88b4a9 Add phase commandline option for interacting with phases
Instead of having a lot different arbitrary named commands
such as "cluster initinfra", "cluster control-plane", add
a generic phase command that would provide capabilities
for interacting with phases (e.g. getting list or applying
specific one).

Change-Id: I1ab797121aecbfd2348933dfd993c1a5974edaf9
Relates-To: #162
2020-04-30 12:34:07 -05:00
Zuul
9be116e3c4 Merge "Add clusterctl resource type." 2020-04-30 10:55:39 +00:00
Zuul
383dfdf26b Merge "Add metadata support for clusterctl repository implementation" 2020-04-30 10:54:14 +00:00
Zuul
e6a6a655e1 Merge "Replacement transformer improvement" 2020-04-29 22:46:14 +00:00
Kostiantyn Kalynovskyi
aa3067cbb7 Add clusterctl resource type.
This type would allow us to control behavior of clusterctl modules
and choose components we want to install, if we put document of kind
Clusterctl into Bundle. In near future we need to generate deepcopy
methods for this object, so it can be used with kubernetes schema

Relates-To: #170

Change-Id: I2f40a7e9b66e0f7f0bbc8b6874ec45f767416c69
2020-04-29 14:39:30 -05:00
Dmitry Ukov
c458e8cde4 Add template based generator plugin
Plugin is responsible for resource generation based on template
(go template format is used) supplied to its configuration.
Go template can use an object (map) specified under 'spec'
configuration parameter.

Change-Id: I8c1b7dc5676bfff2f9b1a47f6433177ae796a15f
2020-04-29 23:38:58 +04:00
Kostiantyn Kalynovskyi
1969866b0a Add metadata support for clusterctl repository implementation
Change-Id: Ie9285cbba0fb590392b975be1eb2b02b28de56d1
2020-04-29 13:45:54 -05:00
Yasin, Siraj (SY495P)
07f8a5e311 Fix for Lint warnings
* Adding comments for missing exported functions

Change-Id: I40c5861a5c076b654e5a3fc660c3c9c6d7b80c8f
Relates-To: #148
2020-04-28 09:40:57 -05:00
Dmitry Ukov
47d8d8c886 Replacement transformer improvement
* Add ability to to replace whole list element filtered by key and
  value
* Raise an error in if user has specified wrong list filer in JSON path
* Enables support for a dot character in a value used in list filtering

Closes: #191
Closes: #192

Change-Id: I889550ee88d5738a1fb808ddef6478c3731b1561
2020-04-28 17:12:29 +04:00
Dmitry Ukov
c30930114a Replacement transformer refactoring
* Apply recommendations from "Effective Go"
* Simplify applySubstringPattern logic
* Reduce complexity of updateMapField
* Switch to typed errors
* Increased test coverage

Change-Id: I8e53a251a43c8f31c286284c77452fbf43ce4e43
2020-04-24 19:48:59 +04:00
Dmitry Ukov
6c716e1a57 Add replacement transformer
* Replace paramter specified by JSON path with predefined value or
  with the value from another resource
* Replace substring in a paramter specified by JSON path with predefined
  value or with the value from another resource

Transformer copied from
https://github.com/mattmceuen/kustomize/tree/substring-subst

Closes: #174
Change-Id: I3a958a0df724fb2eb81bb199a02cf1db81bb0d2f
Co-Authored-By: Matt McEuen <matt.mceuen@att.com>
2020-04-22 11:51:05 +04:00
Dmitry Ukov
39ee048451 Introduce document plugin subcommand
airship document plugin is intended to be executed as an exec plugin
for kustomize document model.
Environment variable is used to gather plugin configuration. Plugin to
execute is determined based on group-version-kind specified in plugin
configuration. Each airship plugin must implement plugin interface.

Relates-To: #173
Change-Id: I4f6c3b5be140c0d8fd7519f1cedd33de1cef662c
2020-04-21 15:00:22 +04:00
Zuul
1ec7a24b54 Merge "Uplift go-git" 2020-04-15 13:08:33 +00:00
Alexander Hughes
c70f4a2742 Refactor of codebase addressing IDE errors
Goland is complaining of a number of bad practices, and unused code.
This patchset addresses some of those complaints, such as:
- struct initialization without field names
- error string should not be capitalized or end with punctuation
- name starts with package name
- duplicate yaml keys
- snake case
- redundant parentheses

For the http.Transport redundant parentheses, the lint target fails
without the use of //nolint:errcheck so it has been added as
(currently) there is no potential for there to be an error returned
see [0] for example.

[0] https://golang.org/src/net/http/transport.go#L42

Change-Id: Ib1b20639c9b6e9be097ef1f158ecd7472f578488
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
2020-04-14 15:55:04 -04:00
Ian Howell
7c6de22354 Uplift go-git
The go-git module (and helper modules)  recently moved from
`gopkg.in/src-d/go-git` to `github.com/go-git/go-git` [0]. This commit
switches to the new module, as well as uplifts from v4 to v5.

[0] https://github.com/src-d/go-git/issues/1295#issuecomment-592965250

Change-Id: I2e73ba7bb1220a3c75c0d81aa5ab0f1f13479161
2020-04-14 09:07:28 -05:00
Zuul
8438935cfe Merge "Update Kustomize integration to api/v0.3.1" 2020-04-14 11:36:33 +00:00
Zuul
f520dc1190 Merge "Change Label and Annotate to accept map of string" 2020-04-13 19:07:47 +00:00
Zuul
e7380b3ba6 Merge "Remove getters/setters of AirshipCTLSettings" 2020-04-13 16:15:02 +00:00
Matt McEuen
e52ed80363 Update Kustomize integration to api/v0.3.1
This updates the Kustomize dependency for airshipctl to
api/v0.3.1, which is the oldest version which will support the
Replacement Transformer plugin.

Some changes were needed to accomodate the fact that various apis
that airshipctl was relying on were moved to kustomize-insternal
packages, namely:
- Integrated with the krusty.Kustomizer to drive kustomization
- Removed the custom plugin loader which leveraged the Unknown type
- Worked around NoFieldError becoming private, inc. removing a test

As a follow on we'll need to re-integrate plugin functionality somehow.

Also, in this release Kustomize has implemented support for the
"config.kubernetes.io/local-config" annotation, which we'd planned
to use to to avoid deploying some documents to the Kubernetes API.
It turns out the semantics are different than we anticipated;
Kustomize also fails to return these docs via document *selection*.
Therefore, this change reverts to an earlier approach which uses
a custom airshipit.org/deploy-k8s label.

Change-Id: I7022e12464ea7b6a3ca8609f99f3699bf8da0edd
2020-04-13 10:06:41 -05:00
Zuul
121adcddc8 Merge "Fixed TODO about error definitions" 2020-04-13 13:34:51 +00:00
Kostiantyn Kalynovskyi
b485f00c58 Change Label and Annotate to accept map of string
The methods are useful when incoming data are not key value pairs,
rather than iterating two times, one time when we call Annotate()
or Label() over the map, and then iterate again inside the function,
it would be better if we could skip iterating the first time, and
let everything to be handled by Annotate or Label functions

Relates-To: #162

Change-Id: Ia9d989a0534d15e4024317c4606a187ac021b751
2020-04-09 16:34:05 -05:00
Yasin, Siraj (SY495P)
c25d223c7b Add copyright for missing files
* added license templates for go, bash & yaml in tools dir
* added a script that will add license information for all
    missing files. Type:  go, yaml, yml, sh
* skip adding license for all files within testdata
* Syntax:
   > ./tools/add_license.sh

* Skip license for manifests folder
* Added one extra line after licene for yaml files
* Added License after Hashbang for bash.
* Add an extra line after hashbang and before license
* Updated the go template to use multiline comments

New Files:
  1. tools/add_license.sh
  2. tools/license_go.txt
  3. tools/license_yaml.txt
  4. tools/license_bash.txt

Change-Id: Ia4da5b261e7cd518d446896b72c810421877472a
Realtes-To:#147
2020-04-09 08:35:59 -05:00
Zuul
780c6b8459 Merge "Update Kustomize to v3.3.1" 2020-04-09 10:07:30 +00:00
Ian Howell
5efc26d9f0 Remove getters/setters of AirshipCTLSettings
This change removes the getters and setters for the following attributes
of pkg/environment.AirshipCTLSettings:

* airshipConfigPath
* kubeConfigPath
* config

The above attributes have been exported to account for the removal of
these methods.

Change-Id: I4e7d38bf6771f7521179a2fe2990475b936b5d95
2020-04-08 09:00:32 -05:00
Yasin, Siraj (SY495P)
f696ed5f11 Fix for Lint warnings
* This commit handles only warnings of mentioned type
  warning:
    comment on exported function <funcname> should be
      of the form "<funcname> ..."
* In this case, there were comments for the exported
  function/constant/var for which warnings raised,
  but not in expected format.
* So this fix is mostly formating the comment to avoid
  lint warnings.

  Observations:
    * comment did not had function name
    * function name in comment was immediately follwed by comma
        => function followed by space and any description is expected
    * function was not mentioned in the first line of the comment
        => when there is multiline comment, first line should start with
           function name

Change-Id: Ife97104ebc1054f4e34259dca51e8bdb42b33bbd
Relates-To:#148
2020-04-07 14:07:45 +00:00
Stanislav Egorov
f7a2e33b8b Fixed TODO about error definitions
Use SelectOne to replace Select and additional checks.

Change-Id: I40b4780d62fa4b3a0e01cc54ef7dbe4b32a4763d
2020-04-06 11:23:42 -07:00
Matt McEuen
e690a4c453 Update Kustomize to v3.3.1
This updates the Kustomize module dependency to v3.3.1,
which has replacement functionality required by the Replacement Transformer
plugin, among other enhancements.

Change-Id: I058114c398464394cfb6b2626cb475d763c1bacc
2020-04-03 12:11:33 -05:00
Ian Howell
bc949f259f Improve error messages related to Selectors
This change modifies the error messages related to Selectors by
delegating formatting to the Selector's String method.

Change-Id: I88acb70751509e588139ea26a8388661b3f63511
Relates-To: #145
2020-04-02 08:07:05 -05:00
Ian Howell
714496fc1f Improve document.Selector's String method
This change adds additional output to a document.Selector's String
method, making it easier to find information about the selector being
used. This is particularly helpful when reporting errors, such as those
in #145.

Change-Id: Ic3e67e7597776feec328ad4f54fdf06df863d928
Closes: #145
2020-04-01 10:24:32 -05:00
Alan Meadows
e2cca32748 Add Redfish Authentication Support
This commit also introduces a dochelper concept.  This provides
some convenience methods to the document pkg that help extract data
from known document types as well as walk document relationships to
discover related information such as BMC credentials for baremetal
hosts.

Once merged, a follow up patchset will leverage these within the
cloud-init code to deduplicate some of these lookups.

Change-Id: Ie6a770ce4b34adbea30281917f0cb2fdc460b4fb
2020-03-27 15:29:07 -07:00
Kostiantyn Kalynovskyi
ad1e160c1a Add selector to filter docs to be deplyed to k8s
By default every document is to be deployed to kubernetes however
some documents may need to be skipped, because they serve only
informational purposes and have no consumers in kubernetes cluster

Change-Id: If2ea5659acb319c15cf3f55e9f3d9e86dc647e9f
Relates-To: #130
Closes: #130
2020-03-26 12:17:00 -05:00
Kostiantyn Kalynovskyi
7a2d01789e Add ordering of the documents
By default kustomize cli orders k8s resources, in a way, that they
can easily by applied to cluster, namespaces and crds first. In this
patch set we implementing same behavior

Relates-To: #131
Closes: #131

Change-Id: I4fc75366627ed361ac1da48e89a35949bcb79801
2020-03-26 09:54:42 -05:00
Alan Meadows
69fefba69f Remove label filtering from initinfra
Now that initinfra leverages a specific entrypoint using
the cluster type, and the Config.Initinfra subdirectory
there is no need to also require all documents be labeled
with a specific label.  That would be redundant.

Change-Id: I41ff1ea01d5f04d7d987d2308636adfff48016ea
2020-03-25 10:47:20 -07:00
Zuul
14f52863e0 Merge "Add filtering method for document bundle" 2020-03-25 13:29:14 +00:00
Pavel Basov
af90464e66 Add selector to filter initinfra docs
airpship.org/phase: "initinfra" labeled documents are deployed with initinfra

Change-Id: Ie4c9421b9c4b3fe44e20a843f7a3b6551c5a7e6d
Closes: #125
2020-03-24 20:51:59 +01:00
Dmitry Ukov
09ef460e31 Add filtering method for document bundle
SelectByFieldValue method returns new bundle with documents which
values referenced by JSON path comply with function passed as input
argument to the method

Related: #14
Change-Id: I31bb6beee96b0d65719b6976caac63e54c65dcd2
2020-03-24 23:23:58 +04:00
Ian Howell
cdc6ba66f1 [#102] Add function pass forwarding for annotating
* Expose the GetAnnotations method of the document.Document interface.
* Add the Annotate method for adding a single annotation to a
  document.Document
* Clean up the implementation of Label and GetLabels

Change-Id: I94af2380f50848bfcd64af9d5c2c991305d9746f
Relates-To: #102
2020-03-17 12:18:52 -05:00
Stanislav Egorov
3eacfc6c71 [#102]: removed methods for bundle interface
Removed two unused methods from bundle interface.

Two other methods are used internally and there is no need to keep it in the interface.

Change-Id: I5d7999e4b2c83dd43025442e5549ac7826009a87
2020-03-16 15:08:19 -07:00
Stanislav Egorov
5224ed1b62 [#103]: replaced external methods for document interface
Removed external methods from document interface.
Created new methods to supply document labelling.

Change-Id: I4c20ad515999a6c049a2feddafdb62af8e1dfefb
2020-03-16 15:08:19 -07:00