44 Commits

Author SHA1 Message Date
Zuul
0fce7c4895 Merge "Update Transformer to accept blank source string" 2021-10-04 23:57:43 +00:00
Graham Steffaniak
5f9ddcc62a Update Transformer to accept blank source string
ADD: return condition for blank source string.
ADD: test case 26 for example.
CHG: Updated source value variable type

Closes: #620
Change-Id: Id2bac427cd0570eac3e87b98ded665d70613d2d6
2021-10-01 21:05:37 +00:00
Alexey Odinokov
e2c56108ee Nextgen secrets implementation with separation per cluster
1. Extending templater with kyaml functions and creating combined catalogue
to be able to request/update the existing resources.
This is based on 'everything is transformer' concept introduced in kustomize 4.x
That includes gathering all secrets into 1 variable catalogue and
special mechanism to regenerate/merge with manual secrets.

2. Implementing 'catalogue per cluster' approach for secrets.

3. Rearranging secrets so it's possible to use:
pgp (each person may have his own key), age, Hachicorp Vault and etc
and the list of people who can decrypt documents is set in a special file.
Since in some cases there should be a separate list of people who can decrypt
data - this list is set for each cluster (ephemeral and target) separatelly.

Closes: #586
Change-Id: I038f84dd138d5ad4a35f4862c61ff2124c2fd530
2021-09-03 20:46:15 +00:00
Alexey Odinokov
37c2854cce Switching airshipctl and krm-functions to kustomize 4.2.0
Reworked build process of dependent krm functions
to use the latest airshipctl

Change-Id: If4c7f992123b86442980f381ae69fb003f4851e3
2021-07-22 17:06:32 +00:00
Zuul
d844fdceca Merge "Generate additional CRDs from API types" 2021-06-29 00:15:35 +00:00
Kostiantyn Kalynovskyi
df55f50cb6 Allow rendering by label for replacement transformer
Change-Id: Ie2d68d735cb2b95987f5c95cbc024d608696f028
2021-06-28 18:42:49 +00:00
Matthew Fuller
52c5aedc63 Generate additional CRDs from API types
'make manifests' attempts to generate all airshipctl API types,
but several types throw errors and the resulting CRDs can't be
properly generated. This change:

- updates the controller-gen version to pull in a bug fix for the
  error 'map values must be a named type, not *ast.StarExpr'
- modifies the Templater type's Values member to be of type JSON
  since controller-gen does not support map[string]interface{},
  and unmarshals it separately in the templater itself
- adds 'image_repositories' to the VersionsCatalogueSpec struct
  to ensure it doesn't get deleted from the schema when manifests
  get regenerated
- checks in generated copies of all CRDs from airshipctl's API
  types, not just catalogues, to be used for validation

Closes: #574
Change-Id: I89a12cfd307a08da9aaec9773eac01169ea43ace
2021-06-25 19:10:25 +00:00
Zuul
7b851d9af2 Merge "Add kyaml Label selector to document plugins" 2021-04-29 22:02:56 +00:00
Kostiantyn Kalynovskyi
b75a06fcb6 Add kyaml Label selector to document plugins
This is needed to be able to select documents inside toolbox
and maybe other KRM functions in the future

Relates-To: #517
Change-Id: I0c0c3552d98bcbe1ea02fb7b582d844a3d8f1c16
2021-04-28 20:22:45 +00:00
Alexey Odinokov
10b0385691 Extended replacement plugin with Targets field
The user can use either `target` or `targets` that is list of
several `target` objects.

Change-Id: I38e457842e1250a2d048f93ba2aac9258758d163
2021-04-28 16:36:19 +00:00
Sreejith Punnapuzha
ab85f2236f Add function to generate SSH KeyPair via templater
This commit adds functionality to generate SSH keypair via templater
plugins. ssh keypairs can be generated via generate-secrets phase.

Signed-off-by: Sreejith Punnapuzha <Sreejith.Punnapuzha@outlook.com>
Change-Id: I83720df5f934caf65dab201a1d0894ed3fee6cb5
2021-03-30 18:00:23 -05:00
Alexey Odinokov
b70848b57b Revert "Adding Filesystem check function to templater"
This reverts commit 154bcec95cc207339f05dd4d62e987efb25d0366

We don't use this function anywhere, moreover
fs related functions are way beyond the reasonable
functionality of templater. Especially taking
into account that we have only 1 function.

Change-Id: I919d8466ff1eab2d2687db7055bf974b7b608d8e
2021-02-26 20:29:36 +00:00
Zuul
e77bac1571 Merge "Implement genCAEx and genSignedCertEx with Subj support" 2021-02-01 20:22:53 +00:00
Alexey Odinokov
47df361761 Implement genCAEx and genSignedCertEx with Subj support
The original sprig-library implementation of
genCA and genSignedCert may accept only CN parameter
which isn't enough for K8s admin certificate.
That certification must have O=system:masters, e.g.
/CN=admin/O=system:masters

This PR introduces the set of functions that
insted of cn accept subj argument that may have
a form compatible with openssl -subj parameter.
If the first symbol isn't '/' subj behaves as
cn argument.

The set of new functions that accept subj arg is:
genCAEx
genCAWithKeyEx
genSignedCertEx
genSignedCertWithKeyEx

Since the implementaion required to copy some
non-exported helper functions from sprig,
the decision was made to separate all go-template
extension functions into a separate package:
extlib. This package can be reused in other
go-applications, it's just necessary to use
GenericFuncMap function to get function-map.

Change-Id: I0ffddee2e597323803bf5f1b54f315ded424b7be
2021-01-29 16:43:32 +00:00
bijayasharma
661ae70fd0 Removed capitalization in Error string message
* This commit will removed capitalizaion in Error message
  and follow the Golang code rule.
* Golang errors should not begin with capitalization:
  https://github.com/golang/go/wiki/CodeReviewComments#error-strings

Signed-off-by: bijayasharma <vetbijaya@gmail.com>
Change-Id: I053cfa9e21ad56b0f30e476ec36320cb7cea4da4
2021-01-26 19:42:08 +00:00
Sirisha Gopigiri
154bcec95c Adding Filesystem check function to templater
The following code uses the documentFs interface and adds
simple fileExists function to templater plugin.

Change-Id: Ia53c573e54188960eaf99f7c48469359d4e3688b
2021-01-05 05:51:01 +00:00
Sirisha Gopigiri
91b2164a4f Adding Regex support for Templater
Adding go code to templater to support generation of strings
based on the regular expression passed.

Closes: #438
Change-Id: Ife0c4f8d5dd8032408875c73ba098919470be9e9
2021-01-04 18:18:27 +05:30
Sirisha Gopigiri
247047f2a3 Update Sprig library and add Uint32 support
Below code updates sprig library from v2.22 to
v3.2. Also added support for uint32 coversion.

Change-Id: I263cf6f1915d2f892f6f77efdcdd477ad902f527
2020-12-21 17:40:11 +05:30
Dmitry Ukov
9c69138990 Remove document plugin subcommand
This change finalises shift to new version of kustomize plugins

Change-Id: Ie47f42a9e2c4dc03fc4dca437020ad71a564d894
2020-11-11 14:59:14 +04:00
Sirisha Gopigiri
d82e04ea5e Adding decoding and encoding feature to Replacement Transformer
This PS adds decoding features to ReplacementTransformer to decode
the source objectRef when source is of `kind: Secret` and has `data`
field. It also encodes the value in the target fieldRefs before
replacement if the target is `kind: Secret` and has `data` fields.
Throws an error if the target fieldRefs have both `data` and `stringData`
for replacement.

Change-Id: I1d918058409b3511faa9a99512d25574027bda86
2020-10-27 17:45:29 +05:30
Dmitry Ukov
96cbb045a4 Remove legacy code from ReplacementTransformer
Change-Id: Ia247ee15efd5f66b5e36cf49339474548d94765d
2020-10-23 14:27:55 +04:00
Stanislav Egorov
71f04ed976 Proxy config for control plane
Along with the change to support proxy configuration for control plane
there is a change for replacement transformer to support arrays of strings.

The current implementation of cloud-init configuration for the control plane
based on KubeadmControlPlane object and uses an array of strings for commands.

Change-Id: Id0a49cc1a0fdbc392c7c7a56859b21737065b0d6
2020-10-20 21:38:18 -07:00
Dmitry Ukov
d783d71f05 Introduce Replacement Transformer plugin container
Relates-To: #341
Change-Id: I65a5b8cf3bcf94321fc02f240b1d8075bca45c0c
2020-10-16 16:00:56 +04:00
Dmitry Ukov
ca1a3a2d0b Migrate Replacement Transformer plugin
Plugin extended to support new kustomize plugin framework which
consider each plugin as a container

Change-Id: If55b7093f711401165b7d4fd3f5b1059fde464ff
Relates-To: #340
2020-10-13 20:43:55 +04:00
Dmitry Ukov
025c2172d6 Implement JSON path filter for kyaml.RNode
Change introduces kyaml.RNode Pipe Filter which uses k8s go-client JSON
path parser. This enables to use JSON queries defined by
https://goessner.net/articles/JsonPath/

Change-Id: I6c2276f27652190ed9d183cea0e45eb118346c6b
Relates-To: #340
2020-10-07 15:27:19 +04:00
Dmitry Ukov
0d475ec6c5 Extend plugin interface with execute method
Change introduces evolution of airshipctl document plugins which
are compatible to the new approach for kustomize plugins.

Change-Id: I422110ba523b97c8a80a0b82cb43b70a4ee8e558
Relates-To: #322
2020-10-05 13:24:03 +04:00
Zuul
233b87a103 Merge "Move plugins configurations to API module" 2020-09-24 15:34:08 +00:00
Ruslan Aliev
2e85043c20 Fix typos
Change-Id: I0ce87ec00d2b5974cf861d9c9685f23d7d0f5fc1
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
2020-09-21 02:49:33 -05:00
Dmitry Ukov
034efc3682 Move plugins configurations to API module
Data structure representing plugin configurations should be a part of
airshipctl API module. Plugin implementations will reside in document
package.

Change-Id: Id2e359b747a16a5573052cfb05c1148d346db508
Relates-To: #322
2020-09-17 19:22:07 +04:00
Matt McEuen
9997e7427e Allow substring replacement of numerics
This change adds the ability to use substring substitution to inject
numeric source values into string targets.  This is required for our
networking catalogue, since a numeric k8s API port needs to be inserted into
numeric and (sub)string targets in different phases.

Co-Authored-By: Ian Howell <ian.howell0@gmail.com>
Change-Id: I24beb46a2bda4e118406129a0a922b0c56142c76
2020-09-14 16:03:51 -05:00
Zuul
ad21e64a98 Merge "Substring replacement fix in ReplacementTranformer" 2020-09-01 16:56:23 +00:00
Ruslan Aliev
78364d7d44 Refactor document plugin command
AirshipCTLSettings are not used here, so it was removed.

Change-Id: If4bf64b9991b4ac05898632c79bacbacd7872635
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Relates-To: #327
2020-09-01 10:15:58 -05:00
Sirajudeen
7e391bfc1b Substring replacement fix in ReplacementTranformer
* Substring replacement failing in ReplacementTranformer when
target substring contains periods (.)

* This is temporary fix till the kpt-function way of replacement
  transformer is implemented

* In this fix it is assumed that the substring condition can only occur at
  the end of the path.

  example:
   expected: replace the nginx version from 1.7.9 to 1.17.0 for container named
             `nginx-tagged`

   target condition:
      spec.template.spec.containers[name=nginx-tagged].image%1.7.9%

Change-Id: I76fb65f69a6eedf3cbdd692d3bc1835a214dc8fa
Relates-To: #336
Closes: #336
2020-08-31 17:43:21 +00:00
muly
9d4347b0c1 documentation changes
1) modifications to exported functions/variables documentation
2) moved some methods into a different folder

Change-Id: I212442516424bde3284f75b326d152c290d20c80
2020-07-29 09:17:00 -04:00
Matt McEuen
0324993d60 Add an m3 host generator function
This PS has a function which constructs a collection of Metal3 BareMetalHost
resources, along with associated configuration Secrets.
It solves for a couple of things:

1. pulling the nitty gritty details for generating BMH into one reusable place,
2. allowing the site-specific details to be filled in via catalogues of values

This function leverages a couple of different plugins in sequence:
The airshipctl Replacement plugin, which pulls the site-specific data from
the catalogue documents into a Templater plugin configuration; and then
the airshipctl Templater plugin, which generates a variable number of
BMHs in a data-driven fashion.

More details can be found in the README.md in this patchset.

Closes: #245

Change-Id: I3ddbd36dc53ea6afbd633098c985f4b28bcbb793
2020-07-17 15:30:26 -05:00
Yasin, Siraj (SY495P)
fffc42ad8a Enable golint as part of golangci-lint.
Issues raised after enabling golint as part of golangci-lint

  * error var ToDiscoveryError should have name of the
	form ErrFoo (golint)
  * error strings should not be capitalized or end with punctuation
	or a newline (golint)

Issues not raised by golint embedded within golangci-lint

  * comment on exported function <funcname> should be
      of the form "<funcname> ..."

So kept the golint standalone tool also in Makefile

Change-Id: I7f2ce66d1e757bc24c042fc212226cecea297f27
2020-05-20 13:29:11 -05:00
Yasin, Siraj (SY495P)
8d16fffb7c Fix Lint warnings: Missing comments for functions
* Added comments wherever missing for exported functions/constants

Change-Id: I7d0b79b5f56fc6c3edb60a3b58c0962ebd76314c
Relates-To: #148
2020-05-19 16:54:09 +00:00
Matt McEuen
2e7224a7bf RT: add error handling for missing target docs
The ReplacementTransformer already checks for the error condition where
a replacement source doesn't match any of the input documents.
This change adds similar error handling for missing replacement
target documents, and tweaks the naming/messages to make it clear
which situation is encountered.

This will make it easier for manifest engineers to catch typos
in their RT rules, or documents that have failed to be kustomized in
or authored.

Change-Id: I0efbcfbc2952afae05e48b9ff8e0b3c67df7e7df
2020-05-11 17:45:34 -05:00
Zuul
e6a6a655e1 Merge "Replacement transformer improvement" 2020-04-29 22:46:14 +00:00
Dmitry Ukov
c458e8cde4 Add template based generator plugin
Plugin is responsible for resource generation based on template
(go template format is used) supplied to its configuration.
Go template can use an object (map) specified under 'spec'
configuration parameter.

Change-Id: I8c1b7dc5676bfff2f9b1a47f6433177ae796a15f
2020-04-29 23:38:58 +04:00
Dmitry Ukov
47d8d8c886 Replacement transformer improvement
* Add ability to to replace whole list element filtered by key and
  value
* Raise an error in if user has specified wrong list filer in JSON path
* Enables support for a dot character in a value used in list filtering

Closes: #191
Closes: #192

Change-Id: I889550ee88d5738a1fb808ddef6478c3731b1561
2020-04-28 17:12:29 +04:00
Dmitry Ukov
c30930114a Replacement transformer refactoring
* Apply recommendations from "Effective Go"
* Simplify applySubstringPattern logic
* Reduce complexity of updateMapField
* Switch to typed errors
* Increased test coverage

Change-Id: I8e53a251a43c8f31c286284c77452fbf43ce4e43
2020-04-24 19:48:59 +04:00
Dmitry Ukov
6c716e1a57 Add replacement transformer
* Replace paramter specified by JSON path with predefined value or
  with the value from another resource
* Replace substring in a paramter specified by JSON path with predefined
  value or with the value from another resource

Transformer copied from
https://github.com/mattmceuen/kustomize/tree/substring-subst

Closes: #174
Change-Id: I3a958a0df724fb2eb81bb199a02cf1db81bb0d2f
Co-Authored-By: Matt McEuen <matt.mceuen@att.com>
2020-04-22 11:51:05 +04:00
Dmitry Ukov
39ee048451 Introduce document plugin subcommand
airship document plugin is intended to be executed as an exec plugin
for kustomize document model.
Environment variable is used to gather plugin configuration. Plugin to
execute is determined based on group-version-kind specified in plugin
configuration. Each airship plugin must implement plugin interface.

Relates-To: #173
Change-Id: I4f6c3b5be140c0d8fd7519f1cedd33de1cef662c
2020-04-21 15:00:22 +04:00