Containerization: Helm Chart Override Generation
Proposing specification on how Helm chart overrides will be managed in a StarlingX deployment. Change-Id: Ibfce3c39742511b4108205612e979dcc81f9beb9 Signed-off-by: Robert Church <robert.church@windriver.com>
This commit is contained in:
Robert Church
parent
f7c04656d8
commit
e2a7fbf95d
@ -0,0 +1,509 @@
|
||||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
|
||||
==================================================
|
||||
Containerization: Helm Chart Override Generation
|
||||
==================================================
|
||||
|
||||
Storyboard: `#2003909`_
|
||||
|
||||
This spec is part of the larger effort to containerize the StarlingX
|
||||
infrastructure. Please see the background information regarding this effort on
|
||||
the StarlingX `wiki`_.
|
||||
|
||||
For managing the life-cycle of containerized services, StarlingX will be using
|
||||
two open source technologies: `Helm`_, the Kubernetes package manager and
|
||||
`Armada`_, a tool for managing multiple Helm charts with dependencies.
|
||||
|
||||
This specification will outline the use of `Helm`_ in a StarlingX containerized
|
||||
platform and the need to generate system compatible `Helm chart`_ overrides.
|
||||
|
||||
`Armada`_ integration and use will be discussed in an additional `Armada
|
||||
integration`_ story.
|
||||
|
||||
.. _#2003909: https://storyboard.openstack.org/#!/story/2003909
|
||||
.. _wiki: https://wiki.openstack.org/wiki/Containerizing_StarlingX_Infrastructure
|
||||
.. _Helm: https://helm.sh
|
||||
.. _Armada: https://airship-armada.readthedocs.io/en/latest/
|
||||
.. _Armada integration: https://storyboard.openstack.org/#!/story/2003908
|
||||
.. _Helm chart: https://docs.helm.sh/developing_charts
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Each StarlingX native service that will be containerized shall be described by
|
||||
a `Helm chart`_.
|
||||
|
||||
The `Helm chart`_ for a given service will be built from a chart provided by
|
||||
one of the following open source projects: `openstack-helm`_ (for openstack
|
||||
services), `openstack-helm-infra`_ (for openstack support services),
|
||||
`external-storage`_ (for volume provisioners), and StarlingX (nova-api proxy,
|
||||
fault manager, etc)
|
||||
|
||||
The current set of native services that will be containerized are: AODH,
|
||||
Ceilometer, Cinder, Fault Manager, Glance, Gnocchi, Heat, Horizon, Ironic,
|
||||
Keystone, Libvirt, Magnum, Murano, Neutron, Nova, Nova-API Proxy, Panko, and
|
||||
Rabbitmq.
|
||||
|
||||
To support these services, the following charts and corresponding services are
|
||||
also required: Ingress, MariaDB, Memcached, and an RBD volume provisioner.
|
||||
|
||||
A chart comes with a default set of values that will describe the behavior of
|
||||
the service installed within the Kubernetes cluster. `Helm`_ provides a
|
||||
mechanism to override these default values so the service can be updated based
|
||||
on the desired service configuration.
|
||||
|
||||
In the current StarlingX code-base, native services are configured via puppet
|
||||
modules with supplied hiera data derived from the contents of the system
|
||||
inventory database. The values in the database reflect the current physical
|
||||
components of installation along with the configuration choices of the
|
||||
Deployer.
|
||||
|
||||
In order to migrate the native services to containerized services, the
|
||||
following problems need to be addressed:
|
||||
|
||||
* Need a sysinv based framework to examine the system inventory database and
|
||||
generate chart specific system overrides for each native service.
|
||||
|
||||
* The framework should be easily extensible for adding chart support.
|
||||
|
||||
* Need an API for the Deployer to easily override or complement the system
|
||||
generated overrides for a specific chart
|
||||
|
||||
* Need an API to generate the final overrides for a specific chart or set of
|
||||
charts. The final generated set of overrides should be overrides with the
|
||||
following superseding priority: default values (base) <- system generated <-
|
||||
Deployer provided.
|
||||
|
||||
It should be noted that this does not preclude any `Helm chart`_ from being
|
||||
installed on the system and used with standard `Helm`_ mechanisms for managing
|
||||
the life-cycle of a chart release.
|
||||
|
||||
.. _openstack-helm: https://github.com/openstack/openstack-helm
|
||||
.. _openstack-helm-infra: https://github.com/openstack/openstack-helm-infra
|
||||
.. _external-storage: https://github.com/kubernetes-incubator/external-storage
|
||||
|
||||
|
||||
Use Cases
|
||||
=========
|
||||
|
||||
The Deployer needs to be able manage and generate the helm overrides for a
|
||||
given `Helm chart`_. The following API actions need to be enabled
|
||||
|
||||
* List system supported helm charts.
|
||||
* Show overrides for a chart.
|
||||
* Delete overrides for a chart.
|
||||
* Update helm chart user overrides.
|
||||
|
||||
|
||||
.. _proposed_change:
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
In order to properly configure a `Helm chart`_ for a containerized service the
|
||||
following is proposed:
|
||||
|
||||
* Provide a plugin framework as part of sysinv to allow adding support for new
|
||||
charts as required.
|
||||
* Provide a basic plugin for each chart in the `openstack-helm`_ project.
|
||||
* Provide specific plugin logic to generate chart overrides for each of the
|
||||
currently supported StarlingX native services.
|
||||
|
||||
With the plugin framework and chart plugins deployed as part of sysinv, a
|
||||
mechanism is required to generate the overrides for a specific chart. For this
|
||||
the following is proposed:
|
||||
|
||||
* Implement sysinv API/CLI for managing system and Deployer provided
|
||||
overrides:
|
||||
|
||||
* helm-override-list - List system helm charts.
|
||||
* helm-override-show - Show overrides for a chart.
|
||||
* helm-override-delete - Delete overrides for a chart.
|
||||
* helm-override-update - Update helm chart user overrides.
|
||||
|
||||
* Additional options will be provided to the helm-overrides-show CLI to
|
||||
extract the combined overrides in a yaml form to feed into `Helm`_ or
|
||||
`Armada`_.
|
||||
|
||||
* --format [ helm | armada ]
|
||||
|
||||
* Specifying armada will add the `Armada`_ compliant header to the yaml
|
||||
output.
|
||||
|
||||
* --file=<filename>
|
||||
|
||||
* An optional parameter, provided with the use of the --format option
|
||||
will generate a yaml file containing the overrides. If this option is
|
||||
not specified, then the resulting yaml file will be delivered to
|
||||
stdout.
|
||||
|
||||
Alternatives
|
||||
============
|
||||
|
||||
An alternative approach would be to leave the override configuration for each
|
||||
charts as an exercise up to the Deployer using standard `Helm`_ and/or
|
||||
`Armada`_ mechanisms. This approach would increase initial system provisioning
|
||||
times and would lead to a hit-or-miss integration strategy to find the correct
|
||||
overrides for a give configuration.
|
||||
|
||||
|
||||
Data model impact
|
||||
=================
|
||||
|
||||
To support managing chart overrides, a new table will be added to the sysinv
|
||||
DB, with the following columns:
|
||||
|
||||
* created_at, updated_at, deleted_at: standard columns
|
||||
* id: primary key
|
||||
* name: chart name
|
||||
* namespace: chart namespace
|
||||
* user_overrides: store deltas as compared to system overrides
|
||||
* unique constraint is a combination of name and namespace
|
||||
|
||||
|
||||
As usual, a new data migration will be added to create this table in the sysinv
|
||||
DB.
|
||||
|
||||
|
||||
REST API impact
|
||||
===============
|
||||
|
||||
This impacts the sysinv REST API:
|
||||
|
||||
* The new resource /helm_charts is added and the GET method would return all
|
||||
the charts that provide system overrides along with their namespaces.
|
||||
|
||||
* URLS:
|
||||
|
||||
* /v1/helm_charts/
|
||||
|
||||
* Request Methods:
|
||||
|
||||
* GET
|
||||
|
||||
* JSON response example::
|
||||
|
||||
{"charts": [
|
||||
{"name": "ingress", "namespaces": ["kube-system", "openstack"]},
|
||||
{"name": "rbd-provisioner", "namespaces": ["kube-system"]},
|
||||
{"name": "openvswitch", "namespaces": ["openstack"]},
|
||||
{"name": "rabbitmq", "namespaces": ["openstack"]},
|
||||
{"name": "libvirt", "namespaces": ["openstack"]},
|
||||
{"name": "heat", "namespaces": ["openstack"]},
|
||||
{"name": "keystone", "namespaces": ["openstack"]},
|
||||
{"name": "nova", "namespaces": ["openstack"]},
|
||||
{"name": "horizon", "namespaces": ["openstack"]},
|
||||
{"name": "cinder", "namespaces": ["openstack"]},
|
||||
{"name": "glance", "namespaces": ["openstack"]},
|
||||
{"name": "mariadb", "namespaces": ["openstack"]},
|
||||
{"name": "memcached", "namespaces": ["openstack"]},
|
||||
{"name": "neutron", "namespaces": ["openstack"]}]}
|
||||
|
||||
* By providing the GET method a specific chart name along with key/value pair
|
||||
specifying a namespace, the values of a chart can be retrieved
|
||||
|
||||
* URLS:
|
||||
|
||||
* /v1/helm_charts/{name}?namespace={namespace}
|
||||
|
||||
* Request Methods:
|
||||
|
||||
* GET
|
||||
|
||||
* Example:
|
||||
|
||||
* Request: GET /v1/helm_charts/rabbitmq?namespace=openstack
|
||||
* JSON response example::
|
||||
|
||||
{"namespace": "openstack",
|
||||
"name": "rabbitmq",
|
||||
"system_overrides": "pod:\n replicas: {server: 1}\n",
|
||||
"user_overrides": "",
|
||||
"combined_overrides": "pod:\n replicas:\n server: 1\n"}
|
||||
|
||||
* Using the PATCH method will allow the Deployer to update override values
|
||||
|
||||
* URLS:
|
||||
|
||||
* /v1/helm_charts/{name}?namespace={namespace}
|
||||
|
||||
* Request Methods:
|
||||
|
||||
* PATCH
|
||||
|
||||
* Example:
|
||||
|
||||
* Request: PATH /v1/helm_charts/rabbitmq?namespace=openstack
|
||||
* JSON request example::
|
||||
|
||||
{"flag": "reuse", "values": {"files": [], "set": ["pod.replicas=2"]}
|
||||
|
||||
* JSON response example::
|
||||
|
||||
{"user_overrides": "pod:\n replicas: 2\n", "namespace": "openstack",
|
||||
"name": "rabbitmq"}
|
||||
|
||||
* Using the DELETE method will allow the Deployer to delete all Deployer
|
||||
provided overrides. Once deleted, the system generated override will still be
|
||||
applied for the chart.
|
||||
|
||||
* URLS:
|
||||
|
||||
* /v1/helm_charts/{name}?namespace={namespace}
|
||||
|
||||
* Request Methods:
|
||||
|
||||
* DELETE
|
||||
|
||||
* Example:
|
||||
|
||||
* Request: DELETE /v1/helm_charts/rabbitmq?namespace=openstack
|
||||
|
||||
|
||||
Security impact
|
||||
===============
|
||||
|
||||
Passwords may be provided in the overrides. Considerations need to be made when
|
||||
displaying/setting these override values. One potential solution is to prevent
|
||||
overriding these values via the API.
|
||||
|
||||
Other end user impact
|
||||
=====================
|
||||
|
||||
As mentioned in the :ref:`proposed_change` section, new CLI commands will be
|
||||
provided and will behave as follows:
|
||||
|
||||
* helm-override-list - List system helm charts.::
|
||||
|
||||
$ system helm-override-list
|
||||
+-----------------+--------------------------------+
|
||||
| chart name | overrides namespaces |
|
||||
+-----------------+--------------------------------+
|
||||
| ceilometer | [u'openstack'] |
|
||||
| cinder | [u'openstack'] |
|
||||
| glance | [u'openstack'] |
|
||||
| gnocci | [u'openstack'] |
|
||||
| heat | [u'openstack'] |
|
||||
| horizon | [u'openstack'] |
|
||||
| ingress | [u'kube-system', u'openstack'] |
|
||||
| keystone | [u'openstack'] |
|
||||
| libvirt | [u'openstack'] |
|
||||
| mariadb | [u'openstack'] |
|
||||
| memcached | [u'openstack'] |
|
||||
| neutron | [u'openstack'] |
|
||||
| nova | [u'openstack'] |
|
||||
| openvswitch | [u'openstack'] |
|
||||
| rabbitmq | [u'openstack'] |
|
||||
| rbd-provisioner | [u'kube-system'] |
|
||||
+-----------------+--------------------------------+
|
||||
|
||||
* helm-override-update - Update helm chart user overrides.::
|
||||
|
||||
$ system helm-override-update rabbitmq openstack \
|
||||
--reuse-values \
|
||||
--set pod.replicas=2
|
||||
+----------------+---------------+
|
||||
| Property | Value |
|
||||
+----------------+---------------+
|
||||
| name | rabbitmq |
|
||||
| namespace | openstack |
|
||||
| user_overrides | pod: |
|
||||
| | replicas: 2 |
|
||||
| | |
|
||||
+----------------+---------------+
|
||||
|
||||
* helm-override-show - Show overrides for a chart.::
|
||||
|
||||
$ system helm-override-show rabbitmq openstack
|
||||
+--------------------+-------------------------+
|
||||
| Property | Value |
|
||||
+--------------------+-------------------------+
|
||||
| combined_overrides | pod: |
|
||||
| | replicas: 2 |
|
||||
| | |
|
||||
| name | rabbitmq |
|
||||
| namespace | openstack |
|
||||
| system_overrides | pod: |
|
||||
| | replicas: {server: 1} |
|
||||
| | |
|
||||
| user_overrides | pod: |
|
||||
| | replicas: 2 |
|
||||
| | |
|
||||
+--------------------+-------------------------+
|
||||
|
||||
* helm-override-delete - Delete overrides for a chart.::
|
||||
|
||||
$ system helm-override-delete rabbitmq openstack
|
||||
Deleted chart overrides for rabbitmq:openstack
|
||||
|
||||
|
||||
|
||||
Performance Impact
|
||||
==================
|
||||
|
||||
Minimal impact to the system controllers is to be expected. The proposed
|
||||
changes to generate and manage the chart overrides will access the sysinv
|
||||
database and execute helm commands to determined the combined overrides.
|
||||
|
||||
Other deployer impact
|
||||
=====================
|
||||
|
||||
The API and ability to extract the combined overrides will enable Armada to be
|
||||
integrated as a mechanism to manage and launch all the current bare metal
|
||||
native service as a kubetnetes application.
|
||||
|
||||
Developer impact
|
||||
================
|
||||
|
||||
Developers working in StarlingX will need to use the API changes provided here
|
||||
to view, adjust, and generate chart overrides for containerized service
|
||||
deployment via `Helm`_.
|
||||
|
||||
Upgrade impact
|
||||
==============
|
||||
|
||||
None. This is the first StarlingX release with `Helm`_ support. No previous
|
||||
release supports this feature and no upgrade support is required.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
As the system inventory database contains all the pertinent information about
|
||||
the installed system, changes will be required in stx-config/sysinv to generate
|
||||
the overrides for the `Helm`_ charts and provide the APIs to access them.
|
||||
|
||||
The implementation will duplicate the plugin framework and associated
|
||||
constructs used for extracting and generating the puppet hiera for existing
|
||||
native services.
|
||||
|
||||
Once fully implemented, the puppet manifests for the native services and the
|
||||
plugins that generated the hiera data for those manifests will be removed. In
|
||||
their stead, `Helm`_ chart overrides will be generated and provided `Armada`_
|
||||
for launching the containerized services.
|
||||
|
||||
Assignee(s)
|
||||
===========
|
||||
|
||||
Primary assignee:
|
||||
|
||||
* Robert Church (rchurch)
|
||||
|
||||
Other contributors:
|
||||
|
||||
* Chris Friesen (cbf123)
|
||||
* Gerry Kopec (gerry-kopec)
|
||||
* Joseph Richard (josephrichard)
|
||||
* Tyler Smith (tyler.smith)
|
||||
* Angie Wang (angiewang)
|
||||
* Irina Mihai (irina.mihai.wrs)
|
||||
* Ovidiu Poncea (ovidiu.poncea)
|
||||
* Al Bailey (albailey1974)
|
||||
* Lachlan Plant (lachlan.plant)
|
||||
|
||||
|
||||
Repos Impacted
|
||||
==============
|
||||
|
||||
stx-config
|
||||
|
||||
Work Items
|
||||
==========
|
||||
|
||||
* stx-config/sysinv:
|
||||
|
||||
* Provide a plugin framework as part of sysinv to allow adding support for
|
||||
new charts as required.
|
||||
* Provide a basic plugin for each chart in the `openstack-helm`_ project.
|
||||
* Provide specific plugin logic to generate chart overrides for each of the
|
||||
currently supported StarlingX native services.
|
||||
* Implement API changes to support listing supported charts that provide
|
||||
system overrides.
|
||||
* Implement API changes to support showing the system, Deployer, and combined
|
||||
overrides.
|
||||
* Implement API changes to support adding and modifying Deployer provided
|
||||
overrides.
|
||||
* Implement API changes to support deleting Deployer provided overrides.
|
||||
|
||||
* stx-config/cgts-client:
|
||||
|
||||
* Implement sysinv CLI for managing system and Deployer provided overrides by
|
||||
calling the sysinv API:
|
||||
|
||||
* helm-override-list - List system helm charts.
|
||||
* helm-override-show - Show overrides for a chart.
|
||||
* helm-override-delete - Delete overrides for a chart.
|
||||
* helm-override-update - Update helm chart user overrides.
|
||||
* Add helm-override-show options for generating the final view of overrides
|
||||
|
||||
* Add: --format [ helm | armada ]
|
||||
* Add: --file=<filename>
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
This requires new functionality being developed under the following stories:
|
||||
|
||||
* Kubernetes Platform Support: `#2002843`_
|
||||
* CEPH persistent storage backend for Kubernetes: `#2002844`_
|
||||
* Local Docker Registry: `#2002840`_
|
||||
* Docker Image Generation: `#2003907`_
|
||||
* Infrastructure HELM Chart Override Generation: `#2003909`_
|
||||
* Armada Integration: `#2003908`_
|
||||
|
||||
.. _#2002843: https://storyboard.openstack.org/#!/story/2002843
|
||||
.. _#2002844: https://storyboard.openstack.org/#!/story/2002844
|
||||
.. _#2002840: https://storyboard.openstack.org/#!/story/2002840
|
||||
.. _#2003907: https://storyboard.openstack.org/#!/story/2003907
|
||||
.. _#2003909: https://storyboard.openstack.org/#!/story/2003909
|
||||
.. _#2003908: https://storyboard.openstack.org/#!/story/2003908
|
||||
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
The following testing will be performed in association with these proposed
|
||||
changes:
|
||||
|
||||
* The sysinv REST API will be exercised using curl to verify/validate it's
|
||||
operation. It will be used for documenting API access.
|
||||
* The cgts-client commands will be exercised along with any supported options.
|
||||
* The resulting combined overrides yaml files will be used with `Helm`_ and
|
||||
`Armada`_ to ensure compatibility.
|
||||
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
This story affects the following StarlingX documentation:
|
||||
|
||||
* Installation and configuration of containerized services.
|
||||
* sysinv REST API documentation.
|
||||
|
||||
Specific details of the documentation changes will be addressed once the
|
||||
implementation is complete.
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
References are provided throughout this document at the point when terms or
|
||||
items are introduced. No additional references are needed at this time.
|
||||
|
||||
|
||||
History
|
||||
=======
|
||||
|
||||
.. list-table:: Revisions
|
||||
:header-rows: 1
|
||||
|
||||
* - Release Name
|
||||
- Description
|
||||
* - 2019.03
|
||||
- Introduced
|
||||
Loading…
Reference in New Issue
Block a user