780afe54e4
Adds basic expectations and explanations of openstack-helms testing and gating approaches. This will continue to be expanded upon but serves to provide a base on which to build Change-Id: I689446b7124c4e11a92c73ef04e95d1840c6dc0a
145 lines
6.6 KiB
ReStructuredText
145 lines
6.6 KiB
ReStructuredText
====================
|
|
OpenStack-Helm Gates
|
|
====================
|
|
|
|
To facilitate ease of testing and debugging, information regarding gates and
|
|
their functionality can be found here.
|
|
|
|
OpenStack-Helm's single node and multinode gates leverage the kubeadm-aio
|
|
environment created and maintained for use as a development environment. All
|
|
information regarding the kubeadm-aio environment can be found here_.
|
|
|
|
.. _here: https://github.com/openstack/openstack-helm/tree/master/tools/kubeadm-aio
|
|
|
|
Gate Checks
|
|
-----------
|
|
|
|
OpenStack-Helm currently checks the following scenarios:
|
|
|
|
- Testing any documentation changes and impacts
|
|
- Running Make on each chart, which lints and packages the charts. This gate
|
|
does not stand up a Kubernetes cluster
|
|
- Provisioning a single node cluster and deploying the OpenStack services. This
|
|
check is provided for: Ubuntu-1604, CentOS-7, and Fedora-25
|
|
- Provisioning a multi-node Ubuntu-1604 cluster and deploying the OpenStack
|
|
services. This check is provided for both a two node cluster and a three
|
|
node cluster.
|
|
|
|
|
|
Gate Functions
|
|
--------------
|
|
|
|
To provide reusable components for gate functionality, functions have been
|
|
provided in the gates/funcs directory. These functions include:
|
|
|
|
- Functions for common host preparation operations, found in common.sh
|
|
- Functions for Helm specific operations, found in helm.sh. These functions
|
|
include: installing Helm, serving a Helm repository locally, linting and
|
|
building all Helm charts, running Helm tests on a release, installing the
|
|
helm template plugin, and running the helm template plugin against a chart.
|
|
- Functions for Kubernetes specific operations, found in kube.sh. These
|
|
functions include: waiting for pods in a specific namespace to register as
|
|
ready, waiting for all nodes to register as ready, install the requirements
|
|
for the kubeadm-aio container used in the gates, building the kubeadm-aio
|
|
container, launching the kubeadm-aio container, and replacing the
|
|
kube-controller-manager with a specific image necessary for ceph functionality.
|
|
- Functions for network specific operations, found in network.sh. These
|
|
functions include: creating a backup of the host's resolv.conf file before
|
|
deploying the kubeadm environments, restoring the original resolv.conf
|
|
settings, creating a backup of the host's /etc/hosts file before adding the
|
|
hosts interface and address, and restoring the original /etc/hosts file.
|
|
- Functions for OpenStack specific operations, found in openstack.sh. These
|
|
functions include: waiting for a successful ping, and waiting for a booted
|
|
virtual machine's status to return as ACTIVE.
|
|
|
|
Any additional functions required for testing new charts or improving the gate
|
|
workflow should be placed in the appropriate location.
|
|
|
|
|
|
Gate Output
|
|
-----------
|
|
|
|
To provide meaningful output from the gates, all information pertaining to the
|
|
components of the cluster and workflow are output to the logs directory inside
|
|
each gate. The contents of the log directory are as follows:
|
|
|
|
- The dry-runs directory contains the rendered output of Helm dry-run installs
|
|
on each of the OpenStack service charts. This gives visibility into the
|
|
manifests created by the templates with the supplied values. When the dry-run
|
|
gate fails, the reason should be apparent in the dry-runs output. The logs
|
|
found here are helpful in identifying issues resulting from using helm-toolkit
|
|
functions incorrectly or other rendering issues with gotpl.
|
|
- The k8s directory contains the logs and output of the Kubernetes objects. It
|
|
includes: pods, nodes, secrets, services, namespaces, configmaps, deployments,
|
|
daemonsets, and statefulsets. Descriptions for the state of all resources
|
|
during execution are found here, and this information can prove valuable when
|
|
debugging issues raised during a check. When a single node or multi-node
|
|
check fails, this is the first place to look. The logs found here are helpful
|
|
when the templates render correctly, but the services are not functioning
|
|
correctly, whether due to service configuration issues or issues with the
|
|
pods themselves.
|
|
- The nodes directory contains information about the node the gate tests are
|
|
running on in openstack-infra. This includes: the network interfaces, the
|
|
contents of iptables, the host's resolv.conf, and the kernel IP routing table.
|
|
These logs can be helpful when trying to identify issues with host networking
|
|
or other issues at the node level.
|
|
|
|
|
|
Adding Services
|
|
---------------
|
|
|
|
As charts for additional services are added to OpenStack-Helm, they should be
|
|
included in the gates. Adding new services to the gates allows a chart
|
|
developer and the review team to identify any potential issues associated with
|
|
a new service. All services are currently launched in the gate via
|
|
basic_launch.sh. Any new services are added to this file, and the required
|
|
format for a basic service is:
|
|
|
|
::
|
|
|
|
helm install --namespace=openstack ${WORK_DIR}/mistral --name=mistral
|
|
|
|
Some services in the gate require specific overrides to the default values in
|
|
the chart's values.yaml file. If a service requires multiple overrides to
|
|
function in the gate, the service should include a separate values.yaml file
|
|
placed in the tools/overrides/mvp directory. The <service>.yaml MVP files
|
|
provide a configuration file to use for overriding default configuration values
|
|
in the chart's values.yaml as an alternative to overriding individual values
|
|
during installation. A chart that requires a MVP overrides file
|
|
requires the following format:
|
|
|
|
::
|
|
|
|
helm install --namespace=openstack ${WORK_DIR}/cinder --name=cinder \
|
|
--values=${WORK_DIR}/tools/overrides/mvp/cinder.yaml
|
|
|
|
|
|
Adding Tests
|
|
------------
|
|
|
|
As new charts are developed and the services are added to the gate, an
|
|
associated Helm test should be introduced to the gates. The appropriate place
|
|
for executing these tests is in basic_launch.sh, and must be placed after the
|
|
entry for installing the service and any associated overrides. Any tests that
|
|
use the Rally testing framework should leverage the helm_test_deployment
|
|
function in the aforementioned funcs/helm.sh file. For example, a Helm test for
|
|
Mistral might look like:
|
|
|
|
::
|
|
|
|
helm_test_deployment mistral 600
|
|
|
|
This results in the gate running the following:
|
|
|
|
::
|
|
|
|
helm test --timeout 600 mistral
|
|
mkdir -p mistral/rally
|
|
kubectl logs -n openstack mistral-rally-test > logs/rally/mistral
|
|
kubectl delete -n openstack pod mistral-rally-test
|
|
|
|
Any tests that do not use the Rally testing framework would need to be handled
|
|
in the appropriate manner in basic_launch.sh. This would ideally result in new
|
|
functions that could be reused, or expansion of the gate scripts to include
|
|
scenarios beyond basic service launches.
|