Phase execution engine
======================

General phase architecture
--------------------------

Airshipctl performs operations such as ``kubectl apply``,
``clusterctl init`` etc... using `phases <#phase>`__ defined in a `phase
bundle <#phase-bundle>`__. You can think of a phase as a step that has
to be performed in order to achieve a desired state of the managed site.

A phase is usually run against an `executor bundle <#executor-bundle>`__,
which is a document set containing kubernetes resources.

`Executor bundles <#executor-bundle>`__ and `phase
bundles <#phase-bundle>`__ are separate bundles and it is important to
understand the difference between them:

- A `phase bundle <#phase-bundle>`__ can be thought of as an airshipctl configuration
  bundle, it defines what will be executed and against which `executor bundles
  <#executor-bundle>`__. **Important**: path to phase bundle is constructed
  using the airship config current manifest TargetPath, phaseRepositoryName directory
  and phase.path field taken from metadata file.
- An `executor bundle <#executor-bundle>`__ defines a document set that the `executor
  <#executor>`__ will run against, for example kubernetes manifests, that will
  be applied to the kubernetes cluster by the KubernetesApply executor. The path to an
  executor bundle is constructed using targetPath, phaseRepoDir,
  docEntryPointPrefix(optional), documentEntryPoint.

.. figure:: img/phase_overview.svg
   :alt: architecture diagram

   architecture diagram

Steps performed during phase execution based on diagram above:

1. Load metadata file based on airshipctl config from manifest
2. Build `phase bundle <#phase-bundle>`__ (please see phase bundle section
   for the details on how the path to the phase bundle is constructed using manifest
   and metadata.yaml file)
3. Prepare executor config
    - Find requested `phase <#phase>`__ in phase document bundle
    - Load `kubeconfig <#kubeconfig>`__ and `cluster map <#cluster-map>`__ from
      phase document bundle
    - Build `executor bundle <#executor-bundle>`__ based on phase document
4. Find `executor <#executor>`__ document in `phase Bundle <#phase-bundle>`__ and
   load executor providing config from step 3
5. Run `executor <#executor>`__, (validate/render available as well)

Phase
-----

Phase is a basic step that is executed by airshipctl to get a site to a
desired state. Phases are defined in `phase document
bundles <#phase-bundle>`__ as a `phase
document <https://godoc.org/opendev.org/airship/airshipctl/pkg/api/v1alpha1#Phase>`__
containing:

- `type meta <https://godoc.org/k8s.io/apimachinery/pkg/apis/meta/v1#TypeMeta>`__

    .. code:: yaml

        apiVersion: airshipit.org/v1alpha1
        kind: Phase

- `object meta <https://godoc.org/k8s.io/apimachinery/pkg/apis/meta/v1#ObjectMeta>`__

    **Note** that the clusterName that the phase belongs to comes from objects
    metadata and not from phase config. ClusterName is used to identify kubeconfig
    context and if needed cluster name of a parent cluster to be used during
    phase execution, also this clusterName must be present in the `cluster
    map <#cluster-map>`__.

    .. code:: yaml

        metadata:
          name: initinfra-ephemeral
          clusterName: ephemeral-cluster

- `config <https://godoc.org/opendev.org/airship/airshipctl/pkg/api/v1alpha1#PhaseConfig>`__

    Can include documentEntryPoint relative to **TargetPath** + **phaseRepoDir** +
    **docEntryPointPrefix**, some executors don't need a bundle, for example
    ``clusterctl move``, and in such cases documentEntryPoint field can be
    omitted. Also it includes an executorRef, which is a k8s
    `ObjectReference <https://godoc.org/k8s.io/kubernetes/pkg/apis/core#ObjectReference>`__,
    to an executor document, which defines an executor that will run
    the phase. The executor document referenced must exist in `phase bundle <#phase-bundle>`__

    .. code:: yaml

        config:
          executorRef:
            apiVersion: airshipit.org/v1alpha1
            kind: KubernetesApply
            name: kubernetes-apply
          documentEntryPoint: ephemeral/initinfra

Complete phase example:

.. code:: yaml

    apiVersion: airshipit.org/v1alpha1
    kind: Phase
    metadata:
      name: initinfra-ephemeral
      clusterName: ephemeral-cluster
    config:
      executorRef:
        apiVersion: airshipit.org/v1alpha1
        kind: KubernetesApply
        name: kubernetes-apply
      documentEntryPoint: ephemeral/initinfra

Phase Bundle
~~~~~~~~~~~~

A phase bundle controls how airshipctl will run, it contains but not limited to objects
depicted in the following table:

+-------------------------------------+---------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------+
| Name                                | Description                                 | API object source                                                                                                                    |
+=====================================+=============================================+======================================================================================================================================+
| `Cluster map <#cluster-map>`__      | Defines relationship between clusters       | `Cluster map API object source code <https://github.com/airshipit/airshipctl/blob/master/pkg/api/v1alpha1/cluster_map_types.go>`__   |
+-------------------------------------+---------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------+
| `Phase <#phase>`__                  | Basic step that is executed by airshipctl   | `phase document <https://github.com/airshipit/airshipctl/blob/master/pkg/api/v1alpha1/phase_types.go#L25-L36>`__                     |
+-------------------------------------+---------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------+
| `Kubeconfig <#kubeconfig>`__        | Credentials for kubernetes clusters         | `kubeconfig api object <https://github.com/airshipit/airshipctl/blob/master/pkg/api/v1alpha1/kubeconfig_types.go>`__                 |
+-------------------------------------+---------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------+
| `Phase plan <#phase-plan>`__        | Sequence of phase execution                 | `plan api object <https://github.com/airshipit/airshipctl/blob/master/pkg/api/v1alpha1/phase_types.go>`__                            |
+-------------------------------------+---------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------+
| `Executor document <#executor>`__   | Define what is going to be used by phases   | see `executor section <#executor>`__                                                                                                 |
+-------------------------------------+---------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------+

How is the path to a phase bundle constructed?
- A phase bundle is constructed
  using kustomize root that consists of TargetPath defined in current
  context manifest, and phase path taken from metadata file that is
  located inside the manifest: ``TargetPath + Metadata.Phase.Path``

Executor Bundle
~~~~~~~~~~~~~~~

- Executor bundle may contain any documents, they are available to
  executor during execution. For example these documents can be k8s
  resources, such as Deployments, Secrets, Services etc, and
  KubernetesApply executor will apply those resources to k8s cluster
  that phase belongs to.
- Some executors may require an executor bundle, some may not.
  ``Clusterctl move`` is an example of an executor that doesn't need a bundle
- Each executor may perform different actions based on the document set
- The path to the executor bundle is currently constructed using
  ``TargetPath + phaseRepositoryName + metadata.Phase.DocEntryPointPrefix
  + DocumentEntrypoint``

Executor
--------

Executor is what actually performs execution of the phase.
If a phase is considered as an abstract step of execution process,
then Executor is an implementation of this step. Executors are
defined as executor documents inside phase bundle, these documents
contain configuration for the executor, Kind and apiVersion that is
mapped by airshipctl internally to execution module that is invoked
during ``airshipctl phase`` commands.

Executor has access to kubeconfig, cluster map and its cluster name,
which allows the executor to find its context in kubeconfig, and if needed
parent cluster name and context as well.

Currently there is a limited, but growing number of executors:

-  `KubernetesApply <#kubernetesapply-executor-document-example>`__:
   applies resources to kubernetes.
-  `Clusterctl <#clusterctl-executor-document-example>`__: performs
   clusterctl commands based on its config.

**Note**: for more information about each executor please refer to the code
base, in the future more documentation will be developed for each
executor.

Clusterctl executor document example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-  `Executor source
   code <https://pkg.go.dev/opendev.org/airship/airshipctl/pkg/phase/executors#ClusterctlExecutor>`__
-  `Executor API object source
   code <https://godoc.org/opendev.org/airship/airshipctl/pkg/api/v1alpha1#Clusterctl>`__

.. code:: yaml

    apiVersion: airshipit.org/v1alpha1
    kind: Clusterctl
    metadata:
      labels:
        airshipit.org/deploy-k8s: "false"
      name: clusterctl_init
    init-options:
      core-provider: "cluster-api:v0.3.7"
      bootstrap-providers:
        - "kubeadm:v0.3.7"
      infrastructure-providers:
        - "metal3:v0.3.2"
      control-plane-providers:
        - "kubeadm:v0.3.7"
    action: init
    providers:
      - name: "metal3"
        type: "InfrastructureProvider"
        variable-substitution: true
        versions:
          v0.3.2: airshipctl/manifests/function/capm3/v0.3.2
      - name: "kubeadm"
        type: "BootstrapProvider"
        variable-substitution: true
        versions:
          v0.3.7: airshipctl/manifests/function/cabpk/v0.3.7
      - name: "cluster-api"
        type: "CoreProvider"
        variable-substitution: true
        versions:
          v0.3.7: airshipctl/manifests/function/capi/v0.3.7
      - name: "kubeadm"
        type: "ControlPlaneProvider"
        variable-substitution: true
        versions:
          v0.3.7: airshipctl/manifests/function/cacpk/v0.3.7
    additional-vars:
      CONTAINER_CAPM3_MANAGER: quay.io/metal3-io/cluster-api-provider-metal3:v0.3.2

KubernetesApply executor document example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-  `Executor source code
   <https://godoc.org/opendev.org/airship/airshipctl/pkg/k8s/applier#Executor>`__
-  `Executor API object source code
   <https://godoc.org/opendev.org/airship/airshipctl/pkg/api/v1alpha1#KubernetesApply>`__

.. code:: yaml

    apiVersion: airshipit.org/v1alpha1
    kind: KubernetesApply
    metadata:
      labels:
        airshipit.org/deploy-k8s: "false"
      name: kubernetes-apply
    config:
      waitOptions:
        timeout: 2000
      pruneOptions:
        prune: false

Kubeconfig
----------

Executors have access to site's kubeconfig. In kubeconfig, context names
must correspond to cluster names, for example if cluster name is
``workload01``, then kubeconfig should have context named ``workload01``
to be able to access this kubernetes cluster. (TODO: update
  this later when https://github.com/airshipit/airshipctl/issues/380 is closed)

Kubeconfig can come from various sources:

- `Phase bundle <#phase-bundle>`__, standard way of supplying kubeconfig
  is putting it into phase bundle as `kubeconfig api object
  <https://github.com/airshipit/airshipctl/blob/master/pkg/api/v1alpha1/kubeconfig_types.go>`__,
  `Kubeconfig api object example <#kubeconfig-api-object-example>`__.
  This is a good option when you are supplying predefined certificates
  for some of the clusters (ephemeral and target clusters) while still
  leaving possibility to use dynamic kubeconfig from a parent cluster
  for worker clusters.
- **User provided** , if a user wants to override kubeconfig, and use their own,
  they can do so using **--kubeconfig** flag in airshipctl, but they **MUST** be
  aware that kubeconfig context names must correspond to cluster names. (TODO: update
  this later when https://github.com/airshipit/airshipctl/issues/380 is closed)
- **Parent cluster** (*not implemented*), airshipctl uses clusterctl to deploy
  k8s clusters. And can use parent cluster to get kubeconfig secret from it, in
  such case dynamic kubeconfig should be allowed to be used in `cluster map
  <#cluster-map>`__, this is done by setting **dynamicKubeConf: true** in cluster
  map for specific cluster, default is false.

Kubeconfig api object example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: yaml

    apiVersion: airshipit.org/v1alpha1
    kind: KubeConfig
    metadata:
      name: default
      labels:
        airshipit.org/deploy-k8s: "false"
    config:
      apiVersion: v1
      clusters:
      - cluster:
          certificate-authority-data: <ca-data>
          server: https://10.23.25.102:6443
        name: target-cluster
      - cluster:
          certificate-authority-data: <ca-data>
          server: https://10.23.25.101:6443
        name: ephemeral-cluster
      contexts:
      - context:
          cluster: target-cluster
          user: target-cluster-admin
        name: target-cluster
      - context:
          cluster: ephemeral-cluster
          user: ephemeral-cluster-admin
        name: ephemeral-cluster
      current-context: ""
      kind: Config
      preferences: {}
      users:
      - name: ephemeral-cluster-admin
        user:
          client-certificate-data: <client-cert-data>
          client-key-data: <client-key-data>
      - name: target-cluster-admin
        user:
          client-certificate-data: <client-cert-data>
          client-key-data: <client-key-data>

Cluster Map
-----------

Cluster map defines parent-child relationship between clusters, allows
dynamic kubeconfig for clusters. When kubeconfig must be sourced from
a parent cluster, cluster map will be used to find parent cluster and
request kubeconfig from it.

Also, ClusterAPIRef is a part of Cluster Map and will be used to find
cluster object in kubernetes parent cluster. It also maps a clusterapi
name and namespaces for a given cluster. In the below example object,
the cluster api ref describes the reference to the cluster api object.
The cluster `workload01` in the cluster map has an clusterAPIRef to
corresponding cluster-api cluster object (kind: cluster) with name workload01
and inside namespace tenant01-namespace.

Cluster map is defined in `Phase bundle <#phase-bundle>`__ as a document.

- `Cluster map API object source code
  <https://godoc.org/opendev.org/airship/airshipctl/pkg/api/v1alpha1#ClusterMap>`__
- `Cluster map interface source code
  <https://pkg.go.dev/opendev.org/airship/airshipctl/pkg/phase/executors#ClusterctlExecutor>`__

Example of cluster map
~~~~~~~~~~~~~~~~~~~~~~

.. code:: yaml

    apiVersion: airshipit.org/v1alpha1
    kind: ClusterMap
    metadata:
      labels:
        airshipit.org/deploy-k8s: "false"
      name: main-map
    map:
      target-cluster:
        parent: ephemeral-cluster
        # dynamicKubeConf: false # default value
      ephemeral-cluster: {}
      workload01:
        parent: target-cluster
        dynamicKubeConf: true
        clusterAPIRef:
          name: workload01
          namespace: tenant01-namespace

Metadata file
-------------

Phase plan
----------

TODO expand this part of documentation when we utilize phase plan