openstack/tacker-specs
Code Issues Proposed changes
f9af3dcfa0
tacker-specs/specs/xena/cir-k8s-cluster.rst
Masaki Ueno d3848c7339 Support CIR for k8s cluster with MgmtDriver 
The Tacker Wallaby release supports spec "Support deploying Kubernetes
cluster with MgmtDriver".
Deployed Kubernetes cluster supports only Images registered in
the Docker public registry but doesn't support Images in Docker
private registry.
This specification adds functionality to enable the use of Docker
private registry images in a Kubernetes Cluster environment.
This specification assumes spec "Support deploying Kubernetes cluster
with MgmtDriver".

Implements: blueprint cir-k8s-cluster
Change-Id: I19d63d48c47eacd24552eaa47d63abc3ce6a22bd
2021-07-19 17:31:09 +09:00

59 KiB
Raw Blame History

Support Docker private registry for Kubernetes cluster with MgmtDriver

https://blueprints.launchpad.net/tacker/+spec/cir-k8s-cluster

Problem description

The Tacker Wallaby release supports spec "Support deploying Kubernetes cluster with MgmtDriver"1. Deployed Kubernetes cluster supports only Images registered in the Docker public registry but doesn't support Images in Docker private registry. This specification adds functionality to enable the use of Docker private registry images in a Kubernetes Cluster environment. This specification assumes spec "Support deploying Kubernetes cluster with MgmtDriver"2.

Proposed change

The following operations are supported:

  1. Deploy Docker private registry VNF for Kubernetes cluster before deploying Kubernetes cluster VNF.
  2. Deploy Kubernetes cluster VNF that can support both the Docker private registry created above and any others created outside of Tacker.

The LCM operations for Docker private registry VNF requires the following changes:

  • Instantiate operation with MgmtDriver to install Docker private registry.
  • Heal operation with MgmtDriver to re-install Docker private registry.

Note

  • Scale operation is not supported.
  • Terminate operation is out of scope in this spec. There is no specific processes in the termination of Docker private registry VNF, so it is already supported by existing implementation.

The LCM operations for Kubernetes cluster VNF requires the following changes:

  • Instantiate operation for Kubernetes cluster with MgmtDriver3.

    Changes: MgmtDriver configures connections to Docker private registries on newly created all Master/Worker VMs.

  • Scale-out operation for Kubernetes cluster Worker-nodes with MgmtDriver4.

    Changes: MgmtDriver configures connections to Docker private registries on newly created Worker VMs.

  • Heal operation for the entire Kubernetes cluster with MgmtDriver5.

    Changes: MgmtDriver configures connections to Docker private registries on the created all Master/Worker VMs.

  • Heal operation for a single node (Master/Worker) in Kubernetes cluster with MgmtDriver6.

    Changes: MgmtDriver configures connections to Docker private registries on the created Master/Worker VM.

Deploy Docker private registry VNF for Kubernetes cluster

Deploy the Docker private registry for Kubernetes cluster as a VNF. The diagram below shows creating VM and set up Docker private registry:

+--------------+ +---------+
| Docker Image | |  VNFD   |
| (Optional)   | |         |
+------------+-+ +-+-------+
             |     |
             v     v    +---------------+
          +----------+  | Instantiation |
          |          |  | Request with  |
          |   CSAR   |  | Additional    |
          |          |  | Params        |
          +----+-----+  +-+-------------+
               |          | 1. Instantiate VNF
               |          |
         +-----+----------+-------------+
         |     v          v        VNFM |
         |  +-------------------+       |
         |  |   Tacker-server   |       |
         |  +-------+-----------+       |
         |          |                   |
         |          v                   |
+--------------------------+ 4. Run Private  |  +------------------------+  |
|                          |    registry     |  |                        |  |
|  +--------------------+  |    container    |  |    +--------------+    |  |
|  | Docker container   |<-+-----------------+--+----+              |    |  |
|  | (Private registry) |  |                 |  |    |  MgmtDriver  |    |  |
|  +--------------------+  | 3. Docker       |  |    |              |    |  |
|  +--------------------+  |    Installation |  |    |              |    |  |
|  |       Docker       |<-+-----------------+--+----+              |    |  |
|  +--------------------+  |                 |  |    +--------------+    |  |
|                          |                 |  |    +--------------+    |  |
|                          | 2. Create VM    |  |    | OpenStack    |    |  |
|                          |<----------------+--+----+ Infra Driver |    |  |
|                          |                 |  |    +--------------+    |  |
|            VM            |                 |  |                        |  |
+--------------------------+                 |  |    Tacker-conductor    |  |
+--------------------------+                 |  +------------------------+  |
|    Hardware Resources    |                 |                              |
+--------------------------+                 +------------------------------+

VNFD for Docker private registry VNF

One VDU and one or more CPs, and the interface definitions of instantiate_end and heal_end are required in VNFD as the following sample:

node_templates:
 VNF:
   ...
   interfaces:
     Vnflcm:
       ...
       instantiate_end:
         implementation: mgmt-drivers-private-registry
       heal_end:
         implementation: mgmt-drivers-private-registry
   artifacts:
     mgmt-drivers-private-registry:
       description: Management driver for Docker private registry
       type: tosca.artifacts.Implementation.Python
       file: Scripts/private_registry_mgmt.py

 PrivateRegistryVDU:
   type: tosca.nodes.nfv.Vdu.Compute
   ...

 CP1:
   type: tosca.nodes.nfv.VduCp
   ...
   requirements:
     - virtual_binding: PrivateRegistryVDU
   ...

Request parameters for Instantiate Docker private registry VNF operation

User gives InstantiateVnfRequest data type defined in ETSI NFV-SOL003 v2.6.17 as request parameters. It is basically the same parameters as described in the spec "REST API for VNF based on ETSI NFV-SOL specification"8. User can use the following additionalParams:

Attribute name Cardinality Parameter description
private_registry_installation_param 1 Configuration for private registry installation.
>ssh_cp_name 1 CP name that MgmtDriver uses when SSH/SFTP access to the Private registry VM.
>ssh_username 1 User name that MgmtDriver uses when SSH/SFTP access to the Private registry VM.
>ssh_password 1 User password that MgmtDriver uses when SSH/SFTP access to the Private registry VM.
>image_path 0..1 Optional. Path of the Docker image file in the VNF Package for the Private registry container to run on Docker. If this attribute is omitted, the image for the Private registry container is pulled from the Docker public registry. If the Private registry VM is unable to connect to the Docker public registry, put the file created using "docker save" command into the VNF Package and specify the path of the file in this attribute.
>port_no 0..1 Optional. The default value is 5000. TCP port number that provides the Private registry service.

The following is a sample of body provided in the Instantiate VNF request `POST /vnflcm/v1/vnf_instances/{vnfInstanceId}/instantiate`:

{
  "flavourId": "simple",
  "extVirtualLinks": [
    {
      "id": "net0",
      "resourceId": "f0c82461-36b5-4d86-8322-b0bc19cda65f",
      "extCps": [
        {
          "cpdId": "CP1",
          "cpConfig": [
            {
              "cpProtocolData": [
                {
                  "layerProtocol": "IP_OVER_ETHERNET"
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "additionalParams": {
    "private_registry_installation_param": {
      "ssh_cp_name": "CP1",
      "ssh_username": "ubuntu",
      "ssh_password": "ubuntu",
      "image_path": "Files/images/private_registry.tar"
    }
  },
  "vimConnectionInfo": [
    {
      "id": "8a3adb69-0784-43c7-833e-aab0b6ab4470",
      "vimId": "8d8373fe-6977-49ff-83ac-7756572ed186",
      "vimType": "openstack"
    }
  ]
}

Sequence for Instantiate Docker private registry VNF operation

seqdiag {

node_width = 80; edge_length = 100;

"Client" "Tacker-server" "Tacker-conductor" "VnfLcmDriver" "OpenStackInfraDriver" "Heat" "MgmtDriver" "VnfInstance(Tacker DB)" "RemoteCommandExecutor" "SFTPClient"

Client -> "Tacker-server"

[label = "POST /vnf_instances/{vnfInstanceId}/instantiate"];

Client <-- "Tacker-server"

[label = "Response 202 Accepted"];

"Tacker-server" -> "Tacker-conductor"

[label = "trigger asynchronous task"];

"Tacker-conductor" -> "VnfLcmDriver"

[label = "execute VnfLcmDriver"];

"VnfLcmDriver" -> "OpenStackInfraDriver"

[label = "execute OpenStackInfraDriver"];

"OpenStackInfraDriver" -> "Heat"

[label = "create stack"];

"OpenStackInfraDriver" <-- "Heat"

[label = "return stack id"];

"VnfLcmDriver" <-- "OpenStackInfraDriver"

[label = "return instance_id"];

"VnfLcmDriver" -> "MgmtDriver"

[label = "instantiate_end"];

"MgmtDriver" -> "VnfInstance(Tacker DB)"

[label = "get stack id"];

"MgmtDriver" <-- "VnfInstance(Tacker DB)"

[label = ""];

"MgmtDriver" -> "Heat"

[label = "get SSH/SFTP IP address using stack id"];

"MgmtDriver" <-- "Heat"

[label = ""];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "install Docker on the VM"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"MgmtDriver" -> "SFTPClient"

[label = "transfer a Docker image file to the VM if specified by request parameter"];

"MgmtDriver" <-- "SFTPClient"

[label = ""];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "run Private registry container on Docker"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"VnfLcmDriver" <-- "MgmtDriver"

[label = ""];

"Tacker-conductor" <-- "VnfLcmDriver"

[label = ""];

}

The procedure consists of the following steps as illustrated in above sequence:

  1. Client sends a POST Instantiate VNF request.
  2. It is basically the same sequence as described in the "2) Flow of Instantiation of a VNF instance" chapter of spec "REST API for VNF based on ETSI NFV-SOL specification"9, except for the MgmtDriver.
  3. The following processes are performed in instantiate_end.

    1. MgmtDriver gets IP address of Private registry VM from Heat.

    2. MgmtDriver installs Docker on Private registry VM.

    3. MgmtDriver transfers a Docker image file to Private registry VM if specified by request parameter "image_path".

    4. MgmtDriver uses a Docker image to run Private registry container on Docker.

      MgmtDriver uses a command like the following to start the Private registry container:

      sudo docker run -d -p 5000:5000 -v /home/ubuntu/registry:/var/lib/registry --restart=always --name private_registry registry:latest

      Note

      The Docker image name for the Private registry container is registry. If the request parameter "image_path" is specified, "docker load" command is used to load the registry image from the transferred image file. If the request parameter "image_path" is omitted, "docker pull" command is used to pull the registry image from the Docker public registry.

      Note

      The above sample command assumes that the OS of the Private registry VM is Ubuntu 20.04.

Heal operation for Docker private registry VNF with MgmtDriver

The diagram below shows the Heal VNF operation for Docker private registry VNF:

+--------------+
| Heal Request |
+-+------------+
  | 1. Heal VNF
  |
+-----+------------------------+
|     v                   VNFM |
|  +-------------------+       |
|  |   Tacker-server   |       |
|  +-------+-----------+       |
|          |                   |
+--------------------------+                  |          v                   |
|                          | 5. Run Private   |  +------------------------+  |
|                          |    registry      |  |                        |  |
|  +--------------------+  |    container     |  |    +--------------+    |  |
|  | Docker container   |<-+------------------+--+----+              |    |  |
|  | (Private registry) |  |                  |  |    |  MgmtDriver  |    |  |
|  +--------------------+  | 4. Docker        |  |    |              |    |  |
|  +--------------------+  |    Installation  |  |    |              |    |  |
|  |       Docker       |<-+------------------+--+----+              |    |  |
|  +--------------------+  |                  |  |    +--------------+    |  |
|                          | 3. Create new VM |  |    +--------------+    |  |
|            VM            |<-----------------+--+----+ OpenStack    |    |  |
+--------------------------+                  |  |    | Infra Driver |    |  |
2. Delete     +--+--+----+              |    |  |
+--------------------------+    failed VM  |  |  |    +--------------+    |  |
|            VM            |<--------------+  |  |                        |  |
+--------------------------+                  |  |    Tacker-conductor    |  |
+--------------------------+                  |  +------------------------+  |
|    Hardware Resources    |                  |                              |
+--------------------------+                  +------------------------------+

VNFD for Heal operation

Same as "VNFD for Docker private registry VNF" chapter.

Request parameters for Heal Docker private registry VNF operation

Same parameters as described in the spec "REST API for VNF based on ETSI NFV-SOL specification"10.

Sequence for Heal Docker private registry VNF operation

seqdiag {

node_width = 80; edge_length = 100;

"Client" "Tacker-server" "Tacker-conductor" "VnfLcmDriver" "OpenStackInfraDriver" "Heat" "MgmtDriver" "VnfInstance(Tacker DB)" "RemoteCommandExecutor" "SFTPClient"

Client -> "Tacker-server"

[label = "POST /vnf_instances/{vnfInstanceId}/heal"];

Client <-- "Tacker-server"

[label = "Response 202 Accepted"];

"Tacker-server" -> "Tacker-conductor"

[label = "trigger asynchronous task"];

"Tacker-conductor" -> "VnfLcmDriver"

[label = "execute VnfLcmDriver"];

"VnfLcmDriver" -> "OpenStackInfraDriver"

[label = "execute OpenStackInfraDriver"];

"OpenStackInfraDriver" -> "Heat"

[label = "resource mark unhealthy"];

"OpenStackInfraDriver" <-- "Heat"

[label = ""];

"OpenStackInfraDriver" -> "Heat"

[label = "update stack"];

"OpenStackInfraDriver" <-- "Heat"

[label = ""];

"VnfLcmDriver" <-- "OpenStackInfraDriver"

[label = ""];

"VnfLcmDriver" -> "MgmtDriver"

[label = "heal_end"];

"MgmtDriver" -> "VnfInstance(Tacker DB)"

[label = "get stack id"];

"MgmtDriver" <-- "VnfInstance(Tacker DB)"

[label = ""];

"MgmtDriver" -> "Heat"

[label = "get SSH/SFTP IP address using stack id"];

"MgmtDriver" <-- "Heat"

[label = ""];

"MgmtDriver" -> "VnfInstance(Tacker DB)"

[label = "get additionalParams of Instantiate VNF request"];

"MgmtDriver" <-- "VnfInstance(Tacker DB)"

[label = ""];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "install Docker on the VM"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"MgmtDriver" -> "SFTPClient"

[label = "transfer a Docker image file to the VM if specified by Instantiate VNF request parameter"];

"MgmtDriver" <-- "SFTPClient"

[label = ""];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "run Private registry container on Docker"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"VnfLcmDriver" <-- "MgmtDriver"

[label = ""];

"Tacker-conductor" <-- "VnfLcmDriver"

[label = ""];

}

The procedure consists of the following steps as illustrated in above sequence:

  1. Client sends a POST Heal VNF request.
  2. It is basically the same sequence as described in the "3) Flow of Heal of a VNF instance" chapter of spec "REST API for VNF based on ETSI NFV-SOL specification"11, except for the MgmtDriver.
  3. The following processes are performed in heal_end.
    1. MgmtDriver gets IP address of the new Private registry VM from Heat.
    2. MgmtDriver gets additionalParams of Instantiate VNF request from VnfInstance(Tacker DB).
    3. MgmtDriver installs Docker on the new Private registry VM.
    4. MgmtDriver transfers a Docker image file to the new Private registry VM if specified by Instantiate VNF request parameter "image_path".
    5. MgmtDriver uses a Docker image to run the Private registry container on Docker.

Deploy Kubernetes cluster VNF that can connect to Docker private registries

Add the function to connect to Docker private registries based on spec "Support deployment Kubernetes cluster with MgmtDriver"12.

The diagram below shows creating Kubernetes cluster and configure connections to Docker private registries:

+---------+ +---------+
| Cluster | |         |
| Install | |  VNFD   |
| Script  | |         |
+----------------+  +-------+-+ +-+-------+
| CA certificate +--+       |     |
+----------------+  |       v     v
+---------------+  |    +----------+  +---------------+
| LCM operation |  +--->|          |  | Instantiation |
| UserData      +------>|   CSAR   |  | Request with  |
+---------------+       |          |  | Additional    |
+------------+  +--->|          |  | Params        |
| Heat       |  |    +----+-----+  +-+-------------+
| Template   +--+         |          | 1. Instantiate VNF
| (Base HOT) |            |          |
+------------+      +-----+----------+-------------+
    |     v          v        VNFM |
    |  +------------------+        |
    |  |   Tacker-server  |        |
    |  +-------+----------+        |
3. Kubernetes Cluster              |          |                   |
Installation                    |          v                   |
4. Configure connections           |  +------------------------+  |
to Docker private registries    |  |    +--------------+    |  |
+-------------+--------------------+--+----+  MgmtDriver  |    |  |
|             |                    |  |    +--------------+    |  |
+-------+-------------+--------+           |  |                        |  |
|       |             |        |           |  |                        |  |
|  +----+-----+  +----+-----+  |           |  |                        |  |
|  |    v     |  |    v     |  |           |  |    +--------------+    |  |
|  | +------+ |  | +------+ |  |           |  |    | OpenStack    |    |  |
|  | |Worker| |  | |Master| |<-+-----------+--+----+ Infra Driver |    |  |
|  | +------+ |  | +------+ |  | 2. Create |  |    +--------------+    |  |
|  |    VM    |  |    VM    |  |    VMs    |  |                        |  |
|  +----------+  +----------+  |           |  |                        |  |
+------------------------------+           |  |    Tacker-conductor    |  |
+------------------------------+           |  +------------------------+  |
|      Hardware Resources      |           |                              |
+------------------------------+           +------------------------------+

VNFD for Kubernetes cluster VNF

No change from user guide "How to use Mgmt Driver for deploying Kubernetes Cluster"13.

Request parameters with Docker private registries

Add the following attributes to additionalParams described in the user guide "How to use Mgmt Driver for deploying Kubernetes Cluster"14.

Attribute name Cardinality Parameter description
k8s_cluster_installation_param 1 Configuration for Kubernetes cluster installation.
>private_registry_connection_info 0..N Optional. Configuring to connect to Docker private registries.
>>connection_type 1
Type of connection. Set one of the following values.
0 : HTTP, 1 : HTTPS
Set to 0 if connecting to the deployed Docker private registry VNF or a Docker private registry outside of Tacker over HTTP.
Set to 1 if connecting to a Docker private registry outside of Tacker over HTTPS.
>>server 1 Server name of the Docker private registry to connect to. For example, "192.168.0.10:5000"
>>username 0..1 Optional. Username to log in to the Docker private registry.
>>password 0..1 Optional. Password to log in to the Docker private registry.
>>certificate_path 0..1 Optional. The path of the CA certificate file to use for HTTPS connection.
>>hosts_string 0..1 Optional. String to add to /etc/hosts. The base Kubernetes cluster environment does not have a DNS server and must be added to /etc/hosts. The value consists of "<IP address> <FQDN>". For example, "192.168.0.20 registry.example.com"

The following is a sample of body provided in the Instantiate VNF request `POST /vnflcm/v1/vnf_instances/{vnfInstanceId}/instantiate`:

{
  "flavourId": "simple",
  "additionalParams": {
    "k8s_cluster_installation_param": {
      "script_path": "Scripts/install_k8s_cluster.sh",
      "vim_name": "kubernetes_vim",
      "master_node": {
        "aspect_id": "master_instance",
        "ssh_cp_name": "masterNode_CP1",
        "nic_cp_name": "masterNode_CP1",
        "username": "ubuntu",
        "password": "ubuntu",
        "pod_cidr": "192.168.3.0/24",
        "cluster_cidr": "10.199.187.0/24",
        "cluster_cp_name": "masterNode_CP1"
      },
      "worker_node": {
        "aspect_id": "worker_instance",
        "ssh_cp_name": "workerNode_CP2",
        "nic_cp_name": "workerNode_CP2",
        "username": "ubuntu",
        "password": "ubuntu"
      },
      "proxy": {
        "http_proxy": "http://user1:password1@host1:port1",
        "https_proxy": "https://user2:password2@host2:port2",
        "no_proxy": "192.168.246.0/24,10.0.0.1,registry.example.com",
        "k8s_node_cidr": "192.168.0.0/24"
      },
      "private_registry_connection_info": [
        {
          "connection_type": "0",
          "server": "192.168.0.10:5000"
        },
        {
          "connection_type": "1",
          "server": "registry.example.com",
          "username": "user01",
          "password": "user01",
          "certificate_path": "Files/registries/ca.crt",
          "hosts_string": "192.168.0.20 registry.example.com"
        }
      ]
    },
    "lcm-operation-user-data": "./UserData/k8s_cluster_user_data.py",
    "lcm-operation-user-data-class": "KubernetesClusterUserData"
  },
  "extVirtualLinks": [
    {
      "id": "net0_master",
      "resourceId": "f0c82461-36b5-4d86-8322-b0bc19cda65f",
      "extCps": [
        {
          "cpdId": "masterNode_CP1",
          "cpConfig": [
            {
              "cpProtocolData": [
                {
                  "layerProtocol": "IP_OVER_ETHERNET"
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "id": "net0_worker",
      "resourceId": "f0c82461-36b5-4d86-8322-b0bc19cda65f",
      "extCps": [
        {
          "cpdId": "workerNode_CP2",
          "cpConfig": [
            {
              "cpProtocolData": [
                {
                  "layerProtocol": "IP_OVER_ETHERNET"
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "vimConnectionInfo": [
    {
      "id": "8a3adb69-0784-43c7-833e-aab0b6ab4470",
      "vimId": "8d8373fe-6977-49ff-83ac-7756572ed186",
      "vimType": "openstack"
    }
  ]
}

Sequence for Instantiate Kubernetes cluster VNF with Docker private registries

seqdiag {

node_width = 80; edge_length = 100;

"Client" "Tacker-server" "Tacker-conductor" "VnfLcmDriver" "OpenStackInfraDriver" "Heat" "MgmtDriver" "VnfInstance(Tacker DB)" "RemoteCommandExecutor" "SFTPClient" "NfvoPlugin"

Client -> "Tacker-server"

[label = "POST /vnf_instances/{vnfInstanceId}/instantiate"];

Client <-- "Tacker-server"

[label = "Response 202 Accepted"];

"Tacker-server" -> "Tacker-conductor"

[label = "trigger asynchronous task"];

"Tacker-conductor" -> "VnfLcmDriver"

[label = "execute VnfLcmDriver"];

"VnfLcmDriver" -> "OpenStackInfraDriver"

[label = "execute OpenStackInfraDriver"];

"OpenStackInfraDriver" -> "Heat"

[label = "create stack"];

"OpenStackInfraDriver" <-- "Heat"

[label = "return stack id"];

"VnfLcmDriver" <-- "OpenStackInfraDriver"

[label = "return instance_id"];

"VnfLcmDriver" -> "MgmtDriver"

[label = "instantiate_end"];

"MgmtDriver" -> "VnfInstance(Tacker DB)"

[label = "get stack id"];

"MgmtDriver" <-- "VnfInstance(Tacker DB)"

[label = ""];

"MgmtDriver" -> "Heat"

[label = "get SSH/SFTP IP addresses and Kubernetes addresses using stack id"];

"MgmtDriver" <-- "Heat"

[label = ""];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "install Kubernetes and create cluster on new Master/Worker VMs"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "get identification token from Kubernetes cluster"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"MgmtDriver" -> "SFTPClient"

[label = "transfer CA certificate files to all Master/Worker VMs if specified by request parameter"];

"MgmtDriver" <-- "SFTPClient"

[label = ""];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "configure connections to the Docker private registries on all Master/Worker VMs"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"MgmtDriver" -> "NfvoPlugin"

[label = "register Kubernetes VIM to tacker"];

"MgmtDriver" <-- "NfvoPlugin"

[label = ""]

"MgmtDriver" -> "VnfInstance(Tacker DB)"

[label = "append Kubernetes cluster VIM info to VimConnectionInfo"]

"MgmtDriver" <-- "VnfInstance(Tacker DB)"

[label = ""]

"VnfLcmDriver" <-- "MgmtDriver"

[label = ""];

"Tacker-conductor" <-- "VnfLcmDriver"

[label = ""];

}

The procedure consists of the following steps as illustrated in above sequence.

  1. Client sends a POST Instantiate VNF request.
  2. It is basically the same sequence as described in spec "Support deployment Kubernetes cluster with MgmtDriver"15, except for following additional processes.
  3. The following processes are added to instantiate_end.
    1. MgmtDriver transfers CA certificate files to all Master/Worker VMs if specified by request parameter.
    2. MgmtDriver configure connections to the Docker private registries on all Master/Worker VMs.

Scale-out operation for Kubernetes cluster Worker-nodes with MgmtDriver

Add the function to connect to Docker private registries based on spec "Support scaling Kubernetes Worker-nodes with Mgmtdriver"16.

The diagram below shows Scale-out VNF operation:

+---------------+
| Scale Request |
+--+------------+
   | 1. Scale VNF
   |    (Scale-out)
+----------------+------------+
|                v       VNFM |
|  +------------------+       |
|  |   Tacker-server  |       |
|  +-------+----------+       |
3. Kubernetes Cluster           |          |                  |
Installation                 |          v                  |
4. Configure connections        |  +-----------------------+  |
to Docker private registries |  |    +-------------+    |  |
+----------------------+--+----+ MgmtDriver  |    |  |
|                      |  |    +-------------+    |  |
+---------------------+--------+             |  |                       |  |
|                     |        |             |  |                       |  |
|  +----------+  +----+-----+  |             |  |                       |  |
|  |          |  |    v     |  | 2. Add      |  |    +-------------+    |  |
|  | +------+ |  | +------+ |  |    new VM   |  |    | OpenStack   |    |  |
|  | |Master| |  | |Worker| |<-+-------------+--+----+ InfraDriver |    |  |
|  | +------+ |  | +------+ |  |             |  |    +-------------+    |  |
|  |    VM    |  |    VM    |  |             |  |                       |  |
|  +----------+  +----------+  |             |  |                       |  |
|                +----------+  |             |  |                       |  |
|                | +------+ |  |             |  |                       |  |
|                | |Worker| |  |             |  |                       |  |
|                | +------+ |  |             |  |                       |  |
|                |    VM    |  |             |  |                       |  |
|                +----------+  |             |  |                       |  |
+------------------------------+             |  |    Tacker-conductor   |  |
+------------------------------+             |  +-----------------------+  |
|      Hardware Resources      |             |                             |
+------------------------------+             +-----------------------------+

VNFD for Scaling operation

No change from spec "Support scaling Kubernetes Worker-nodes with Mgmtdriver"17.

Request parameters for Scaling operation

No change from spec "Support scaling Kubernetes Worker-nodes with Mgmtdriver"18.

Sequence for Scale-out operation

seqdiag {

node_width = 80; edge_length = 100;

"Client" "Tacker-server" "Tacker-conductor" "VnfLcmDriver" "OpenStackInfraDriver" "Heat" "MgmtDriver" "VnfInstance(Tacker DB)" "RemoteCommandExecutor" "SFTPClient"

Client -> "Tacker-server"

[label = "POST /vnf_instances/{vnfInstanceId}/scale"];

Client <-- "Tacker-server"

[label = "Response 202 Accepted"];

"Tacker-server" -> "Tacker-conductor"

[label = "trigger asynchronous task"];

"Tacker-conductor" -> "VnfLcmDriver"

[label = "execute VnfLcmDriver"];

"VnfLcmDriver" -> "OpenStackInfraDriver"

[label = "execute OpenStackInfraDriver"];

"OpenStackInfraDriver" -> "Heat"

[label = "resource signal"];

"OpenStackInfraDriver" <-- "Heat"

[label = ""];

"OpenStackInfraDriver" -> "Heat"

[label = "update stack"];

"OpenStackInfraDriver" <-- "Heat"

[label = ""];

"VnfLcmDriver" <-- "OpenStackInfraDriver"

[label = ""];

"VnfLcmDriver" -> "MgmtDriver"

[label = "scale_end"];

"MgmtDriver" -> "VnfInstance(Tacker DB)"

[label = "get stack id"];

"MgmtDriver" <-- "VnfInstance(Tacker DB)"

[label = ""];

"MgmtDriver" -> "Heat"

[label = "get the new VM info created by scale out operation"];

"MgmtDriver" <-- "Heat"

[label = ""];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "install Kubernetes and join cluster on the new Worker VM"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"MgmtDriver" -> "VnfInstance(Tacker DB)"

[label = "get private_registry_connection_info of Instantiate VNF request"];

"MgmtDriver" <-- "VnfInstance(Tacker DB)"

[label = ""];

"MgmtDriver" -> "SFTPClient"

[label = "transfer CA certificate files to the new Worker VM if specified by Instantiate VNF request parameter"];

"MgmtDriver" <-- "SFTPClient"

[label = ""];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "configure connections to Docker private registries on the new Worker VM"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"VnfLcmDriver" <-- "MgmtDriver"

[label = ""];

"Tacker-conductor" <-- "VnfLcmDriver"

[label = ""];

}

The procedure consists of the following steps as illustrated in above sequence.

  1. Client sends a POST Scale VNF request.
  2. It is basically the same sequence as described in spec "Support scaling Kubernetes Worker-nodes with Mgmtdriver"19, except for following additional processes.
  3. The following processes are added to scale_end.
    1. MgmtDriver gets private_registry_connection_info of Instantiate VNF request from VnfInstance(Tacker DB).
    2. MgmtDriver transfers CA certificate files to the new Worker VM if specified by Instantiate VNF request parameter.
    3. MgmtDriver configure connections to Docker private registries on the new Worker VM.

Heal operation for the entire Kubernetes cluster with MgmtDriver

Add the function to connect to Docker private registries based on spec "Support Healing Kubernetes Master/Worker-nodes with Mgmtdriver"20.

The diagram below shows Heal VNF (entire Kubernetes cluster) operation:

+--------------+
| Heal Request |
+-+------------+
  | 1. Heal VNF
  |
+----------------+------------+
|                v       VNFM |
|  +-------------------+      |
|  |   Tacker-server   |      |
|  +---------+---------+      |
5. Kubernetes Cluster            |            |                |
Installation                  |            v                |
6. Configure connections         |  +-----------------------+  |
to Docker private registries  |  |    +-------------+    |  |
+-------------+---------------------+--+----+ MgmtDriver  |    |  |
|             |                     |  |    +-------------+    |  |
|             |                     |  |  2. Delete old        |  |
|             |                     |  |  Kubernetes cluster   |  |
|             |                     |  |  information          |  |
|             |                     |  |  7. Register new      |  |
+-------+-------------+--------+            |  |  Kubernetes cluster   |  |
|       |             |        |            |  |  information          |  |
|  +----+-----+  +----+-----+  |            |  |                       |  |
|  |    v     |  |    v     |  | 4. Create  |  |    +-------------+    |  |
|  | +------+ |  | +------+ |  | new VMs    |  |    | OpenStack   |    |  |
|  | |Master| |  | |Worker| |  |<-----------+--+----+ InfraDriver |    |  |
|  | +------+ |  | +------+ |  |            |  |    +---+---------+    |  |
|  |    VM    |  |    VM    |  |            |  |        |              |  |
|  +----------+  +----------+  |            |  |        |              |  |
|   Kubernetes cluster(new)    |            |  |        |              |  |
+------------------------------+            |  |        |              |  |
+------------------------------+            |  |        |              |  |
|  +----------+  +----------+  | 3. Delete  |  |        |              |  |
|  | +------+ |  | +------+ |  | failed VMs |  |        |              |  |
|  | |Master| |  | |Worker| |  |<-----------+--+--------+              |  |
|  | +------+ |  | +------+ |  |            |  |                       |  |
|  |    VM    |  |    VM    |  |            |  |                       |  |
|  +----------+  +----------+  |            |  |                       |  |
|   Kubernetes cluster(old)    |            |  |                       |  |
+------------------------------+            |  |   Tacker-conductor    |  |
+------------------------------+            |  +-----------------------+  |
|      Hardware Resources      |            |                             |
+------------------------------+            +-----------------------------+

VNFD for Healing operation

No change from spec "Support Healing Kubernetes Master/Worker-nodes with Mgmtdriver"21.

Request parameters for Healing operation

No change from spec "Support Healing Kubernetes Master/Worker-nodes with Mgmtdriver"22.

Sequence for Healing (entire Kubernetes cluster) operation

seqdiag {

node_width = 80; edge_length = 100;

"Client" "Tacker-server" "Tacker-conductor" "VnfLcmDriver" "OpenStackInfraDriver" "Heat" "MgmtDriver" "VnfInstance(Tacker DB)" "RemoteCommandExecutor" "SFTPClient" "NfvoPlugin"

Client -> "Tacker-server"

[label = "POST /vnf_instances/{vnfInstanceId}/heal"];

Client <-- "Tacker-server"

[label = "Response 202 Accepted"];

"Tacker-server" -> "Tacker-conductor"

[label = "trigger asynchronous task"];

"Tacker-conductor" -> "VnfLcmDriver"

[label = "execute VnfLcmDriver"];

"VnfLcmDriver" -> "MgmtDriver"

[label = "heal_start"];

"MgmtDriver" -> "NfvoPlugin"

[label = "delete the failed VIM information"];

"MgmtDriver" <-- "NfvoPlugin"

[label = ""];

"MgmtDriver" -> "VnfInstance(Tacker DB)"

[label = "Clear the old Kubernetes cluster information stored in the vim_connection_info of the VNF Instance"];

"MgmtDriver" <-- "VnfInstance(Tacker DB)"

[label = ""];

"VnfLcmDriver" <-- "MgmtDriver"

[label = ""];

"VnfLcmDriver" -> "OpenStackInfraDriver"

[label = "execute OpenStackInfraDriver"];

"OpenStackInfraDriver" -> "Heat"

[label = "delete stack"];

"OpenStackInfraDriver" <-- "Heat"

[label = ""];

"OpenStackInfraDriver" -> "Heat"

[label = "create stack"];

"OpenStackInfraDriver" <-- "Heat"

[label = ""];

"VnfLcmDriver" <-- "OpenStackInfraDriver"

[label = ""];

"VnfLcmDriver" -> "MgmtDriver"

[label = "heal_end"];

"MgmtDriver" -> "VnfInstance(Tacker DB)"

[label = "get stack id"];

"MgmtDriver" <-- "VnfInstance(Tacker DB)"

[label = ""];

"MgmtDriver" -> "Heat"

[label = "get SSH/SFTP IP addresses and Kubernetes addresses using stack id"];

"MgmtDriver" <-- "Heat"

[label = ""];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "install Kubernetes and create cluster on new Master/Worker VMs"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "get identification token from Kubernetes cluster"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"MgmtDriver" -> "VnfInstance(Tacker DB)"

[label = "get private_registry_connection_info of Instantiate VNF request"];

"MgmtDriver" <-- "VnfInstance(Tacker DB)"

[label = ""];

"MgmtDriver" -> "SFTPClient"

[label = "transfer CA certificate files to all Master/Worker VMs if specified by Instantiate VNF request parameter"];

"MgmtDriver" <-- "SFTPClient"

[label = ""];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "configure connections to Docker private registries on all Master/Worker VMs"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"MgmtDriver" -> "NfvoPlugin"

[label = "register Kubernetes VIM to tacker"];

"MgmtDriver" <-- "NfvoPlugin"

[label = ""]

"MgmtDriver" -> "VnfInstance(Tacker DB)"

[label = "append Kubernetes cluster VIM info to VimConnectionInfo"]

"MgmtDriver" <-- "VnfInstance(Tacker DB)"

[label = ""]

"VnfLcmDriver" <-- "MgmtDriver"

[label = ""];

"Tacker-conductor" <-- "VnfLcmDriver"

[label = ""];

}

The procedure consists of the following steps as illustrated in above sequence.

  1. Client sends a POST Heal VNF request.
  2. It is basically the same sequence as described in spec "Support Healing Kubernetes Master/Worker-nodes with Mgmtdriver"23, except for following additional processes.
  3. The following processes are added to heal_end.
    1. MgmtDriver gets private_registry_connection_info of Instantiate VNF request from VnfInstance(Tacker DB).
    2. MgmtDriver transfers CA certificate files to all new Master/Worker VMs if specified by Instantiate VNF request parameter.
    3. MgmtDriver configure connections to Docker private registries on all new Master/Worker VMs.

Heal operation for a single node (Master/Worker) in Kubernetes cluster with MgmtDriver

Add the function to connect to Docker private registries based on spec "Support Healing Kubernetes Master/Worker-nodes with Mgmtdriver"24.

The diagram below shows Heal VNF (single node) operation:

+--------------+
| Heal Request |
+-+------------+
  | 1. Heal VNF
  |
+----------------+------------+
|                v       VNFM |
|  +-------------------+      |
|  |   Tacker-server   |      |
|  +---------+---------+      |
5. Kubernetes Cluster           |            |                |
Installation                 |            v                |
6. Configure connections        |  +-----------------------+  |
to Docker private registries |  |    +-------------+    |  |
+------------------------+--+----+ MgmtDriver  |    |  |
|                        |  |    +------------++    |  |
|                        |  |  2. Evacuating  |     |  |
|                        |  |  Pods(Worker)   |     |  |
+------------+------------+           |  |  and removing   |     |  |
|            |            |           |  |  failed node    |     |  |
|            |            |           |  |  from cluster   |     |  |
|            |            |           |  | (Master/Worker) |     |  |
|  +---------+---------+  |           |  |                 |     |  |
|  |         v         |  | 4. Create |  | +-------------+ |     |  |
|  | +---------------+ |  | new VM    |  | | OpenStack   | |     |  |
|  | | Master/Worker | |<-+-----------+--+-+ InfraDriver | |     |  |
|  | +---------------+ |  |           |  | +------+------+ |     |  |
|  |         VM        |  |           |  |        |        |     |  |
|  +-------------------+  |           |  |        |        |     |  |
|  +-------------------+  | 3. Delete |  |        |        |     |  |
|  | +---------------+ |  | failed VM |  |        |        |     |  |
|  | | Master/Worker | |<-+-----------+--+--------+        |     |  |
|  | +---------------+ |  |           |  |                 |     |  |
|  |         VM        |<-+-----------+--+-----------------+     |  |
|  +-------------------+  |           |  |                       |  |
+-------------------------+           |  |   Tacker-conductor    |  |
+-------------------------+           |  +-----------------------+  |
|   Hardware Resources    |           |                             |
+-------------------------+           +-----------------------------+

VNFD for Healing operation

No change from spec "Support Healing Kubernetes Master/Worker-nodes with Mgmtdriver"25.

Request parameters for Healing operation

No change from spec "Support Healing Kubernetes Master/Worker-nodes with Mgmtdriver"26.

Sequence for Healing (single node) operation

seqdiag {

node_width = 80; edge_length = 100;

"Client" "Tacker-server" "Tacker-conductor" "VnfLcmDriver" "OpenStackInfraDriver" "Heat" "MgmtDriver" "VnfInstance(Tacker DB)" "RemoteCommandExecutor" "SFTPClient"

Client -> "Tacker-server"

[label = "POST /vnf_instances/{vnfInstanceId}/heal"];

Client <-- "Tacker-server"

[label = "Response 202 Accepted"];

"Tacker-server" -> "Tacker-conductor"

[label = "trigger asynchronous task"];

"Tacker-conductor" -> "VnfLcmDriver"

[label = "execute VnfLcmDriver"];

"VnfLcmDriver" -> "MgmtDriver"

[label = "heal_start"];

"MgmtDriver" -> "VnfInstance(Tacker DB)"

[label = "get the vim connection info"];

"MgmtDriver" <-- "VnfInstance(Tacker DB)"

[label = ""];

"MgmtDriver" -> "MgmtDriver"

[label = "get the vm info to be deleted by the request data"];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "(Worker only)evacuating the pod running on the failed VM to another worker node"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "remove the failed node from cluster"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "(Master only)remove the failed node from HA proxy"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"VnfLcmDriver" <-- "MgmtDriver"

[label = ""];

"VnfLcmDriver" -> "OpenStackInfraDriver"

[label = "execute OpenStackInfraDriver"];

"OpenStackInfraDriver" -> "Heat"

[label = "resource mark unhealthy"];

"OpenStackInfraDriver" <-- "Heat"

[label = ""];

"OpenStackInfraDriver" -> "Heat"

[label = "update stack"];

"OpenStackInfraDriver" <-- "Heat"

[label = ""];

"VnfLcmDriver" <-- "OpenStackInfraDriver"

[label = ""];

"VnfLcmDriver" -> "MgmtDriver"

[label = "heal_end"];

"MgmtDriver" -> "VnfInstance(Tacker DB)"

[label = "get stack id"];

"MgmtDriver" <-- "VnfInstance(Tacker DB)"

[label = ""];

"MgmtDriver" -> "Heat"

[label = "get the new VM info created by Heal operation"];

"MgmtDriver" <-- "Heat"

[label = ""];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "install Kubernetes and join cluster on the new VM"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "(Master only)Changes HA proxy configuration"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"MgmtDriver" -> "VnfInstance(Tacker DB)"

[label = "get private_registry_connection_info of Instantiate VNF request"];

"MgmtDriver" <-- "VnfInstance(Tacker DB)"

[label = ""];

"MgmtDriver" -> "SFTPClient"

[label = "transfer CA certificate files to the new VM if specified by Instantiate VNF request parameter"];

"MgmtDriver" <-- "SFTPClient"

[label = ""];

"MgmtDriver" -> "RemoteCommandExecutor"

[label = "configure connections to Docker private registries on the new VM"];

"MgmtDriver" <-- "RemoteCommandExecutor"

[label = ""];

"VnfLcmDriver" <-- "MgmtDriver"

[label = ""];

"Tacker-conductor" <-- "VnfLcmDriver"

[label = ""];

}

The procedure consists of the following steps as illustrated in above sequence.

  1. Client sends a POST Heal VNF request.
  2. It is basically the same sequence as described in spec "Support Healing Kubernetes Master/Worker-nodes with Mgmtdriver"27, except for following additional processes.
  3. The following processes are added to heal_end.
    1. MgmtDriver gets private_registry_connection_info of Instantiate VNF request from VnfInstance(Tacker DB).
    2. MgmtDriver transfers CA certificate files to the new Master/Worker VM if specified by Instantiate VNF request parameter.
    3. MgmtDriver configure connections to Docker private registries on the new Master/Worker VM.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

Masaki Ueno <masaki.ueno.up@hco.ntt.co.jp>

Other contributors:

Yoshito Ito <yoshito.itou.dr@hco.ntt.co.jp>

Yoshiyuki Katada <katada.yoshiyuk@fujitsu.com>

Ayumu Ueha <ueha.ayumu@fujitsu.com>

Liang Lu <lu.liang@fujitsu.com>

Work Items

  • Provide the following sample scripts executed by MgmtDriver:

    • For Docker private registry VNF:

      Install the Docker private registry in instantiate_end and heal_end.

    • For Kubernetes cluster VNF:

      Add processes to connect to Docker private registries based on the Wallaby released Kubernetes cluster script.

  • Add new unit and functional tests.

Dependencies

LCM operations for the Kubernetes cluster depend on the following specifications:

  • Instantiate operation for the Kubernetes cluster

    Depends on spec "Support deploying Kubernetes cluster with MgmtDriver"28.

  • Scale operation for the Kubernetes cluster

    Depends on spec "Support scaling Kubernetes Worker-nodes with Mgmtdriver"29.

  • Heal operation for the Kubernetes cluster

    Depends on spec "Support Healing Kubernetes Master/Worker-nodes with Mgmtdriver"30.

Testing

Unit and functional tests will be added to cover cases required in the spec.

Documentation Impact

Complete user guide will be added to explain how to deploy Docker private registry VNF and how to connect to Docker private registry from Kubernetes cluster.

References

  1. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-cluster.html↩︎

  2. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-cluster.html↩︎

  3. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-cluster.html↩︎

  4. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-scale.html↩︎

  5. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-heal.html↩︎

  6. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-heal.html↩︎

  7. https://www.etsi.org/deliver/etsi_gs/NFV-SOL/001_099/003/02.06.01_60/gs_NFV-SOL003v020601p.pdf↩︎

  8. https://specs.openstack.org/openstack/tacker-specs/specs/ussuri/etsi-nfv-sol-rest-api-for-VNF-deployment.html↩︎

  9. https://specs.openstack.org/openstack/tacker-specs/specs/ussuri/etsi-nfv-sol-rest-api-for-VNF-deployment.html↩︎

  10. https://specs.openstack.org/openstack/tacker-specs/specs/ussuri/etsi-nfv-sol-rest-api-for-VNF-deployment.html↩︎

  11. https://specs.openstack.org/openstack/tacker-specs/specs/ussuri/etsi-nfv-sol-rest-api-for-VNF-deployment.html↩︎

  12. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-cluster.html↩︎

  13. https://docs.openstack.org/tacker/latest/user/mgmt_driver_deploy_k8s_usage_guide.html↩︎

  14. https://docs.openstack.org/tacker/latest/user/mgmt_driver_deploy_k8s_usage_guide.html↩︎

  15. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-cluster.html↩︎

  16. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-scale.html↩︎

  17. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-scale.html↩︎

  18. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-scale.html↩︎

  19. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-scale.html↩︎

  20. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-heal.html↩︎

  21. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-heal.html↩︎

  22. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-heal.html↩︎

  23. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-heal.html↩︎

  24. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-heal.html↩︎

  25. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-heal.html↩︎

  26. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-heal.html↩︎

  27. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-heal.html↩︎

  28. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-cluster.html↩︎

  29. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-scale.html↩︎

  30. https://specs.openstack.org/openstack/tacker-specs/specs/wallaby/mgmt-driver-for-k8s-heal.html↩︎