40 KiB
How to use Mgmt Driver for deploying Kubernetes Cluster with Private Registries
Overview
It is supported downloading image files from Docker public registry
in the Kubernetes Cluster built with the user guide How
to use Mgmt Driver for deploying Kubernetes Cluster, but not
supported from Docker private registry. In this user guide, we provide a
way to support the use of Docker private registry for Kubernetes
cluster. The Docker private registry can either be created in the Tacker
or created outside. If you want to use Tacker to create a Docker private
registry, we provide a sample user script run by Mgmt Driver to install
a Docker private registry. You can refer to chapter Install Docker Private Registry
for this usage. If you
want to use a private registry created outside of the tacker, please
create it and then add the configuration of your private registry into
the parameters of instantiation request body of Kubernetes cluster.
Install Docker Private Registry
1. Preparations
The preparations of installing Docker private registry is the same as the one in How to use Mgmt Driver for deploying Kubernetes Cluster. You can refer to it for how to set OpenStack configuration, how to download ubuntu image and how to register Mgmt Driver and usage of VNF Package.
The sample structure of VNF Package is shown below.
Note
You can also find them in the samples/mgmt_driver/kubernetes/private_registry_vnf_package/ directory of the tacker.
The directory structure:
- TOSCA-Metadata/TOSCA.meta
- Definitions/
- Files/images/
- Scripts/
!----TOSCA-Metadata
!---- TOSCA.meta
!----Definitions
!---- etsi_nfv_sol001_common_types.yaml
!---- etsi_nfv_sol001_vnfd_types.yaml
!---- sample_vnfd_df_simple.yaml
!---- sample_vnfd_top.yaml
!---- sample_vnfd_types.yaml
!----Files
!---- images
!---- ubuntu-20.04-server-cloudimg-amd64.img
!----Scripts
!---- private_registry_mgmt.py
2. Deploy Docker Private Registry VNF
Docker private registry can be installed and set up in
instantiate_end
operation, which allows you to execute any
scripts after its instantiation, and it's enabled with Mgmt Driver
support.
You must create the parameter file which is used to instantiate correctly. The following are the methods of creating the parameter file and CLI commands of OpenStack.
1. Create the Parameter File
Create a private_registry_param_file.json
file with the
following format. This is the file that defines the parameters for an
instantiate request. These parameters will be set in the body of the
instantiate request.
Required parameter:
- flavourId
- additionalParams
Optional parameters:
- instantiationLevelId
- extVirtualLinks
- extManagedVirtualLinks
- vimConnectionInfo
In this guide, the VMs need to have extCPs to be accessed via SSH by
Tacker. Therefore, extVirtualLinks
parameter is required.
You can skip vimConnectionInfo
only when you have the
default VIM described in cli-legacy-vim.
Explanation of the parameters for deploying Docker private registry VNF
For deploying Docker private registry VNF, you must set the
private_registry_installation_param
key in
additionalParams. The KeyValuePairs is shown in table below:
parameter | data type | description | required/optional |
---|---|---|---|
ssh_cp_name | string | CP name that MgmtDriver uses when SSH/SFTP access to the Private Registry VM. | required |
ssh_username | string | User name that MgmtDriver uses when SSH/SFTP access to the Private Registry VM. | required |
ssh_password | string | User password that MgmtDriver uses when SSH/SFTP access to the Private Registry VM. | required |
image_path | string | Path of the Docker image file in the VNF Package for the Private Registry container to run on Docker. If this attribute is omitted, the image for the Private Registry container is pulled from the Docker public registry. If the Private Registry VM is unable to connect to the Docker public registry, put the file created using "docker save" command into the VNF Package and specify the path of the file in this attribute. | optional |
port_no | string | The default value is 5000. TCP port number provides the private registry service. | optional |
proxy | dict | Information for proxy setting on VM | optional |
parameter | data type | description | required/optional |
---|---|---|---|
http_proxy | string | Http proxy server address | optional |
https_proxy | string | Https proxy server address | optional |
no_proxy | string | User-customized, proxy server-free IP address or segment | optional |
private_registry_param_file.json
{
"flavourId": "simple",
"extVirtualLinks": [{
"id": "net0",
"resourceId": "f0c82461-36b5-4d86-8322-b0bc19cda65f", #Set the uuid of the network to use
"extCps": [{
"cpdId": "CP1",
"cpConfig": [{
"cpProtocolData": [{
"layerProtocol": "IP_OVER_ETHERNET"
}]
}]
}]
}],
"additionalParams": {
"private_registry_installation_param": {
"ssh_cp_name": "CP1",
"ssh_username": "ubuntu",
"ssh_password": "ubuntu",
"proxy": {
"http_proxy": "http://user1:password1@host1:port1",
"https_proxy": "https://user2:password2@host2:port2",
"no_proxy": "192.168.246.0/24,10.0.0.1"
}
}
},
"vimConnectionInfo": [{
"id": "8a3adb69-0784-43c7-833e-aab0b6ab4470",
"vimId": "8d8373fe-6977-49ff-83ac-7756572ed186", #Set the uuid of the VIM to use
"vimType": "openstack"
}]
}
2. Execute the Instantiation Operations
Execute the following CLI command to instantiate the VNF instance.
Create VNF with VNFD ID:
$ openstack vnflcm create c1bb0ce7-ebca-4fa7-95ed-4840d70a118f
+--------------------------+---------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------+---------------------------------------------------------------------------------------------+
| ID | f93ed99c-e2f2-4f18-8377-37a171ea344f |
| Instantiation State | NOT_INSTANTIATED |
| Links | { |
| | "self": { |
| | "href": "/vnflcm/v1/vnf_instances/f93ed99c-e2f2-4f18-8377-37a171ea344f" |
| | }, |
| | "instantiate": { |
| | "href": "/vnflcm/v1/vnf_instances/f93ed99c-e2f2-4f18-8377-37a171ea344f/instantiate" |
| | } |
| | } |
| VNF Instance Description | None |
| VNF Instance Name | vnf-f93ed99c-e2f2-4f18-8377-37a171ea344f |
| VNF Product Name | Sample Private registry VNF |
| VNF Provider | Company |
| VNF Software Version | 1.0 |
| VNFD ID | c1bb0ce7-ebca-4fa7-95ed-4840d70a118f |
| VNFD Version | 1.0 |
| vnfPkgId | |
+--------------------------+---------------------------------------------------------------------------------------------+
Instantiate VNF with VNF ID:
$ openstack vnflcm instantiate f93ed99c-e2f2-4f18-8377-37a171ea344f ./private_registry_param_file.json
Instantiate request for VNF Instance f93ed99c-e2f2-4f18-8377-37a171ea344f has been accepted.
Check instantiation state:
$ openstack vnflcm show f93ed99c-e2f2-4f18-8377-37a171ea344f
+--------------------------+-------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------+-------------------------------------------------------------------------------------------------+
| ID | f93ed99c-e2f2-4f18-8377-37a171ea344f |
| Instantiated Vnf Info | { |
| | "flavourId": "simple", |
| | "vnfState": "STARTED", |
| | "extCpInfo": [ |
| | { |
| | "id": "187cc96a-7577-4156-8b0a-efcda82d56fc", |
| | "cpdId": "CP1", |
| | "extLinkPortId": null, |
| | "associatedVnfcCpId": "52431660-dde4-49de-99a4-9d593b17b9c4", |
| | "cpProtocolInfo": [ |
| | { |
| | "layerProtocol": "IP_OVER_ETHERNET" |
| | } |
| | ] |
| | } |
| | ], |
| | "extVirtualLinkInfo": [ |
| | { |
| | "id": "net0", |
| | "resourceHandle": { |
| | "vimConnectionId": null, |
| | "resourceId": "1642ac54-642c-407c-9c7d-e94c55ba5d33", |
| | "vimLevelResourceType": null |
| | } |
| | } |
| | ], |
| | "vnfcResourceInfo": [ |
| | { |
| | "id": "52431660-dde4-49de-99a4-9d593b17b9c4", |
| | "vduId": "PrivateRegistryVDU", |
| | "computeResource": { |
| | "vimConnectionId": "c3369b54-e376-4423-bb61-afd255900fea", |
| | "resourceId": "f93edf04-07ac-410e-96aa-7fd64774f951", |
| | "vimLevelResourceType": "OS::Nova::Server" |
| | }, |
| | "storageResourceIds": [], |
| | "vnfcCpInfo": [ |
| | { |
| | "id": "8355de52-61ec-495e-ac81-537d0c676915", |
| | "cpdId": "CP1", |
| | "vnfExtCpId": null, |
| | "vnfLinkPortId": "2b7fa3dc-35a8-4d46-93ba-0c11f39ccced", |
| | "cpProtocolInfo": [ |
| | { |
| | "layerProtocol": "IP_OVER_ETHERNET" |
| | } |
| | ] |
| | } |
| | ] |
| | } |
| | ], |
| | "vnfVirtualLinkResourceInfo": [ |
| | { |
| | "id": "245b35c0-7cf1-4470-87c7-5927eb0ad2ee", |
| | "vnfVirtualLinkDescId": "net0", |
| | "networkResource": { |
| | "vimConnectionId": null, |
| | "resourceId": "1642ac54-642c-407c-9c7d-e94c55ba5d33", |
| | "vimLevelResourceType": "OS::Neutron::Net" |
| | }, |
| | "vnfLinkPorts": [ |
| | { |
| | "id": "2b7fa3dc-35a8-4d46-93ba-0c11f39ccced", |
| | "resourceHandle": { |
| | "vimConnectionId": "c3369b54-e376-4423-bb61-afd255900fea", |
| | "resourceId": "2eb5d67b-fe24-40ca-b25a-8c4e47520aee", |
| | "vimLevelResourceType": "OS::Neutron::Port" |
| | }, |
| | "cpInstanceId": "8355de52-61ec-495e-ac81-537d0c676915" |
| | } |
| | ] |
| | } |
| | ], |
| | "vnfcInfo": [ |
| | { |
| | "id": "49330b17-bb00-44df-a1e1-34ea0cd09307", |
| | "vduId": "PrivateRegistryVDU", |
| | "vnfcState": "STARTED" |
| | } |
| | ], |
| | "additionalParams": { |
| | "private_registry_installation_param": { |
| | "proxy": { |
| | "http_proxy": "http://user1:password1@host1:port1", |
| | "https_proxy": "https://user2:password2@host2:port2", |
| | "no_proxy": "192.168.246.0/24,10.0.0.1" |
| | }, |
| | "ssh_cp_name": "CP1", |
| | "ssh_username": "ubuntu", |
| | "ssh_password": "ubuntu" |
| | } |
| | } |
| | } |
| Instantiation State | INSTANTIATED |
| Links | { |
| | "self": { |
| | "href": "/vnflcm/v1/vnf_instances/f93ed99c-e2f2-4f18-8377-37a171ea344f" |
| | }, |
| | "terminate": { |
| | "href": "/vnflcm/v1/vnf_instances/f93ed99c-e2f2-4f18-8377-37a171ea344f/terminate" |
| | }, |
| | "heal": { |
| | "href": "/vnflcm/v1/vnf_instances/f93ed99c-e2f2-4f18-8377-37a171ea344f/heal" |
| | }, |
| | "changeExtConn": { |
| | "href": "/vnflcm/v1/vnf_instances/f93ed99c-e2f2-4f18-8377-37a171ea344f/change_ext_conn" |
| | } |
| | } |
| VIM Connection Info | [ |
| | { |
| | "id": "8a3adb69-0884-43c7-833e-aab0b6ab4470", |
| | "vimId": "c3369b54-e376-4423-bb61-afd255900fea", |
| | "vimType": "openstack", |
| | "interfaceInfo": {}, |
| | "accessInfo": {} |
| | } |
| | ] |
| VNF Instance Description | None |
| VNF Instance Name | vnf-f93ed99c-e2f2-4f18-8377-37a171ea344f |
| VNF Product Name | Sample Private registry VNF |
| VNF Provider | Company |
| VNF Software Version | 1.0 |
| VNFD ID | c1bb0ce7-ebca-4fa7-95ed-4840d70a118f |
| VNFD Version | 1.0 |
| vnfPkgId | |
+--------------------------+-------------------------------------------------------------------------------------------------+
3. Heal Docker Private Registry VNF
According to NFV-SOL001 v2.6.1, heal_start and heal_end operation allows users to execute any scripts in the heal operation, and healing operations on the private registry server is supported with Mgmt Driver.
After instantiating Docker private registry VNF, if it is not running properly, you can heal it. The following are the methods of creating the parameter file and CLI commands of OpenStack.
Note
Since the heal entire operation will cause the server's ip to change, user should avoid using it when the Docker private registry service has already been used.
Note
The image information registered in the Docker private registry that is the target of Heal is not retained after Heal.
1. Create the Parameter File
The following is heal parameter to "POST /vnf_instances/{id}/heal" as
HealVnfRequest
data type. It is not the same in SOL002 and
SOL003.
Attribute name | Parameter description |
---|---|
vnfcInstanceId | User specify heal target, user can know "vnfcInstanceId" by
InstantiatedVnfInfo.vnfcResourceInfo that contained in the
response of "GET /vnf_instances/{id}". |
cause | Not needed |
additionalParams | Not needed |
healScript | Not needed |
Attribute name | Parameter description |
---|---|
cause | Not needed |
additionalParams | Not needed |
If the vnfcInstanceId parameter is null, this means that healing operation is required for the entire Kubernetes cluster, which is the case in SOL003.
Following is a sample of healing request body for SOL002:
{
"vnfcInstanceId": "52431660-dde4-49de-99a4-9d593b17b9c4"
}
Note
In chapter of Deploy Docker Private Registry VNF
, the
result of VNF instance instantiated has shown in CLI command openstack vnflcm show VNF INSTANCE ID.
You can get the vnfcInstanceId from
Instantiated Vnf Info
in above result. The
vnfcResourceInfo.id
is vnfcInstanceId.
2. Execute Heal Operations
1. Heal Docker Private Registry VNF (Specify VNFC)
When healing specified with VNFC instances, Heat APIs are called from Tacker.
- stack resource mark unhealthy
- stack update
To confirm whether healing is successful, execute Heat CLI command and check physical_resource_id and resource_status of private registry VNF before and after healing.
Note
Note that 'vnfc-instance-id' managed by Tacker and
'physical-resource-id' managed by Heat are different. The
physical_resource_id
is the same as
vnfcResourceInfo.computeResource.resourceId
.
Private registry VNF information before healing:
$ openstack stack resource list vnf-f93ed99c-e2f2-4f18-8377-37a171ea344f -n 2 \
--filter type=OS::Nova::Server -c resource_name -c physical_resource_id -c \
resource_type -c resource_status
+--------------------+--------------------------------------+------------------+-----------------+
| resource_name | physical_resource_id | resource_type | resource_status |
+--------------------+--------------------------------------+------------------+-----------------+
| PrivateRegistryVDU | f93edf04-07ac-410e-96aa-7fd64774f951 | OS::Nova::Server | CREATE_COMPLETE |
+--------------------+--------------------------------------+------------------+-----------------+
We heal the private registry VNF with
physical_resource_id
f93edf04-07ac-410e-96aa-7fd64774f951
, its
vnfc_instance_id
is
52431660-dde4-49de-99a4-9d593b17b9c4
.
Healing private registry vnf of the vnf_instance:
$ openstack vnflcm heal f93ed99c-e2f2-4f18-8377-37a171ea344f --vnfc-instance 52431660-dde4-49de-99a4-9d593b17b9c4
Heal request for VNF Instance f93ed99c-e2f2-4f18-8377-37a171ea344f has been accepted.
private registry vnf information after healing:
$ openstack stack resource list vnf-c5215213-af4b-4080-95ab-377920474e1a -n 2 \
--filter type=OS::Nova::Server -c resource_name -c physical_resource_id -c \
resource_type -c resource_status
+--------------------+--------------------------------------+------------------+-----------------+
| resource_name | physical_resource_id | resource_type | resource_status |
+--------------------+--------------------------------------+------------------+-----------------+
| PrivateRegistryVDU | c8a67180-f49b-492c-a2a2-1ac668a80453 | OS::Nova::Server | CREATE_COMPLETE |
+--------------------+--------------------------------------+------------------+-----------------+
2. Heal Docker Private Registry VNF (Entire Heal)
When healing of the entire VNF, the following APIs are executed from Tacker to Heat.
- stack delete
- stack create
1. Execute Heat CLI command and check 'ID' and 'Stack Status' of the stack before and after healing.
2. All the information of Private Registry VNF will be changed.
This is to confirm that stack 'ID' has changed before and after healing.
Stack information before healing:
$ openstack stack list -c 'ID' -c 'Stack Name' -c 'Stack Status'
+--------------------------------------+---------------------------------------------+-----------------+
| ID | Stack Name | Stack Status |
+--------------------------------------+---------------------------------------------+-----------------+
| fb03d2f0-3bd1-4382-a303-b7619484a4fa | vnf-f93ed99c-e2f2-4f18-8377-37a171ea344f | CREATE_COMPLETE |
+--------------------------------------+---------------------------------------------+-----------------+
Healing execution of the entire VNF:
$ openstack vnflcm heal f93ed99c-e2f2-4f18-8377-37a171ea344f
Heal request for VNF Instance f93ed99c-e2f2-4f18-8377-37a171ea344f has been accepted.
Stack information after healing:
$ openstack stack list -c 'ID' -c 'Stack Name' -c 'Stack Status'
+--------------------------------------+---------------------------------------------+-----------------+
| ID | Stack Name | Stack Status |
+--------------------------------------+---------------------------------------------+-----------------+
| 98ef6003-5422-4b04-bfc8-d56614d23fcc | vnf-f93ed99c-e2f2-4f18-8377-37a171ea344f | CREATE_COMPLETE |
+--------------------------------------+---------------------------------------------+-----------------+
Deploy Kubernetes Cluster with Docker Private Registry
1. Instantiate Kubernetes Cluster
The VNF Package for Kubernetes Cluster VNF used here has no change with the one used in How to use Mgmt Driver for deploying Kubernetes Cluster.
To use Docker private registry for Kubernetes Cluster, the operation
steps and methods of instantiating are the same as those in Deploy
Kubernetes Cluster in How to use Mgmt Driver
for deploying Kubernetes Cluster. The difference is the parameter
file. The following attributes should be added into k8s_cluster_installation_param of
additionalParams
described in
Deploy Kubernetes Cluster
.
parameter | data type | description | required/optional |
---|---|---|---|
connection_type | string | Type of connection. Set one of the following values. 0 : HTTP, 1 : HTTPS. Set to 0 if connecting to the deployed Docker private registry VNF or a Docker private registry outside of Tacker over HTTP. Set to 1 if connecting to a Docker private registry outside of Tacker over HTTPS. | required |
server | string | Server name of the Docker private registry to connect to. For example, "192.168.0.10:5000" | required |
username | string | Username to log in to the Docker private registry. | optional |
password | string | Password to log in to the Docker private registry. | optional |
certificate_path | string | The path of the CA certificate file to use for HTTPS connection. | optional |
hosts_string | string | String to add to /etc/hosts. The base Kubernetes cluster environment does not have a DNS server and must be added to /etc/hosts. The value consists of "<IP address> <FQDN>". For example, "192.168.0.20 registry.example.com". | optional |
The private_registry_kubernetes_param_file.json
is shown
below.
private_registry_kubernetes_param_file.json
{
"flavourId": "simple",
"vimConnectionInfo": [{
"id": "3cc2c4ff-525c-48b4-94c9-29247223322f",
"vimId": "05ef7ca5-7e32-4a6b-a03d-52f811f04496", #Set the uuid of the VIM to use
"vimType": "openstack"
}],
"additionalParams": {
"k8s_cluster_installation_param": {
"script_path": "Scripts/install_k8s_cluster.sh",
"vim_name": "kubernetes_vim",
"master_node": {
"aspect_id": "master_instance",
"ssh_cp_name": "masterNode_CP1",
"nic_cp_name": "masterNode_CP1",
"username": "ubuntu",
"password": "ubuntu",
"pod_cidr": "192.168.0.0/16",
"cluster_cidr": "10.199.187.0/24",
"cluster_cp_name": "vip_CP"
},
"worker_node": {
"aspect_id": "worker_instance",
"ssh_cp_name": "workerNode_CP2",
"nic_cp_name": "workerNode_CP2",
"username": "ubuntu",
"password": "ubuntu"
},
"proxy": {
"http_proxy": "http://user1:password1@host1:port1",
"https_proxy": "https://user2:password2@host2:port2",
"no_proxy": "192.168.246.0/24,10.0.0.1",
"k8s_node_cidr": "10.10.0.0/24"
},
"private_registry_connection_info": [
{
"connection_type": "0",
"server": "192.168.246.10:5000" #Set the server of Docker private registry to use
}
]
},
"lcm-operation-user-data": "./UserData/k8s_cluster_user_data.py",
"lcm-operation-user-data-class": "KubernetesClusterUserData"
},
"extVirtualLinks": [{
"id": "net0_master",
"resourceId": "71a3fbd1-f31e-4c2c-b0e2-26267d64a9ee", #Set the uuid of the network to use
"extCps": [{
"cpdId": "masterNode_CP1",
"cpConfig": [{
"cpProtocolData": [{
"layerProtocol": "IP_OVER_ETHERNET"
}]
}]
}]
}, {
"id": "net0_worker",
"resourceId": "71a3fbd1-f31e-4c2c-b0e2-26267d64a9ee", #Set the uuid of the network to use
"extCps": [{
"cpdId": "workerNode_CP2",
"cpConfig": [{
"cpProtocolData": [{
"layerProtocol": "IP_OVER_ETHERNET"
}]
}]
}]
}]
}
2. Scale and Heal Operation
For users, other operations such as scale and heal are the same as the ones in How to use Mgmt Driver for deploying Kubernetes Cluster.
Limitations
- Heal entire operation will cause the server's ip to change, user should avoid using it when the Docker private registry service is in use.
- The image information registered in the Docker private registry that is the target of Heal is not retained after Heal.