The following is a summary of the fixes for this patch.
* The description of k8s environment construction in kuryr-kubernetes
was changed to the description of using devstack-plugin-container.
* Updated k8s support version to 1.30.
* Updated helm support version to 3.15.
This patch also contains the following fixes:
coredns pods not in RUNNING state:
- modified file: roles/restart-kubelet-service/tasks/main.yaml
- The coredns pods did not go into RUNNING state after reconfiguring
the "cni0" interface. Restarting the coredns pods fixed the problem.
Change-Id: Id6d9b3279780a88b15c38f008d1e68e1ae466976
This patch replaces the "kuryr-kubernetes" handling used by tacker's
FT to build the k8s environment with "devstack-plugin-container".
Also, with the update of devstack-plugin-container, k8s, cri-o and
helm will be upgraded.
k8s: 1.26.8 -> 1.30.5
crio: 1.26 -> 1.30.5
helm: 3.11.3 -> 3.15.4
The following is a summary of the fixes in this patch.
* Remove plugins and settings related to kuryr-kubernetes
* Rename parameters with "kuryr"
* Modify devstack-plugin-container to be used in FT k8s environment
build
* Add parameters required by devstack-plugin-container
Also, the following is a list of problems that occurred when setting
up the k8s environment with devstack-plugin-container and how to fix
them.
Cannot get bearer_token value:
- modified file: roles/setup-default-vim/tasks/main.yaml
- The task "Get admin token from described secret" of the Ansible
role "setup-default-vim" failed to obtain the value of
bearer_token, which is set as a parameter when creating vim,
causing an error. Retrying to obtain token fixed the problem.
Unknown error in "Create clusterrolebinding on k8s server" task:
- modified file: roles/setup-k8s-nodes/tasks/main.yaml
- In task "Create clusterrolebinding on k8s server" in Ansible role
"setup-k8s-oidc", `failed to download openapi: unknown;` error
occurred. The cause was that the pod status of kube-apiserver was
"Pending" after executing the previous "Wait for k8s apiserver to
restart" task. The error was fixed by waiting for the Pod status
to reach the "Running" state.
"cni0" is not assigned the intended IP address:
- added file: roles/restart-kubelet-service/tasks/main.yaml
- When using devstack-plugin-container to create a k8s environment
and deploy a Pod, the Pod deployment fails with the error `network:
failed to set bridge addr: "cni0" already has an IP address
different from 10.x.x.x`. Removing the associated interface and
restarting the service cleared the error.
Depends-On: https://review.opendev.org/c/openstack/devstack-plugin-container/+/926709
Change-Id: I596a2339f6a3c78fee99b92d7bfb65a6b0244901
This patch obsoletes Legacy APIs excluding VIM feature.
And this patch mainly contains the following changes:
* Drop the implementation/db of NS and VNFFG functions.
* Remove the implementation that was used only by Legacy VNFM features.
* Remove the following components used only by Legacy features.
* ceilometer
* fenix
* blazar
* Drop the functional test jobs for Legacy features.
* Delete the unnecessary files that were used in Legacy tests.
Implements: blueprint deprecate-legacy-apis
Change-Id: I76ba79f42cf7c3f04c48a87de4ae893f2d53e467
Previously there are a lot of warning messages of setuptools
in the log files of tox jobs. This is mainly caused by test
samples because these are in python codes directories.
This patch moves test samples from under {tacker_root}/tacker/
to under {tacker_root}/samples/.
This patch reduces log messages drastically. For example,
the log file of py38 tox job decreased about 34,000 lines.
Change-Id: I8187ef892c6fe8be323fa5cc20969d298843f1ea
This patch removes documentation for the following legacy features.
- Extensions
- Virtualized Network Function Descriptors (VNFDs)
- Virtualized Network Functions (VNFs)
- Events
- VNF Forwarding Graph Descriptors (VNFFGDs)
- VNF Forwarding Graphs (VNFFGs)
- Network Forwarding Paths (NFPs)
- Service Function Chains (SFCs)
- Classifiers
- Network Service Descriptors (NSD)
- Network Services (NSs)
And also remove the following tools that are currently not in use.
- tools/vnfc/build_image.sh
- tools/check_i18n_test_case.txt
- tools/check_i18n.py
- tools/i18n_cfg.py
- tools/clean.sh
- tools/prepare_function_test.sh
- tools/test-setup.sh
- tools/install_venv_common.py
- tools/install_venv.py
- tools/with_venv.sh
For the following tools used in FT, move them under
`tacker/tests/functional/tools`.
- tools/test-setup-default-vim.sh
- tools/test-setup-k8s-vim.sh
- tools/test-setup-mgmt.sh
- tools/test-setup-fake-prometheus-server.sh
Implements: blueprint deprecate-legacy-apis
Change-Id: Iea89c32f69ccbe47badcfddcf77430abda98362b
This patch provides terraform infra-driver with several unit and
functional tests and is build an environment for terraform. The
supported version of terraform is v1.4.0 or later.
To build the terraform environment that need to install as follow
components:
- awscli
- docker
- localstack or moto server
- terraform
Implements: blueprint terraform-infra-driver
Change-Id: I14414c42229dcdb8e0083d7c51d6be6b5f2fc841
For fine-grained access control based on user and VNF information
for API resources, this patch does the following things:
1.Add three comparison attributes of area, vendor, and tenant
for the enhanced Tacker policy.
2.Convert special roles to API attributes in context.
3.Modify the API process to support Tacker policy authorize.
4.Add the Tacker policy filter to the list API processes.
Implements: blueprint enhance-api-policy
Change-Id: I5b4c39387860133a3bcf4544f18a6353c80773f6
Provide the option to verify the SSL certificate when accessing an
external server from Tacker. Several parameters have been added to
config to allow verification of SSL certificates when accessing
external NFVO servers, heat servers, and notification endpoints from
Tacker.
Implements: blueprint enhance-http-client
Change-Id: I55b2b53cfe0dc794040d0e46ac13a20524b1d9f0
In v2 API, when using vim with `ETSINFV.HELM.V_3` type, you must
set the `ssl_ca_cert` information. Currently, when registering vim
with kubernetes type and use_helm parameter, it will succeed even
if you don't set `ssl_ca_cert` information. This causes v2 APIs to
fail when you use the vimConnectionInfo stored in the database.
This patch added a check in register vim to fix this issue. If the
helm is used to register the vim of kubernetes type, the
`ssl_ca_cert` parameter will be checked. If it's not set, the
registration will fail.
At the same time, FT test items have been added to verify that
v2 API operations perform properly when using the `ETSINFV.HELM.V_3`
type vim that exists in the database.
Implements: blueprint helmchart-k8s-vim
Change-Id: I629e347413b242ab9e1a5db16c52ca222adc3873
This patch enables availability zone reselected by stack retry
when "instantiate" "scale" "heal" operations in v2 API fails
due to availability zone is unavailable.
Note that precondition of using these functions is to use
StandardUserData as UserData class.
Implements: blueprint enhance-placement
Change-Id: Icc9eb7a8fffbf35d2e005a9703dcefe66f097584
This patch modifies the task replacement rules in set-default-vim,
enabling zuul to create kubernetes vim using oidc. At the same time,
the FT code of v1-related oidc is modified so that it can use
oidc-related vim during testing.
Closes-Bug: #2007054
Change-Id: Idc27bd2609582ed949041d2f8c730eec686f5102
For Jammy migration, podman that is used in cri-o instead of dockershim
as container runtime need to configure unqualified search registries.
This patch sets the registries of image used in kubernetes jobs to
unqualified search registries.
Change-Id: I41c5bba802a91a9b9059325d26b35430ac13817f
This patch updates the k8s version to 1.25.2 and helm version to 3.10.1.
The following changes have been made in kubernetes 1.24, so this patch
supports them.
- Dockershim Removed from kubelet
-> Changed from Dockershim to CRIO
- Service account tokens are no longer auto-generated
for every ServiceAccount
-> Changed secrets to be created manually
Also fixes the following API versions that have already been removed
in kubernetes 1.25.
- autoscaling/v2beta1
- extensions/v1beta1
Implements: blueprint update-k8s-helm-prometheus
Change-Id: Ic9c2f66251c5d11a652184be5908d91e0ee3fb7d
Support container based VNF AutoHeal and AutoScale operation with
External Monitoring Tools.
Add the Fault Management interfaces and CLI to support AutoHeal.
Add the Performance Management interfaces and CLI to support
AutoScale. The Fault Management and Performance Management
interfaces are based on ETSI NFV-SOL 002 v3.3.1 and ETSI NFV-SOL
003 v3.3.1, which are Version "2.0.0" API of Tacker. Add the
Prometheus Plugin that has a interface between tacker and Prometheus
as a sample of External Monitoring Tool.
Implements: blueprint support-auto-lcm
Change-Id: Ib74305f0b1da4eb8d996ebae400e75902aaa1321
This patch enables CNF v2 API to operate using Helm chart.
New vimType 'ETSINFV.HELM.V_3' is introduced.
Since helm VIM uses existing functions of k8s VIM, k8s VIM code
is refactored to share between k8s VIM and helm VIM.
Implements: blueprint helmchart-k8s-vim
Change-Id: I0329a0d43294181b7ffb1494bb5dd2d0528eb5dc
This patch adds openid token auth support when calling k8s APIs.
Openid token auth of k8s relies on an external openid provider,
and Keycloak acts as the openid provider in this implementation.
Implements: blueprint support-openid-k8s-vim
Change-Id: Ie5e080a20cba3ba0ed514ede7955eb16729d797c
When initializing k8s client in InfraDriverV2, the SSL CA
certificate is set incorrectly.
To fix the issue, the following modifies are made in this patch:
* A temp file for ssl_ca_cert is created before initializing
k8s client and the temp file path is set to k8s_config.ssl_ca_cert,
* The temp file is deleted until the lifetime of k8s client ends.
Note: This references the implementation in InfraDriverV1.
If set the ssl_ca_cert in instantiate request, the validation of
request is failed because of the length of ssl_ca_cert exceeds 1024.
For this issue, add a new type `keyvalue_pairs_no_length_limit`
which has no max length limitation to verify the request.
And the interfaceInfo, accessInfo, extra are all set to the new type
for unity.
In Zuul test environment, when registering default vim, ssl_ca_cert
is not set. So the case with ssl_ca_cert is not tested.
In this patch ssl_ca_cert is set into the default vim.
Closes-Bug: #1979413
Change-Id: I61dbd70690b737a72fc619e5a08b4bab51160a27
This patch adds functional test cases to validate instantiate CNF
functionality in a multi-tenant environment.
Validates CNF instantiation is only allowed when CNF and VIM
belongs to same tenant.
Implements: blueprint k8s-namespace
Change-Id: I800b497ea6fa8f4978eb551558f0257e3d82a4ee
This patch supports MgmtDriver in the operation of modifying VNF.
It provides a sample script MgmtDriver, when modifying CNF,
If the ConfigMap and Secret are updated, the Pod and Deployment will
also be updated (image only).
Implements: blueprint container-update
Change-Id: I1e7a1b03fef13f4c7a83488f6d2fdd7efc2e454b
To validate functional test cases in Zuul environment this
patch adds a new Ansible playbook. This playbook helps in
creating two different OpenStack projects, users to
validate multi tenant policy in Lifecycle Management.
In current design, tacker uses an administrator role user
"nfv_user" to execute functional test cases. Whereas this
patch adds member role (non administrator user) to newly
created users.
Generates OpenStack VIM config files using helper script and
register default VIMs to respective tenants.
Additionally copies newly generated VIM config files to
"tacker/tacker/tests/etc/samples" folder as these are required
in functional test cases.
Partial Implement: blueprint multi-tenant-policy
Change-Id: I20491eb294e5653bcdc2864885f55d04b21696a1
kuryr-kubernetes patch [1] that changes to use kubeadm for
installing Kubernetes on devstack.
A patch [2] was previously created to address the change, but the fix
used kuryr-Kubernetes stable/wallaby as a temporary fix.
This patch fixes to use latest kuryr-kubernetes.
* Change the ".zuul.yaml" setting to use the latest kuryr-kubernetes.
* Add the process of creating a ServiceAccount and change the vim
authentication method to bearer_token. This is because
kuryr-kubernetes does not create an admin ServiceAccount by default.
[1] https://review.opendev.org/c/openstack/kuryr-kubernetes/+/779250
[2] https://review.opendev.org/c/openstack/tacker/+/791252
Change-Id: Ib64183b5e978774811f51f8af0f4590a20ced856
This patch changes the following to support helm chart spec and
to test instantiate/terminate cnf with helm chart.
* Add `extra` field to vims db.
* Add `setup-helm` task to ansible-playbook roles.
[On controller-k8s node]
* Create and setup helm user for executing helm command.
* Install helm.
* Create folder for putting local helm chart.
* Enable password authentication in sshd_config and restart sshd.
[On controller node]
* Update Vims DB of vim-kubernetes to modify extra field that include
helm access information.
Implements: blueprint helmchart-k8s-vim
Change-Id: Iaf7c11c5bedb77e9cd21074be2b4f73528aa2ce7
This patch add a kubernetes cluster for the kubernetes related
functional tests of the VNF LCM in the zuul environment.
There is no impact to the existing jobs because this patch only
add a new job, however we may need to watch the load on the Zuul
environment due to its parallel jobs.
A new node-set consists of four nodes;
* Controller: Keystone, Nova, Neutron, Glance, Cinder, Octavia,
MySQL, MQ, ETCD
* Controller-tacker: Tacker, Tacker-conductor
* Controller-k8s: kuryr-k8s, kuryr-CNI, k8s-api, kubelet
* Compute: Nova-compute
All kubernetes resources are created on the controller-k8s node.
This patch includes the following changes:
* Added a execution command for the functional test of related
k8s for the VNF LCM in tox.ini.
* Registered a vim of the `kubernetes` type by ansible. Also
added related materials.
* Moved the functional test files for k8s to other new directory.
* Fixed a minor invalid definition in the definition file used
for functional testing.
Change-Id: I1621b904450e94d6793b4c524de6785520f2e805
The number of required plugins has increased to the extent that
controller on Zuul FT infrastructure almost runs out of memory [1].
This potentially induces various problems such as FT failure noises
or POST_FAILURE at the ansible task 'export-devstack-journal'.
To request expanded images such as ubuntu-focal-expanded (16GB) or
ubuntu-focal-32GB would be an option, but it turns out that would
lead us to another problematic situation. [2]
This patch, instead, addresses the issue by subdividing the memory
load. As a first step, introduce a new subnode 'tacker-controller',
on which both tacker-server and tacker-conductor are located.
Note:
* when we re-locate some other components to this new subnode, it might
better to rename it.
* `devstack_local_conf: {}` in .zuul.yaml is to cancel out the global
job.vars devstack_local_conf.post-config.$NEUTRON_DHCP_CONF, which
is not present on 'tacker-controller' in the first place.
* TACKER_MODE is set to 'standalone'. 'all-in-one' supposes core
services like nova, neutron, keystone, etc. api servers are located
on the same host as tacker-server.
* in devstack/lib/tacker:create_tacker_accounts, SERVICE_HOST should
have been TACKER_HOST. this minor fix is included.
* in roles/setup-default-vim/tasks/main.yaml, the same where conditions
were scattered but all tasks in it just needed to run on 'controller'
only. so let us wrap them all in a block.
* renamed devstack/plugin.sh:tacker_register_default_vim for clarity.
* policy file modification for Heat is now done by an ansible task.
it frees us from the co-location requirements for Tacker and Heat.
* drop devstack/lib/tacker:is_tacker_enabled as it's no longer needed.
[1]: we investigated how severe the memory load on 'controller' was
on Zuul FT infrastructure:
* The highest memory-consuming processes in desc order:
808.70MB (9.87%) 828112 /usr/sbin/mysqld
179.81MB (2.19%) 184124 ... /usr/local/bin/tacker-server ...
152.57MB (1.86%) 156232 ... /usr/local/bin/tacker-conductor .
146.67MB (1.79%) 150188 ... /usr/local/bin/neutron-server ...
132.96MB (1.62%) 136148 ... /usr/local/bin/neutron-server ...
129.08MB (1.58%) 132180 ... /usr/local/bin/heat-engine ...
127.48MB (1.56%) 130544 ... /usr/local/bin/heat-engine ...
122.16MB (1.49%) 125092 nova-apiuWSGI worker 1
121.00MB (1.48%) 123900 neutron-openvswitch-agent ...
119.50MB (1.46%) 122368 cinder-apiuWSGI worker 1
---(snip)---
* `free -m` output
total used free shared buff/cache available
Mem: 7955 7427 196 16 331 219
Swap: 1022 1019 3
[2]: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/
%23openstack-infra.2020-11-25.log.html
Change-Id: I030ffd5fd11b7ca9abca56e85e449ed4c4d709bd
There are some placeholder files for running functional test in zuul.
The contents of the files, such as `auth_url` of the following files,
are updated while running devstack `stack.sh`.
* tacker/tests/etc/samples/local-vim.yaml
* tools/test-setup-default-vim.sh
These Git management files including the above, should not be
modified. Therefore, I will be fix as follow.
The need files to be change for Zuul are deploy by Ansible,
so the deploy process is run with Ansible instead of the
`devstack/lib/tacker` script.
Allow `tools/test-setup-default-vim.sh` script to be executed
by itself, so the git management files aren't updated by running
devstack `stack.sh`.
Also, as other improvements, replace the file path included in
Ansible role to definition value.
Change-Id: Iad88c7adfe56e926ee0324d94787577ba066989b
Closes-Bug: #1879303
In patch [1], the mode of tools/test-setup-default-vim.sh shell
script file was changed from 755 to 644 because of which it's fails to
run the script which registers a default vim 'VIM0' needed to run
for some of the functional tests.
This patch reverts back the mode of test-setup-default-vim.sh
from 644 to 755 and logs meaningful error message in case
the script file doesn't exists or it doesn't have execute permission.
[1] : https://review.opendev.org/#/c/674761
Change-Id: I4651bdd89da12720416e630173f80cc788911307
Closes-Bug: #1874007
Add devstack multinode job.
This job installs nova in a single cell as "late anti-affinity
[1] doesn't work in multi-cells environment.
[1] : https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#operations-requiring-upcalls
Co-Author: tpatil <tushar.vitthal.patil@gmail.com>
Co-Author: yong sheng gong <gong.yongsheng@99cloud.net>
Change-Id: I56a9cf4bb553c8026eec73212a3742d5eab17420
