OpenStack host maintenance and upgrade in interaction with application on top of it
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

11 KiB

This directory contains tools and instructions to test Fenix workflows. Currently OPNFV Doctor has been used to test OpenStack related workflows. As Doctor is at the moment only for OpenStack and Fenix itself needs a way to be tested, the Kubernetes workflow (fenix/workflow/workflows/ testing is implemented here.


  • 'demo-ha.yaml': demo-ha ReplicaSet to make 2 anti-affinity PODS.
  • 'demo-nonha.yaml': demo-nonha ReplicaSet to make n nonha PODS.
  • '': VNFM to test workflow.

Kubernetes workflow (

First version of workflow towards Kubernetes use cases.

Requirements for testing

This workflow assumes ReplicaSet used for PODs. A Kubernetes cluster with 1 master and at least 3 workers are required for testing. Master node needs DevStack to have Fenix and OpenStack services it still uses. Later on there can be a version of Fenix not needing Keystone and AODH event alarming, but using native Kubernetes services for RBAC and events.

As in Doctor testing, there is a pair of anti-affinity PODs (demo-ha) and rest of the worker node capacity is filled with (demo-nonha) PODs. Scaling of PODs is done via ReplicaSet number of replicas. Idea is that each POD is taking ("number of worker node CPUs" - 2) / 2. That will make sure scheduler fits 2 PODs on each node and 2 CPUs capacity is assumed for other node services. This should require at least 6 CPUs for each node to work.

Install Kubernetes cluster with 1 Manager and 3 Worker nodes.

Here is instructions:

On Manager node, install DevStack including Fenix and its minimum services

Note! There is no conflict with Kubernetes as limiting to only Fenix needed services.

Clone DevStack. Tested to work with latest stable release Train.

git clone -b stable/train

Make local.conf. 'HOST_IP' should be the master node IP.

cd devstack vi local.conf




enable_plugin ceilometer stable/train
enable_plugin aodh stable/train
enable_plugin gnocchi
enable_plugin fenix master

Deploy needed OpenStack services with Fenix


Now you should have Kubernetes cluster and Fenix via DevStack. Any hacking of Fenix can be done under '/opt/stack/fenix'.

Running test

Use 3 terminal windows (Term1, Term2 and Term3) to test Fenix with Kubernetes kluster. Under here is what you can run in different terminals. Terminals should be running in master node. Here is short description:

  • Term1: Used for logging Fenix
  • Term2: Infrastructure admin commands
  • Term3: VNFM logging for testing and setting up the VNF

Term1: Fenix-engine logging

If any changes to Fenix make them under '/opt/stack/fenix'; restart Fenix and see logs

sudo systemctl restart devstack@fenix*;sudo journalctl -f --unit devstack@fenix-engine

API logs can also be seen

sudo journalctl -f --unit devstack@fenix-api

Debugging and other configuration changes to '.conf' files under '/etc/fenix'

Term2: Infrastructure admin window

Use DevStack admin as user. Set your variables needed accordingly

. ~/devstack/operc admin admin
USER_ID=`openstack user list | grep admin | awk '{print $2}'`

Authenticate to Keystone as admin user before calling Fenix. If you will have some not authorized error later on, you need to do this again.

OS_AUTH_TOKEN=`openstack token issue | grep " id " |awk '{print $4}'`

After you have first: Fenix running in Term1; Next: VNF created a in Term3 Next: VNFM running in Term3, you can create maintenance session utilizing those

DATE=`date -d "+15 sec" "+%Y-%m-%d %H:%M:%S"`;MSESSION=`curl -g -i -X POST http://$HOST:$PORT/v1/maintenance -H "Accept: application/json" -H "Content-Type: application/json" -d '{"workflow": "k8s", "state": "MAINTENANCE","metadata": {} ,"maintenance_at": "'"$DATE"'"}' -H "X-Auth-Token: $OS_AUTH_TOKEN" -H "X-User-Id: $USER_ID" | grep session_id | jq -r '.session_id'`

After maintenance workflow is 'MAINTENANCE_DONE', you should first press "ctrl + c" in VNFM window (Term3), so it removes constraints from Fenix and dies. Then you can remove the finished session from Fenix

curl -g -i -X DELETE http://$HOST:$PORT/v1/maintenance/$MSESSION -H "Accept: application/json" -H "Content-Type: application/json" -H "X-Auth-Token: $OS_AUTH_TOKEN" -H "X-User-Id: $USER_ID"

If maintenance run till the end with 'MAINTENANCE_DONE', you are ready to run it again if you wish. 'MAINTENANCE_FAILED' or in case of exceptions, you should recover system before trying to test again. This is covered in Term3 below.

Term3: VNFM (fenix/tools/

Use DevStack admin as user.

. ~/devstack/operc admin admin

Go to Fenix Kubernetes tool directory for testing

cd /opt/stack/fenix/fenix/tools

Create demo namespace (we use demo namespace and demo user and project in Keystone)

kubectl create namespace demo

Start VNFM (when done in this order, we make sure demo-ha has nodes for anti-affinity):

kubectl apply -f demo-ha.yaml --namespace=demo;sleep 1;kubectl apply -f demo-nonha.yaml  --namespace=demo

Note you should modify above yaml files so that "cpu:" has value of '(workernode.status.capacity["cpu"] - 2) / 2'. Default is expecting that there is 32 cpus, so value is "15" in both yaml files. Replicas can be changed in demo-nonha.yaml. Minimum 2 (if minimum of 3 worker nodes) to maximum '(amount_of_worker_nodes-1)*2'. Greater amount means more scaling needed and longer maintenance window as less parallel actions possible. Surely constraints in also can be changed for different behavior.

You can delete pods used like this

kubectl delete replicaset.apps demo-ha demo-nonha --namespace=demo

Start Kubernetes VNFM that we need for testing


Now you can start maintenance session in Term2. When workflow failed or completed; you first kill with "ctrl+c" and delete maintenance session in Term2.

If workflow failed something might need to be manually fixed. Here you uncordon your 3 worker nodes, if maintenance workflow did not run to the end.

kubectl uncordon worker-node3 worker-node2 worker-node1

You can check your pods matches to amount of replicas mentioned in demo-nonha.yaml and demo-ha.yaml:

kubectl get pods --all-namespaces --output=wide

If not matching, delete and create again as easiest solution

kubectl delete replicaset.apps demo-ha demo-nonha --namespace=demo;sleep 15;kubectl apply -f demo-ha.yaml --namespace=demo;sleep 1;kubectl apply -f demo-nonha.yaml  --namespace=demo

OpenStack workflows ( and

OpenStack workflows can be tested by using OPNFV Doctor project for testing. Workflows:

  • is the first example workflow with VNFM interaction.
  • is enhanced Telco workflow that utilizes the ETSI FEAT03 constraints. This workflow can optimize parallel nodes and migrations according to ETSI constraints.

Requirements for testing

  • Multinode DevStack environment with 1 controller and at least 3 computes.
  • Each compute needs to have at least 2 VCPUs
  • DevStack local.conf for controller node need Fenix enabled. Also Heat, AODH and Ceilometer are needed. VCPUs needs one to one to mapping to CPU.

Example controller baremetal local.conf:



## Neutron options
#This is wrong, but if right devstack deletes the host ip

# Open vSwitch provider networking configuration


disable_service ceilometer-alarm-notifier,ceilometer-alarm-evaluator,ceilometer-acompute
enable_service aodh-evaluator,aodh-notifier,aodh-api

enable_plugin heat stable/train
enable_plugin ceilometer stable/train
enable_plugin aodh stable/train
enable_plugin gnocchi
enable_plugin fenix master
enable_service fenix-engine
enable_service fenix-api

disable_service n-cpu

cpu_allocation_ratio = 1.0
allow_resize_to_same_host = False


On controller node clone Doctor to be able to test. Doctor currently requires Python 3.6:

git clone
export INSTALLER_TYPE=devstack
export ADMIN_TOOL_TYPE=fenix
export TEST_CASE=maintenance

Use DevStack admin as user.

. ~/devstack/operc admin admin

Goto Doctor and start testing

cd doctor
sudo -E tox -e py36

Use journalctl to track the progress in Fenix

sudo journalctl -f --unit devstack@fenix-engine

If any changes to Fenix make them under '/opt/stack/fenix' and restart Fenix

sudo systemctl restart devstack@fenix*

You can also make changed to Doctor before running Doctor test


This workflow differs from above as it expects ETSI FEAT03 constraints. In Doctor testing it means we also need to use different application manager (VNFM)

Where worklow used the application manager workflow uses workflow (doctor/doctor_tests/app_manager/

Only change to testing is that you should export variable to use different application manager.

export APP_MANAGER_TYPE=vnfm

If again want to use, you can export the default value for application manager

export APP_MANAGER_TYPE=sample

Doctor modifies the message where it calls maintenance accordingly to use either 'default' or 'nfv' as workflow in Fenix side