Browse Source

Detailed session information and enhancements

- Add GET /v1/maintenance/{session_id}/detail
- Add 'maintenance.session' event. This can be used
  to track workflow. It gives you percent of hosts
  maintained.

Other enhancements:
- Add Sample VNFM for OpenStack: vnfm.py
  (Kubernetes renamed to vnfm_k8s.py)
- Add Sample VNF for OpenStack:
  maintenance_hot_tpl.yaml
- Update testing instructions (tools)
- Update documentation
- Add more tools for testing:
  - fenix_db_reset (flushed the database)
  - set_config.py (set the AODH / Ceilometer config)
- Add admin tool: infra_admin.py
  This tool can run maintenance workflow and
  track its progress
- Make sure everything is written in database.
  If Fenix is restarted, it initialise existing
  'ongoing' workflows from database. More functions
  to database API and utilization in example workflows.

story: 2004336
Task: #27922

Change-Id: I794b11a8684f5fc513cb8f5affcd370ec70f3dbc
Signed-off-by: Tomi Juvonen <tomi.juvonen@nokia.com>
changes/72/720672/2
Tomi Juvonen 2 years ago
parent
commit
244fb3ced0
  1. 1
      README.rst
  2. 6
      doc/source/api-ref/index.rst
  3. 31
      doc/source/api-ref/v1/index.rst
  4. 60
      doc/source/api-ref/v1/maintenance.inc
  5. 85
      doc/source/api-ref/v1/parameters.yaml
  6. 18
      doc/source/api-ref/v1/project.inc
  7. 212
      doc/source/api-ref/v1/samples/maintenance-session-detail-get-200.json
  8. 0
      doc/source/api-ref/v1/samples/maintenance-session-get-200.json
  9. 0
      doc/source/api-ref/v1/samples/maintenance-sessions-get-200.json
  10. 0
      doc/source/api-ref/v1/samples/project-maintenance-session-post-200.json
  11. 44
      doc/source/api-ref/v1/status.yaml
  12. 34
      doc/source/user/notifications.rst
  13. 6
      fenix/api/v1/controllers/__init__.py
  14. 13
      fenix/api/v1/controllers/maintenance.py
  15. 6
      fenix/api/v1/maintenance.py
  16. 52
      fenix/db/api.py
  17. 2
      fenix/db/migration/alembic_migrations/versions/001_initial.py
  18. 75
      fenix/db/sqlalchemy/api.py
  19. 2
      fenix/db/sqlalchemy/models.py
  20. 4
      fenix/tests/db/sqlalchemy/test_sqlalchemy_api.py
  21. 180
      fenix/tools/README.md
  22. 9
      fenix/tools/fenix_db_reset
  23. 320
      fenix/tools/infra_admin.py
  24. 108
      fenix/tools/maintenance_hot_tpl.yaml
  25. 6
      fenix/tools/session.json
  26. 185
      fenix/tools/set_config.py
  27. 574
      fenix/tools/vnfm.py
  28. 561
      fenix/tools/vnfm_k8s.py
  29. 52
      fenix/utils/service.py
  30. 4
      fenix/workflow/actions/dummy.py
  31. 86
      fenix/workflow/workflow.py
  32. 41
      fenix/workflow/workflows/default.py
  33. 29
      fenix/workflow/workflows/k8s.py
  34. 58
      fenix/workflow/workflows/vnf.py

1
README.rst

@ -27,6 +27,7 @@ would also be telling about adding or removing a host.
* Documentation: https://fenix.readthedocs.io/en/latest/index.html
* Developer Documentation: https://wiki.openstack.org/wiki/Fenix
* Source: https://opendev.org/x/fenix
* Running sample workflows: https://opendev.org/x/fenix/src/branch/master/fenix/tools/README.md
* Bug tracking and Blueprints: https://storyboard.openstack.org/#!/project/x/fenix
* How to contribute: https://docs.openstack.org/infra/manual/developers.html
* `Fenix Specifications <specifications/index.html>`_

6
doc/source/api-ref/index.rst

@ -1,6 +1,6 @@
####################
Host Maintenance API
####################
###
API
###
.. toctree::
:maxdepth: 2

31
doc/source/api-ref/v1/index.rst

@ -1,28 +1,29 @@
:tocdepth: 2
#######################
Host Maintenance API v1
#######################
######
API v1
######
.. rest_expand_all::
#####
Admin
#####
#########
Admin API
#########
These APIs are meant for infrastructure admin who is in charge of triggering
the rolling maintenance and upgrade workflows.
the rolling maintenance and upgrade workflow sessions.
.. include:: maintenance.inc
#######
Project
#######
###########
Project API
###########
These APIs are meant for projects having instances on top of the infrastructure
under corresponding rolling maintenance or upgrade session. Usage of these APIs
expects there is an application manager (VNFM) that can interact with Fenix
workflow via these APIs. If this is not the case, workflow should have a default
behavior for instances owned by projects, that are not interacting with Fenix.
These APIs are meant for projects (tenant/VNF) having instances on top of the
infrastructure under corresponding rolling maintenance or upgrade session.
Usage of these APIs expects there is an application manager (VNFM) that can
interact with Fenix workflow via these APIs. If this is not the case, workflow
should have a default behavior for instances owned by projects, that are not
interacting with Fenix.
.. include:: project.inc

60
doc/source/api-ref/v1/maintenance.inc

@ -1,13 +1,13 @@
.. -*- rst -*-
===========
Maintenance
===========
==========================
Admin workflow session API
==========================
Create maintenance session
==========================
.. rest_method:: POST /v1/maintenance/
.. rest_method:: POST /v1/maintenance
Create a new maintenance session. You can specify a list of 'hosts' to be
maintained or have an empty list to indicate those should be self-discovered.
@ -49,7 +49,7 @@ Response codes
Update maintenance session (planned future functionality)
=========================================================
.. rest_method:: PUT /v1/maintenance/{session_id}/
.. rest_method:: PUT /v1/maintenance/{session_id}
Update existing maintenance session. This can be used to continue a failed
session after manually fixing what failed. Workflow should then run
@ -79,7 +79,7 @@ Response codes
Get maintenance sessions
========================
.. rest_method:: GET /v1/maintenance/
.. rest_method:: GET /v1/maintenance
Get all ongoing maintenance sessions.
@ -88,7 +88,7 @@ Response codes
.. rest_status_code:: success status.yaml
- 200: get-maintenance-sessions-get
- 200: maintenance-sessions-get
.. rest_status_code:: error status.yaml
@ -98,7 +98,7 @@ Response codes
Get maintenance session
=======================
.. rest_method:: GET /v1/maintenance/{session_id}/
.. rest_method:: GET /v1/maintenance/{session_id}
Get a maintenance session state.
@ -114,7 +114,38 @@ Response codes
.. rest_status_code:: success status.yaml
- 200: get-maintenance-session-get
- 200: maintenance-session-get
.. rest_status_code:: error status.yaml
- 400
- 404
- 422
- 500
Get maintenance session details
===============================
.. rest_method:: GET /v1/maintenance/{session_id}/detail
Get a maintenance session details. This information can be usefull to see
detailed status of a maintennace session or to troubleshoot a failed session.
Usually session should fail on simple problem, that can be fast manually
fixed. Then one can update maintenance session state to continue from 'prev_state'.
Request
-------
.. rest_parameters:: parameters.yaml
- session_id: session_id
Response codes
--------------
.. rest_status_code:: success status.yaml
- 200: maintenance-session-detail-get
.. rest_status_code:: error status.yaml
@ -126,7 +157,7 @@ Response codes
Delete maintenance session
==========================
.. rest_method:: DELETE /v1/maintenance/{session_id}/
.. rest_method:: DELETE /v1/maintenance/{session_id}
Delete a maintenance session. Usually called after the session is successfully
finished.
@ -141,12 +172,3 @@ finished.
- 400
- 422
- 500
Future
======
On top of some expected changes mentioned above, it will also be handy to get
detailed information about the steps run already in the maintenance session.
This will be helpful when need to figure out any correcting actions to
successfully finish a failed session. For now admin can update failed session
state to previous or his wanted state to try continue a failed session.

85
doc/source/api-ref/v1/parameters.yaml

@ -36,7 +36,7 @@ uuid-path:
#############################################################################
action-metadata:
description: |
Metadata; hints to plug-ins
Metadata; hints to plug-ins.
in: body
required: true
type: dictionary
@ -44,7 +44,17 @@ action-metadata:
action-plugin-name:
description: |
plug-in name. Default workflow executes same type of plug-ins in an
alphabetical order
alphabetical order.
in: body
required: true
type: string
action-plugin-state:
description: |
Action plug-in state. This is workflow and action plug-in specific
information to be passed from action plug-in to workflow. Helps
understanding how action plug-in was executed and to troubleshoot
accordingly.
in: body
required: true
type: string
@ -77,6 +87,20 @@ boolean:
required: true
type: boolean
datetime-string:
description: |
Date and time string according to ISO 8601.
in: body
required: true
type: string
details:
description: |
Workflow internal special usage detail. Example nova-compute service id.
in: body
required: true
type: string
group-uuid:
description: |
Instance group uuid. Should match with OpenStack server group if one exists.
@ -84,6 +108,21 @@ group-uuid:
required: true
type: string
host-type:
description: |
Host type as it is wanted to be used in workflow implementation.
Example workflows uses values as compute and controller.
in: body
required: false
type: list of strings
hostname:
description: |
Name of the host.
in: body
required: true
type: string
hosts:
description: |
Hosts to be maintained. An empty list can indicate hosts are to be
@ -102,7 +141,7 @@ instance-action:
instance-actions:
description: |
instance ID : action string. This variable is not needed in reply to state
MAINTENANCE, SCALE_IN or MAINTENANCE_COMPLETE
MAINTENANCE, SCALE_IN or MAINTENANCE_COMPLETE.
in: body
required: true
type: dictionary
@ -128,6 +167,14 @@ instance-name:
required: true
type: string
instance-state:
description: |
State of the instance as in underlying cloud. Can be different in
different clouds like OpenStack or Kubernetes.
in: body
required: true
type: string
lead-time:
description: |
How long lead time VNF needs for 'migration_type' operation. VNF needs to
@ -177,30 +224,50 @@ max-interruption-time:
metadata:
description: |
Metadata; like hints to projects
Hint to project/tenant/VNF to know what capability the infrastructure
is offering to instance when it moves to already maintained host in
'PLANNED_MAINTENANCE' state action. This may have impact on how
the instance is to be moved or if instance is to be upgraded and
VNF needs to re-instantiate it as its 'OWN_ACTION'. This could be the
case with new hardware or instance could be wanted to be upgraded
anyhow at the same time of the infrastructure maintenance.
in: body
required: true
type: dictionary
migration-type:
description: |
LIVE_MIGRATION, MIGRATION or OWN_ACTION
'LIVE_MIGRATE', 'MIGRATE' or 'OWN_ACTION'
Own action is create new and delete old instance.
Note! VNF need to obey resource_mitigation with own action
This affects to order of delete old and create new to not over
commit the resources. In Kubernetes also EVICTION supported. There admin
commit the resources. In Kubernetes also 'EVICTION' supported. There admin
will delete instance and VNF automation like ReplicaSet will make a new
instance
instance.
in: body
required: true
type: string
percent_done:
description: |
How many percent of hosts are maintained.
in: body
required: true
type: dictionary
plugin:
description: |
Action plugin name.
in: body
required: true
type: dictionary
recovery-time:
description: |
VNF recovery time after operation to instance. Workflow needs to take
into account recovery_time for previous instance moved and only then
start moving next obyeing max_impacted_members
Note! regardless anti_affinity group or not
Note! regardless anti_affinity group or not.
in: body
required: true
type: integer
@ -255,7 +322,7 @@ workflow-name:
workflow-state:
description: |
Maintenance workflow state.
Maintenance workflow state (States explained in the user guide)
in: body
required: true
type: string

18
doc/source/api-ref/v1/project.inc

@ -1,8 +1,8 @@
.. -*- rst -*-
=======
Project
=======
============================
Project workflow session API
============================
These APIs are generic for any cloud as instance ID should be something that can
be matched to virtual machines or containers regardless of the cloud underneath.
@ -10,7 +10,7 @@ be matched to virtual machines or containers regardless of the cloud underneath.
Get project maintenance session
===============================
.. rest_method:: GET /v1/maintenance/{session_id}/{project_id}/
.. rest_method:: GET /v1/maintenance/{session_id}/{project_id}
Get project instances belonging to the current state of maintenance session.
the Project-manager receives an AODH event alarm telling about different
@ -31,7 +31,7 @@ Response codes
.. rest_status_code:: success status.yaml
- 200: get-project-maintenance-session-post
- 200: project-maintenance-session-post
.. rest_status_code:: error status.yaml
@ -42,7 +42,7 @@ Response codes
Input from project to maintenance session
=========================================
.. rest_method:: PUT /v1/maintenance/{session_id}/{project_id}/
.. rest_method:: PUT /v1/maintenance/{session_id}/{project_id}
Project having instances on top of the infrastructure handled by a maintenance
session might need to make own action for its instances on top of a host going
@ -78,9 +78,9 @@ Response codes
- 422
- 500
============================
Project with NFV constraints
============================
===========================
Project NFV constraints API
===========================
These APIs are for VNFs, VNMF and EM that are made to support ETSI defined
standard VIM interface for sophisticated interaction to optimize rolling

212
doc/source/api-ref/v1/samples/maintenance-session-detail-get-200.json

@ -0,0 +1,212 @@
{
"session_id": "47479bca-7f0e-11ea-99c9-2c600c9893ee",
"instances": [
{
"instance_id": "da8f96ae-a1fe-4e6b-a852-6951d513a440",
"action_done": false,
"host": "overcloud-novacompute-2",
"created_at": "2020-04-15T11:43:09.000000",
"project_state": "INSTANCE_ACTION_DONE",
"updated_at": null,
"session_id": "47479bca-7f0e-11ea-99c9-2c600c9893ee",
"instance_name": "demo_nonha_app_2",
"state": "active",
"details": null,
"action": null,
"project_id": "444b05e6f4764189944f00a7288cd281",
"id": "73190018-eab0-4074-bed0-4b0c274a1c8b"
},
{
"instance_id": "22d869d7-2a67-4d70-bb3c-dcc14a014d78",
"action_done": false,
"host": "overcloud-novacompute-4",
"created_at": "2020-04-15T11:43:09.000000",
"project_state": "ACK_PLANNED_MAINTENANCE",
"updated_at": null,
"session_id": "47479bca-7f0e-11ea-99c9-2c600c9893ee",
"instance_name": "demo_nonha_app_3",
"state": "active",
"details": null,
"action": "MIGRATE",
"project_id": "444b05e6f4764189944f00a7288cd281",
"id": "c0930990-65ac-4bca-88cb-7cb0e7d5c420"
},
{
"instance_id": "89467f5c-d5f8-461f-8b5c-236ce54138be",
"action_done": false,
"host": "overcloud-novacompute-2",
"created_at": "2020-04-15T11:43:09.000000",
"project_state": "INSTANCE_ACTION_DONE",
"updated_at": null,
"session_id": "47479bca-7f0e-11ea-99c9-2c600c9893ee",
"instance_name": "demo_nonha_app_1",
"state": "active",
"details": null,
"action": null,
"project_id": "444b05e6f4764189944f00a7288cd281",
"id": "c6eba3ae-cb9e-4a1f-af10-13c66f61e4d9"
},
{
"instance_id": "5243f1a4-9f7b-4c91-abd5-533933bb9c90",
"action_done": false,
"host": "overcloud-novacompute-3",
"created_at": "2020-04-15T11:43:09.000000",
"project_state": "INSTANCE_ACTION_DONE",
"updated_at": null,
"session_id": "47479bca-7f0e-11ea-99c9-2c600c9893ee",
"instance_name": "demo_ha_app_0",
"state": "active",
"details": "floating_ip",
"action": null,
"project_id": "444b05e6f4764189944f00a7288cd281",
"id": "d67176ff-e2e4-45e3-9a52-c069a3a66c5e"
},
{
"instance_id": "4e2e24d7-0e5d-4a92-8edc-e343b33b9f10",
"action_done": false,
"host": "overcloud-novacompute-3",
"created_at": "2020-04-15T11:43:09.000000",
"project_state": "INSTANCE_ACTION_DONE",
"updated_at": null,
"session_id": "47479bca-7f0e-11ea-99c9-2c600c9893ee",
"instance_name": "demo_nonha_app_0",
"state": "active",
"details": null,
"action": null,
"project_id": "444b05e6f4764189944f00a7288cd281",
"id": "f2f7fd7f-8900-4b24-91dc-098f797790e1"
},
{
"instance_id": "92aa44f9-7ce4-4ba4-a29c-e03096ad1047",
"action_done": false,
"host": "overcloud-novacompute-4",
"created_at": "2020-04-15T11:43:09.000000",
"project_state": "ACK_PLANNED_MAINTENANCE",
"updated_at": null,
"session_id": "47479bca-7f0e-11ea-99c9-2c600c9893ee",
"instance_name": "demo_ha_app_1",
"state": "active",
"details": null,
"action": "MIGRATE",
"project_id": "444b05e6f4764189944f00a7288cd281",
"id": "f35c9ba5-e5f7-4843-bae5-7df9bac2a33c"
},
{
"instance_id": "afa2cf43-6a1f-4508-ba59-12b773f8b926",
"action_done": false,
"host": "overcloud-novacompute-0",
"created_at": "2020-04-15T11:43:09.000000",
"project_state": "ACK_PLANNED_MAINTENANCE",
"updated_at": null,
"session_id": "47479bca-7f0e-11ea-99c9-2c600c9893ee",
"instance_name": "demo_nonha_app_4",
"state": "active",
"details": null,
"action": "MIGRATE",
"project_id": "444b05e6f4764189944f00a7288cd281",
"id": "fea38e9b-3d7c-4358-ba2e-06e9c340342d"
}
],
"state": "PLANNED_MAINTENANCE",
"session": {
"workflow": "vnf",
"created_at": "2020-04-15T11:43:09.000000",
"updated_at": "2020-04-15T11:44:04.000000",
"session_id": "47479bca-7f0e-11ea-99c9-2c600c9893ee",
"maintenance_at": "2020-04-15T11:43:28.000000",
"state": "PLANNED_MAINTENANCE",
"prev_state": "START_MAINTENANCE",
"meta": "{'openstack': 'upgrade'}"
},
"hosts": [
{
"created_at": "2020-04-15T11:43:09.000000",
"hostname": "overcloud-novacompute-3",
"updated_at": null,
"session_id": "47479bca-7f0e-11ea-99c9-2c600c9893ee",
"disabled": false,
"maintained": true,
"details": "3de22382-5500-4d13-b9a2-470cc21002ee",
"type": "compute",
"id": "426ea4b9-4438-44ee-9849-1b3ffcc42ad6",
},
{
"created_at": "2020-04-15T11:43:09.000000",
"hostname": "overcloud-novacompute-2",
"updated_at": null,
"session_id": "47479bca-7f0e-11ea-99c9-2c600c9893ee",
"disabled": false,
"maintained": true,
"details": "91457572-dabf-4aff-aab9-e12a5c6656cd",
"type": "compute",
"id": "74f0f6d1-520a-4e5b-b69c-c3265d874b14",
},
{
"created_at": "2020-04-15T11:43:09.000000",
"hostname": "overcloud-novacompute-5",
"updated_at": null,
"session_id": "47479bca-7f0e-11ea-99c9-2c600c9893ee",
"disabled": false,
"maintained": true,
"details": "87921762-0c70-4d3e-873a-240cb2e5c0bf",
"type": "compute",
"id": "8d0f764e-11e8-4b96-8f6a-9c8fc0eebca2",
},
{
"created_at": "2020-04-15T11:43:09.000000",
"hostname": "overcloud-novacompute-1",
"updated_at": null,
"session_id": "47479bca-7f0e-11ea-99c9-2c600c9893ee",
"disabled": false,
"maintained": true,
"details": "52c7270a-cfc2-41dd-a574-f4c4c54aa78d",
"type": "compute",
"id": "be7fd08c-0c5f-4bf4-a95b-bc3b3c01d918",
},
{
"created_at": "2020-04-15T11:43:09.000000",
"hostname": "overcloud-novacompute-0",
"updated_at": null,
"session_id": "47479bca-7f0e-11ea-99c9-2c600c9893ee",
"disabled": true,
"maintained": false,
"details": "ea68bd0d-a5b6-4f06-9bff-c6eb0b248530",
"type": "compute",
"id": "ce46f423-e485-4494-8bb7-e1a2b038bb8e",
},
{
"created_at": "2020-04-15T11:43:09.000000",
"hostname": "overcloud-novacompute-4",
"updated_at": null,
"session_id": "47479bca-7f0e-11ea-99c9-2c600c9893ee",
"disabled": true,
"maintained": false,
"details": "d5271d60-db14-4011-9497-b1529486f62b",
"type": "compute",
"id": "efdf668c-b1cc-4539-bdb6-aea9afbcc897",
},
{
"created_at": "2020-04-15T11:43:09.000000",
"hostname": "overcloud-controller-0",
"updated_at": null,
"session_id": "47479bca-7f0e-11ea-99c9-2c600c9893ee",
"disabled": false,
"maintained": true,
"details": "9a68c85e-42f7-4e40-b64a-2e7a9e2ccd03",
"type": "controller",
"id": "f4631941-8a51-44ee-b814-11a898729f3c",
}
],
"percent_done": 71,
"action_plugin_instances": [
{
"created_at": "2020-04-15 11:12:16",
"updated_at": null,
"id": "4e864972-b692-487b-9204-b4d6470db266",
"session_id": "47479bca-7f0e-11ea-99c9-2c600c9893ee",
"hostname": "overcloud-novacompute-4",
"plugin": "dummy",
"state": null
}
]
}

0
doc/source/api-ref/v1/samples/get-maintenance-session-get-200.json → doc/source/api-ref/v1/samples/maintenance-session-get-200.json

0
doc/source/api-ref/v1/samples/get-maintenance-sessions-get-200.json → doc/source/api-ref/v1/samples/maintenance-sessions-get-200.json

0
doc/source/api-ref/v1/samples/get-project-maintenance-session-post-200.json → doc/source/api-ref/v1/samples/project-maintenance-session-post-200.json

44
doc/source/api-ref/v1/status.yaml

@ -19,28 +19,60 @@
.. literalinclude:: samples/maintenance-session-put-200.json
:language: javascript
get-maintenance-sessions-get: |
maintenance-sessions-get: |
.. rest_parameters:: parameters.yaml
- session_id: uuid-list
.. literalinclude:: samples/get-maintenance-sessions-get-200.json
.. literalinclude:: samples/maintenance-sessions-get-200.json
:language: javascript
get-maintenance-session-get: |
maintenance-session-get: |
.. rest_parameters:: parameters.yaml
- state: workflow-state
.. literalinclude:: samples/get-maintenance-session-get-200.json
.. literalinclude:: samples/maintenance-session-get-200.json
:language: javascript
get-project-maintenance-session-post: |
maintenance-session-detail-get: |
.. rest_parameters:: parameters.yaml
- action: migration-type
- action_done: boolean
- created_at: datetime-string
- details: details
- disabled: boolean
- host: hostname
- hostname: hostname
- id: uuid
- instance_id: uuid
- instance_name: instance-name
- maintained: boolean
- maintenance_at: datetime-string
- meta: metadata
- percent_done: percent_done
- plugin: plugin
- prev_state: workflow-state
- project_id: uuid
- project_state: workflow-state-reply
- session_id: uuid
- state(action_plugin_instances): action-plugin-state
- state(instances): instance-state
- state: workflow-state
- type: host-type
- updated_at: datetime-string
- workflow: workflow-name
.. literalinclude:: samples/maintenance-session-detail-get-200.json
:language: javascript
project-maintenance-session-post: |
.. rest_parameters:: parameters.yaml
- instance_ids: instance-ids
.. literalinclude:: samples/get-project-maintenance-session-post-200.json
.. literalinclude:: samples/project-maintenance-session-post-200.json
:language: javascript
201:

34
doc/source/user/notifications.rst

@ -77,12 +77,38 @@ Example:
Event type 'maintenance.session'
--------------------------------
--Not yet implemented--
This event type is meant for infrastructure admin to know the changes in the
ongoing maintenance workflow session. When implemented, there will not be a need
for polling the state through an API.
ongoing maintenance workflow session. This can be used instead of polling API.
Via API you will get more detailed information if you need to troubleshoot.
payload
~~~~~~~~
+--------------+--------+------------------------------------------------------------------------------+
| Name | Type | Description |
+==============+========+==============================================================================+
| service | string | Origin service name: Fenix |
+--------------+--------+------------------------------------------------------------------------------+
| state | string | Maintenance workflow state (States explained in the user guide) |
+--------------+--------+------------------------------------------------------------------------------+
| session_id | string | UUID of the related maintenance session |
+--------------+--------+------------------------------------------------------------------------------+
| percent_done | string | How many percent of hosts are maintained |
+--------------+--------+------------------------------------------------------------------------------+
| project_id | string | workflow admin project ID |
+--------------+--------+------------------------------------------------------------------------------+
Example:
.. code-block:: json
{
"service": "fenix",
"state": "IN_MAINTENANCE",
"session_id": "76e55df8-1c51-11e8-9928-0242ac110002",
"percent_done": 34,
"project_id": "ead0dbcaf3564cbbb04842e3e54960e3"
}
Project
=======

6
fenix/api/v1/controllers/__init__.py

@ -66,7 +66,11 @@ class V1Controller(rest.RestController):
else:
args[0] = 'http404-nonexistingcontroller'
elif depth == 3 and route == "maintenance":
args[0] = "project"
last = self._routes.get(args[2], args[2])
if last == "detail":
args[0] = "session"
else:
args[0] = "project"
elif depth == 4 and route == "maintenance":
args[0] = "project_instance"
else:

13
fenix/api/v1/controllers/maintenance.py

@ -160,9 +160,10 @@ class SessionController(BaseController):
self.engine_rpcapi = maintenance.EngineRPCAPI()
# GET /v1/maintenance/<session_id>
# GET /v1/maintenance/<session_id>/detail
@policy.authorize('maintenance:session', 'get')
@expose(content_type='application/json')
def get(self, session_id):
def get(self, session_id, detail=None):
try:
jsonschema.validate(session_id, schema.uid)
except jsonschema.exceptions.ValidationError as e:
@ -173,7 +174,15 @@ class SessionController(BaseController):
LOG.error("Unexpected data")
abort(400)
try:
session = self.engine_rpcapi.admin_get_session(session_id)
if detail:
if detail != "detail":
description = "Invalid path %s" % detail
LOG.error(description)
abort(400, six.text_type(description))
session = (
self.engine_rpcapi.admin_get_session_detail(session_id))
else:
session = self.engine_rpcapi.admin_get_session(session_id)
except RemoteError as e:
self.handle_remote_error(e)
if session is None:

6
fenix/api/v1/maintenance.py

@ -37,9 +37,13 @@ class EngineRPCAPI(service.RPCClient):
return self.call('admin_create_session', data=data)
def admin_get_session(self, session_id):
"""Get maintenance workflow session details"""
"""Get maintenance workflow session state"""
return self.call('admin_get_session', session_id=session_id)
def admin_get_session_detail(self, session_id):
"""Get maintenance workflow session details"""
return self.call('admin_get_session_detail', session_id=session_id)
def admin_delete_session(self, session_id):
"""Delete maintenance workflow session thread"""
return self.call('admin_delete_session', session_id=session_id)

52
fenix/db/api.py

@ -115,11 +115,23 @@ def create_session(values):
return IMPL.create_session(values)
def update_session(values):
return IMPL.update_session(values)
def remove_session(session_id):
"""Remove a session from the tables."""
return IMPL.remove_session(session_id)
def get_session(session_id):
return IMPL.maintenance_session_get(session_id)
def get_sessions():
return IMPL.maintenance_session_get_all()
def create_action_plugin(values):
"""Create a action from the values."""
return IMPL.create_action_plugin(values)
@ -129,10 +141,22 @@ def create_action_plugins(session_id, action_dict_list):
return IMPL.create_action_plugins(action_dict_list)
def get_action_plugins(session_id):
return IMPL.action_plugins_get_all(session_id)
def create_action_plugin_instance(values):
return IMPL.create_action_plugin_instance(values)
def get_action_plugin_instances(session_id):
return IMPL.action_plugin_instances_get_all(session_id)
def update_action_plugin_instance(values):
return IMPL.update_action_plugin_instance(values)
def remove_action_plugin_instance(ap_instance):
return IMPL.remove_action_plugin_instance(ap_instance)
@ -141,11 +165,19 @@ def create_downloads(download_dict_list):
return IMPL.create_downloads(download_dict_list)
def get_downloads(session_id):
return IMPL.download_get_all(session_id)
def create_host(values):
"""Create a host from the values."""
return IMPL.create_host(values)
def update_host(values):
return IMPL.update_host(values)
def create_hosts(session_id, hostnames):
hosts = []
for hostname in hostnames:
@ -174,6 +206,10 @@ def create_hosts_by_details(session_id, hosts_dict_list):
return IMPL.create_hosts(hosts)
def get_hosts(session_id):
return IMPL.hosts_get(session_id)
def create_projects(session_id, project_ids):
projects = []
for project_id in project_ids:
@ -185,6 +221,18 @@ def create_projects(session_id, project_ids):
return IMPL.create_projects(projects)
def update_project(values):
return IMPL.update_project(values)
def get_projects(session_id):
return IMPL.projects_get(session_id)
def update_instance(values):
return IMPL.update_instance(values)
def create_instance(values):
"""Create a instance from the values."""
return IMPL.create_instance(values)
@ -199,6 +247,10 @@ def remove_instance(session_id, instance_id):
return IMPL.remove_instance(session_id, instance_id)
def get_instances(session_id):
return IMPL.instances_get(session_id)
def update_project_instance(values):
return IMPL.update_project_instance(values)

2
fenix/db/migration/alembic_migrations/versions/001_initial.py

@ -58,8 +58,6 @@ def upgrade():
sa.Column('maintained', sa.Boolean, default=False),
sa.Column('disabled', sa.Boolean, default=False),
sa.Column('details', sa.String(length=255), nullable=True),
sa.Column('plugin', sa.String(length=255), nullable=True),
sa.Column('plugin_state', sa.String(length=32), nullable=True),
sa.UniqueConstraint('session_id', 'hostname', name='_session_host_uc'),
sa.PrimaryKeyConstraint('id'))

75
fenix/db/sqlalchemy/api.py

@ -135,6 +135,15 @@ def maintenance_session_get(session_id):
return _maintenance_session_get(get_session(), session_id)
def _maintenance_session_get_all(session):
query = model_query(models.MaintenanceSession, session)
return query
def maintenance_session_get_all():
return _maintenance_session_get_all(get_session())
def create_session(values):
values = values.copy()
msession = models.MaintenanceSession()
@ -152,6 +161,18 @@ def create_session(values):
return maintenance_session_get(msession.session_id)
def update_session(values):
session = get_session()
session_id = values.session_id
with session.begin():
msession = _maintenance_session_get(session,
session_id)
msession.update(values)
msession.save(session=session)
return maintenance_session_get(session_id)
def remove_session(session_id):
session = get_session()
with session.begin():
@ -276,6 +297,22 @@ def action_plugin_instances_get_all(session_id):
return _action_plugin_instances_get_all(get_session(), session_id)
def update_action_plugin_instance(values):
session = get_session()
session_id = values.session_id
plugin = values.plugin
hostname = values.hostname
with session.begin():
ap_instance = _action_plugin_instance_get(session,
session_id,
plugin,
hostname)
ap_instance.update(values)
ap_instance.save(session=session)
return action_plugin_instance_get(session_id, plugin, hostname)
def create_action_plugin_instance(values):
values = values.copy()
ap_instance = models.MaintenanceActionPluginInstance()
@ -402,6 +439,18 @@ def create_host(values):
return host_get(mhost.session_id, mhost.hostname)
def update_host(values):
session = get_session()
session_id = values.session_id
hostname = values.hostname
with session.begin():
mhost = _host_get(session, session_id, hostname)
mhost.update(values)
mhost.save(session=session)
return host_get(session_id, hostname)
def create_hosts(values_list):
for values in values_list:
vals = values.copy()
@ -468,6 +517,18 @@ def create_project(values):
return project_get(mproject.session_id, mproject.project_id)
def update_project(values):
session = get_session()
session_id = values.session_id
project_id = values.project_id
with session.begin():
mproject = _project_get(session, session_id, project_id)
mproject.update(values)
mproject.save(session=session)
return project_get(session_id, project_id)
def create_projects(values_list):
for values in values_list:
vals = values.copy()
@ -476,7 +537,7 @@ def create_projects(values_list):
mproject = models.MaintenanceProject()
mproject.update(vals)
if _project_get(session, mproject.session_id,
mproject.project_id):
mproject.project_id):
selected = ['project_id']
raise db_exc.FenixDBDuplicateEntry(
model=mproject.__class__.__name__,
@ -512,6 +573,18 @@ def instances_get(session_id):
return _instances_get(get_session(), session_id)
def update_instance(values):
session = get_session()
session_id = values.session_id
instance_id = values.instance_id
with session.begin():
minstance = _instance_get(session, session_id, instance_id)
minstance.update(values)
minstance.save(session=session)
return instance_get(session_id, instance_id)
def create_instance(values):
values = values.copy()
minstance = models.MaintenanceInstance()

2
fenix/db/sqlalchemy/models.py

@ -99,8 +99,6 @@ class MaintenanceHost(mb.FenixBase):
maintained = sa.Column(sa.Boolean, default=False)
disabled = sa.Column(sa.Boolean, default=False)
details = sa.Column(sa.String(length=255), nullable=True)
plugin = sa.Column(sa.String(length=255), nullable=True)
plugin_state = sa.Column(sa.String(length=32), nullable=True)
def to_dict(self):
return super(MaintenanceHost, self).to_dict()

4
fenix/tests/db/sqlalchemy/test_sqlalchemy_api.py

@ -117,9 +117,7 @@ def _get_fake_host_values(uuid=_get_fake_uuid(),
'type': 'compute',
'maintained': False,
'disabled': False,
'details': None,
'plugin': None,
'plugin_state': None}
'details': None}
return hdict

180
fenix/tools/README.md

@ -10,7 +10,18 @@ Files:
- 'demo-ha.yaml': demo-ha ReplicaSet to make 2 anti-affinity PODS.
- 'demo-nonha.yaml': demo-nonha ReplicaSet to make n nonha PODS.
- 'vnfm.py': VNFM to test k8s.py workflow.
- 'vnfm_k8s.py': VNFM to test k8s.py (Kubernetes example) workflow.
- 'vnfm.py': VNFM to test nfv.py (OpenStack example) workflow.
- 'infra_admin.py': Tool to act as infrastructure admin. Tool catch also
the 'maintenance.session' and 'maintenance.host' events to keep track
where the maintenance is going. You will see when certain host is maintained
and how many percent of hosts are maintained.
- 'session.json': Example to define maintenance session parameters as JSON
file to be given as input to 'infra_admin.py'. Example if for nfv.py workflow.
This could be used for any advanced workflow testing giving software downloads
and real action plugins.
- 'set_config.py': You can use this to set Fenix AODH/Ceilometer configuration.
- 'fenix_db_reset': Flush the Fenix database.
## Kubernetes workflow (k8s.py)
@ -92,7 +103,7 @@ kluster. Under here is what you can run in different terminals. Terminals
should be running in master node. Here is short description:
- Term1: Used for logging Fenix
- Term2: Infrastructure admin commands
- Term2: Infrastructure admin
- Term3: VNFM logging for testing and setting up the VNF
#### Term1: Fenix-engine logging
@ -114,6 +125,8 @@ Debugging and other configuration changes to '.conf' files under '/etc/fenix'
#### Term2: Infrastructure admin window
##### Admin commands as command line and curl
Use DevStack admin as user. Set your variables needed accordingly
```sh
@ -148,12 +161,42 @@ If maintenance run till the end with 'MAINTENANCE_DONE', you are ready to run it
again if you wish. 'MAINTENANCE_FAILED' or in case of exceptions, you should
recover system before trying to test again. This is covered in Term3 below.
#### Term3: VNFM (fenix/tools/vnfm.py)
##### Admin commands using admin tool
Use DevStack admin as user.
Go to Fenix tools directory
```sh
. ~/devstack/operc admin admin
cd /opt/stack/fenix/fenix/tools
```
Call admin tool and it will run the maintenance workflow. Admin tool defaults
to 'OpenStack' and 'nfv' workflow, so you can override those by exporting
environmental variables
```sh
. ~/devstack/openrc admin admin
export WORKFLOW=k8s
export CLOUD_TYPE=k8s
python infra_admin.py
```
If you want to choose freely parameters for maintenance workflow session,
you can give session.json file as input. With this option infra_admin.py
will only override the 'maintenance_at' to be 20seconds in future when
Fenix is called.
```sh
python infra_admin.py --file session.json
```
Maintenance will start by pressing enter, just follow instructions on the
console.
#### Term3: VNFM (fenix/tools/vnfm_k8s.py)
Use DevStack as demo user for testing demo application
```sh
. ~/devstack/operc demo demo
```
Go to Fenix Kubernetes tool directory for testing
@ -181,7 +224,7 @@ is 32 cpus, so value is "15" in both yaml files. Replicas can be changed in
demo-nonha.yaml. Minimum 2 (if minimum of 3 worker nodes) to maximum
'(amount_of_worker_nodes-1)*2'. Greater amount means more scaling needed and
longer maintenance window as less parallel actions possible. Surely constraints
in vnfm.py also can be changed for different behavior.
in vnfm_k8s.py also can be changed for different behavior.
You can delete pods used like this
@ -192,11 +235,11 @@ kubectl delete replicaset.apps demo-ha demo-nonha --namespace=demo
Start Kubernetes VNFM that we need for testing
```sh
python vnfm.py
python vnfm_k8s.py
```
Now you can start maintenance session in Term2. When workflow failed or
completed; you first kill vnfm.py with "ctrl+c" and delete maintenance session
completed; you first kill vnfm_k8s.py with "ctrl+c" and delete maintenance session
in Term2.
If workflow failed something might need to be manually fixed. Here you
@ -221,7 +264,8 @@ kubectl delete replicaset.apps demo-ha demo-nonha --namespace=demo;sleep 15;kube
## OpenStack workflows (default.py and nvf.py)
OpenStack workflows can be tested by using OPNFV Doctor project for testing.
OpenStack workflows can be tested by using OPNFV Doctor project for testing
or to use Fenix own tools.
Workflows:
- default.py is the first example workflow with VNFM interaction.
@ -290,7 +334,7 @@ cpu_allocation_ratio = 1.0
allow_resize_to_same_host = False
```
### Workflow default.py
### Workflow default.py testing with Doctor
On controller node clone Doctor to be able to test. Doctor currently requires
Python 3.6:
@ -331,13 +375,13 @@ sudo systemctl restart devstack@fenix*
You can also make changed to Doctor before running Doctor test
### Workflow vnf.py
### Workflow vnf.py testing with Doctor
This workflow differs from above as it expects ETSI FEAT03 constraints.
In Doctor testing it means we also need to use different application manager (VNFM)
Where default.py worklow used the sample.py application manager vnf.py
workflow uses vnfm.py workflow (doctor/doctor_tests/app_manager/vnfm.py)
workflow uses vnfm_k8s.py workflow (doctor/doctor_tests/app_manager/vnfm_k8s.py)
Only change to testing is that you should export variable to use different
application manager.
@ -354,3 +398,115 @@ export APP_MANAGER_TYPE=sample
```
Doctor modifies the message where it calls maintenance accordingly to use
either 'default' or 'nfv' as workflow in Fenix side
### Workflow vnf.py testing with Fenix
Where Doctor is made to automate everything as a test case, Fenix provides
different tools for admin and VNFM:
- 'vnfm.py': VNFM to test nfv.py.
- 'infra_admin.py': Tool to act as infrastructure admin.
Use 3 terminal windows (Term1, Term2 and Term3) to test Fenix with Kubernetes
kluster. Under here is what you can run in different terminals. Terminals
should be running in master node. Here is short description:
- Term1: Used for logging Fenix
- Term2: Infrastructure admin
- Term3: VNFM logging for testing and setting up the VNF
#### Term1: Fenix-engine logging
If any changes to Fenix make them under '/opt/stack/fenix'; restart Fenix and
see logs
```sh
sudo systemctl restart devstack@fenix*;sudo journalctl -f --unit devstack@fenix-engine
```
API logs can also be seen
```sh
sudo journalctl -f --unit devstack@fenix-api
```
Debugging and other configuration changes to '.conf' files under '/etc/fenix'
#### Term2: Infrastructure admin window
Go to Fenix tools directory for testing
```sh
cd /opt/stack/fenix/fenix/tools
```
Make flavor for testing that takes the half of the amount of VCPUs on single
compute node (here we have 48 VCPUs on each compute) This is required by
the current example 'vnfm.py' and the vnf 'maintenance_hot_tpl.yaml' that
is used in testing. 'vnf.py' workflow is not bind to these in any way, but
can be used with different VNFs and VNFM.
```sh
openstack flavor create --ram 512 --vcpus 24 --disk 1 --public demo_maint_flavor
```
Call admin tool and it will run the nvf.py workflow.
```sh
. ~/devstack/openrc admin admin
python infra_admin.py
```
If you want to choose freely parameters for maintenance workflow session,
you can give 'session.json' file as input. With this option 'infra_admin.py'
will only override the 'maintenance_at' to be 20 seconds in future when
Fenix is called.
```sh
python infra_admin.py --file session.json
```
Maintenance will start by pressing enter, just follow instructions on the
console.
In case you failed to remove maintenance workflow session, you can do it
manually as instructed above in 'Admin commands as command line and curl'.
#### Term3: VNFM (fenix/tools/vnfm.py)
Use DevStack as demo user for testing demo application
```sh
. ~/devstack/openrc demo demo
```
Go to Fenix tools directory for testing
```sh
cd /opt/stack/fenix/fenix/tools
```
Start VNFM that we need for testing
```sh
python vnfm.py
```
Now you can start maintenance session in Term2. When workflow failed or
completed; you first kill vnfm.py with "ctrl+c" and then delete maintenance
session in Term2.
If workflow failed something might need to be manually fixed.
Here you can remove the heat stack if vnfm.py failed to sdo that:
```sh
openstack stack delete -y --wait demo_stack
```
It may also be that workflow failed somewhere in the middle and some
'nova-compute' are disabled. You can enable those. Here you can see the
states:
```sh
openstack compute service list
```

9
fenix/tools/fenix_db_reset

@ -0,0 +1,9 @@
MYSQLPW=admin
# Fenix DB
[ `mysql -uroot -p$MYSQLPW -e "SELECT host, user FROM mysql.user;" | grep fenix | wc -l` -eq 0 ] && {
mysql -uroot -p$MYSQLPW -hlocalhost -e "CREATE USER 'fenix'@'localhost' IDENTIFIED BY 'fenix';"
mysql -uroot -p$MYSQLPW -hlocalhost -e "GRANT ALL PRIVILEGES ON fenix.* TO 'fenix'@'' identified by 'fenix';FLUSH PRIVILEGES;"
}
mysql -ufenix -pfenix -hlocalhost -e "DROP DATABASE IF EXISTS fenix;"
mysql -ufenix -pfenix -hlocalhost -e "CREATE DATABASE fenix CHARACTER SET utf8;"

320
fenix/tools/infra_admin.py

@ -0,0 +1,320 @@
# Copyright (c) 2020 Nokia Corporation.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import aodhclient.client as aodhclient
import argparse
import datetime
from flask import Flask
from flask import request
import json
from keystoneauth1 import loading
from keystoneclient import client as ks_client
import logging as lging
import os
from oslo_config import cfg
from oslo_log import log as logging
import requests
import sys
from threading import Thread
import time
import yaml
try:
import fenix.utils.identity_auth as identity_auth
except ValueError:
sys.path.append('../utils')
import identity_auth
try:
input = raw_input
except NameError:
pass
LOG = logging.getLogger(__name__)
streamlog = lging.StreamHandler(sys.stdout)
formatter = lging.Formatter("%(asctime)s: %(message)s")
streamlog.setFormatter(formatter)
LOG.logger.addHandler(streamlog)
LOG.logger.setLevel(logging.INFO)
def get_identity_auth(conf, project=None, username=None, password=None):
loader = loading.get_plugin_loader('password')
return loader.load_from_options(
auth_url=conf.service_user.os_auth_url,
username=(username or conf.service_user.os_username),
password=(password or conf.service_user.os_password),
user_domain_name=conf.service_user.os_user_domain_name,
project_name=(project or conf.service_user.os_project_name),
tenant_name=(project or conf.service_user.os_project_name),
project_domain_name=conf.service_user.os_project_domain_name)
class InfraAdmin(object):
def __init__(self, conf, log):
self.conf = conf
self.log = log
self.app = None
def start(self):
self.log.info('InfraAdmin start...')
self.app = InfraAdminManager(self.conf, self.log)
self.app.start()
def stop(self):
self.log.info('InfraAdmin stop...')
if not self.app:
return
headers = {
'Content-Type': 'application/json',
'Accept': 'application/json',
}
url = 'http://%s:%d/shutdown'\
% (self.conf.host,
self.conf.port)
requests.post(url, data='', headers=headers)
class InfraAdminManager(Thread):
def __init__(self, conf, log, project='service'):
Thread.__init__(self)
self.conf = conf
self.log = log
self.project = project
# Now we are as admin:admin:admin by default. This means we listen
# notifications/events as admin
# This means Fenix service user needs to be admin:admin:admin
# self.auth = identity_auth.get_identity_auth(conf,
# project=self.project)
self.auth = get_identity_auth(conf,
project='service',
username='fenix',
password='admin')
self.session = identity_auth.get_session(auth=self.auth)
self.keystone = ks_client.Client(version='v3', session=self.session)
self.aodh = aodhclient.Client(2, self.session)
self.headers = {
'Content-Type': 'application/json',
'Accept': 'application/json'}
self.project_id = self.keystone.projects.list(name=self.project)[0].id
self.headers['X-Auth-Token'] = self.session.get_token()
self.create_alarm()
services = self.keystone.services.list()
for service in services:
if service.type == 'maintenance':
LOG.info('maintenance service: %s:%s type %s'
% (service.name, service.id, service.type))
maint_id = service.id
self.endpoint = [ep.url for ep in self.keystone.endpoints.list()
if ep.service_id == maint_id and
ep.interface == 'public'][0]
self.log.info('maintenance endpoint: %s' % self.endpoint)
if self.conf.workflow_file:
with open(self.conf.workflow_file) as json_file:
self.session_request = yaml.safe_load(json_file)
else:
if self.conf.cloud_type == 'openstack':
metadata = {'openstack': 'upgrade'}
elif self.conf.cloud_type in ['k8s', 'kubernetes']:
metadata = {'kubernetes': 'upgrade'}
else:
metadata = {}
self.session_request = {'state': 'MAINTENANCE',
'workflow': self.conf.workflow,
'metadata': metadata,
'actions': [
{"plugin": "dummy",
"type": "host",
"metadata": {"foo": "bar"}}]}
self.start_maintenance()
def create_alarm(self):
alarms = {alarm['name']: alarm for alarm in self.aodh.alarm.list()}
alarm_name = "%s_MAINTENANCE_SESSION" % self.project
if alarm_name not in alarms:
alarm_request = dict(
name=alarm_name,
description=alarm_name,
enabled=True,
alarm_actions=[u'http://%s:%d/maintenance_session'
% (self.conf.host,
self.conf.port)],
repeat_actions=True,
severity=u'moderate',
type=u'event',
event_rule=dict(event_type=u'maintenance.session'))
self.aodh.alarm.create(alarm_request)
alarm_name = "%s_MAINTENANCE_HOST" % self.project
if alarm_name not in alarms:
alarm_request = dict(
name=alarm_name,
description=alarm_name,
enabled=True,
alarm_actions=[u'http://%s:%d/maintenance_host'
% (self.conf.host,
self.conf.port)],
repeat_actions=True,
severity=u'moderate',
type=u'event',
event_rule=dict(event_type=u'maintenance.host'))
self.aodh.alarm.create(alarm_request)
def start_maintenance(self):
self.log.info('Waiting AODH to initialize...')
time.sleep(5)
input('--Press ENTER to start maintenance session--')
maintenance_at = (datetime.datetime.utcnow() +
datetime.timedelta(seconds=20)
).strftime('%Y-%m-%d %H:%M:%S')
self.session_request['maintenance_at'] = maintenance_at
self.headers['X-Auth-Token'] = self.session.get_token()
url = self.endpoint + "/maintenance"
self.log.info('Start maintenance session: %s\n%s\n%s' %
(url, self.headers, self.session_request))
ret = requests.post(url, data=json.dumps(self.session_request),
headers=self.headers)
session_id = ret.json()['session_id']
self.log.info('--== Maintenance session %s instantiated ==--'
% session_id)
def _alarm_data_decoder(self, data):
if "[" in data or "{" in data:
# string to list or dict removing unicode
data = yaml.load(data.replace("u'", "'"))