Management Driver Improvement for Error Message

This patch defined interface between Tacker and Management Driver,
and resolved lack of information in error message of
Ansible Management Driver.

Co-Authored-By: Masaki Oyama <ma-ooyama@kddi.com>
Co-Authored-By: Hitomi Koba <hi-koba@kddi.com>
Co-Authored-By: Yukihiro Kinjo <yu-kinjou@kddi.com>
Implements: blueprint improving-mgmt-driver-log
Change-Id: I59154647fc75d149ce835a74a8332462d2ff9581
This commit is contained in:
Hongjin Xu 2023-03-07 05:32:22 +00:00
parent af88b178a9
commit 8200eac023
11 changed files with 281 additions and 51 deletions

View File

@ -18,6 +18,9 @@
Tacker Admin Guide
==================
Command List
------------
The following list covers the commands for Tacker services available in
**openstack** command.
@ -75,3 +78,12 @@ of individual command can be referred by **openstack help <command-name>**.
openstack vnf network forwarding path show Show information of a given NFP.
openstack nfv event show Show event given the event id.
openstack nfv event list List events of resources.
Mgmt Driver
-----------
.. toctree::
:maxdepth: 1
mgmt_driver_error_handling

View File

@ -0,0 +1,77 @@
==============================
Error Handling for Mgmt Driver
==============================
Overview
--------
This document aims to explain the definition of error handling interface
between Tacker and Mgmt Driver for developers.
Error Handling Interface between Tacker and Mgmt Driver
-------------------------------------------------------
The definition of error handling interface between Tacker and Mgmt Driver
follows below rules.
- Conform to the format of ``ProblemDetails``
defined in `ETSI GS NFV-SOL 013 v2.6.1`_
- Tacker-conductor must be able to catch the type of exception.
.. note:: For backward compatibility reasons, exceptions of any type
can also be received. In that case, the exception is converted to a
string and stored in the error field of VnfLcmOpOccs
as the current implementation.
The definition of ``ProblemDetails`` is as follows.
.. csv-table:: Table 6.3-1: Definition of the ``ProblemDetails`` data type `ETSI GS NFV-SOL 013 v2.6.1`_
:header: Attribute name, Data type, Cardinality, Description
type,Uri,0..1, "A URI reference according to `IETF RFC 3986`_ that identifies
the problem type. It is encouraged that the URI provides
human-readable documentation for the problem (e.g. using
HTML) when dereferenced. When this member is not present,
its value is assumed to be 'about:blank'."
title,String,0..1, "A short, human-readable summary of the problem type.
It should not change from occurrence to occurrence of the problem,
except for purposes of localization. If type is given and other
than 'about:blank', this attribute shall also be provided."
status,Integer,1, "The HTTP status code for this
occurrence of the problem."
detail,String,1, "A human-readable explanation specific
to this occurrence of the problem."
instance,Uri,0..1, "A URI reference that identifies the specific
occurrence of the problem. It may yield further
information if dereferenced"
(additional attributes),Not specified.,0..N, "Any number of additional
attributes, as defined in a specification or by an implementation."
To implement the rule, there is a base exception class
complied with ``ProblemDetails`` in Tacker side, which can be
used by the developers of Mgmt Driver to make exceptions in their Mgmt Driver
that compatible with ``ProblemDetails``.
The base exception class is defined in `tacker/common/exceptions.py`,
and the format is shown below.
.. code-block:: python
class MgmtDriverException(TackerException):
def __init__(self, type=None, title=None, status, detail, instance=None)
self.type = type
self.title = title
self.status = status
self.detail = detail
self.instance = instance
In order to realize the compliance of the exceptions made by developers
with ``ProblemDetails``, the exceptions in Mgmt Driver should inherit
the base exception class as below.
.. code-block:: python
class SampleException(MgmtDriverException):
.. _ETSI GS NFV-SOL 013 v2.6.1: https://www.etsi.org/deliver/etsi_gs/NFV-SOL/001_099/013/02.06.01_60/gs_nfv-sol013v020601p.pdf
.. _IETF RFC 3986: https://www.rfc-editor.org/rfc/rfc3986

View File

@ -17,6 +17,7 @@ from tacker.vnfm.mgmt_drivers import constants as mgmt_constants
from tacker.vnfm.mgmt_drivers import vnflcm_abstract_driver
from tacker.vnfm.mgmt_drivers.ansible import ansible_driver
from tacker.vnfm.mgmt_drivers.ansible import exceptions
LOG = logging.getLogger(__name__)
@ -51,6 +52,8 @@ class DeviceMgmtAnsible(vnflcm_abstract_driver.VnflcmMgmtAbstractDriver):
driver._driver_process_flow(context, vnf_instance,
mgmt_constants.ACTION_INSTANTIATE_VNF,
instantiate_vnf_request, **kwargs)
except exceptions.AnsibleDriverException as e:
raise e
except Exception as e:
raise Exception("Ansible Driver Error: %s",
encodeutils.exception_to_unicode(e))
@ -65,6 +68,8 @@ class DeviceMgmtAnsible(vnflcm_abstract_driver.VnflcmMgmtAbstractDriver):
driver._driver_process_flow(context, vnf_instance,
mgmt_constants.ACTION_TERMINATE_VNF,
terminate_vnf_request, **kwargs)
except exceptions.AnsibleDriverException as e:
raise e
except Exception as e:
raise Exception("Ansible Driver Error: %s",
encodeutils.exception_to_unicode(e))
@ -84,6 +89,8 @@ class DeviceMgmtAnsible(vnflcm_abstract_driver.VnflcmMgmtAbstractDriver):
driver._driver_process_flow(context, vnf_instance,
mgmt_constants.ACTION_SCALE_IN_VNF,
scale_vnf_request, **kwargs)
except exceptions.AnsibleDriverException as e:
raise e
except Exception as e:
raise Exception("Ansible Driver Error: %s",
encodeutils.exception_to_unicode(e))
@ -98,6 +105,8 @@ class DeviceMgmtAnsible(vnflcm_abstract_driver.VnflcmMgmtAbstractDriver):
driver._driver_process_flow(context, vnf_instance,
mgmt_constants.ACTION_SCALE_OUT_VNF,
scale_vnf_request, **kwargs)
except exceptions.AnsibleDriverException as e:
raise e
except Exception as e:
raise Exception("Ansible Driver Error: %s",
encodeutils.exception_to_unicode(e))
@ -117,6 +126,8 @@ class DeviceMgmtAnsible(vnflcm_abstract_driver.VnflcmMgmtAbstractDriver):
driver._driver_process_flow(context, vnf_instance,
mgmt_constants.ACTION_HEAL_VNF,
heal_vnf_request, **kwargs)
except exceptions.AnsibleDriverException as e:
raise e
except Exception as e:
raise Exception("Ansible Driver Error: %s",
encodeutils.exception_to_unicode(e))

View File

@ -130,18 +130,19 @@ class AnsibleDriver(object):
action_interface = 'heal_end'
action_value = interfaces_vnflcm_value.get(action_interface, {})
action_dependencies = (action_value.get('implementation', {})
.get('dependencies', []))
# NOTE: Currently action_dependencies is having the value of
# last element in the list because in the current specification
# only a single value is available for dependencies.
if isinstance(action_dependencies, list):
for arti in action_dependencies:
action_dependencies = arti
filename = artifacts_vnflcm_value.get(action_dependencies,
{}).get('file', {})
implimentation_value = action_value.get('implementation', {})
if isinstance(implimentation_value, dict):
action_dependencies = implimentation_value.get('dependencies', [])
# NOTE: Currently action_dependencies is having the value of
# last element in the list because in the current specification
# only a single value is available for dependencies.
if isinstance(action_dependencies, list):
action_dependencies = action_dependencies[-1]
filename = artifacts_vnflcm_value.get(action_dependencies,
{}).get('file', {})
else:
filename = {}
# load the configuration file
config_yaml = self._load_ansible_config(request_obj, filename)
@ -289,6 +290,8 @@ class AnsibleDriver(object):
return yaml.dump(config_data)
def _load_ansible_config(self, request_obj, filename):
if not filename:
return
# load vnf package path
vnf_package_path = vnflcm_utils._get_vnf_package_path(self._context,
self._vnf_instance.vnfd_id)

View File

@ -96,8 +96,9 @@ class AnsiblePlaybookExecutor(executor.Executor):
return path
def _get_final_command(self, playbook_cmd):
init_cmd = ("cd {} ; ansible-playbook -i {} -vvv {} "
"--extra-vars \"host={} node_pair_ip={}".format(
init_cmd = ("cd {} ;ANSIBLE_DISPLAY_FAILED_STDERR=yes ansible-playbook"
" -i {} -vvv {} --extra-vars \"host={} node_pair_ip={}"
.format(
os.path.dirname(self._get_playbook_path(playbook_cmd)),
self._get_playbook_target_hosts(playbook_cmd),
self._get_playbook_path(playbook_cmd),

View File

@ -170,7 +170,7 @@ class Executor(config_walker.VmAppConfigWalker):
res_code = -1
try:
res_code, host = self._execute_cmd(
res_code, host, std_error = self._execute_cmd(
cmd,
retry_count,
retry_interval,
@ -179,16 +179,26 @@ class Executor(config_walker.VmAppConfigWalker):
)
except exceptions.AnsibleDriverException:
raise
except Exception as ex:
raise exceptions.CommandExecutionError(vdu=self._vdu,
details=ex)
except Exception:
raise exceptions.CommandExecutionError(
type=None,
title="VNF configuration failed: [VDU: {}, cmd: {}]".
format(self._vdu, playbook_cmd),
status=500,
detail=std_error,
instance=None
)
self._post_execution(cmd, res_code, host)
if self._is_execution_error(res_code):
raise exceptions.CommandExecutionError(
vdu=self._vdu,
details="Non-zero return code"
type=None,
title="VNF configuration failed: [VDU: {}, cmd: {}]".
format(self._vdu, playbook_cmd),
status=500,
detail=std_error,
instance=None
)
def _sort_playbook_cmd_list(self, playbook_cmd_list):
@ -241,10 +251,10 @@ class Executor(config_walker.VmAppConfigWalker):
LOG.debug("command execution result code: {}".format(
result.returncode))
LOG.debug("command execution result code: {}".format(std_out))
LOG.debug("command execution result code: {}".format(std_err))
LOG.debug("command execution result stdout: {}".format(std_out))
LOG.debug("command execution result stderr: {}".format(std_err))
return result.returncode, host
return result.returncode, host, std_err
def _post_execution(self, cmd, res_code, host):

View File

@ -10,15 +10,22 @@
# License for the specific language governing permissions and limitations
# under the License.
from tacker.common.exceptions import MgmtDriverException
class AnsibleDriverException(Exception):
def __init__(self, vdu=None, **kwargs):
if "details" not in kwargs or not kwargs["details"]:
kwargs["details"] = "No error information available."
self.vdu = vdu
self.message = self.message % kwargs
super(AnsibleDriverException, self).__init__(self.message)
class AnsibleDriverException(MgmtDriverException):
def __init__(self, vdu=None, status=500, detail='',
type=None, title=None, instance=None, **kwargs):
if not title and vdu:
title = "VNF configuration failed:[VDU: {}]".format(vdu)
if detail == '':
if "details" not in kwargs or not kwargs["details"]:
kwargs["details"] = "No error information available."
detail = self.message % kwargs
super().__init__(status, detail, type=type,
title=title, instance=instance)
class InternalErrorException(AnsibleDriverException):
@ -37,8 +44,8 @@ class ConfigParserConfigurationError(AnsibleDriverException):
- ex_type: the exception type
- details: the exception message or error information
"""
message = "Parameter conversion error. "
"Error encountered in configuring parser: [%(ex_type)s, %(details)s]"
message = "Parameter conversion error. "\
+ "Error encountered in configuring parser: [%(ex_type)s, %(details)s]"
class ConfigParserParsingError(AnsibleDriverException):
@ -49,8 +56,8 @@ class ConfigParserParsingError(AnsibleDriverException):
- ex_type: the exception type
- details: the exception message or error information
"""
message = "Parameter conversion error. "
"Encountered error in parsing '%(cmd)s': [%(ex_type)s, %(details)s]"
message = "Parameter conversion error. "\
+ "Encountered error in parsing '%(cmd)s': [%(ex_type)s, %(details)s]"
class ConfigValidationError(AnsibleDriverException):
@ -68,8 +75,8 @@ class MandatoryKeyNotDefinedError(AnsibleDriverException):
Define the following upon using this exception:
- key: the offending key
"""
message = "Config file validation error. "
"The key '%(key)s' is not defined."
message = "Config file validation error. "\
+ "The key '%(key)s' is not defined."
class InvalidValueError(AnsibleDriverException):
@ -78,8 +85,8 @@ class InvalidValueError(AnsibleDriverException):
Define the following upon using this exception:
- key: the offending key
"""
message = "Config file validation error. "
"Invalid value of '%(key)s' is defined."
message = "Config file validation error. "\
+ "Invalid value of '%(key)s' is defined."
class PlaybooksCommandsNotFound(AnsibleDriverException):
@ -88,8 +95,8 @@ class PlaybooksCommandsNotFound(AnsibleDriverException):
Define the following upon using this exception:
- key: the offending action key
"""
message = "Config file validation error. "
"Playbooks or commands not found for action key: %(key)s"
message = "Config file validation error. "\
+ "Playbooks or commands not found for action key: %(key)s"
class InvalidKeyError(AnsibleDriverException):
@ -116,7 +123,7 @@ class CommandExecutionError(AnsibleDriverException):
Define the following upon using this exception:
- details: the exception message or error information
"""
message = "Command execution error: %(details)s"
pass
class CommandExecutionTimeoutError(AnsibleDriverException):
@ -126,8 +133,8 @@ class CommandExecutionTimeoutError(AnsibleDriverException):
- host: the target host for execution
- cmd: the command executed that caused the error
"""
message = "Command execution has reached timeout. "
"Target: %(host)s Command: %(cmd)s"
message = "Command execution has reached timeout. "\
+ "Target: %(host)s Command: %(cmd)s"
class CommandConnectionLimitReached(AnsibleDriverException):

View File

@ -176,6 +176,19 @@ class DuplicatedExtension(TackerException):
class MgmtDriverException(TackerException):
"""Management Driver Exception.
These attributes are defined in "ProblemDetails"
in ETSI GS NFV-SOL013 v2.6.1.
"""
def __init__(self, status, detail, type=None, title=None, instance=None):
self.type = type
self.title = title
self.status = status
self.detail = detail
self.instance = instance
super().__init__(self.title)
message = _("VNF configuration failed")

View File

@ -1661,11 +1661,18 @@ class Conductor(manager.Manager, v2_hook.ConductorV2Hook):
fields.LcmOccsOperationState.FAILED_TEMP
or operation_state == fields.LcmOccsOperationState.FAILED):
vnf_lcm_op_occ.error_point = error_point
error_details = objects.ProblemDetails(
context=context,
status=500,
detail=error
)
if isinstance(error, exceptions.MgmtDriverException):
error_details = objects.ProblemDetails(
title=error.title,
status=error.status,
detail=error.detail
)
else:
error_details = objects.ProblemDetails(
context=context,
status=500,
detail=str(error)
)
vnf_lcm_op_occ.error = error_details
vnf_lcm_op_occ.save()
except exceptions.VnfInstanceNotFound:
@ -2034,7 +2041,7 @@ class Conductor(manager.Manager, v2_hook.ConductorV2Hook):
vnf_instance=vnf_instance,
operation=fields.LcmOccsOperationType.INSTANTIATE,
operation_state=fields.LcmOccsOperationState.FAILED_TEMP,
error=str(ex),
error=ex,
error_point=vnf_dict['current_error_point']
)

View File

@ -952,6 +952,93 @@ class TestConductor(SqlTestCase, unit_base.FixturedTestCase):
vnf_instance.id, mock.ANY, 'ERROR')
mock_update_vnf_attributes.assert_called_once()
@mock.patch('tacker.conductor.conductor_server.Conductor'
'._change_vnf_status')
@mock.patch('tacker.conductor.conductor_server.Conductor'
'._build_instantiated_vnf_info')
@mock.patch.object(objects.VnfLcmOpOcc, "save")
@mock.patch.object(coordination.Coordinator, 'get_lock')
@mock.patch('tacker.vnflcm.utils._get_vnfd_dict')
@mock.patch('tacker.vnflcm.utils._convert_desired_capacity')
@mock.patch('tacker.conductor.conductor_server.LOG')
@mock.patch.object(objects.VnfLcmOpOcc, "get_by_id")
@mock.patch('tacker.vnflcm.utils._get_affected_resources')
def test_instantiate_vnf_instance_error_mgmt_driver_exception(
self, mock_res, mock_vnf_by_id, mock_log,
mock_des, mock_vnfd_dict,
mock_get_lock, mock_save,
mock_build_instantiated_vnf_info,
mock_change_vnf_status):
lcm_op_occs_data = fakes.get_lcm_op_occs_data()
mock_vnf_by_id.return_value = \
objects.VnfLcmOpOcc(context=self.context,
**lcm_op_occs_data)
vnf_package_vnfd = self._create_and_upload_vnf_package()
vnf_instance_data = fake_obj.get_vnf_instance_data(
vnf_package_vnfd.vnfd_id)
vnf_instance = objects.VnfInstance(context=self.context,
**vnf_instance_data)
vnf_instance.create()
instantiate_vnf_req = vnflcm_fakes.get_instantiate_vnf_request_obj()
vnf_lcm_op_occs_id = uuidsentinel.vnf_lcm_op_occs_id
vnf_lcm_op_occ = objects.VnfLcmOpOcc.get_by_id(context,
vnf_lcm_op_occs_id)
vnf_dict = db_utils.get_dummy_vnf_etsi(instance_id=self.instance_uuid,
flavour=instantiate_vnf_req.flavour_id)
vnf_dict['before_error_point'] = fields.ErrorPoint.INITIAL
# MgmtDriverException type ex test
ex = exceptions.MgmtDriverException(title='test_title',
status=501, detail='test_detail')
self.vnflcm_driver.instantiate_vnf.side_effect = ex
self.conductor.instantiate(self.context, vnf_instance, vnf_dict,
instantiate_vnf_req, vnf_lcm_op_occs_id)
self.assertEqual(ex.title, vnf_lcm_op_occ.error.title)
self.assertEqual(ex.status, vnf_lcm_op_occ.error.status)
self.assertEqual(ex.detail, vnf_lcm_op_occ.error.detail)
@mock.patch('tacker.conductor.conductor_server.Conductor'
'._change_vnf_status')
@mock.patch('tacker.conductor.conductor_server.Conductor'
'._build_instantiated_vnf_info')
@mock.patch.object(objects.VnfLcmOpOcc, "save")
@mock.patch.object(coordination.Coordinator, 'get_lock')
@mock.patch('tacker.vnflcm.utils._get_vnfd_dict')
@mock.patch('tacker.vnflcm.utils._convert_desired_capacity')
@mock.patch('tacker.conductor.conductor_server.LOG')
@mock.patch.object(objects.VnfLcmOpOcc, "get_by_id")
@mock.patch('tacker.vnflcm.utils._get_affected_resources')
def test_instantiate_vnf_instance_error_invalid(
self, mock_res, mock_vnf_by_id, mock_log,
mock_des, mock_vnfd_dict,
mock_get_lock, mock_save,
mock_build_instantiated_vnf_info,
mock_change_vnf_status):
lcm_op_occs_data = fakes.get_lcm_op_occs_data()
mock_vnf_by_id.return_value = \
objects.VnfLcmOpOcc(context=self.context,
**lcm_op_occs_data)
vnf_package_vnfd = self._create_and_upload_vnf_package()
vnf_instance_data = fake_obj.get_vnf_instance_data(
vnf_package_vnfd.vnfd_id)
vnf_instance = objects.VnfInstance(context=self.context,
**vnf_instance_data)
vnf_instance.create()
instantiate_vnf_req = vnflcm_fakes.get_instantiate_vnf_request_obj()
vnf_lcm_op_occs_id = uuidsentinel.vnf_lcm_op_occs_id
vnf_lcm_op_occ = objects.VnfLcmOpOcc.get_by_id(context,
vnf_lcm_op_occs_id)
vnf_dict = db_utils.get_dummy_vnf_etsi(instance_id=self.instance_uuid,
flavour=instantiate_vnf_req.flavour_id)
vnf_dict['before_error_point'] = fields.ErrorPoint.INITIAL
# string type ex test
ex = exceptions.Invalid()
self.vnflcm_driver.instantiate_vnf.side_effect = ex
self.conductor.instantiate(self.context, vnf_instance, vnf_dict,
instantiate_vnf_req, vnf_lcm_op_occs_id)
self.assertEqual(500, vnf_lcm_op_occ.error.status)
@mock.patch('tacker.conductor.conductor_server.Conductor'
'.send_notification')
@mock.patch('tacker.conductor.conductor_server.Conductor'

View File

@ -562,7 +562,8 @@ class TestVNFMPlugin(db_base.SqlTestCase):
def test_create_vnf_mgmt_driver_exception(self, mock_mgmt_call):
self._insert_dummy_vnf_template()
vnf_obj = utils.get_dummy_vnf_obj()
mock_mgmt_call.side_effect = exceptions.MgmtDriverException
mock_mgmt_call.side_effect = exceptions.MgmtDriverException(
title="test_title", status=500, detail="test_detail")
vnf_dict = self.vnfm_plugin.create_vnf(self.context, vnf_obj)
self.assertEqual(constants.ERROR,
vnf_dict['status'])
@ -822,7 +823,8 @@ class TestVNFMPlugin(db_base.SqlTestCase):
self._insert_dummy_vnf_template()
dummy_vnf_obj = self._insert_dummy_vnf()
vnf_config_obj = utils.get_dummy_vnf_config_obj()
mock_mgmt_call.side_effect = exceptions.MgmtDriverException
mock_mgmt_call.side_effect = exceptions.MgmtDriverException(
title="test_title", status=500, detail="test_detail")
vnf_dict = self.vnfm_plugin.update_vnf(self.context,
dummy_vnf_obj['id'],
vnf_config_obj)
@ -830,7 +832,7 @@ class TestVNFMPlugin(db_base.SqlTestCase):
vnf_dict['status'])
mock_set_vnf_error_status_reason.assert_called_once_with(self.context,
dummy_vnf_obj['id'],
'VNF configuration failed')
'test_title')
@patch('tacker.db.vnfm.vnfm_db.VNFMPluginDb.set_vnf_error_status_reason')
def test_update_vnf_fail_update_wait_error(self,