17 KiB

Raw Blame History

Enhance Placement Process

https://blueprints.launchpad.net/tacker/+spec/enhance-placement

This specification supports the placement functionality for VNF instance and enhances the applicability of Tacker to various systems.

Problem description

The placement constraints are defined in ETSI NFV-SOL 003 v3.3.1 ¹ and that VNFM sends to NFVO in order to the resource placement decision. In VNF Lifecycle Management (LCM), there are some cases that VNFs are not deployed due to placement constraints or lack of availability zone's resources. From a high-availability perspective, Tacker needs to implement a fallbackBestEffort option in Grant Request and availability zone reselection functions.

Proposed change

1. Add fallbackBestEffort parameter

fallbackBestEffort is to look for an alternate best effort placement for specified resources cannot be allocated based on specified placement constraint and defined in ETSI NFV-SOL 003 v3.3.1². On this specification, Tacker will support additional parameter fallbackBestEffort in Grant Request (PlacementConstrains) and Tacker configuration settings. We will modify Tacker conductor and tacker.conf. Add definitions details are described in 3. Add definitions to the configuration file<configuration-file>.

GrantRequest.PlacementConstrains

Attribute name Data type Cardinality Description

fallbackBestEffort Boolean 0..1 Indication if the constraint is handled with fall back best effort. Default value is "false".

Attribute name	Data type	Cardinality	Description
fallbackBestEffort	Boolean	0..1	Indication if the constraint is handled with fall back best effort. Default value is "false".

Note

If fallbackBestEffort is present in placement constraints and set to "true", the NFVO shall process the Affinity/Anti-Affinity constraint in a best effort manner, in which case, if specified resources cannot be allocated based on specified placement constraint, the NFVO looks for an alternate best effort placement for the specified resources to be granted.

2. Availability zone reselection

The VNFLCM v2 API (instantiate/heal/scale for VNF) process can change the availability zone to be used from the one notified by the NFVO if necessary. If the availability zone notified by the NFVO has insufficient resources, the VNF is created/updated in a different availability zone. The availability zone is reselected considering Affinity/Anti-Affinity and re-create/update until there are no more candidates.

1) Flowchart of availability zone reselection

blockdiag {

orientation = portrait; edge_layout = flowchart;

create [shape=flowchart.condition,label='stack create/update']; start [shape=beginpoint]; lin [shape=flowchart.loopin,label='stack create/updatenin other AZ']; recreate [shape=flowchart.condition,label='stack create/update']; lout [shape=flowchart.loopout,label='no more AZncandidates']; failure [shape=endpoint,label='end']; success [shape=endpoint,label='end'];

class none [shape=none]; n [class=none];

start -> create [label='NFVO AZ'] create -> lin [label=error] lin -> recreate [label='other AZ'] recreate -> lout [label='error'] lout -> failure [label='failure'];

create -> success;

recreate -> n [dir=none] n -> success [label='success'];

}

The procedure consists of the following steps as illustrated in above flow:

Execute "stack create/update" in the availability zone notified by the NFVO.
If an error occurs in 1, get the availability zone list. Availability zone list details are described in 2) Get and manage availability zone list<az-list>.
Select an availability zone (excluding the Availability Zone where the error occurred) randomly in compliance with Affinity/Anti-Affinity from the availability zone list obtained in 2, and re-execute “stack create/update”. Reselection policy details are described in 3) Availability zone reselection policy<reselection-policy>.
If "stack create/update" in the availability zone reselected in 3 becomes an error, reselect the availability zone and repeat until "stack create/update" succeeds or until all availability zone candidates fail. Detecting error details are described in 4) Detection method of insufficient resource error<detection-method>.

2) Get and manage availability zone list

Get availability zone list

Extract all availability zones as candidates for reselection without limiting the availability zones to be reselected. Although it is possible to extract only availability zones permitted by Grant as candidates for reselection, this is not adopted in this Spec because it depends on the NFVO product specifications.
The concept of availability zone exists for Compute/Volume/Network, but this Spec targets only Compute. The reason is that SOL(SOL003 v3.3.1 Type: GrantInfo ³) specifies that the zoneId of GrantInfo, which is the data type of addResources, etc., is usually specified as a COMPUTE resource.

Note

SOL003 v3.3.1 Type: GrantInfo

Reference to the identifier of the "ZoneInfo" structure in the "Grant" structure defining the resource zone into which this resource is to be placed. Shall be present for new resources if the zones concept is applicable to them (typically, Compute resources) and shall be absent for resources that have already been allocated. Shall be present for new resources if the zones concept is applicable to them (typically, Compute resources) and shall be absent for resources that have been allocated.
Call the Compute-API "GetDetailedAvailabilityZoneInformation" ⁴ to get the availability zones from the "hosts" response associated with "nova-compute".

Compute endpoints are obtained in the following way.
1. Get Keystone endpoint from VnfInstance.vimConnectionInfo.interfaceInfo.endpoint
2. Call "List endpoints"⁵ and "List services"⁶ of Keystone-API to link obtained Compute's services and endpoint

Manage availability zone list

Keep in on-memory until availability zone reselection iterations are completed, and discard after completion (no storage in DB).

Note

Error-Handling Retry consideration

Since the availability zone list is not saved and is retrieved again, there is no guarantee that the availability zone is reselected in the same order when Retry is executed.

3) Availability zone reselection policy

Availability zones in error are excluded from the reselection candidates, and Availability zones are reselected randomly in compliance with Affinity/Anti-Affinity of PlacementConstraint.

The availability zone in error can be identified in the following way.

Call Heat-API "Show stack details" after an error occurs in "stack create/update"
Identify the VDU where the error occurred due to insufficient resource by the stack_status_reason in the response of 1.
Identify the availability zone by the VDU identified in 2.

Note

Insufficient resource in availability zones that once failed during reselection attempts may be resolved, but the availability zones will not be reselected. In Scale/Heal operations, VDUs that have already been deployed will not be re-created.

Availability zone reselection for each PlacementConstraint is as follows.

Precondition: availability zone AZ-1/AZ-2/AZ-3 exist and VNF VDU-1/VDU-2 are deployed

PlacementConstraint is Anti-Affinity
- Before reselection, the following attempts to deploy failed (AZ-1 has insufficient resource)
  - VDU-1: AZ-1
  - VDU-2: AZ-2
- Reselect the following (except AZ-1, select AZ-2/AZ-3 in compliance with Anti-Affinity)
  - VDU-1: AZ-2
  - VDU-2: AZ-3
Note

The above is an example, and it is possible that the reverse availability zones are selected for VDU-1 and VDU-2, but it is guaranteed that they will not be the same availability zone.
PlacementConstraint is Affinity
- Before reselection, attempt to deploy in the following and fail (AZ-1 has insufficient resource)
  - VDU-1: AZ-1
  - VDU-2: AZ-1
- Reselect the following (except AZ-1, select AZ-2/AZ-3 in compliance with Affinity)
  - VDU-1: AZ-2
  - VDU-2: AZ-2
Note

The above is an example, and it is possible that the availability zone AZ-3 is selected for VDU-1 and VDU-2, but it is guaranteed that they will be the same availability zone.

4) Detection method of insufficient resource error

When "stack create/update" fails, it is detected from "Show stack details" ⁷ of Heat-API response whether the failure is due to insufficient resources. The error message that indicates insufficient resources is extracted from the parameter "stack_status_reason" in the response.

Note

In the case of insufficient resources, the error occurs after "stack create/update" returns an acceptance response, so the "Show stack details" response is used to detect the cause.

The following is an example of an error message stored in "stack_status_reason" when resources are insufficient.

ex1) Set the flavor defined in “OS::Nova::Server” to a large value that cannot be deployed (not enough storage/not enough vcpu/not enough memory).
- Resource CREATE failed: ResourceInError: resources.<VDU-name>: Went to status ERROR due to “Message: No valid host was found. , Code: 500”
ex2) Specifies an extra-spec that cannot be assigned for the flavor defined in "OS::Nova::Server."
- Resource CREATE failed: ResourceInError: resources.<VDU-name>: Went to status ERROR due to “Message: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance <server-UUID>., Code: 500”

Error messages that Tacker detects as insufficient resources are specified by a regular expression in the configuration file. Add definitions details are described in 3. Add definitions to the configuration file<configuration-file>.

By changing the method of specifying this regular expression in accordance with the operational policy, it is possible to flexibly set a policy to detect more error messages as insufficient resource with a higher tolerance for misdetection, or to detect only specific error messages as insufficient resource.

ex1) Regular expression for a policy to detect more error messages as insufficient resource by increasing the tolerance for misclassification
- Resource CREATE failed:(. *)
ex2) Regular expression to specify the policy to detect more error messages as insufficient resource with higher tolerance for false positives
- Resource CREATE failed: ResourceInError: resources(. *): Went to status ERROR due to "Message: No valid host was found. *): Went to status ERROR due to "Message: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance(. *). , Code: 500".

5) AutoScalingGroup consideration

In BaseHOT which includes AutoScalingGroup definitions, there is a constraint that each VNFC associated with a VDU under AutoScalingGroup cannot be set to Anti-Affinity for the availability zone. This constraint is due to the constraint in the HOT specification that availability zones can only be set for each VDU under the AutoScalingGroup. This constraint occurs not only at the time of reselection, but also at the time of initial execution. Therefore, BaseHOT which includes AutoScalingGroup definitions, ignores the PlacementConstraint for each VNFC associated with a VDU and reselects a single availability zone for each VDU under the AutoScalingGroup. (Always set to Affinity.)

top HOT:

resources:
  VDU1_scale_group:
    type: OS::Heat::AutoScalingGroup
    properties:
      min_size: 1
      max_size: 3
      desired_capacity: { get_param: [ nfv, VDU, VDU1, desired_capacity ] }
      resource:
        type: VDU1.yaml
        properties:
          flavor: { get_param: [ nfv, VDU, VDU1, computeFlavourId ] }
          image: { get_param: [ nfv, VDU, VDU1-VirtualStorage, vcImageId ] }
          zone: { get_param: [ nfv, VDU, VDU1, locationConstraints] }
          net1: { get_param: [ nfv, CP, VDU1_CP1, network] }
          net2: { get_param: [ nfv, CP, VDU1_CP2, network ] }
          subnet1: { get_param: [nfv, CP, VDU1_CP1, fixed_ips, 0, subnet ]}
          subnet2: { get_param: [nfv, CP, VDU1_CP2, fixed_ips, 0, subnet ]}
          net3: { get_resource: internalVL1 }
          net4: { get_resource: internalVL2 }
          net5: { get_resource: internalVL3 }

nested HOT:

resources:
  VDU1:
    type: OS::Nova::Server
    properties:
      flavor: { get_param: flavor }
      name: VDU1
      block_device_mapping_v2: [{"volume_id": { get_resource: VDU1-VirtualStorage }}]
      networks:
      - port:
          get_resource: VDU1_CP1
      - port:
          get_resource: VDU1_CP2
      - port:
          get_resource: VDU1_CP3
      - port:
          get_resource: VDU1_CP4
      - port:
          get_resource: VDU1_CP5
      availability_zone: { get_param: zone }

As shown above, top HOT specifies a single "zone" (availability zone) for each VDU under the AutoScalingGroup, so each VNFC associated with a VDU under the AutoScalingGroup is in the same availability zone.

3. Add definitions to the configuration file

Add the following definition to the tacker.conf file.

Boolean value of "GrantRequest.PlacementConstrains.fallbackBestEffort"

Default value: "false"
Whether or not to reselect availability zone

Default value: not to reselect
Regular expression for detecting insufficient resource error

Default value: regular expression for insufficient resource error

Note

Consider the regular expression that can catch stack create and stack update errors.
Maximum number of retries for reselection of availability zone

Default value: no upper limit

Note

Consider the case where there are a large number of availability zones and the availability zone reselection process takes too long.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

Yuta Kazato <yuta.kazato.nw@hco.ntt.co.jp>

Hirofumi Noguchi <hirofumi.noguchi.rs@hco.ntt.co.jp>

Other contributors:

Hiroo Kitamura <hiroo.kitamura@ntt-at.co.jp>

Ai Hamano <ai.hamano@ntt-at.co.jp>

Work Items

Implement availability zone reselection functions.
Add new parameter fallbackBestEffort in GrantRequest API.
Add new definitions to the Tacker configuration file tacker.conf.
Add new unit and functional tests.
Add new examples to the Tacker User Guide.

Dependencies

VNF Lifecycle Operation Granting interface (Grant Lifecycle Operation)⁸
VNF Lifecycle Management interface (Instantiate/Heal/Scale VNF)⁹

Testing

Unit and functional test cases will be added for the new placement functionalities.

Documentation Impact

New supported functions need to be added into the Tacker User Guide.

17 KiB Raw Blame History

Enhance Placement Process

Problem description

Proposed change

1. Add fallbackBestEffort parameter

2. Availability zone reselection

1) Flowchart of availability zone reselection

2) Get and manage availability zone list

3) Availability zone reselection policy

4) Detection method of insufficient resource error

5) AutoScalingGroup consideration

3. Add definitions to the configuration file

Data model impact

REST API impact

Security impact

Notifications impact

Other end user impact

Performance Impact

Other deployer impact

Developer impact

Implementation

Assignee(s)

Work Items

Dependencies

Testing

Documentation Impact

References

17 KiB

Raw Blame History