1. What is the problem Problems in the current smoke test: (1) Mix use of bash and python scripts (2) Resources are not cleaned at the end of the test 2. What is the solution for the problem A proposed solution is discussed in the spec. The basic idea is to define tests using YAML and use a engine to parse YAML and run tests. 3. What features need to be implemented to the Tricircle to realize the solution A task engine and a task runner. Change-Id: Ib7ca2242529e147e9edd6292eec3003967ea83f0
7.0 KiB
Smoke Test Engine
Problems
Currently we are running a simple smoke test in the CI job. Several resources are created to build a simple topology, then we query to check whether the resources are also created in local Neutron servers as expected. The problems exist are:
- 1 Bash scripts are used to invoke client to send API request while python scripts are used to check the result. Mix use of bash and python makes us hard to write new tests.
- 2 Resources are not cleaned at the end of the test so we can't proceed other tests in the same environment.
Proposal
Using bash scripts to do both API request and result validation is tricky and hard to read, working with python is a better choice. We have several python libraries that can help us to send API request: openstackclient, neutronclient and openstacksdk. The main goal of the first two libraries is providing command line interface(CLI), so they don't expose methods for us to send API request, but we can still use them by calling internal functions that are used by their CLI instance. The drawback of using internal functions is that those internal functions are undocumented and are possible to be changed or removed someday. Compare to openstackclient and neutronclient, openstacksdk is a library that aims for application building and is well-documented. Actually openstackclient uses openstacksdk for some of its commands' implementation. The limitation of openstacksdk is that some service extensions like trunk and service function chaining have not been supported yet, but it's easy to extend by our own.
Before starting to write python code to prepare, validate and finally clean resources for each test scenario, let's hold on and move one step forward. Heat uses template to define resources and networking topologies that need to be created, we can also use YAML file to describe our test tasks.
Schema
A task can be defined as a dict that has the following basic fields:
Field | Type | Description | Required or not |
---|---|---|---|
task_id | string | user specified task ID | required |
region | string | keystone region to send API | required |
type | string | resource type | required |
depend | list | task IDs the current task depends on | optional |
params | dict | parameters to run the task, usage differs in different task types | optional |
Currently four type of tasks are defined. The usage of "params" field for each type of task is listed below:
Task type | Usage of "params" field |
---|---|
create | used as the post body of the create request |
query | used as the query filter |
action | used as the put body of the action request |
validate | used as the filter to query resources that need to be validated |
Task doesn't have "task type" field, but it can have an extra dict type field to include extra needed information for that task. This extra field differs in different task types. "Create" task doesn't have an extra field.
Extra field | Sub field | Type | Description | Required or not |
---|---|---|---|---|
query(for query task) | get_one | bool | whether to return an element or a list | required |
action(for action task) | target | string | target resource ID | required |
method | string | action method, "update" and "delete" are also included | required | |
retries | int | times to retry the current task | optional | |
validate(for validate task) | predicate | string | value should be "any" or "all", "any" means that for each condition, there exists an resource satisfying that condition; "all" means that every condition is satisfied by all the resources | required |
condition | list | each condition is a dict, key of the dict is the field of the resource, value of the dict is the expected value of the resource field | required | |
retries | int | times to retry the current task | optional |
Several related tasks can be grouped to form a task set. A task set is a dict with the following fields:
Field | Type | Description | Required or not |
---|---|---|---|
task_set_id | string | user specified task set ID | required |
depend | list | task set IDs the current task set depends on | optional |
tasks | list | task dicts of the task set | required |
So the YAML file contains a list of task sets.
Result and Reference
"Create" and "query" type tasks will return results, which can be
used in the definition of other tasks that depend on them. Use
task_id@resource_field
to refer to "resource_field" of the
resource returned by "task_id". If the task relied on belongs to other
task set, use task_set_id@task_id@resource_field
to specify
the task set ID. The reference can be used in the "params", "action
target" and "validate condition" field. If reference is used, task_id
needs to be in the list of task's "depend" field, and task_set_id needs
to be in the list of task set's "depend" field. For the "query" type
task which is depended on, "get_one" field needs to be true.
Example
Give an example to show how to use the above schema to define tasks:
- task_set_id: preparation
tasks:
- task_id: image1
region: region1
type: image
query:
get_one: true
- task_id: net1
region: central
type: network
params:
name: net1
- task_id: subnet1
region: central
type: subnet
depend: [net1]
params:
name: subnet1
ip_version: 4
cidr: 10.0.1.0/24
network_id: net1@id
- task_id: vm1
region: region1
type: server
depend:
- net1
- subnet1
- image1
params:
flavor_id: 1
image_id: image1@id
name: vm1
networks:
- uuid: net1@id
- task_set_id: wait-for-job
tasks:
- task_id: check-job
region: central
type: job
validate:
predicate: all
retries: 10
condition:
- status: SUCCESS
- task_set_id: check
depend: [preparation]
tasks:
- task_id: check-servers1
region: region1
type: server
validate:
predicate: any
condition:
- status: ACTIVE
name: vm1
The above YAML content define three task sets. "Preparation" task set create network, subnet and server, then "wait-for-job" task set waits for asynchronous jobs to finish, finally "check" task set check whether the server is active.
Implementation
A task engine needs to be implemented to parse the YAML file, analyse the task and task set dependency and then run the tasks. A runner based on openstacksdk will also be implemented.
Dependencies
None