Spec for smoke test engine
1. What is the problem Problems in the current smoke test: (1) Mix use of bash and python scripts (2) Resources are not cleaned at the end of the test 2. What is the solution for the problem A proposed solution is discussed in the spec. The basic idea is to define tests using YAML and use a engine to parse YAML and run tests. 3. What features need to be implemented to the Tricircle to realize the solution A task engine and a task runner. Change-Id: Ib7ca2242529e147e9edd6292eec3003967ea83f0
This commit is contained in:
zhiyuan_cai
parent
b0a5c6c684
commit
91137e7eaa
219
specs/pike/smoke-test-engine.rst
Normal file
219
specs/pike/smoke-test-engine.rst
Normal file
@ -0,0 +1,219 @@
|
||||
=================
|
||||
Smoke Test Engine
|
||||
=================
|
||||
|
||||
Problems
|
||||
========
|
||||
Currently we are running a simple smoke test in the CI job. Several resources
|
||||
are created to build a simple topology, then we query to check whether the
|
||||
resources are also created in local Neutron servers as expected. The problems
|
||||
exist are:
|
||||
|
||||
- 1 Bash scripts are used to invoke client to send API request while python
|
||||
scripts are used to check the result. Mix use of bash and python makes us
|
||||
hard to write new tests.
|
||||
- 2 Resources are not cleaned at the end of the test so we can't proceed other
|
||||
tests in the same environment.
|
||||
|
||||
Proposal
|
||||
========
|
||||
Using bash scripts to do both API request and result validation is tricky and
|
||||
hard to read, working with python is a better choice. We have several python
|
||||
libraries that can help us to send API request: openstackclient, neutronclient
|
||||
and openstacksdk. The main goal of the first two libraries is providing command
|
||||
line interface(CLI), so they don't expose methods for us to send API request,
|
||||
but we can still use them by calling internal functions that are used by their
|
||||
CLI instance. The drawback of using internal functions is that those internal
|
||||
functions are undocumented and are possible to be changed or removed someday.
|
||||
Compare to openstackclient and neutronclient, openstacksdk is a library that
|
||||
aims for application building and is well-documented. Actually openstackclient
|
||||
uses openstacksdk for some of its commands' implementation. The limitation of
|
||||
openstacksdk is that some service extensions like trunk and service function
|
||||
chaining have not been supported yet, but it's easy to extend by our own.
|
||||
|
||||
Before starting to write python code to prepare, validate and finally clean
|
||||
resources for each test scenario, let's hold on and move one step forward. Heat
|
||||
uses template to define resources and networking topologies that need to be
|
||||
created, we can also use YAML file to describe our test tasks.
|
||||
|
||||
Schema
|
||||
------
|
||||
|
||||
A task can be defined as a dict that has the following basic fields:
|
||||
|
||||
.. csv-table::
|
||||
:header: Field, Type, Description, Required or not
|
||||
:widths: 10, 10, 40, 10
|
||||
|
||||
task_id, string, user specified task ID, required
|
||||
region, string, keystone region to send API, required
|
||||
type, string, resource type, required
|
||||
depend, list, task IDs the current task depends on, optional
|
||||
params, dict, "parameters to run the task, usage differs in different task types", optional
|
||||
|
||||
Currently four type of tasks are defined. The usage of "params" field for each
|
||||
type of task is listed below:
|
||||
|
||||
.. csv-table::
|
||||
:header: Task type, Usage of "params" field
|
||||
:widths: 10, 50
|
||||
|
||||
create, used as the post body of the create request
|
||||
query, used as the query filter
|
||||
action, used as the put body of the action request
|
||||
validate, used as the filter to query resources that need to be validated
|
||||
|
||||
Task doesn't have "task type" field, but it can have an extra dict type field
|
||||
to include extra needed information for that task. This extra field differs in
|
||||
different task types. "Create" task doesn't have an extra field.
|
||||
|
||||
.. list-table::
|
||||
:widths: 15, 10, 10, 40, 10
|
||||
:header-rows: 1
|
||||
|
||||
* - Extra field
|
||||
- Sub field
|
||||
- Type
|
||||
- Description
|
||||
- Required or not
|
||||
* - query(for query task)
|
||||
- get_one
|
||||
- bool
|
||||
- whether to return an element or a list
|
||||
- required
|
||||
* - action(for action task)
|
||||
- target
|
||||
- string
|
||||
- target resource ID
|
||||
- required
|
||||
* -
|
||||
- method
|
||||
- string
|
||||
- action method, "update" and "delete" are also included
|
||||
- required
|
||||
* -
|
||||
- retries
|
||||
- int
|
||||
- times to retry the current task
|
||||
- optional
|
||||
* - validate(for validate task)
|
||||
- predicate
|
||||
- string
|
||||
- value should be "any" or "all", "any" means that for each condition,
|
||||
there exists an resource satisfying that condition; "all" means that
|
||||
every condition is satisfied by all the resources
|
||||
- required
|
||||
* -
|
||||
- condition
|
||||
- list
|
||||
- each condition is a dict, key of the dict is the field of the resource,
|
||||
value of the dict is the expected value of the resource field
|
||||
- required
|
||||
* -
|
||||
- retries
|
||||
- int
|
||||
- times to retry the current task
|
||||
- optional
|
||||
|
||||
Several related tasks can be grouped to form a task set. A task set is a dict
|
||||
with the following fields:
|
||||
|
||||
.. csv-table::
|
||||
:header: Field, Type, Description, Required or not
|
||||
:widths: 10, 10, 40, 10
|
||||
|
||||
task_set_id, string, user specified task set ID, required
|
||||
depend, list, task set IDs the current task set depends on, optional
|
||||
tasks, list, task dicts of the task set, required
|
||||
|
||||
So the YAML file contains a list of task sets.
|
||||
|
||||
Result and Reference
|
||||
--------------------
|
||||
|
||||
"Create" and "query" type tasks will return results, which can be used in the
|
||||
definition of other tasks that depend on them. Use ``task_id@resource_field``
|
||||
to refer to "resource_field" of the resource returned by "task_id". If the task
|
||||
relied on belongs to other task set, use ``task_set_id@task_id@resource_field``
|
||||
to specify the task set ID. The reference can be used in the "params", "action
|
||||
target" and "validate condition" field. If reference is used, task_id needs to
|
||||
be in the list of task's "depend" field, and task_set_id needs to be in the
|
||||
list of task set's "depend" field. For the "query" type task which is depended
|
||||
on, "get_one" field needs to be true.
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
Give an example to show how to use the above schema to define tasks::
|
||||
|
||||
- task_set_id: preparation
|
||||
tasks:
|
||||
- task_id: image1
|
||||
region: region1
|
||||
type: image
|
||||
query:
|
||||
get_one: true
|
||||
- task_id: net1
|
||||
region: central
|
||||
type: network
|
||||
params:
|
||||
name: net1
|
||||
- task_id: subnet1
|
||||
region: central
|
||||
type: subnet
|
||||
depend: [net1]
|
||||
params:
|
||||
name: subnet1
|
||||
ip_version: 4
|
||||
cidr: 10.0.1.0/24
|
||||
network_id: net1@id
|
||||
- task_id: vm1
|
||||
region: region1
|
||||
type: server
|
||||
depend:
|
||||
- net1
|
||||
- subnet1
|
||||
- image1
|
||||
params:
|
||||
flavor_id: 1
|
||||
image_id: image1@id
|
||||
name: vm1
|
||||
networks:
|
||||
- uuid: net1@id
|
||||
- task_set_id: wait-for-job
|
||||
tasks:
|
||||
- task_id: check-job
|
||||
region: central
|
||||
type: job
|
||||
validate:
|
||||
predicate: all
|
||||
retries: 10
|
||||
condition:
|
||||
- status: SUCCESS
|
||||
- task_set_id: check
|
||||
depend: [preparation]
|
||||
tasks:
|
||||
- task_id: check-servers1
|
||||
region: region1
|
||||
type: server
|
||||
validate:
|
||||
predicate: any
|
||||
condition:
|
||||
- status: ACTIVE
|
||||
name: vm1
|
||||
|
||||
The above YAML content define three task sets. "Preparation" task set create
|
||||
network, subnet and server, then "wait-for-job" task set waits for asynchronous
|
||||
jobs to finish, finally "check" task set check whether the server is active.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
A task engine needs to be implemented to parse the YAML file, analyse the task
|
||||
and task set dependency and then run the tasks. A runner based on openstacksdk
|
||||
will also be implemented.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
None
|
||||
Loading…
Reference in New Issue
Block a user