Implement additional properties Defcore waiver
Spec describing the features to be added into refstack-client in order to implement Defcore waiver in regards of additional properties on Nova API. Change-Id: I27b55073ea36ea2c12c2893d33e8c8558505ac98
This commit is contained in:
Luz Cazares
parent
7523bc5dbc
commit
5e4553c8e7
264
specs/newton/approved/refstack-waiver.rst
Normal file
264
specs/newton/approved/refstack-waiver.rst
Normal file
@ -0,0 +1,264 @@
|
||||
======================================================================
|
||||
Refstack-client implementation of Defcore waiver regarding additional
|
||||
properties on Nova API response
|
||||
======================================================================
|
||||
|
||||
Launchpad blueprint: https://blueprints.launchpad.net/refstack/+spec/refstack-waiver
|
||||
|
||||
Defcore waiver: https://review.openstack.org/#/c/333067/
|
||||
|
||||
Defcore committee approved above waiver which allows vendors who are using the
|
||||
Nova 2.0 API with additional properties to disable strict response checking
|
||||
when testing products for the OpenStack Powered program in 2016.
|
||||
|
||||
This spec defines the changes needed for refstack-client to optionally bypass
|
||||
Tempest strict validation.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Vendors need an automated way to apply the waiver. The proposed method is to
|
||||
run Tempest from the RefStack client, identify tests that fail because of
|
||||
strict response checking, and rerun those tests with strict checking disabled.
|
||||
|
||||
APIs and test cases using the waiver must be clearly identified.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
1. Workflow
|
||||
|
||||
- Vendor run Tempest suite as usual: via refstack-client test, ostestr, testr
|
||||
or with any other test runner. Some test cases failed due to additional
|
||||
properties in Nova response.
|
||||
|
||||
- Vendor have the subunit test results file from the Tempest test execution.
|
||||
|
||||
- Vendor have the Tempest configuration file.
|
||||
|
||||
- Vendor rerun failed test cases by running ``refstack-client bypass-extras``
|
||||
command. Command identifies failed test cases, disables Tempest strict
|
||||
validations, rerun test cases, and enables strict validations again.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ refstack-client bypass-extras --subunit-file <results> --conf-file <tempest-conf-file>
|
||||
|
||||
- Output of bypass-extras command is a zip bundle containing the following
|
||||
files:
|
||||
|
||||
- tests_list - List of failed test cases due to additional properties.
|
||||
|
||||
- patched_schemas - List of tempest schemas which value was set to True
|
||||
(to allow additional properties).
|
||||
|
||||
- api_details - API call details from each failed test case (due to
|
||||
additional properties).
|
||||
|
||||
- rerun_test_results - The subunit result file for the re-run test cases.
|
||||
|
||||
- combined_test_results.json - The Refstack JSON file with the combined
|
||||
passed TCs from both initial and rerun subunit files.
|
||||
|
||||
2. Implement "bypass-extras" Refstack command:
|
||||
Assume Tempest test suite was run independently.
|
||||
Subunit test results and Tempest configuration files are available.
|
||||
|
||||
*bypass-extras* command is the helper tool for vendors to bypass the strict
|
||||
validation of additional properties in Tempest. Process steps and
|
||||
implementation details are explained on step 3.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ refstack-client --help
|
||||
|
||||
usage: refstack-client [-h] <ARG> ...
|
||||
...
|
||||
bypass-extras Apply Defcore waiver to identify additional properties
|
||||
on Nova API response. Re-runs failed test cases
|
||||
without Tempest strict response validations.
|
||||
|
||||
$ refstack-client bypass-extras --help
|
||||
|
||||
usage: refstack-client [-h] <ARG> ...
|
||||
|
||||
To see help on specific argument, do:
|
||||
refstack-client <ARG> -h waiver
|
||||
[-h] [-s | -v] [-y] [--url URL] [-k] [-i PRIV_KEY] file
|
||||
|
||||
optional arguments:
|
||||
-h, --help Show this help message and exit
|
||||
-s, --silent Suppress output except warnings and errors.
|
||||
-v, --verbose Show verbose output.
|
||||
-y Assume Yes to all prompt queries
|
||||
--subunit-file Path to subunit test result file.
|
||||
-c, --conf-file Path of the Tempest configuration file to use.
|
||||
|
||||
3. Flow for ``bypass-extras`` command.
|
||||
|
||||
Having as input a subunit test results file and a Tempest configuration file:
|
||||
|
||||
3.1 Find failed test cases and its details
|
||||
|
||||
Integrate code from find_additional_properties.py into Refstack-client to
|
||||
analyze subunit stream (from input results file). Find failed test cases
|
||||
due to additional properties in the response. Reconstruct the tempest schema
|
||||
causing the test case failure. Run subunit-describe-calls
|
||||
filter command to get test cases API call details.
|
||||
|
||||
Input: subunit-results
|
||||
|
||||
Output files:
|
||||
|
||||
- tests_list - List of failed test cases due to additional properties.
|
||||
|
||||
- patched_schemas - List of tempest schemas causing errors
|
||||
|
||||
- api_details - API calls from each test case.
|
||||
|
||||
3.2 Patch Tempest:
|
||||
|
||||
Create patch for .tempest virtual environment which lives under refstack-client
|
||||
installation.
|
||||
|
||||
- Modify tempest/lib/api_schema/response/compute/v2_1/__init__.py:
|
||||
|
||||
- Import module where schema lives.
|
||||
|
||||
- Set schema addtionalProperties key to True so that additional properties
|
||||
are accepted - bypass strict validation.
|
||||
|
||||
3.3 Rerun failed test cases using patched refstack-client .tempest environment
|
||||
|
||||
Use tests_list as withelist for ostestr in order to re-run failed test cases.
|
||||
|
||||
.. code-block:: bash
|
||||
ostestr --serial -w test_list
|
||||
|
||||
Input: test_list and conf-file files.
|
||||
|
||||
Output: rerun_test_results subunit file
|
||||
|
||||
3.4 Remove Tempest patch
|
||||
|
||||
Regardless of previous steps outcome, unpatching Tempest step will be
|
||||
attempted.
|
||||
|
||||
Clean __init__.py by opening with access mode 'w'
|
||||
|
||||
3.5 Create refstack JSON format files
|
||||
|
||||
Transform subunit result files - The one provided as input and the rerun
|
||||
test results - into a combined refstack JSON format.
|
||||
|
||||
Input: initial_results_file, rerun_test_results files.
|
||||
|
||||
Output: combined_test_results.json files.
|
||||
|
||||
3.6 Create zip bundle
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
- Add additional property to Tempest config file
|
||||
|
||||
- For Tempest patch
|
||||
Comment the validate_response call by looking into the service_client.py file
|
||||
for the corresponding method (search through code files).
|
||||
|
||||
- Manual process
|
||||
Products applying for the OpenStack Powered Trademark in 2016 may
|
||||
request the waiver by submitting subunit data from their Tempest run
|
||||
that can be manually analyzed by the `find_additional_properties.py` script
|
||||
from the DefCore repository. This script will identify tests that
|
||||
failed because of additional properties. The vendor will then need
|
||||
to manually modify tempest-lib to remove additional checks on the impacted
|
||||
APIs.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
None
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
None
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
None
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
|
||||
luz-cazares
|
||||
|
||||
Other contributors:
|
||||
|
||||
Chris Hoge
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
- Add *bypass-extras* command to refstack-client.
|
||||
- Integrate find_additional_properties code
|
||||
- Method to call subunit-describe-calls filter
|
||||
- Implement JSON schema gathering from test exception
|
||||
- Implement Tempest patch
|
||||
- Implement Tempest unpatch
|
||||
- Create zip bundle with file results
|
||||
- Add *bypass-extras* command usage documentation
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
None
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Add unit testing for the new command, verify expected outcomes are met.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
Add refstack-client bypass-extras usage information under refstack-client/
|
||||
README.rst
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
None
|
||||
Loading…
Reference in New Issue
Block a user