Implement additional properties Defcore waiver
Spec describing the features to be added into refstack-client in order to implement Defcore waiver in regards of additional properties on Nova API. Change-Id: I27b55073ea36ea2c12c2893d33e8c8558505ac98
This commit is contained in:
Luz Cazares
parent
7523bc5dbc
commit
5e4553c8e7
264
specs/newton/approved/refstack-waiver.rst
Normal file
264
specs/newton/approved/refstack-waiver.rst
Normal file
@ -0,0 +1,264 @@
|
|||||||
|
======================================================================
|
||||||
|
Refstack-client implementation of Defcore waiver regarding additional
|
||||||
|
properties on Nova API response
|
||||||
|
======================================================================
|
||||||
|
|
||||||
|
Launchpad blueprint: https://blueprints.launchpad.net/refstack/+spec/refstack-waiver
|
||||||
|
|
||||||
|
Defcore waiver: https://review.openstack.org/#/c/333067/
|
||||||
|
|
||||||
|
Defcore committee approved above waiver which allows vendors who are using the
|
||||||
|
Nova 2.0 API with additional properties to disable strict response checking
|
||||||
|
when testing products for the OpenStack Powered program in 2016.
|
||||||
|
|
||||||
|
This spec defines the changes needed for refstack-client to optionally bypass
|
||||||
|
Tempest strict validation.
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
Vendors need an automated way to apply the waiver. The proposed method is to
|
||||||
|
run Tempest from the RefStack client, identify tests that fail because of
|
||||||
|
strict response checking, and rerun those tests with strict checking disabled.
|
||||||
|
|
||||||
|
APIs and test cases using the waiver must be clearly identified.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
1. Workflow
|
||||||
|
|
||||||
|
- Vendor run Tempest suite as usual: via refstack-client test, ostestr, testr
|
||||||
|
or with any other test runner. Some test cases failed due to additional
|
||||||
|
properties in Nova response.
|
||||||
|
|
||||||
|
- Vendor have the subunit test results file from the Tempest test execution.
|
||||||
|
|
||||||
|
- Vendor have the Tempest configuration file.
|
||||||
|
|
||||||
|
- Vendor rerun failed test cases by running ``refstack-client bypass-extras``
|
||||||
|
command. Command identifies failed test cases, disables Tempest strict
|
||||||
|
validations, rerun test cases, and enables strict validations again.
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
$ refstack-client bypass-extras --subunit-file <results> --conf-file <tempest-conf-file>
|
||||||
|
|
||||||
|
- Output of bypass-extras command is a zip bundle containing the following
|
||||||
|
files:
|
||||||
|
|
||||||
|
- tests_list - List of failed test cases due to additional properties.
|
||||||
|
|
||||||
|
- patched_schemas - List of tempest schemas which value was set to True
|
||||||
|
(to allow additional properties).
|
||||||
|
|
||||||
|
- api_details - API call details from each failed test case (due to
|
||||||
|
additional properties).
|
||||||
|
|
||||||
|
- rerun_test_results - The subunit result file for the re-run test cases.
|
||||||
|
|
||||||
|
- combined_test_results.json - The Refstack JSON file with the combined
|
||||||
|
passed TCs from both initial and rerun subunit files.
|
||||||
|
|
||||||
|
2. Implement "bypass-extras" Refstack command:
|
||||||
|
Assume Tempest test suite was run independently.
|
||||||
|
Subunit test results and Tempest configuration files are available.
|
||||||
|
|
||||||
|
*bypass-extras* command is the helper tool for vendors to bypass the strict
|
||||||
|
validation of additional properties in Tempest. Process steps and
|
||||||
|
implementation details are explained on step 3.
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
$ refstack-client --help
|
||||||
|
|
||||||
|
usage: refstack-client [-h] <ARG> ...
|
||||||
|
...
|
||||||
|
bypass-extras Apply Defcore waiver to identify additional properties
|
||||||
|
on Nova API response. Re-runs failed test cases
|
||||||
|
without Tempest strict response validations.
|
||||||
|
|
||||||
|
$ refstack-client bypass-extras --help
|
||||||
|
|
||||||
|
usage: refstack-client [-h] <ARG> ...
|
||||||
|
|
||||||
|
To see help on specific argument, do:
|
||||||
|
refstack-client <ARG> -h waiver
|
||||||
|
[-h] [-s | -v] [-y] [--url URL] [-k] [-i PRIV_KEY] file
|
||||||
|
|
||||||
|
optional arguments:
|
||||||
|
-h, --help Show this help message and exit
|
||||||
|
-s, --silent Suppress output except warnings and errors.
|
||||||
|
-v, --verbose Show verbose output.
|
||||||
|
-y Assume Yes to all prompt queries
|
||||||
|
--subunit-file Path to subunit test result file.
|
||||||
|
-c, --conf-file Path of the Tempest configuration file to use.
|
||||||
|
|
||||||
|
3. Flow for ``bypass-extras`` command.
|
||||||
|
|
||||||
|
Having as input a subunit test results file and a Tempest configuration file:
|
||||||
|
|
||||||
|
3.1 Find failed test cases and its details
|
||||||
|
|
||||||
|
Integrate code from find_additional_properties.py into Refstack-client to
|
||||||
|
analyze subunit stream (from input results file). Find failed test cases
|
||||||
|
due to additional properties in the response. Reconstruct the tempest schema
|
||||||
|
causing the test case failure. Run subunit-describe-calls
|
||||||
|
filter command to get test cases API call details.
|
||||||
|
|
||||||
|
Input: subunit-results
|
||||||
|
|
||||||
|
Output files:
|
||||||
|
|
||||||
|
- tests_list - List of failed test cases due to additional properties.
|
||||||
|
|
||||||
|
- patched_schemas - List of tempest schemas causing errors
|
||||||
|
|
||||||
|
- api_details - API calls from each test case.
|
||||||
|
|
||||||
|
3.2 Patch Tempest:
|
||||||
|
|
||||||
|
Create patch for .tempest virtual environment which lives under refstack-client
|
||||||
|
installation.
|
||||||
|
|
||||||
|
- Modify tempest/lib/api_schema/response/compute/v2_1/__init__.py:
|
||||||
|
|
||||||
|
- Import module where schema lives.
|
||||||
|
|
||||||
|
- Set schema addtionalProperties key to True so that additional properties
|
||||||
|
are accepted - bypass strict validation.
|
||||||
|
|
||||||
|
3.3 Rerun failed test cases using patched refstack-client .tempest environment
|
||||||
|
|
||||||
|
Use tests_list as withelist for ostestr in order to re-run failed test cases.
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
ostestr --serial -w test_list
|
||||||
|
|
||||||
|
Input: test_list and conf-file files.
|
||||||
|
|
||||||
|
Output: rerun_test_results subunit file
|
||||||
|
|
||||||
|
3.4 Remove Tempest patch
|
||||||
|
|
||||||
|
Regardless of previous steps outcome, unpatching Tempest step will be
|
||||||
|
attempted.
|
||||||
|
|
||||||
|
Clean __init__.py by opening with access mode 'w'
|
||||||
|
|
||||||
|
3.5 Create refstack JSON format files
|
||||||
|
|
||||||
|
Transform subunit result files - The one provided as input and the rerun
|
||||||
|
test results - into a combined refstack JSON format.
|
||||||
|
|
||||||
|
Input: initial_results_file, rerun_test_results files.
|
||||||
|
|
||||||
|
Output: combined_test_results.json files.
|
||||||
|
|
||||||
|
3.6 Create zip bundle
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
- Add additional property to Tempest config file
|
||||||
|
|
||||||
|
- For Tempest patch
|
||||||
|
Comment the validate_response call by looking into the service_client.py file
|
||||||
|
for the corresponding method (search through code files).
|
||||||
|
|
||||||
|
- Manual process
|
||||||
|
Products applying for the OpenStack Powered Trademark in 2016 may
|
||||||
|
request the waiver by submitting subunit data from their Tempest run
|
||||||
|
that can be manually analyzed by the `find_additional_properties.py` script
|
||||||
|
from the DefCore repository. This script will identify tests that
|
||||||
|
failed because of additional properties. The vendor will then need
|
||||||
|
to manually modify tempest-lib to remove additional checks on the impacted
|
||||||
|
APIs.
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Notifications impact
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
|
||||||
|
luz-cazares
|
||||||
|
|
||||||
|
Other contributors:
|
||||||
|
|
||||||
|
Chris Hoge
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
- Add *bypass-extras* command to refstack-client.
|
||||||
|
- Integrate find_additional_properties code
|
||||||
|
- Method to call subunit-describe-calls filter
|
||||||
|
- Implement JSON schema gathering from test exception
|
||||||
|
- Implement Tempest patch
|
||||||
|
- Implement Tempest unpatch
|
||||||
|
- Create zip bundle with file results
|
||||||
|
- Add *bypass-extras* command usage documentation
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
Add unit testing for the new command, verify expected outcomes are met.
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
Add refstack-client bypass-extras usage information under refstack-client/
|
||||||
|
README.rst
|
||||||
|
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
None
|
||||||
Loading…
Reference in New Issue
Block a user