From 5e4553c8e72d0e06f29140544cf717b868b7cb19 Mon Sep 17 00:00:00 2001 From: Luz Cazares Date: Sat, 30 Jul 2016 22:33:30 +0000 Subject: [PATCH] Implement additional properties Defcore waiver Spec describing the features to be added into refstack-client in order to implement Defcore waiver in regards of additional properties on Nova API. Change-Id: I27b55073ea36ea2c12c2893d33e8c8558505ac98 --- specs/newton/approved/refstack-waiver.rst | 264 ++++++++++++++++++++++ 1 file changed, 264 insertions(+) create mode 100644 specs/newton/approved/refstack-waiver.rst diff --git a/specs/newton/approved/refstack-waiver.rst b/specs/newton/approved/refstack-waiver.rst new file mode 100644 index 0000000..c0b26b7 --- /dev/null +++ b/specs/newton/approved/refstack-waiver.rst @@ -0,0 +1,264 @@ +====================================================================== +Refstack-client implementation of Defcore waiver regarding additional +properties on Nova API response +====================================================================== + +Launchpad blueprint: https://blueprints.launchpad.net/refstack/+spec/refstack-waiver + +Defcore waiver: https://review.openstack.org/#/c/333067/ + +Defcore committee approved above waiver which allows vendors who are using the +Nova 2.0 API with additional properties to disable strict response checking +when testing products for the OpenStack Powered program in 2016. + +This spec defines the changes needed for refstack-client to optionally bypass +Tempest strict validation. + +Problem description +=================== + +Vendors need an automated way to apply the waiver. The proposed method is to +run Tempest from the RefStack client, identify tests that fail because of +strict response checking, and rerun those tests with strict checking disabled. + +APIs and test cases using the waiver must be clearly identified. + +Proposed change +=============== + +1. Workflow + +- Vendor run Tempest suite as usual: via refstack-client test, ostestr, testr + or with any other test runner. Some test cases failed due to additional + properties in Nova response. + +- Vendor have the subunit test results file from the Tempest test execution. + +- Vendor have the Tempest configuration file. + +- Vendor rerun failed test cases by running ``refstack-client bypass-extras`` + command. Command identifies failed test cases, disables Tempest strict + validations, rerun test cases, and enables strict validations again. + +.. code-block:: bash + + $ refstack-client bypass-extras --subunit-file --conf-file + +- Output of bypass-extras command is a zip bundle containing the following + files: + + - tests_list - List of failed test cases due to additional properties. + + - patched_schemas - List of tempest schemas which value was set to True + (to allow additional properties). + + - api_details - API call details from each failed test case (due to + additional properties). + + - rerun_test_results - The subunit result file for the re-run test cases. + + - combined_test_results.json - The Refstack JSON file with the combined + passed TCs from both initial and rerun subunit files. + +2. Implement "bypass-extras" Refstack command: +Assume Tempest test suite was run independently. +Subunit test results and Tempest configuration files are available. + +*bypass-extras* command is the helper tool for vendors to bypass the strict +validation of additional properties in Tempest. Process steps and +implementation details are explained on step 3. + +.. code-block:: bash + + $ refstack-client --help + + usage: refstack-client [-h] ... + ... + bypass-extras Apply Defcore waiver to identify additional properties + on Nova API response. Re-runs failed test cases + without Tempest strict response validations. + + $ refstack-client bypass-extras --help + + usage: refstack-client [-h] ... + + To see help on specific argument, do: + refstack-client -h waiver + [-h] [-s | -v] [-y] [--url URL] [-k] [-i PRIV_KEY] file + + optional arguments: + -h, --help Show this help message and exit + -s, --silent Suppress output except warnings and errors. + -v, --verbose Show verbose output. + -y Assume Yes to all prompt queries + --subunit-file Path to subunit test result file. + -c, --conf-file Path of the Tempest configuration file to use. + +3. Flow for ``bypass-extras`` command. + +Having as input a subunit test results file and a Tempest configuration file: + +3.1 Find failed test cases and its details + +Integrate code from find_additional_properties.py into Refstack-client to +analyze subunit stream (from input results file). Find failed test cases +due to additional properties in the response. Reconstruct the tempest schema +causing the test case failure. Run subunit-describe-calls +filter command to get test cases API call details. + +Input: subunit-results + +Output files: + +- tests_list - List of failed test cases due to additional properties. + +- patched_schemas - List of tempest schemas causing errors + +- api_details - API calls from each test case. + +3.2 Patch Tempest: + +Create patch for .tempest virtual environment which lives under refstack-client +installation. + +- Modify tempest/lib/api_schema/response/compute/v2_1/__init__.py: + + - Import module where schema lives. + + - Set schema addtionalProperties key to True so that additional properties + are accepted - bypass strict validation. + +3.3 Rerun failed test cases using patched refstack-client .tempest environment + +Use tests_list as withelist for ostestr in order to re-run failed test cases. + +.. code-block:: bash + ostestr --serial -w test_list + +Input: test_list and conf-file files. + +Output: rerun_test_results subunit file + +3.4 Remove Tempest patch + +Regardless of previous steps outcome, unpatching Tempest step will be +attempted. + +Clean __init__.py by opening with access mode 'w' + +3.5 Create refstack JSON format files + +Transform subunit result files - The one provided as input and the rerun +test results - into a combined refstack JSON format. + +Input: initial_results_file, rerun_test_results files. + +Output: combined_test_results.json files. + +3.6 Create zip bundle + +Alternatives +------------ + +- Add additional property to Tempest config file + +- For Tempest patch + Comment the validate_response call by looking into the service_client.py file + for the corresponding method (search through code files). + +- Manual process + Products applying for the OpenStack Powered Trademark in 2016 may + request the waiver by submitting subunit data from their Tempest run + that can be manually analyzed by the `find_additional_properties.py` script + from the DefCore repository. This script will identify tests that + failed because of additional properties. The vendor will then need + to manually modify tempest-lib to remove additional checks on the impacted + APIs. + +Data model impact +----------------- + +None + +REST API impact +--------------- + +None + +Security impact +--------------- + +None + +Notifications impact +-------------------- + +None + +Other end user impact +--------------------- + +None + +Performance Impact +------------------ + +None + +Other deployer impact +--------------------- + +None + +Developer impact +---------------- + +None + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + + luz-cazares + +Other contributors: + + Chris Hoge + +Work Items +---------- + +- Add *bypass-extras* command to refstack-client. +- Integrate find_additional_properties code +- Method to call subunit-describe-calls filter +- Implement JSON schema gathering from test exception +- Implement Tempest patch +- Implement Tempest unpatch +- Create zip bundle with file results +- Add *bypass-extras* command usage documentation + +Dependencies +============ + +None + +Testing +======= + +Add unit testing for the new command, verify expected outcomes are met. + +Documentation Impact +==================== + +Add refstack-client bypass-extras usage information under refstack-client/ +README.rst + + +References +========== + +None