From 4556979745a9ad0aee329afe8ca29cd563855913 Mon Sep 17 00:00:00 2001 From: Chris Hoge Date: Mon, 26 Jan 2015 09:17:41 -0800 Subject: [PATCH] Test Results JSON Schema This schema proposes the development of a json schema for test results to aid in client development and server side data validation. Blueprint: https://blueprints.launchpad.net/refstack/+spec/test-result-json-schema Change-Id: Ic8ba94e9646285f827be5a07c309baf8aa197267 --- .../refstack-org-test-result-json-schema.rst | 176 ++++++++++++++++++ 1 file changed, 176 insertions(+) create mode 100644 specs/proposed/refstack-org-test-result-json-schema.rst diff --git a/specs/proposed/refstack-org-test-result-json-schema.rst b/specs/proposed/refstack-org-test-result-json-schema.rst new file mode 100644 index 00000000..f2655a3b --- /dev/null +++ b/specs/proposed/refstack-org-test-result-json-schema.rst @@ -0,0 +1,176 @@ +========================================== +Example Spec - The title of your blueprint +========================================== + +https://blueprints.launchpad.net/refstack/+spec/test-result-json-schema + +Refstack accepts connections from the refstac-client to post test results +for storage and scoring. Test results are in json and follow a simple +format. However there is no defined schema for the test results, allowing +for a greater possibility of invalid or malicious data to be injected +into the refstack database. This spec proposes the creation of a json +schema that can be used by client developers to generate their own valid +test results. The spec will also be used by the server to validate +incoming test results. + +Problem description +=================== + +* Refstack accepts test results in json format. However, there is no + explicit definition of the structure of this json data. + +* Data sent to the Refstack server is not validated against any schema, + increasing the possibility of invalid or malicious data. + +* Clients developers do not have a specific schema to develop against. + + +Proposed change +=============== + +This change proposes the following additions: + +* A json schema file to capture the format of the test results. + +* Validation of incoming results against the json schema in Refstack. + +Alternatives +------------ + +Another alternative is manually writing code to validate test results. +This schema does not preclue that alternative, but by creating and +validating against a schema we take advantage of existing libraries +and reduce the possiblity of introducing unintended parsing errors. + +Data model impact +----------------- + +This proposed change has two impacts on the data model: + +* It introduces a schema that must be maintained and versioned + against the current data model. + +* It adds an additional level of validation to the 'create' operation + of the CRUD model. Since running a test is a long operation this + addition level of complexity will not have an impact on application + performance. + +* No database migrations will be necessary. + +* A static page to serve the schema will be necessary. + +REST API impact +--------------- + +The v1 api must be modified to validate test results. The modification +will result in an addition to behavior on the API with no changes +to the user-facing endpoint. + +**description:** Receive tempest test result from a remote test runner. +This function expects json formated pass results. +Update to the api described at: +https://github.com/stackforge/refstack/blob/master/specs/approved/api-v1.md + +**url:** post /v1/results + +**failed response:** http:400 - Malformed data. + + { + 'message': 'malformed json data, see /v1/schema/results.json' + } + +**url:** get /v1/schema/results.json + +**valid response:** http:200 results.json file + +No invalid responses. No accepted parameters. + +Security impact +--------------- + +This change is intented to improve the security of the application +by introducing data validation. No data-changing apis are +introduced. + +Notifications impact +-------------------- + +No known notification impacts. + +Other end user impact +--------------------- + +End users may experience client failures if their client does not produce +valid json. + +Performance Impact +------------------ + +This change introduces a validation step to the POST process. The additional +time used to validate the data is very small compared to the time taken +to generate data through cloud testing. The timing impact is negligible +in the context of the upload use case. + +Other deployer impact +--------------------- + +This change will take immediate effect after being merged. + +Developer impact +---------------- + +No additional developer impact. + +Implementation +============== + +This change will be implemented as a validation funtion in the API POST +pipeline. It will essentially be middleware that takes the input data, +validates the results, and sends back a positive or a negative result. +If the result is negative, the 400 response will be returned. +If the result is positive, the data processing will continue as normal. + +Assignee(s) +----------- + +Primary assignee: + hogepodge + +Work Items +---------- + +* Write json schema +* Write json schema GET endpoint +* Implement validation on results POST endpoint. +* Implement unit tests for validation and endpoint. + +Dependencies +============ + +No additional dependencies will be added. + +Testing +======= + +To the TestRefStackApi class the following tests will be added: +* test_results_valid_data +* test_results_invalid_data + +These results will confirm both positive (200) and negative (400) results. + +To the unit tests the validator function will be tested: +* test_valid_data +* test_invalid_data +* test_empty_data + +These results will provide three modes of schema validation. + +Documentation Impact +==================== + +Documentation will be updated to link to current schema. + +References +========== + +No additional references.