Test Results JSON Schema

This schema proposes the development of a json schema for test results
to aid in client development and server side data validation.

Blueprint:
https://blueprints.launchpad.net/refstack/+spec/test-result-json-schema

Change-Id: Ic8ba94e9646285f827be5a07c309baf8aa197267

specs/proposed/refstack-org-test-result-json-schema.rst

@@ -0,0 +1,176 @@
+==========================================
+Example Spec - The title of your blueprint
+==========================================
+
+https://blueprints.launchpad.net/refstack/+spec/test-result-json-schema
+
+Refstack accepts connections from the refstac-client to post test results
+for storage and scoring. Test results are in json and follow a simple
+format. However there is no defined schema for the test results, allowing
+for a greater possibility of invalid or malicious data to be injected
+into the refstack database. This spec proposes the creation of a json
+schema that can be used by client developers to generate their own valid
+test results. The spec will also be used by the server to validate
+incoming test results.
+
+Problem description
+===================
+
+* Refstack accepts test results in json format. However, there is no
+  explicit definition of the structure of this json data.
+
+* Data sent to the Refstack server is not validated against any schema,
+  increasing the possibility of invalid or malicious data.
+
+* Clients developers do not have a specific schema to develop against.
+
+
+Proposed change
+===============
+
+This change proposes the following additions:
+
+* A json schema file to capture the format of the test results.
+
+* Validation of incoming results against the json schema in Refstack.
+
+Alternatives
+------------
+
+Another alternative is manually writing code to validate test results.
+This schema does not preclue that alternative, but by creating and
+validating against a schema we take advantage of existing libraries
+and reduce the possiblity of introducing unintended parsing errors.
+
+Data model impact
+-----------------
+
+This proposed change has two impacts on the data model:
+
+* It introduces a schema that must be maintained and versioned
+  against the current data model.
+
+* It adds an additional level of validation to the 'create' operation
+  of the CRUD model. Since running a test is a long operation this
+  addition level of complexity will not have an impact on application
+  performance.
+
+* No database migrations will be necessary.
+
+* A static page to serve the schema will be necessary.
+
+REST API impact
+---------------
+
+The v1 api must be modified to validate test results. The modification
+will result in an addition to behavior on the API with no changes
+to the user-facing endpoint.
+
+**description:** Receive tempest test result from a remote test runner. 
+This function expects json formated pass results.
+Update to the api described at:
+https://github.com/stackforge/refstack/blob/master/specs/approved/api-v1.md
+
+**url:** post /v1/results
+
+**failed response:** http:400 - Malformed data.
+
+    {
+     'message': 'malformed json data, see /v1/schema/results.json'
+    }
+
+**url:** get /v1/schema/results.json
+
+**valid response:** http:200 results.json file
+
+No invalid responses. No accepted parameters.
+
+Security impact
+---------------
+
+This change is intented to improve the security of the application
+by introducing data validation. No data-changing apis are
+introduced.
+
+Notifications impact
+--------------------
+
+No known notification impacts.
+
+Other end user impact
+---------------------
+
+End users may experience client failures if their client does not produce
+valid json.
+
+Performance Impact
+------------------
+
+This change introduces a validation step to the POST process. The additional
+time used to validate the data is very small compared to the time taken
+to generate data through cloud testing. The timing impact is negligible
+in the context of the upload use case.
+
+Other deployer impact
+---------------------
+
+This change will take immediate effect after being merged.
+
+Developer impact
+----------------
+
+No additional developer impact.
+
+Implementation
+==============
+
+This change will be implemented as a validation funtion in the API POST
+pipeline. It will essentially be middleware that takes the input data,
+validates the results, and sends back a positive or a negative result.
+If the result is negative, the 400 response will be returned.
+If the result is positive, the data processing will continue as normal.
+
+Assignee(s)
+-----------
+
+Primary assignee:
+    hogepodge
+
+Work Items
+----------
+
+* Write json schema
+* Write json schema GET endpoint
+* Implement validation on results POST endpoint.
+* Implement unit tests for validation and endpoint.
+
+Dependencies
+============
+
+No additional dependencies will be added.
+
+Testing
+=======
+
+To the TestRefStackApi class the following tests will be added:
+* test_results_valid_data
+* test_results_invalid_data
+
+These results will confirm both positive (200) and negative (400) results.
+
+To the unit tests the validator function will be tested:
+* test_valid_data
+* test_invalid_data
+* test_empty_data
+
+These results will provide three modes of schema validation.
+
+Documentation Impact
+====================
+
+Documentation will be updated to link to current schema.
+
+References
+==========
+
+No additional references.