Merge "Test Results JSON Schema"
This commit is contained in:
commit
97c6c53952
176
specs/proposed/refstack-org-test-result-json-schema.rst
Normal file
176
specs/proposed/refstack-org-test-result-json-schema.rst
Normal file
@ -0,0 +1,176 @@
|
||||
==========================================
|
||||
Example Spec - The title of your blueprint
|
||||
==========================================
|
||||
|
||||
https://blueprints.launchpad.net/refstack/+spec/test-result-json-schema
|
||||
|
||||
Refstack accepts connections from the refstac-client to post test results
|
||||
for storage and scoring. Test results are in json and follow a simple
|
||||
format. However there is no defined schema for the test results, allowing
|
||||
for a greater possibility of invalid or malicious data to be injected
|
||||
into the refstack database. This spec proposes the creation of a json
|
||||
schema that can be used by client developers to generate their own valid
|
||||
test results. The spec will also be used by the server to validate
|
||||
incoming test results.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
* Refstack accepts test results in json format. However, there is no
|
||||
explicit definition of the structure of this json data.
|
||||
|
||||
* Data sent to the Refstack server is not validated against any schema,
|
||||
increasing the possibility of invalid or malicious data.
|
||||
|
||||
* Clients developers do not have a specific schema to develop against.
|
||||
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
This change proposes the following additions:
|
||||
|
||||
* A json schema file to capture the format of the test results.
|
||||
|
||||
* Validation of incoming results against the json schema in Refstack.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
Another alternative is manually writing code to validate test results.
|
||||
This schema does not preclue that alternative, but by creating and
|
||||
validating against a schema we take advantage of existing libraries
|
||||
and reduce the possiblity of introducing unintended parsing errors.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
This proposed change has two impacts on the data model:
|
||||
|
||||
* It introduces a schema that must be maintained and versioned
|
||||
against the current data model.
|
||||
|
||||
* It adds an additional level of validation to the 'create' operation
|
||||
of the CRUD model. Since running a test is a long operation this
|
||||
addition level of complexity will not have an impact on application
|
||||
performance.
|
||||
|
||||
* No database migrations will be necessary.
|
||||
|
||||
* A static page to serve the schema will be necessary.
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
The v1 api must be modified to validate test results. The modification
|
||||
will result in an addition to behavior on the API with no changes
|
||||
to the user-facing endpoint.
|
||||
|
||||
**description:** Receive tempest test result from a remote test runner.
|
||||
This function expects json formated pass results.
|
||||
Update to the api described at:
|
||||
https://github.com/stackforge/refstack/blob/master/specs/approved/api-v1.md
|
||||
|
||||
**url:** post /v1/results
|
||||
|
||||
**failed response:** http:400 - Malformed data.
|
||||
|
||||
{
|
||||
'message': 'malformed json data, see /v1/schema/results.json'
|
||||
}
|
||||
|
||||
**url:** get /v1/schema/results.json
|
||||
|
||||
**valid response:** http:200 results.json file
|
||||
|
||||
No invalid responses. No accepted parameters.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
This change is intented to improve the security of the application
|
||||
by introducing data validation. No data-changing apis are
|
||||
introduced.
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
No known notification impacts.
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
End users may experience client failures if their client does not produce
|
||||
valid json.
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
This change introduces a validation step to the POST process. The additional
|
||||
time used to validate the data is very small compared to the time taken
|
||||
to generate data through cloud testing. The timing impact is negligible
|
||||
in the context of the upload use case.
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
This change will take immediate effect after being merged.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
No additional developer impact.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
This change will be implemented as a validation funtion in the API POST
|
||||
pipeline. It will essentially be middleware that takes the input data,
|
||||
validates the results, and sends back a positive or a negative result.
|
||||
If the result is negative, the 400 response will be returned.
|
||||
If the result is positive, the data processing will continue as normal.
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
hogepodge
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
* Write json schema
|
||||
* Write json schema GET endpoint
|
||||
* Implement validation on results POST endpoint.
|
||||
* Implement unit tests for validation and endpoint.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
No additional dependencies will be added.
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
To the TestRefStackApi class the following tests will be added:
|
||||
* test_results_valid_data
|
||||
* test_results_invalid_data
|
||||
|
||||
These results will confirm both positive (200) and negative (400) results.
|
||||
|
||||
To the unit tests the validator function will be tested:
|
||||
* test_valid_data
|
||||
* test_invalid_data
|
||||
* test_empty_data
|
||||
|
||||
These results will provide three modes of schema validation.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
Documentation will be updated to link to current schema.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
No additional references.
|
Loading…
x
Reference in New Issue
Block a user