Test Results JSON Schema
This schema proposes the development of a json schema for test results to aid in client development and server side data validation. Blueprint: https://blueprints.launchpad.net/refstack/+spec/test-result-json-schema Change-Id: Ic8ba94e9646285f827be5a07c309baf8aa197267
This commit is contained in:
parent
d0d108f891
commit
4556979745
|
@ -0,0 +1,176 @@
|
||||||
|
==========================================
|
||||||
|
Example Spec - The title of your blueprint
|
||||||
|
==========================================
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/refstack/+spec/test-result-json-schema
|
||||||
|
|
||||||
|
Refstack accepts connections from the refstac-client to post test results
|
||||||
|
for storage and scoring. Test results are in json and follow a simple
|
||||||
|
format. However there is no defined schema for the test results, allowing
|
||||||
|
for a greater possibility of invalid or malicious data to be injected
|
||||||
|
into the refstack database. This spec proposes the creation of a json
|
||||||
|
schema that can be used by client developers to generate their own valid
|
||||||
|
test results. The spec will also be used by the server to validate
|
||||||
|
incoming test results.
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
* Refstack accepts test results in json format. However, there is no
|
||||||
|
explicit definition of the structure of this json data.
|
||||||
|
|
||||||
|
* Data sent to the Refstack server is not validated against any schema,
|
||||||
|
increasing the possibility of invalid or malicious data.
|
||||||
|
|
||||||
|
* Clients developers do not have a specific schema to develop against.
|
||||||
|
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
This change proposes the following additions:
|
||||||
|
|
||||||
|
* A json schema file to capture the format of the test results.
|
||||||
|
|
||||||
|
* Validation of incoming results against the json schema in Refstack.
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
Another alternative is manually writing code to validate test results.
|
||||||
|
This schema does not preclue that alternative, but by creating and
|
||||||
|
validating against a schema we take advantage of existing libraries
|
||||||
|
and reduce the possiblity of introducing unintended parsing errors.
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
This proposed change has two impacts on the data model:
|
||||||
|
|
||||||
|
* It introduces a schema that must be maintained and versioned
|
||||||
|
against the current data model.
|
||||||
|
|
||||||
|
* It adds an additional level of validation to the 'create' operation
|
||||||
|
of the CRUD model. Since running a test is a long operation this
|
||||||
|
addition level of complexity will not have an impact on application
|
||||||
|
performance.
|
||||||
|
|
||||||
|
* No database migrations will be necessary.
|
||||||
|
|
||||||
|
* A static page to serve the schema will be necessary.
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
The v1 api must be modified to validate test results. The modification
|
||||||
|
will result in an addition to behavior on the API with no changes
|
||||||
|
to the user-facing endpoint.
|
||||||
|
|
||||||
|
**description:** Receive tempest test result from a remote test runner.
|
||||||
|
This function expects json formated pass results.
|
||||||
|
Update to the api described at:
|
||||||
|
https://github.com/stackforge/refstack/blob/master/specs/approved/api-v1.md
|
||||||
|
|
||||||
|
**url:** post /v1/results
|
||||||
|
|
||||||
|
**failed response:** http:400 - Malformed data.
|
||||||
|
|
||||||
|
{
|
||||||
|
'message': 'malformed json data, see /v1/schema/results.json'
|
||||||
|
}
|
||||||
|
|
||||||
|
**url:** get /v1/schema/results.json
|
||||||
|
|
||||||
|
**valid response:** http:200 results.json file
|
||||||
|
|
||||||
|
No invalid responses. No accepted parameters.
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
This change is intented to improve the security of the application
|
||||||
|
by introducing data validation. No data-changing apis are
|
||||||
|
introduced.
|
||||||
|
|
||||||
|
Notifications impact
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
No known notification impacts.
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
End users may experience client failures if their client does not produce
|
||||||
|
valid json.
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
This change introduces a validation step to the POST process. The additional
|
||||||
|
time used to validate the data is very small compared to the time taken
|
||||||
|
to generate data through cloud testing. The timing impact is negligible
|
||||||
|
in the context of the upload use case.
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
This change will take immediate effect after being merged.
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
No additional developer impact.
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
This change will be implemented as a validation funtion in the API POST
|
||||||
|
pipeline. It will essentially be middleware that takes the input data,
|
||||||
|
validates the results, and sends back a positive or a negative result.
|
||||||
|
If the result is negative, the 400 response will be returned.
|
||||||
|
If the result is positive, the data processing will continue as normal.
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
hogepodge
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
* Write json schema
|
||||||
|
* Write json schema GET endpoint
|
||||||
|
* Implement validation on results POST endpoint.
|
||||||
|
* Implement unit tests for validation and endpoint.
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
No additional dependencies will be added.
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
To the TestRefStackApi class the following tests will be added:
|
||||||
|
* test_results_valid_data
|
||||||
|
* test_results_invalid_data
|
||||||
|
|
||||||
|
These results will confirm both positive (200) and negative (400) results.
|
||||||
|
|
||||||
|
To the unit tests the validator function will be tested:
|
||||||
|
* test_valid_data
|
||||||
|
* test_invalid_data
|
||||||
|
* test_empty_data
|
||||||
|
|
||||||
|
These results will provide three modes of schema validation.
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
Documentation will be updated to link to current schema.
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
No additional references.
|
Loading…
Reference in New Issue