refstack/specs/prior/implemented/refstack-org-test-result-json-schema.rst

Example Spec - The title of your blueprint

https://blueprints.launchpad.net/refstack/+spec/test-result-json-schema

Refstack accepts connections from the refstac-client to post test results for storage and scoring. Test results are in json and follow a simple format. However there is no defined schema for the test results, allowing for a greater possibility of invalid or malicious data to be injected into the refstack database. This spec proposes the creation of a json schema that can be used by client developers to generate their own valid test results. The spec will also be used by the server to validate incoming test results.

Problem description

• Refstack accepts test results in json format. However, there is no explicit definition of the structure of this json data.
• Data sent to the Refstack server is not validated against any schema, increasing the possibility of invalid or malicious data.
• Clients developers do not have a specific schema to develop against.

Proposed change

This change proposes the following additions:

• A json schema file to capture the format of the test results.
• Validation of incoming results against the json schema in Refstack.

Alternatives

Another alternative is manually writing code to validate test results. This schema does not preclue that alternative, but by creating and validating against a schema we take advantage of existing libraries and reduce the possiblity of introducing unintended parsing errors.

Data model impact

This proposed change has two impacts on the data model:

• It introduces a schema that must be maintained and versioned against the current data model.
• It adds an additional level of validation to the 'create' operation of the CRUD model. Since running a test is a long operation this addition level of complexity will not have an impact on application performance.
• No database migrations will be necessary.
• A static page to serve the schema will be necessary.

REST API impact

The v1 api must be modified to validate test results. The modification will result in an addition to behavior on the API with no changes to the user-facing endpoint.

description: Receive tempest test result from a remote test runner. This function expects json formated pass results. Update to the api described at: https://github.com/stackforge/refstack/blob/master/specs/approved/api-v1.md

url: post /v1/results

failed response: http:400 - Malformed data.

{

'message': 'Malformed json data, see /v1/results/schema'

}

url: get /v1/results/schema

valid response: http:200 schema.json file

No invalid responses. No accepted parameters.

Security impact

This change is intented to improve the security of the application by introducing data validation. No data-changing apis are introduced.

Notifications impact

No known notification impacts.

Other end user impact

End users may experience client failures if their client does not produce valid json.

Performance Impact

This change introduces a validation step to the POST process. The additional time used to validate the data is very small compared to the time taken to generate data through cloud testing. The timing impact is negligible in the context of the upload use case.

Other deployer impact

This change will take immediate effect after being merged.

Developer impact

No additional developer impact.

Implementation

This change will be implemented as a validation funtion in the API POST pipeline. It will essentially be middleware that takes the input data, validates the results, and sends back a positive or a negative result. If the result is negative, the 400 response will be returned. If the result is positive, the data processing will continue as normal.

Assignee(s)

Primary assignee:

hogepodge

Work Items

• Write json schema
• Write json schema GET endpoint
• Implement validation on results POST endpoint.
• Implement unit tests for validation and endpoint.

Dependencies

No additional dependencies will be added.

Testing

To the TestRefStackApi class the following tests will be added: * test_results_valid_data * test_results_invalid_data

These results will confirm both positive (200) and negative (400) results.

To the unit tests the validator function will be tested: * test_valid_data * test_invalid_data * test_empty_data

These results will provide three modes of schema validation.

Documentation Impact

Documentation will be updated to link to current schema.

References

No additional references.