110 lines
2.0 KiB
Markdown
110 lines
2.0 KiB
Markdown
# WebSocket compliance testing
|
|
|
|
We are testing on these Pythons
|
|
|
|
* CPython 2.7
|
|
* CPython 3.6
|
|
* PyPy 2.7
|
|
* PyPy3 (3.5 lang. level)
|
|
|
|
using these networking frameworks
|
|
|
|
* Twisted
|
|
* asyncio (or Trollius fallback)
|
|
|
|
for
|
|
|
|
* WebSocket client
|
|
* WebSocket server
|
|
|
|
This gives a total of 16 combinations being tested.
|
|
|
|
The generated reports are hosted here:
|
|
|
|
* [Client](http://autobahn.ws/testsuite/reports/clients/index.html)
|
|
* [Server](http://autobahn.ws/testsuite/reports/servers/index.html)
|
|
|
|
> Note (23.3.2016): the server reports show some fails for some WebSocket compression tests - these are timeouts of the tests, because my machine was too slow / the timeout is too tight. However, it takes too now for me to repeat.
|
|
|
|
|
|
## Testing
|
|
|
|
### Prepare
|
|
|
|
Download Pythons
|
|
|
|
```console
|
|
make downloads
|
|
```
|
|
|
|
Build Pythons and wstest (the command line tool of the Autobahn WebSocket testsuite)
|
|
|
|
```console
|
|
make build
|
|
```
|
|
|
|
Check versions:
|
|
|
|
```console
|
|
oberstet@office-corei7:~/scm/crossbario/autobahn-python/wstest$ make versions
|
|
./cpy2/bin/python -V
|
|
Python 2.7.13
|
|
./cpy3/bin/python -V
|
|
Python 3.6.0
|
|
./pypy2/bin/python -V
|
|
Python 2.7.13 (fa3249d55d15, Mar 19 2017, 20:21:48)
|
|
[PyPy 5.7.0 with GCC 6.2.0 20160901]
|
|
./pypy3/bin/python -V
|
|
Python 3.5.3 (b16a4363e930, Mar 20 2017, 16:13:46)
|
|
[PyPy 5.7.0-beta0 with GCC 6.2.0 20160901]
|
|
```
|
|
|
|
Setup dependencies
|
|
|
|
```console
|
|
make setup
|
|
```
|
|
|
|
### Client Testing
|
|
|
|
In a first console, start the fuzzing server:
|
|
|
|
```console
|
|
make wstest_server
|
|
```
|
|
|
|
In a second console, run the testee clients:
|
|
|
|
```console
|
|
make test_client
|
|
```
|
|
|
|
Reports will be generated under the folder `./reports/clients`, with the summary report `./reports/clients/index.html`.
|
|
|
|
|
|
### Server Testing
|
|
|
|
|
|
In a console, start all the testee servers:
|
|
|
|
```console
|
|
make start_server
|
|
```
|
|
|
|
> Note: this will start 8 servers in the background. To stop: `make stop_server`
|
|
|
|
|
|
Now run the fuzzing client, which will test all servers:
|
|
|
|
```console
|
|
make test_server
|
|
```
|
|
|
|
### Upload reports
|
|
|
|
To upload reports:
|
|
|
|
```console
|
|
make upload_reports
|
|
```
|