WebSocket compliance testing

We are testing on these Pythons

CPython 2.7
CPython 3.6
PyPy 2.7
PyPy3 (3.5 lang. level)

using these networking frameworks

Twisted
asyncio (or Trollius fallback)

for

WebSocket client
WebSocket server

This gives a total of 16 combinations being tested.

The generated reports are hosted here:

Note (23.3.2016): the server reports show some fails for some WebSocket compression tests - these are timeouts of the tests, because my machine was too slow / the timeout is too tight. However, it takes too now for me to repeat.

Testing

Prepare

Download Pythons

make downloads

Build Pythons and wstest (the command line tool of the Autobahn WebSocket testsuite)

make build

Check versions:

oberstet@office-corei7:~/scm/crossbario/autobahn-python/wstest$ make versions
./cpy2/bin/python -V
Python 2.7.13
./cpy3/bin/python -V
Python 3.6.0
./pypy2/bin/python -V
Python 2.7.13 (fa3249d55d15, Mar 19 2017, 20:21:48)
[PyPy 5.7.0 with GCC 6.2.0 20160901]
./pypy3/bin/python -V
Python 3.5.3 (b16a4363e930, Mar 20 2017, 16:13:46)
[PyPy 5.7.0-beta0 with GCC 6.2.0 20160901]

Setup dependencies

make setup

Client Testing

In a first console, start the fuzzing server:

make wstest_server

In a second console, run the testee clients:

make test_client

Reports will be generated under the folder ./reports/clients, with the summary report ./reports/clients/index.html.

Server Testing

In a console, start all the testee servers:

make start_server

Note: this will start 8 servers in the background. To stop: make stop_server

Now run the fuzzing client, which will test all servers:

make test_server

Upload reports

To upload reports:

make upload_reports

2.0 KiB Raw Blame History