deb-python-autobahn/DEVELOPERS.md

Developers Guide

This is a guide for developers hacking on AutobahnPython.

Python 2 / 3

AutobahnPython supports both Python 2 and Python 3.

• http://python-future.org/compatible_idioms.html

Coding Style

The rules and text here follows Django.

Please follow these coding standards when writing code for inclusion in AutobahnPython.

1. Unless otherwise specified, follow PEP 8. However, remember that PEP 8 is only a guide, so respect the style of the surrounding code as a primary goal.
2. Use 4 spaces for indents.
3. Use CamelCase for classes and snake_case for variables, functions and members, and UPPERCASE for constants.
4. Everything that is not part of the public API must be prefixed with a single underscore.
5. Rules 3 and 4 apply to the public API exposed by AutobahnPython for both Twisted and asyncio users as well as everything within the library itself.
6. An exception to PEP 8 is our rules on line lengths. Don’t limit lines of code to 79 characters if it means the code looks significantly uglier or is harder to read. We allow up to 119 characters as this is the width of GitHub code review; anything longer requires horizontal scrolling which makes review more difficult. Documentation, comments, and docstrings should be wrapped at 79 characters, even though PEP 8 suggests 72.
7. Use hanging indents with each argument strictly on a separate line to limit line length (see also here for an explanation why this is PEP8 compliant):

raise ApplicationError(
    u"crossbar.error.class_import_failed",
    u"Session not derived of ApplicationSession"
)

Code must be checked for PEP8 compliance using flake8 with pyflakes and pep8-naming plugins installed:

flake8 autobahn

There is no automatic checker for rule 4, hence reviewers of PRs should manually inspect code for compliance.

Note that AutobahnPython currently does not fully comply to above rules:

(python279_1)oberstet@thinkpad-t430s:~/scm/autobahn/AutobahnPython$ flake8 --statistics -qq autobahn
388     E501 line too long (131 > 119 characters)
4       N801 class names should use CapWords convention
296     N802 function name should be lowercase
80      N803 argument name should be lowercase
1       N805 first argument of a method should be named 'self'
69      N806 variable in function should be lowercase

It also does not comply fully to rule 4. This will get addressed in the next major release (0.11).

API Stability Level

We distinbuish three levels of API in AutobahnPython:

1. public user API
2. private library API
3. private non-API

The Public User API is what (third party) application developers should rely on exclusively. It is the only API we make any kind of stability guarantees. The Public User API is

• defined via ABC interfaces (when that makes sense)
• has docstring including the tag @public
• has docs generated and published
• follow the PEP8 naming convention (stuff does not start with a _)
• is versioned using semver

The Private Library API is for library internal use, crossing files, classes and such. Application developers should not use this API, and we make no guarantees whatsoever. Any minor version bump might change anything here. We might rip out anything here or add stuff. This API may be used from our companion projects (Autobahn <-> Crossbar). The reason we are allowed to use that API is simple: we know what we are doing, and we are able to coordinate across projects and can rectify issues that arise. This Private Library API does NOT mark things with a starting _.

The Private non-API isn't an API at all: like class members which may only be used within that class, or functions which may only be used in the same module where the function is defined.

Public API

The new rule for the public API is simple: if something is exported from the modules below, then it is public. Otherwise not.

• Top
• WebSocket
• WAMP
• Asyncio
• Twisted

Cross-platform Considerations

Autobahn supports many different platforms and both major async frameworks. One thing that helps with this is the txaio library. This is used for all Deferred/Future operations throughout the code and more recently for logging.

Here is a recommended way to do logging:

class Foo(object):
    log = txaio.make_logger()

    def connect(self):
        try:
            self.log.info("Connecting")
            raise Exception("an error")
        except:
            fail = txaio.create_failure()
            self.log.error("Connection failed: {msg}", msg=txaio.failure_message(fail))
            self.log.debug("{traceback}", traceback=txaio.failure_format_traceback(fail))
            # Exception instance in fail.value

Note that create_failure() can (and should) be called without arguments when inside an except block; this will give it a valid traceback instance. The only attribute you can depend on is fail.value which is the Exception instance. Otherwise use txaio.failre_* methods.

How to handler async methods with txaio:

f = txaio.as_future(mightReturnDeferred, 'arg0')

def success(result):
    print("It worked! {}".format(result))

def error(fail):
    print("It failed! {}".format(txaio.failure_message(fail)))
txaio.add_callbacks(f, success, error)

Either the success or error callback can be None (e.g. if you just need to add an error-handler). fail must implement txaio.IFailedFuture (but only that; don't depend on any other methods). You cannot use @asyncio.coroutine or @inlineCallbacks.

Use of assert vs Exceptions

See the discussion here.

assert is for telling fellow programmers: "When I wrote this, I thought X could/would never really happen, and if it does, this code will very likely do the wrong thing".

That is, use an assert if the following holds true: if the assert fails, it means we have a bug within the library itself.

In contrast, to check e.g. for user errors, such as application code using the wrong type when calling into the library, use Exceptions:

import six

def foo(uri):
    if type(uri) != six.text_type:
        raise RuntimeError(u"URIs for foo() must be unicode - got {} instead".format(type(uri)))

In this specific example, we also have a WAMP defined error (which would be preferred compared to the generic exception used above):

import six
from autobahn.wamp import ApplicationError

def foo(uri):
    if type(uri) != six.text_type:
        raise ApplicationError(ApplicationError.INVALID_URI,
                               u"URIs for foo() must be unicode - got {} instead".format(type(uri)))

Release Process

1. Travis is fully green
2. Run AutobahnTestsuite
3. Build and spellcheck the docs
4. Update the doc/changelog.rst
5. Tag the release
6. Publish the package to PyPi
7. Publish the docs to the Autobahn Web site
8. Announce on mailing list

Documentation

Requirements

You will need to have some stuff installed to generate the docs:

cd ~/scm/autobahn/AutobahnPython/doc
install_deps

Additionally, install SCons.

Test

To generate and publish the documentation to here:

cd ~/scm/autobahn/AutobahnPython
make clean
make install
cd doc
make test

and open http://localhost:8080.

Publish

To generate and publish the documentation to here:

cd ~/scm/autobahn/AutobahnPython
make clean
make install
cd doc
make publish

The docs are published to the Autobahn Web site, which is hosted on a AWS S3 bucket is owned by Tavendo, and hence the publish can only be done by Tavendo.

WebSocket Test Reports

AutobahnTestsuite provides a fully automated test suite to verify client and server implementations of the WebSocket Protocol for specification conformance and implementation robustness.

We use that to test AutobahnPython at the WebSocket level, both client and server mode.

AutobahnPython has 100% strictly green passes on all of the >500 tests. If there is only one test failing or not strictly green, then don't do a release and fix the issue before.

The testsuite and the testee should both be run under PyPy on Ubuntu for now, as this is one of the most important targets. In the future, we might extend that to more platforms and Pythons.

Installation

Install PyPy and create a fresh virtualenv:

cd $HOME
wget https://bitbucket.org/pypy/pypy/downloads/pypy-2.6.1-linux64.tar.bz2
tar xvf pypy-2.6.1-linux64.tar.bz2
wget https://bootstrap.pypa.io/get-pip.py
./pypy-2.6.1-linux64/bin/pypy get-pip.py
./pypy-2.6.1-linux64/bin/pip install virtualenv
./pypy-2.6.1-linux64/bin/virtualenv $HOME/pypy261_1

Install AutobahnPython from source (the sources that is to be released!) and AutobahnTestsuite from PyPi:

source $HOME/pypy261_1/bin/activate
cd ~/scm/crossbario/autobahn-python
make install
pip install autobahntestsuite

Prepare for testing

We'll be generating reports in /tmp/reports. Make sure to remove any old stuff:

rm -rf reports
rm -f fuzzing*.json

The json files are the test specs that control what tests to run. When there are no test spec files, the testsuite will generate default ones, which is what we want.

Two test runs need to be performed now: one testing AutobahnPython acting as a client, and one where is acts as a server.

Testing WebSocket client mode

Open a first terminal and run the following to start the WebSocket fuzzing server:

cd /tmp
source $HOME/pypy261_1/bin/activate
wstest -m fuzzingserver

Open a second terminal and run the following to start the WebSocket testee client:

source $HOME/pypy261_1/bin/activate
wstest -m testeeclient -w ws://127.0.0.1:9001

Get some coffee. It'll take some minutes. When it's done, open /tmp/reports/servers/index.html in you browser.

Testing WebSocketserver mode

Open a first terminal and run the following to start the WebSocket testee server:

source $HOME/pypy261_1/bin/activate
wstest -m testeeserver -w ws://127.0.0.1:9001

Open a second terminal and run the following to start the WebSocket fuzzing client:

cd /tmp
source $HOME/pypy261_1/bin/activate
wstest -m fuzzingclient -w ws://127.0.0.1:9001

Get some coffee. It'll take some minutes. When it's done, open /tmp/reports/clients/index.html in you browser.

Upload the results

To upload the results, you will need the AWS CLI.

To install:

pip install --upgrade awscli

We are using the high-level commands.

To upload:

aws --region eu-west-1 s3 sync \
    /tmp/reports \
    s3://autobahn.ws/python/testreport \
    --delete \
    --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers

The above S3 bucket is owned by Tavendo, and hence the upload can only be done by Tavendo.

After the upload has finished, check the live pages:

• Client Mode Reports
• Server Mode Reports

Roadmap

0.10.6

Milestone for 0.10.6

• Automatic reconnection
• Configurable WAMP connecting transports
• WAMP Connection abstration
• Authentication

0.10 will get into "maintenance mode" after 0.10.6. We'll have a maintenance branch for that over some time. The new development will be based on 0.11 (see below).

0.11.0

Milestone for 0.11.0

• new main development line with new, long-term API