telemetry-specs/specs/liberty/remove-web-eventlet.rst

Remove Eventlet from the API Server

https://blueprints.launchpad.net/ceilometer/+spec/remove-web-eventlet

As of Kilo, the WSGI application which provides the Ceilometer API can be run in two fundamentally different ways:

• As a Python command that runs a Werkzeug-based web server that is monkeypatched to use eventlet.
• As a WSGI application hosted by any WSGI server, often Apache + mod wsgi.

Hosting the server application in a dedicate WSGI host has performance, configuration and scaling advantages. Running the command line service, while convenient for simple testing, can result in difficult to diagnose bugs and unusual behaviors when it is monkeypatched to use eventlet. Since the Werkzeug server can (and as written, does) provide support for multi-process and multi-threaded interactions, the inclusion of eventlet should be removed.

Problem description

Eventlet monkeypatches the socket module to provide non-blocking network I/O. This can work well most of the time, but when unusual socket situations arise (for example the client frequently closes the connection before reading all the data the server would like to send) the produced tracebacks can be more difficult to use as debugging aids than those that are produced without eventlet in place. We want our tooling to be as useful as possible especially when there are situations that require debugging.

When using the Werkzeug service, using eventlet is fairly redundant as the web server is configured to run multiple processes.

Proposed change

We can resolve this problem relatively easily. At the moment eventlet monkey patching is turned on for everything that is imported under ceilometer.cmd (through ceilometer/cmd/__init__.py). This is overkill. We should evaluate which services benefit from it and turn it on only for those. We can start with the API server.

Further, for the sake of guiding people to the best solutions, we should update documentation and guidance to downstreams to indicate that using a WSGI host is the better way to host the API service in production.

Alternatives

• Do nothing. This is a hygienic change that will benefit users and developers but is not strictly required.

Data model impact

None

REST API impact

None. If there is anything we've messed up.

Security impact

None.

Pipeline impact

None.

Other end user impact

None.

Performance/Scalability Impacts

For production environments, the guidance to use a WSGI host will provide more options for adjusting configuration for scaling and performance.

Other deployer impact

Deployers who wish to take advantage of the more direct guidance to use a WSGI host will have to do what it says.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

chdent

Other contributors:

None

Ongoing maintainer:

chdent

Work Items

• Separate eventlet and non-eventlet commands into two different module directories, starting with the api module.
• Compare and contrast performance of the API server with and without eventlet paying specific attention to the impact on accessing the storage layer. Mike Bayer has pointed out that removing eventlet has little impact on a properly configured web server with a properly configured storage layer. We need to confirm this across the three styles of presenting the APU: WSGI host, Werkzeug with eventlet, Werkzeug without eventlet.
• Update developer and admin documentation.

Future lifecycle

This is a core service which will receive ongoing maintenance from the Ceilometer project. Once the API server is not using eventlet we can see if any of the other console scripts would benefit from the change.

Dependencies

No new dependencies.

Testing

Existing API tests will cover these changes.

Documentation Impact

As mentioned above developer and admin documentation will need to be updated to reflect the new guidance about the best service configuration.

References

None