Include the URL of your story:
Currently in all Python Monasca components the copy of kafka-python library in version 0.9.5 (released on Feb 16, 2016) is used1. This specification describes the process of upgrading the Apache Kafka client to confluent-kafka-python2. This will improve the performance and reliability. Sticking with the old frozen client version is also unacceptable in terms of security.
The use of KeyedProducer and SimpleConsumer in kafka-python library has been deprecated as of version 1.0.03. Further use of this code poses a security risk. Additionally, profiling of
monasca-persister has shown that most of the time is spent during the consumption of Kafka messages4. Thus, there is a big potential on improving overall Monasca performance by upgrading the used Kafka client.
The wiki page hosted by Apache Software Foundation lists available Python clients5. There are currently three actively maintained and supported clients: confluent-kafka-python, kafka-python and pykafka. Several benchmarks have shown6,7 that the client maintained by Confluent is both the fastest and most complete.
There is significant performance improvement when using asynchronous producer (~50x). Sending messages asynchronously will require more care to avoid duplicating the persisted data but performance gain justifies that.
confluent-kafka-python is also the only client which offers support for Apache Avro serialization which reduces the size of messages and thus additionally speeds up communication.
The proposed change includes using:
Code changes will affect following components:
Java components (monasca-thresh and monasca-persister) are out of scope of this specification. Client upgrading in these components should be handled separately.
This client has an external dependency on librdkafka, a finely tuned C client.
No data model impact.
No REST API impact.
This change will improve the security because of removing the deprecated and unmaintained code.
No end user impact.
This change should dramatically improve the performance of the complete solution. In particular performance of monasca-persister and monasca-api is expected to improve.
New libraries should be packaged and deployed:
confluent-kafka-python has to be used instead of kafka-python in all affected components.
New packages have to be build for:
We should test the implementation using existing integration tests (tempest). Additionally we should test the scenario when the producer fails to receive response from Kafka for some of the messages in the bulk. It should be avoided that duplicate entries are created in the database.
The implantation should be followed by executing following tests on the complete stack:
No documentation impact.