3.0 KiB
kafka-python
>>> pip install kafka-python
kafka-python is a client for the Apache Kafka distributed stream processing system. It is designed to function much like the official java client, with a sprinkling of pythonic interfaces (e.g., iterators).
KafkaConsumer
>>> from kafka import KafkaConsumer >>> consumer = KafkaConsumer('my_favorite_topic') >>> for msg in consumer: ... print (msg)
~kafka.consumer.KafkaConsumer is a full-featured,
high-level message consumer class that is similar in design and function
to the new 0.9 java consumer. Most configuration parameters defined by
the official java client are supported as optional kwargs, with
generally similar behavior. Gzip and Snappy compressed messages are
supported transparently.
In addition to the standard ~kafka.consumer.KafkaConsumer.poll interface (which
returns micro-batches of messages, grouped by topic-partition),
kafka-python supports single-message iteration, yielding ~kafka.consumer.ConsumerRecord namedtuples, which
include the topic, partition, offset, key, and value of each
message.
By default, ~kafka.consumer.KafkaConsumer will attempt to
auto-commit message offsets every 5 seconds. When used with 0.9 kafka
brokers, ~kafka.consumer.KafkaConsumer will dynamically
assign partitions using the kafka GroupCoordinator APIs and a ~kafka.coordinator.assignors.roundrobin.RoundRobinPartitionAssignor
partitioning strategy, enabling relatively straightforward parallel
consumption patterns. See usage for examples.
KafkaProducer
TBD
Protocol
A secondary goal of kafka-python is to provide an easy-to-use
protocol layer for interacting with kafka brokers via the python repl.
This is useful for testing, probing, and general experimentation. The
protocol support is leveraged to enable a ~kafka.KafkaClient.check_version() method that probes
a kafka broker and attempts to identify which version it is running
(0.8.0 to 0.9).
Low-level
Legacy support is maintained for low-level consumer and producer classes, SimpleConsumer and SimpleProducer.
Usage Overview <usage> API </apidoc/modules> install tests compatibility support license