fdec16dc45Add debug logging for all broker-aware requests/responses
Dana Powers
2015-04-05 18:32:23 -07:00
5cc051353eRefactor KafkaClient._send_broker_aware_request to return a list of responses and include individual (unraised) FailedPayloadsError instances rather than always raising a FailedPayloadsError. This should allow producers to determine which payloads succeeded and which failed, specifically.
Dana Powers
2015-04-05 17:46:13 -07:00
04a3816959Merge pull request #365 from dpkp/kafka_consumer_fixup
Dana Powers
2015-04-06 10:38:32 -07:00
c0fc334612Use kafka_bytestring when decoding message topics in KafkaConsumer.fetch_messages
Dana Powers
2015-04-04 21:34:24 -07:00
811fd4cbb9Use list comprehension on _topics in KafkaConsumer.fetch_messages
Dana Powers
2015-04-04 21:04:58 -07:00
87b2ca8e60call _consume_topic_partition() before storing offset in KafkaConsumer.set_topic_partitions((topic, partition, offset))
Dana Powers
2015-04-04 20:19:40 -07:00
e99a751081Fix python3 str/bytes bug in KafkaConsumer.set_topic_partitions({(topic, partition): offset, })
Dana Powers
2015-04-04 13:57:15 -07:00
46ee816327Merge pull request #360 from dpkp/atexit_cleanup
Dana Powers
2015-04-04 10:48:24 -07:00
45c05d0f40Merge pull request #361 from kecaps/master
Dana Powers
2015-04-04 10:48:09 -07:00
1c856e8400Make external API consistently support python3 strings for topic.
Space
2015-03-31 15:25:38 -07:00
47989db113Register atexit handlers for consumer and producer thread/multiprocess cleanup (not __del__)
Dana Powers
2015-03-31 10:29:55 -07:00
9fd0811917Merge pull request #356 from dpkp/always_fetch_offsets
Dana Powers
2015-03-30 17:42:51 -07:00
1d252bfc20Bulk fetch offset partitions in base consumer -- suggested by ecanzonieri
Dana Powers
2015-03-30 16:55:41 -07:00
6fc6856746Merge pull request #355 from dpkp/correlation_id_modulo
Dana Powers
2015-03-30 15:28:54 -07:00
b6d032cc3fFetch previously committed offsets in base consumer class so long as a group is configured (but document that group must be None for old servers). This fixes multiprocessor consumer issue that prevented access to commit offsets if auto_commit is disabled. Also refactor fetch_last_known_offsets based on KafkaConsumer While still setting unknown offsets to 0
Dana Powers
2015-03-29 17:24:56 -07:00
32dd817aacSkip these tests: no OffsetCommitRequest for 0.8.0
Viktor Shlapakov
2015-03-26 15:21:46 +03:00
92a3737a6bAdded basic tests for load_initial_offsets option
Viktor Shlapakov
2015-03-26 14:04:38 +03:00
4bc30a2ec8Add test case for MP Consumer auto commit Tweak MP Consumer test to use iterator
Ali-Akber Saifee
2015-03-18 10:27:04 +08:00
d05fccb9efMerge pull request #357 from dpkp/del_consumer_stop
Dana Powers
2015-03-30 15:02:11 -07:00
bb68965b9eStop consumers on delete
Dana Powers
2015-03-30 11:05:48 -07:00
1313388662Add test for correlation_id rollover
Dana Powers
2015-03-29 18:52:12 -07:00
a4b439141bRollover KafkaClient correlation ids at 2**31 to keep within int32 protocol encoding
Dana Powers
2015-03-29 16:45:20 -07:00
bb1c11e199Merge pull request #341 from dpkp/kafka_consumer_docs
Dana Powers
2015-03-29 18:09:03 -07:00
35b8f5b5d8Use bootstrap_servers interface in tests
Dana Powers
2015-03-29 16:38:47 -07:00
5652f59563Add sphinx_rtd_theme to docs requirements.txt
Dana Powers
2015-03-09 00:01:00 -07:00
c1ca3bf97eUpdates to KafkaConsumer usage docs
Dana Powers
2015-03-09 00:00:49 -07:00
9be4146532fixup kafka consumer docstring
Dana Powers
2015-03-08 23:54:54 -07:00
484f1a4722Move KafkaConsumer usage examples to docs/usage; Put KeyedProducer usage right after SimpleProducer
Dana Powers
2015-03-08 23:00:50 -07:00
eb5fd4aaa1Improve KafkaConsumer docstrings
Dana Powers
2015-03-08 23:00:00 -07:00
fd204dca17Merge pull request #340 from dpkp/deprecate_metadata_broker_list
Dana Powers
2015-03-29 16:33:09 -07:00
f35995a002Merge pull request #336 from scrapinghub/feature-mp-consumer-params
Dana Powers
2015-03-29 16:32:12 -07:00
9a8cd154fdMerge pull request #350 from ecanzonieri/enable_commit_on_seek
Dana Powers
2015-03-29 15:47:02 -07:00
1dd7b17788Merge pull request #338 from dpkp/structs_reorg
Dana Powers
2015-03-29 15:43:51 -07:00
d02276eb2bMerge pull request #337 from dpkp/gzip_context
Dana Powers
2015-03-29 15:43:31 -07:00
1a5cb0376cMerge pull request #329 from vshlapakov/feature-batch-msg-keys
Dana Powers
2015-03-29 15:36:57 -07:00
fb118fb75cManageable queue.put() operation for MPConsumer processes
Viktor Shlapakov
2015-03-25 14:35:23 +03:00
9d5c93e902Added pylint.rc: ignoring SyncManager for pylint
Viktor Shlapakov
2015-03-24 14:45:33 +03:00
88465f70efUsing mp.manager to solve the issue with join for MPConsumer
Viktor Shlapakov
2015-03-24 14:08:25 +03:00
9641e9fa29Moving to **kwargs for MPConsumer options
Viktor Shlapakov
2015-03-24 12:49:44 +03:00
3b1ccc68b3Deprecate KafkaConsumer config metadata_broker_list in favor of bootstrap_servers
Dana Powers
2015-03-08 22:57:34 -07:00
a049e19a1bIncrease count_since_commit on seek
Enrico Canzonieri
2015-03-20 15:29:19 -07:00
e43f405a0cUsing iter_timeout=0 for MP inner consumer in tests
Viktor Shlapakov
2015-03-13 14:15:21 +03:00
f8012c1a74Moved additional MP consumer options to **kwargs
Viktor Shlapakov
2015-03-11 12:41:34 +03:00
09028f0b22Using additional params for MP consumer child process
Viktor Shlapakov
2015-03-03 13:37:37 +03:00
a5b1c8d0fcMerge pull request #335 from scrapinghub/fix-mp-consumer-distribution
Dana Powers
2015-03-12 16:53:09 -07:00
01ea3bf968Used thread-safe dict.copy().keys() for MP consumer partitions
Viktor Shlapakov
2015-03-12 11:33:07 +03:00
4bab2fa5d1Cleaned code for MP consumer chunking
Viktor Shlapakov
2015-03-11 13:51:07 +03:00
fa43388219Take the linter to kafka/codec.py
Dana Powers
2015-03-08 23:41:46 -07:00
4d59678dd3Gzip context manager not supported in py2.6, so use try/finally instead
Dana Powers
2015-03-08 23:41:10 -07:00
83cee06a33Fixup indentation in kafka/common.py; add links to protocol definitions
Dana Powers
2015-03-08 20:13:13 -07:00
610f01e96dUse context managers in gzip_encode / gzip_decode
Dana Powers
2015-03-08 20:01:57 -07:00
92aa7e9428Merge branch 'vshlapakov-feature-async-threading' PR 330: Threading for async batching
Dana Powers
2015-03-08 16:03:06 -07:00
6ef982c5e8Merge branch 'jobevers-master' PR 302: Add failed_payloads attribute to FailedPayloadsError
Dana Powers
2015-03-08 15:16:17 -07:00
95737d9b89Add failed_payloads attribute to FailedPayloadsError
Job Evers
2015-01-22 17:25:02 -06:00
327b734940Merge pull request #332 from dpkp/kafka_client_edits
Dana Powers
2015-03-07 21:34:17 -08:00
ac66fe9159Fixing distribution for MP Consumer
Viktor Shlapakov
2015-03-03 15:32:32 +03:00
e13eb0de89Merge pull request #324 from chmduquesne/master
Dana Powers
2015-03-02 17:49:07 -08:00
7dea5599b6Merge pull request #328 from wkiser/validate_offset
Dana Powers
2015-03-02 17:36:48 -08:00
2ca10e26b6Merge pull request #296 from ecanzonieri/validate_consumer_offset
Dana Powers
2015-03-02 17:23:03 -08:00
04979248afStopping daemonized threads when references to these threads are lost
Christophe-Marie Duquesne
2015-02-17 00:50:51 +01:00
6de9444b96Fixes consumer/kafka and consumer/simple to only yield messages if the message's offset is greater than or equal to the consumer offset.
wkiser
2015-02-23 12:58:20 -05:00
c80fbd1451KafkaClient.get_partition_ids_for_topic now returns empty list for unknown topics
Dana Powers
2015-03-02 09:37:55 -08:00
83d28ba79aAdd comment about no server response case in _send_broker_aware_request
Dana Powers
2015-03-02 09:35:04 -08:00
082cda6a72Avoid topic_partitions KeyError in KafkaClient
Dana Powers
2015-03-02 09:34:06 -08:00
0f04eba097Remove KafkaConsumer.BYTES_CONFIGURATION_KEYS - rely on KafkaClient to handle client_id string/bytes encoding - accept either string or bytes for group_id - convert group_id string to utf-8 bytes internally
Dana Powers
2015-03-01 11:05:36 -08:00
9f7d61258bAccept KafkaClient client_id as string or bytes; encode string client_id as utf-8 bytes internally
Dana Powers
2015-03-01 11:04:18 -08:00
5137163fa4Async producer stop() fix
Viktor Shlapakov
2015-02-26 13:33:40 +03:00
aedbbb39beUsing threading.Event to stop async producer thread
Viktor Shlapakov
2015-02-26 13:02:48 +03:00
1cce287157Returned original tests, rm dirty flag, name fixes
Viktor Shlapakov
2015-02-25 10:45:47 +03:00