Commit Graph

  • fdec16dc45 Add debug logging for all broker-aware requests/responses Dana Powers 2015-04-05 18:32:23 -07:00
  • 5cc051353e Refactor KafkaClient._send_broker_aware_request to return a list of responses and include individual (unraised) FailedPayloadsError instances rather than always raising a FailedPayloadsError. This should allow producers to determine which payloads succeeded and which failed, specifically. Dana Powers 2015-04-05 17:46:13 -07:00
  • 04a3816959 Merge pull request #365 from dpkp/kafka_consumer_fixup Dana Powers 2015-04-06 10:38:32 -07:00
  • c0fc334612 Use kafka_bytestring when decoding message topics in KafkaConsumer.fetch_messages Dana Powers 2015-04-04 21:34:24 -07:00
  • 811fd4cbb9 Use list comprehension on _topics in KafkaConsumer.fetch_messages Dana Powers 2015-04-04 21:04:58 -07:00
  • 87b2ca8e60 call _consume_topic_partition() before storing offset in KafkaConsumer.set_topic_partitions((topic, partition, offset)) Dana Powers 2015-04-04 20:19:40 -07:00
  • e99a751081 Fix python3 str/bytes bug in KafkaConsumer.set_topic_partitions({(topic, partition): offset, }) Dana Powers 2015-04-04 13:57:15 -07:00
  • 46ee816327 Merge pull request #360 from dpkp/atexit_cleanup Dana Powers 2015-04-04 10:48:24 -07:00
  • 45c05d0f40 Merge pull request #361 from kecaps/master Dana Powers 2015-04-04 10:48:09 -07:00
  • 1c856e8400 Make external API consistently support python3 strings for topic. Space 2015-03-31 15:25:38 -07:00
  • 47989db113 Register atexit handlers for consumer and producer thread/multiprocess cleanup (not __del__) Dana Powers 2015-03-31 10:29:55 -07:00
  • 9fd0811917 Merge pull request #356 from dpkp/always_fetch_offsets Dana Powers 2015-03-30 17:42:51 -07:00
  • 1d252bfc20 Bulk fetch offset partitions in base consumer -- suggested by ecanzonieri Dana Powers 2015-03-30 16:55:41 -07:00
  • 6fc6856746 Merge pull request #355 from dpkp/correlation_id_modulo Dana Powers 2015-03-30 15:28:54 -07:00
  • b6d032cc3f Fetch previously committed offsets in base consumer class so long as a group is configured (but document that group must be None for old servers). This fixes multiprocessor consumer issue that prevented access to commit offsets if auto_commit is disabled. Also refactor fetch_last_known_offsets based on KafkaConsumer While still setting unknown offsets to 0 Dana Powers 2015-03-29 17:24:56 -07:00
  • 32dd817aac Skip these tests: no OffsetCommitRequest for 0.8.0 Viktor Shlapakov 2015-03-26 15:21:46 +03:00
  • 92a3737a6b Added basic tests for load_initial_offsets option Viktor Shlapakov 2015-03-26 14:04:38 +03:00
  • 4bc30a2ec8 Add test case for MP Consumer auto commit Tweak MP Consumer test to use iterator Ali-Akber Saifee 2015-03-18 10:27:04 +08:00
  • d05fccb9ef Merge pull request #357 from dpkp/del_consumer_stop Dana Powers 2015-03-30 15:02:11 -07:00
  • bb68965b9e Stop consumers on delete Dana Powers 2015-03-30 11:05:48 -07:00
  • 1313388662 Add test for correlation_id rollover Dana Powers 2015-03-29 18:52:12 -07:00
  • a4b439141b Rollover KafkaClient correlation ids at 2**31 to keep within int32 protocol encoding Dana Powers 2015-03-29 16:45:20 -07:00
  • bb1c11e199 Merge pull request #341 from dpkp/kafka_consumer_docs Dana Powers 2015-03-29 18:09:03 -07:00
  • 35b8f5b5d8 Use bootstrap_servers interface in tests Dana Powers 2015-03-29 16:38:47 -07:00
  • 5652f59563 Add sphinx_rtd_theme to docs requirements.txt Dana Powers 2015-03-09 00:01:00 -07:00
  • c1ca3bf97e Updates to KafkaConsumer usage docs Dana Powers 2015-03-09 00:00:49 -07:00
  • 9be4146532 fixup kafka consumer docstring Dana Powers 2015-03-08 23:54:54 -07:00
  • 484f1a4722 Move KafkaConsumer usage examples to docs/usage; Put KeyedProducer usage right after SimpleProducer Dana Powers 2015-03-08 23:00:50 -07:00
  • eb5fd4aaa1 Improve KafkaConsumer docstrings Dana Powers 2015-03-08 23:00:00 -07:00
  • fd204dca17 Merge pull request #340 from dpkp/deprecate_metadata_broker_list Dana Powers 2015-03-29 16:33:09 -07:00
  • f35995a002 Merge pull request #336 from scrapinghub/feature-mp-consumer-params Dana Powers 2015-03-29 16:32:12 -07:00
  • 9a8cd154fd Merge pull request #350 from ecanzonieri/enable_commit_on_seek Dana Powers 2015-03-29 15:47:02 -07:00
  • 1dd7b17788 Merge pull request #338 from dpkp/structs_reorg Dana Powers 2015-03-29 15:43:51 -07:00
  • d02276eb2b Merge pull request #337 from dpkp/gzip_context Dana Powers 2015-03-29 15:43:31 -07:00
  • 1a5cb0376c Merge pull request #329 from vshlapakov/feature-batch-msg-keys Dana Powers 2015-03-29 15:36:57 -07:00
  • fb118fb75c Manageable queue.put() operation for MPConsumer processes Viktor Shlapakov 2015-03-25 14:35:23 +03:00
  • 9d5c93e902 Added pylint.rc: ignoring SyncManager for pylint Viktor Shlapakov 2015-03-24 14:45:33 +03:00
  • 88465f70ef Using mp.manager to solve the issue with join for MPConsumer Viktor Shlapakov 2015-03-24 14:08:25 +03:00
  • 9641e9fa29 Moving to **kwargs for MPConsumer options Viktor Shlapakov 2015-03-24 12:49:44 +03:00
  • 3b1ccc68b3 Deprecate KafkaConsumer config metadata_broker_list in favor of bootstrap_servers Dana Powers 2015-03-08 22:57:34 -07:00
  • a049e19a1b Increase count_since_commit on seek Enrico Canzonieri 2015-03-20 15:29:19 -07:00
  • e43f405a0c Using iter_timeout=0 for MP inner consumer in tests Viktor Shlapakov 2015-03-13 14:15:21 +03:00
  • f8012c1a74 Moved additional MP consumer options to **kwargs Viktor Shlapakov 2015-03-11 12:41:34 +03:00
  • 09028f0b22 Using additional params for MP consumer child process Viktor Shlapakov 2015-03-03 13:37:37 +03:00
  • a5b1c8d0fc Merge pull request #335 from scrapinghub/fix-mp-consumer-distribution Dana Powers 2015-03-12 16:53:09 -07:00
  • 01ea3bf968 Used thread-safe dict.copy().keys() for MP consumer partitions Viktor Shlapakov 2015-03-12 11:33:07 +03:00
  • 4bab2fa5d1 Cleaned code for MP consumer chunking Viktor Shlapakov 2015-03-11 13:51:07 +03:00
  • fa43388219 Take the linter to kafka/codec.py Dana Powers 2015-03-08 23:41:46 -07:00
  • 4d59678dd3 Gzip context manager not supported in py2.6, so use try/finally instead Dana Powers 2015-03-08 23:41:10 -07:00
  • 83cee06a33 Fixup indentation in kafka/common.py; add links to protocol definitions Dana Powers 2015-03-08 20:13:13 -07:00
  • 610f01e96d Use context managers in gzip_encode / gzip_decode Dana Powers 2015-03-08 20:01:57 -07:00
  • 92aa7e9428 Merge branch 'vshlapakov-feature-async-threading' PR 330: Threading for async batching Dana Powers 2015-03-08 16:03:06 -07:00
  • 6ef982c5e8 Merge branch 'jobevers-master' PR 302: Add failed_payloads attribute to FailedPayloadsError Dana Powers 2015-03-08 15:16:17 -07:00
  • 95737d9b89 Add failed_payloads attribute to FailedPayloadsError Job Evers 2015-01-22 17:25:02 -06:00
  • 327b734940 Merge pull request #332 from dpkp/kafka_client_edits Dana Powers 2015-03-07 21:34:17 -08:00
  • ac66fe9159 Fixing distribution for MP Consumer Viktor Shlapakov 2015-03-03 15:32:32 +03:00
  • e13eb0de89 Merge pull request #324 from chmduquesne/master Dana Powers 2015-03-02 17:49:07 -08:00
  • 7dea5599b6 Merge pull request #328 from wkiser/validate_offset Dana Powers 2015-03-02 17:36:48 -08:00
  • 2ca10e26b6 Merge pull request #296 from ecanzonieri/validate_consumer_offset Dana Powers 2015-03-02 17:23:03 -08:00
  • 04979248af Stopping daemonized threads when references to these threads are lost Christophe-Marie Duquesne 2015-02-17 00:50:51 +01:00
  • 6de9444b96 Fixes consumer/kafka and consumer/simple to only yield messages if the message's offset is greater than or equal to the consumer offset. wkiser 2015-02-23 12:58:20 -05:00
  • c80fbd1451 KafkaClient.get_partition_ids_for_topic now returns empty list for unknown topics Dana Powers 2015-03-02 09:37:55 -08:00
  • 83d28ba79a Add comment about no server response case in _send_broker_aware_request Dana Powers 2015-03-02 09:35:04 -08:00
  • 082cda6a72 Avoid topic_partitions KeyError in KafkaClient Dana Powers 2015-03-02 09:34:06 -08:00
  • 0f04eba097 Remove KafkaConsumer.BYTES_CONFIGURATION_KEYS - rely on KafkaClient to handle client_id string/bytes encoding - accept either string or bytes for group_id - convert group_id string to utf-8 bytes internally Dana Powers 2015-03-01 11:05:36 -08:00
  • 9f7d61258b Accept KafkaClient client_id as string or bytes; encode string client_id as utf-8 bytes internally Dana Powers 2015-03-01 11:04:18 -08:00
  • 5137163fa4 Async producer stop() fix Viktor Shlapakov 2015-02-26 13:33:40 +03:00
  • aedbbb39be Using threading.Event to stop async producer thread Viktor Shlapakov 2015-02-26 13:02:48 +03:00
  • 1cce287157 Returned original tests, rm dirty flag, name fixes Viktor Shlapakov 2015-02-25 10:45:47 +03:00
  • bc0d5c1e27 Make KafkaConnection copies usable across threads Alexey Borzenkov 2014-04-03 21:39:19 +04:00
  • 250778453f Trying to use threading for async batching Viktor Shlapakov 2015-02-24 21:31:47 +03:00
  • 25ad88cbe6 Correct message keys for async batching mode Viktor Shlapakov 2015-02-20 11:21:12 +03:00
  • 9ad0be662d Merge pull request #314 from dpkp/keyed_producer_failover Dana Powers 2015-02-19 23:25:19 -08:00
  • 4b2807a8a8 Added PYTHONPATH=. when building sphinx doc. debian/0.9.3-2 Thomas Goirand 2015-02-17 15:25:06 +00:00
  • 8b4f754209 Added missing python-mock build-deps. Thomas Goirand 2015-02-17 15:22:30 +00:00
  • f73b734d71 Added debian/patches/do-not-use-sphinxcontrib.napoleon-theme.patch debian/0.9.3-1 Thomas Goirand 2015-02-17 15:14:15 +00:00
  • ccd919d627 Fixed nosetests run Thomas Goirand 2015-02-17 15:11:22 +00:00
  • f92f48d62f python-six as (build-)depends Thomas Goirand 2015-02-17 15:09:10 +00:00
  • 45fa107f8d python-sphinx as build-depends, running nosetests Thomas Goirand 2015-02-17 14:57:05 +00:00
  • 7746850e18 Added initial debian folder. Thomas Goirand 2015-02-17 15:52:31 +01:00
  • 60a73788ee Merge pull request #317 from sontek/update_consumer_docs Dana Powers 2015-02-10 16:25:04 -08:00
  • e94fd22092 Merge pull request #316 from sontek/add_docs_to_tox Dana Powers 2015-02-10 16:24:42 -08:00
  • 2a12fed46f Merge pull request #321 from sontek/switch_to_rst Dana Powers 2015-02-10 16:21:23 -08:00
  • 96bc453da9 Added py3 to classifiers John Anderson 2015-02-10 11:37:55 -08:00
  • 55f5e77421 Final formatting John Anderson 2015-02-10 11:34:12 -08:00
  • 2462b87853 More formatting cleanup John Anderson 2015-02-10 11:32:40 -08:00
  • f6c67aef4f Cleaned up some formatting John Anderson 2015-02-10 11:30:21 -08:00
  • 3193d64e70 Move to .rst and use it in setup.py for long description John Anderson 2015-02-10 11:26:56 -08:00
  • b74e8dd465 Merge pull request #320 from sontek/dont_link_to_mobile David Arthur 2015-02-10 13:18:58 -05:00
  • 960b7a9458 This links to the main page not the mobile site John Anderson 2015-02-10 09:40:06 -08:00
  • 70b3af1b3c Need to commit apidoc for RTD John Anderson 2015-02-10 09:29:32 -08:00
  • e7f52f2aba Use sphinx-apidoc to to generate API docs John Anderson 2015-02-10 08:59:28 -08:00
  • aa19d71f98 Updated documentation for Consumers to prefer KafkaConsumer instead John Anderson 2015-02-09 14:27:42 -08:00
  • c88e3cedf4 Allow better usage of docs locally. John Anderson 2015-02-09 13:50:29 -08:00
  • 6ed6ad53f2 Reorganize producer integration tests; protect against non-sorted partition metadata Dana Powers 2015-02-08 20:07:25 -08:00
  • c6bd6e1fa5 Use a different topic for each producer integration test for isolation Dana Powers 2015-02-08 13:28:06 -08:00
  • 36316c1275 Always use the initial partition list when hashing keys in KeyedProducer Dana Powers 2015-02-06 11:58:54 -08:00
  • fb03c69544 make partitions arg optional in *Partitioner.partition, defaulting to self.partitions from init Dana Powers 2015-02-06 11:57:49 -08:00
  • bb731eb89c Add more comments to test_switch_leader regarding req_acks policy Dana Powers 2015-02-08 10:49:23 -08:00
  • 01f7c0d176 Write a test for issue 313 - keyed producer failover Dana Powers 2015-02-06 11:50:34 -08:00