Mention issue with more than one thread and reduce workers
Due to how it appears the filesystem transport in kombu is not thread-safe we will work around this by not having more than one worker active at the same time in this example. Oddly it appears the memory transport is unaffected (but from looking at the code it doesn't look safe either), this may just be due to how the python memory model works though. Part of blueprint more-examples Change-Id: Idaf04fb1a6a622af292511bbcf25329c9a5aab53
This commit is contained in:
committed by
Joshua Harlow
parent
494003283f
commit
f8fbb30412
@@ -53,7 +53,12 @@ USE_FILESYSTEM = False
|
||||
BASE_SHARED_CONF = {
|
||||
'exchange': 'taskflow',
|
||||
}
|
||||
WORKERS = 2
|
||||
|
||||
# Until https://github.com/celery/kombu/issues/398 is resolved it is not
|
||||
# recommended to run many worker threads in this example due to the types
|
||||
# of errors mentioned in that issue.
|
||||
MEMORY_WORKERS = 2
|
||||
FILE_WORKERS = 1
|
||||
WORKER_CONF = {
|
||||
# These are the tasks the worker can execute, they *must* be importable,
|
||||
# typically this list is used to restrict what workers may execute to
|
||||
@@ -90,6 +95,7 @@ if __name__ == "__main__":
|
||||
|
||||
tmp_path = None
|
||||
if USE_FILESYSTEM:
|
||||
worker_count = FILE_WORKERS
|
||||
tmp_path = tempfile.mkdtemp(prefix='wbe-example-')
|
||||
shared_conf.update({
|
||||
'transport': 'filesystem',
|
||||
@@ -100,6 +106,7 @@ if __name__ == "__main__":
|
||||
},
|
||||
})
|
||||
else:
|
||||
worker_count = MEMORY_WORKERS
|
||||
shared_conf.update({
|
||||
'transport': 'memory',
|
||||
'transport_options': {
|
||||
@@ -115,8 +122,8 @@ if __name__ == "__main__":
|
||||
|
||||
try:
|
||||
# Create a set of workers to simulate actual remote workers.
|
||||
print('Running %s workers.' % (WORKERS))
|
||||
for i in range(0, WORKERS):
|
||||
print('Running %s workers.' % (worker_count))
|
||||
for i in range(0, worker_count):
|
||||
worker_conf['topic'] = 'worker-%s' % (i + 1)
|
||||
worker_topics.append(worker_conf['topic'])
|
||||
w = worker.Worker(**worker_conf)
|
||||
|
||||
Reference in New Issue
Block a user