7994173916
When we run the tripleo-container-image-prepare script, it performs
better under python2 when the process leverages a ProcessPoolExecutor.
Rather than using threading, we should be using processes to handle the
image upload processing. Currently when we're processing the images, we
end up being single threaded due to the GIL when processing the data. By
switching to the ProcessPoolExecutor, we eliminate the locking that is
occuring during the data processing as it'll be handled in each process.
Unfortunately, we cannot leverage the ProcessPoolExecutor when the same
code is run under Mistral. In order to make the code work for both
methods, we need to make the execution type dynamic. This change creates
two types of lock objects that are used to determine what type of
executor to ultimately use when processing the images for uploading.
Additionally this change limits the number of concurrent image upload
processes to 4 if using the ProcessPoolExecutor and caps the number of
threads at a max of 8 based on (cpu count / 2)
Conflicts:
tripleo_common/image/image_uploader.py
Change-Id: I60507eba9884a0660fe269da5ad27b0e57a70ca8
Related-Bug: #1844446
(cherry picked from commit
|
||
|---|---|---|
| .. | ||
| README-tripleo.sh | ||
| bootstrap_host_exec | ||
| bootstrap_host_only_eval | ||
| bootstrap_host_only_exec | ||
| container-update.py | ||
| pull-puppet-modules | ||
| run-validation | ||
| tripleo-build-images | ||
| tripleo-config-download | ||
| tripleo-container-image-prepare | ||
| tripleo-deploy-openshift | ||
| upload-puppet-modules | ||
| upload-swift-artifacts | ||