4fa861fa8a
Distributed serialization is implemented with python distributed library. We have scheduler for jobs management and workers for jobs processing. Scheduler is started on the master node as well as set of workers on it. Also workers are started on all nodes. In the cluster settings we can select the type of serialization and nodes statuses that allows serialization on it. By default nodes with status 'ready' are excluded from the workers list. For data serialization we are using only nodes from the cluster where serialization is performing. Before the computation fresh nailgun code is sent to the workers as zip file and it will be imported for job execution. So we always have fresh nailgun code on the workers. In one job we are processing chunks of tasks on the workers. This approach significantly boosts performance. The tasks chunk size is defined as settings.LCM_DS_TASKS_PER_JOB parameter. For limiting memory consumption on the master node we use parameter settings.LCM_DS_NODE_LOAD_COEFF for calculation max number of jobs in the processing queue. Synthetic tests of distributed serialization for 500 nodes with nubmer of ifaces >= 5 performed on 40 cores (4 different machines) took 6-7 minutes on average. Change-Id: Id8ff8fada2f1ab036775fc01c78d91befdda9ea2 Implements: blueprint distributed-serialization |
||
|---|---|---|
| .. | ||
| examples | ||
| nailgun | ||
| tools | ||
| MANIFEST.in | ||
| conftest.py | ||
| manage.py | ||
| requirements.txt | ||
| setup.cfg | ||
| setup.py | ||
| test-requirements.txt | ||
| tox.ini | ||