f3c440f769
Distributed serialization is implemented with python distributed library. We have scheduler for jobs management and workers for jobs processing. Scheduler is started on the master node as well as set of workers on it. Also workers are started on all nodes. In the cluster settings we can select the type of serialization and nodes statuses that allows serialization on it. By default nodes with status 'ready' are excluded from the workers list. For data serialization we are using only nodes from the cluster where serialization is performing. Before the computation fresh nailgun code is sent to the workers as zip file and it will be imported for job execution. So we always have fresh nailgun code on the workers. In one job we are processing chunks of tasks on the workers. This approach significantly boosts performance. The tasks chunk size is defined as settings.LCM_DS_TASKS_PER_JOB parameter. For limiting memory consumption on the master node we use parameter settings.LCM_DS_NODE_LOAD_COEFF for calculation max number of jobs in the processing queue. Synthetic tests of distributed serialization for 500 nodes with nubmer of ifaces >= 5 performed on 40 cores (4 different machines) took 6-7 minutes on average. Change-Id: Id8ff8fada2f1ab036775fc01c78d91befdda9ea2 Implements: blueprint distributed-serialization |
||
---|---|---|
bin | ||
debian | ||
docs | ||
nailgun | ||
specs | ||
systemd | ||
tools | ||
.gitignore | ||
.gitreview | ||
LICENSE | ||
MAINTAINERS | ||
README.md | ||
build_docs.sh | ||
run_tests.sh | ||
tox.ini |
README.md
fuel-web
fuel-web (nailgun) implements REST API and deployment data management. It manages disk volumes configuration data, networks configuration data and any other environment specific data which are necessary for successful deployment. It has required orchestration logic to build instructions for provisioning and deployment in a right order. Nailgun uses SQL database to store its data and AMQP service to interact with workers.
Project resources
Project status, bugs, and blueprints are tracked on Launchpad: https://launchpad.net/fuel
Development documentation is hosted here: https://docs.fuel-infra.org/fuel-dev
Any additional information can be found on the Fuel's project wiki https://wiki.openstack.org/wiki/Fuel
Anyone wishing to contribute to fuel-web should follow the general OpenStack process. A good reference for it can be found here: https://wiki.openstack.org/wiki/How_To_Contribute