openstack
/
fuel-web
Code Issues Proposed changes
RETIRED, Fuel UI
12,760 Commits 3 Branches 2 Tags 203 MiB
Go to file
Alexander Kislitsky 4fa861fa8a Distributed serialization implementation 
Distributed serialization is implemented with python distributed
library. We have scheduler for jobs management and workers for
jobs processing. Scheduler is started on the master node as well
as set of workers on it. Also workers are started on all nodes.
In the cluster settings we can select the type of serialization
and nodes statuses that allows serialization on it. By default
nodes with status 'ready' are excluded from the workers list.

For data serialization we are using only nodes from the cluster
where serialization is performing.

Before the computation fresh nailgun code is sent to the workers
as zip file and it will be imported for job execution. So we always
have fresh nailgun code on the workers.

In one job we are processing chunks of tasks on the workers. This
approach significantly boosts performance. The tasks chunk size
is defined as settings.LCM_DS_TASKS_PER_JOB parameter.

For limiting memory consumption on the master node we use parameter
settings.LCM_DS_NODE_LOAD_COEFF for calculation max number of jobs
in the processing queue.

Synthetic tests of distributed serialization for 500 nodes with
nubmer of ifaces >= 5 performed on 40 cores (4 different machines)
took 6-7 minutes on average.

Change-Id: Id8ff8fada2f1ab036775fc01c78d91befdda9ea2
Implements: blueprint distributed-serialization
 		2017-03-24 15:15:30 +00:00
bin Remove downloading debian-installer components 2016-04-18 10:18:20 +00:00
debian Introduce proper packaging for fuel-nailgun deb package 2017-03-20 17:29:14 +03:00
docs Move network API handlers to extension 2016-04-27 14:43:42 -07:00
nailgun Distributed serialization implementation 2017-03-24 15:15:30 +00:00
specs Remove version cap for python-networkx-core 2016-09-15 20:47:46 +03:00
systemd Systemd support for nailgun services 2015-11-30 13:47:32 +00:00
tools Prepare for using standard python tests 2017-02-07 19:45:38 +00:00
.gitignore Remove Fuel UI 2016-03-03 12:09:54 +00:00
.gitreview Update .gitreview for new namespace 2015-10-17 22:27:24 +00:00
LICENSE LICENCE added 2014-06-11 17:19:02 +04:00
MAINTAINERS Remove Evgeny from maintainers list for volume_manager 2016-06-23 14:54:54 +00:00
README.md Show team and repo badges on README 2016-11-26 12:16:55 +01:00
build_docs.sh Deprecate shotgun directory 2015-10-27 13:09:34 +03:00
run_tests.sh Move UI-related testing logic to a separate file 2016-02-25 10:53:19 +00:00
tox.ini run extensions tests in 4 threads 2016-09-05 10:44:45 +00:00

README.md

Team and repository tags

Team and repository tags

fuel-web

fuel-web (nailgun) implements REST API and deployment data management. It manages disk volumes configuration data, networks configuration data and any other environment specific data which are necessary for successful deployment. It has required orchestration logic to build instructions for provisioning and deployment in a right order. Nailgun uses SQL database to store its data and AMQP service to interact with workers.

Project resources

Project status, bugs, and blueprints are tracked on Launchpad: https://launchpad.net/fuel

Development documentation is hosted here: https://docs.fuel-infra.org/fuel-dev

Any additional information can be found on the Fuel's project wiki https://wiki.openstack.org/wiki/Fuel

Anyone wishing to contribute to fuel-web should follow the general OpenStack process. A good reference for it can be found here: https://wiki.openstack.org/wiki/How_To_Contribute

http://docs.openstack.org/infra/manual/developers.html