100 nodes support
Change-Id: Ibb8fafac3b6def746f5b731031e652f819b92926 Blueprint: 100-nodes-support
This commit is contained in:
Łukasz Oleś
parent
c4e758890a
commit
d5783f7a8d
219
specs/6.0/100-nodes-support.rst
Normal file
219
specs/6.0/100-nodes-support.rst
Normal file
@ -0,0 +1,219 @@
|
|||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
==========================================
|
||||||
|
100 nodes support (fuel only)
|
||||||
|
==========================================
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/fuel/+spec/100-nodes-support
|
||||||
|
|
||||||
|
Fuel is an enterprise tool for deploying OpenStack, it should be
|
||||||
|
able to deploy large clusters. Fuel also should be fast and responsive.
|
||||||
|
It does not run any processor consuming tasks, so there is no reason
|
||||||
|
for it to be slow.
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
* For large number of nodes Fuel(nailgun, astute) is getting slow.
|
||||||
|
* Probability of failing provisioning is also increasing.
|
||||||
|
* MySQL DB works only as active/standby which has very poor performance.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
For nailgun
|
||||||
|
-----------
|
||||||
|
|
||||||
|
In the first step, it is necessary to write tests which will show places in
|
||||||
|
code which are not optimal. Some of slow parts are already known.
|
||||||
|
Such tests should include(all in fake mode):
|
||||||
|
|
||||||
|
* list 100 nodes
|
||||||
|
* get cluster with 100 nodes
|
||||||
|
* add 100 nodes to environment
|
||||||
|
* remove 100 nodes from environment
|
||||||
|
* run network verification for environment with 100 nodes
|
||||||
|
* change settings in environment with 100 nodes
|
||||||
|
* change network configuration in environment with 100 nodes
|
||||||
|
* run deploy in environment with 100 nodes
|
||||||
|
* run provision in environment with 100 nodes
|
||||||
|
* ...
|
||||||
|
|
||||||
|
In order to detect any specific code that works slow it's necessary to run all
|
||||||
|
the above mentioned tests which measure the time of execution and compare it to
|
||||||
|
specification in order to see which of them are actually slow.
|
||||||
|
Run the operations under a profiler and then analyse and fix all bottlenecks,
|
||||||
|
non-optimal code, etc.
|
||||||
|
To measure and profile code following tools may be used:
|
||||||
|
|
||||||
|
* cprofile - python module
|
||||||
|
* osprofiler - python module
|
||||||
|
* rally - testing framework
|
||||||
|
|
||||||
|
For fuelclient
|
||||||
|
--------------
|
||||||
|
|
||||||
|
There should not be any performance bottlenecks in the fuelclient, it
|
||||||
|
only parses JSON data. There should be tests for fuelclient which should
|
||||||
|
at least include:
|
||||||
|
|
||||||
|
* list nodes
|
||||||
|
* add nodes to environment
|
||||||
|
* list environment with pending changes for 100 nodes
|
||||||
|
* upload nodes from disk
|
||||||
|
|
||||||
|
For astute
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Testing astute is harder because it includes interaction with hardware
|
||||||
|
and other services like cobbler, tftp, dhcp. There is one known problem
|
||||||
|
which can be addressed now. The rest of the problems can be identified after
|
||||||
|
testing on real hardware.
|
||||||
|
|
||||||
|
One known problem is connected with network/storage capabilities of Fuel Master
|
||||||
|
node. When, during provisioning, 100 nodes simultaneously trying to fetch
|
||||||
|
images and packages. Master node can not handle that high load. Astute should
|
||||||
|
detect such situation and handle it.
|
||||||
|
User should be also able to manually tweak astute work. For example to
|
||||||
|
configure it to provision 10 nodes at the time. It will increase provisioning
|
||||||
|
time but will make it more resistant.
|
||||||
|
There should be configuration option to set number nodes to deploy in one run.
|
||||||
|
|
||||||
|
Currently, if provisioning fails on one of the nodes, astute will
|
||||||
|
stop the whole process. It is not an optimal solution for larger deployments.
|
||||||
|
Some nodes may fail because of random failures, provisioning should still
|
||||||
|
continue in this case.
|
||||||
|
Provision will not be restarted for failed nodes. This nodes will be removed
|
||||||
|
from cluster. User can re-add this nodes to cluster after successful
|
||||||
|
deployment.
|
||||||
|
There should be a configuration option to set percent of nodes which can fail
|
||||||
|
during provisioning.
|
||||||
|
In case when for example all controllers failed to provision, provisioning
|
||||||
|
should be stopped.
|
||||||
|
User should be notified about each failure.
|
||||||
|
|
||||||
|
For UI
|
||||||
|
------
|
||||||
|
|
||||||
|
Our tests show that for 100 nodes UI speed is acceptable. In future, for 1000
|
||||||
|
nodes, it will require some speed improvements.
|
||||||
|
|
||||||
|
For puppet manifests library
|
||||||
|
----------------------------
|
||||||
|
|
||||||
|
Configure HAproxy MySQL backends as active/active.
|
||||||
|
There is a patch https://review.openstack.org/#/c/124549/ addressing this
|
||||||
|
change, but it requires additional researching and load testing.
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
Depends on bottlenecks found, but unlikely.
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
No API changes. All optimization have to be backward compatible.
|
||||||
|
|
||||||
|
Upgrade impact
|
||||||
|
--------------
|
||||||
|
|
||||||
|
Only if database is changed, but unlikely.
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Notifications impact
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
If there are failed nodes. User should be informed about this.
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
After blueprint is implemented Fuel should be able to deploy 100 nodes.
|
||||||
|
Active/active load balancing for MySQL connections should improve DB
|
||||||
|
operations.
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
Rules will change. Some nodes can fail now.
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
loles@mirantis.com
|
||||||
|
ksambor@mirantis.com
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
Blueprint will be implemented in several stages:
|
||||||
|
|
||||||
|
* In first stage all tests will be written.
|
||||||
|
* In next stage all known and discovered bottlenecks will be fixed.
|
||||||
|
* After this tests will be run in virtual environment which can create
|
||||||
|
100 nodes.
|
||||||
|
* At the end tests will be run in lab with 100 physical nodes. This test
|
||||||
|
should show us all astute bottlenecks.
|
||||||
|
* To prevent reintroducing bottlenecks in next releases all test
|
||||||
|
will be integrated with our CI infrastructure.
|
||||||
|
* Additional integration with OSProfiler. It can help find bottleneck
|
||||||
|
in production systems
|
||||||
|
* Additional integration with Rally. It will help to test Fuel in real live
|
||||||
|
environment.
|
||||||
|
* Additional Neutron load testing with Rally in HA for active/active MySQL.
|
||||||
|
Even if active/active will fail the testing, at least we could play with
|
||||||
|
tuning related params and provide some output to community.
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
When all bottlenecks are fixed, load test will be added to CI infrastructure,
|
||||||
|
so non optimal code can immediately be noticed.
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
Deployment rules will change, it should be documented. New notifications
|
||||||
|
should be described. Active/active mode for MySQL should be documented.
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
* https://github.com/stackforge/osprofiler
|
||||||
|
* https://github.com/stackforge/rally
|
||||||
|
* https://docs.google.com/a/mirantis.com/document/d/1GJHr4AHw2qA2wYgngoeN2C-6Dhb7wd1Nm1Q9lkhGCag
|
||||||
|
* https://docs.google.com/a/mirantis.com/document/d/1O2G-fTXlEWh0dAbRCtbrFtPVefc5GvEEOhgBIsU_eP0
|
||||||
|
* http://lists.openstack.org/pipermail/openstack-operators/2014-September/005162.html
|
||||||
Loading…
Reference in New Issue
Block a user