Servicegroup foundational refactoring for Control Plane
At present, there are various interfaces through which services data can be manipulated - admin interface(nova-manage), extensions (contrib/services.py), etc. Every interface relies on the servicegroup layer API is_up() to get details about service live-ness. The proposal is keep service data in nova database nova.services table and fetch the live-ness information from the configured servicegroup(SG) driver. Liveness will be a combination of service live-ness and RPC live-ness, where the later will be computed based on information in nova.services. Previously-approved: liberty Implements: blueprint servicegroup-api-control-plane Change-Id: I3cae9495373c4334fedb18b64d907f3a6ac5ab92
This commit is contained in:
Vilobh Meshram
261
specs/mitaka/approved/servicegroup-api-control-plane.rst
Normal file
261
specs/mitaka/approved/servicegroup-api-control-plane.rst
Normal file
@@ -0,0 +1,261 @@
|
||||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
=======================================================
|
||||
Servicegroup foundational refactoring for Control Plane
|
||||
=======================================================
|
||||
|
||||
https://blueprints.launchpad.net/nova/+spec/servicegroup-api-control-plane
|
||||
|
||||
At present, there are various interfaces through which services data can
|
||||
be manipulated - admin interface(nova-manage), extensions
|
||||
(contrib/services.py), etc. Every interface relies on the servicegroup
|
||||
layer API is_up() to get details about service liveness. The proposal
|
||||
is keep service data in nova database nova.services table and fetch
|
||||
the liveness information from the configured servicegroup(SG) driver.
|
||||
Liveness will be a combination of service liveness and RPC liveness,
|
||||
where the latter will be computed based on information in nova.services.
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Nova's way for determining service liveness is not pluggable. In
|
||||
its current state of art, the services information is stored in nova
|
||||
database in nova.services tables. Whereas, the service liveness
|
||||
information is computed by the is_up() call. This is_up() call is
|
||||
implemented depending on what backend servicegroup driver is chosen.
|
||||
|
||||
Right now other SG drivers are not functional and need to be
|
||||
revamped to allow them be involved in giving details about service
|
||||
liveness. That will be covered as part of separate spec.
|
||||
|
||||
The scope of this spec is limited to :-
|
||||
|
||||
1. Making sure the 2 separate interfaces mentioned above namely the
|
||||
REST API interface and admin interface use the servicegroup layer API
|
||||
while fetching service liveness.
|
||||
|
||||
2. Service.is_up will be an attribute for the Service object which
|
||||
will be computed as a combination of service liveness and rpc
|
||||
liveness. Service liveness will be implemented by the respective
|
||||
SG driver and depending whether the service is up/down a boolean will
|
||||
be returned. Whereas to check rpc liveness Nova will still rely on
|
||||
the nova.services table stored in Nova database.
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
|
||||
This is a refactoring effort and can be applicable for following
|
||||
usecases:
|
||||
|
||||
1. As an operator, I want to use zookeeper to achieve quick detection
|
||||
of service outages. Zookeeper servicegroup driver will be used to report
|
||||
service liveness although service data will still reside in
|
||||
nova.services.
|
||||
|
||||
2. As an operator, I want to be sure that when Nova service
|
||||
delete (REST api) is invoked the service record from respective backend
|
||||
either Zookeeper or Memcache is removed. A SG api to leave() the group, which
|
||||
will be added as part of this change, will be invoked.
|
||||
|
||||
Deployment using Database servicegroup driver to manage service
|
||||
liveness will remain the same apart from including the logic to include
|
||||
is_up as service object attribute and computing as proposed in the
|
||||
"Proposed change" section.
|
||||
|
||||
Project Priority
|
||||
----------------
|
||||
|
||||
None
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
1. Proposed change is to fetch service data from DB and verify service
|
||||
liveness using configured SG driver. The details of how the service
|
||||
liveness will be configured by each driver is upto the implementation
|
||||
details of each SG driver. Point #3.2.2 has details for how the Zookeeper
|
||||
SG will compute it.
|
||||
|
||||
2. By storing the Service data in the database but the service liveness
|
||||
information can be managed by the respective servicegroup driver
|
||||
configured. Also, we have things like service version, which will require
|
||||
some efficient version calculations in order to drive things like compute
|
||||
rpc version pinning and object backporting. That can be done efficiently
|
||||
if the services data is stored in database (as database supports max/min
|
||||
functionality) as opposed to storing in Zookeeper or Memcache.
|
||||
|
||||
3. Change for the service liveness API we can have two options like this :
|
||||
|
||||
def service_is_up(self, member):
|
||||
"""Check if the given member is up."""
|
||||
|
||||
A] For DB SG driver:
|
||||
|
||||
#1. Check RPC liveness using the updated_at attribute
|
||||
in nova.services table.
|
||||
|
||||
#2. Check Service liveness depending upon the SG database
|
||||
drivers is_up() method.
|
||||
|
||||
B] For Zookeeper/Memcache SG drivers:
|
||||
|
||||
The Service object will be the interface by which we determine
|
||||
whether a service is up or down. It will necessarily look up the
|
||||
updated_at stamp like it does now, and will optionally consult an
|
||||
external interface (such as zookeeper, memcache) through a defined
|
||||
interface. The external interface depends on the kind of SG driver
|
||||
configured as part of CONF.servicegroup_driver. If either indication
|
||||
results in a "down" verdict, the service will be considered
|
||||
down. Please note both the steps below will be needed to detect
|
||||
service liveness.
|
||||
|
||||
#1. Check RPC liveness using the updated_at attribute
|
||||
in nova.services table.
|
||||
|
||||
#2. Check Service liveness depending upon the Zookeeper
|
||||
SG driver or Memcache SG driver is_up() method.
|
||||
|
||||
#2.1: If Znode for the compute host has not joined
|
||||
the topic 'compute' then the nova-compute service
|
||||
is not running on this compute host. The details
|
||||
on how the Zookeeper ephemeral znode will maintain
|
||||
the service representational state and how to migrate
|
||||
from existing database servicegroup driver to
|
||||
zookeeper/memcache SG driver has been covered
|
||||
in https://review.openstack.org/#/c/138607.
|
||||
|
||||
4. A SG api to leave a group need to be introduced to the SG layer
|
||||
and will be implemented by backend drivers. The drivers that don't need
|
||||
the leave functionality will not provide any additional logic to
|
||||
free up the service record associated with the service. For example
|
||||
the znodes used to keep track of service when using Zookeeper SG driver
|
||||
are ephemeral which means that they will be automatically deleted
|
||||
when the service is deleted. But for other backends like memcache which
|
||||
are key/value stores the record needs to be cleared off explicitly. The
|
||||
api at the SG layer might look like :
|
||||
|
||||
def leave(self, group_id, member)
|
||||
|
||||
As mentioned above depending on the driver used this can be already supported
|
||||
if not need to explicitly call out to clean up the service entry.
|
||||
|
||||
5. The call will now just invoke service.is_up to check rpc liveness and
|
||||
service liveness. Whereas at the object layer is_up will be computed as
|
||||
a combination of when the service record was last updated which will give
|
||||
details about RPC liveness and querying the respective CONF.sg_driver
|
||||
for service liveness.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
None
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
Service liveness is fetched from configured SG driver where as
|
||||
service details will be fetched from nova database nova.services
|
||||
tables. RPC liveness will also be computed based on the data in
|
||||
nova.services table.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
vilobhmm
|
||||
|
||||
Other contributors:
|
||||
jaypipes, harlowja
|
||||
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
- Introduce an additional attribute is_up to nova.objects.service.
|
||||
- Fix admin interface, nova-manage where a service is_up/is_down
|
||||
will depend on the combination of service liveness depending on
|
||||
what SG driver is configured and RPC liveness computed based of
|
||||
information stored in nova.services table.
|
||||
- Introduce leave() API at the SG layer to make sure when a service
|
||||
is deleted in situations where service liveness is maintained by
|
||||
backends other than db, the znode or the associated structure for
|
||||
the service is freed up.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
None
|
||||
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
1. Existing unit tests will be updated to make sure the services
|
||||
data is fetched from nova.services tables and service liveness
|
||||
using servicegroup API.
|
||||
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
None
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
- http://lists.openstack.org/pipermail/openstack-dev/2015-May/063602.html
|
||||
- https://review.openstack.org/#/c/138607
|
||||
- http://lists.openstack.org/pipermail/openstack-dev/2015-September/075267.html
|
||||
|
||||
.. _etherpad: https://etherpad.openstack.org/p/servicegroup-refactoring
|
||||
Reference in New Issue
Block a user