From d5088b7474e4f0a28c82af5d887c162fc242e5da Mon Sep 17 00:00:00 2001 From: Xinran WANG Date: Wed, 11 Apr 2018 04:14:14 +0000 Subject: [PATCH] Introduce Cyborg Resource Quota -- Usage Part The concept of quota gives the possibility to limit the maximum number of acceleration resources that one project/tenant can prossess. It will also give the admin users the ability to increase the resource count on-demand, given that the system constraints are not exceeded. Change-Id: I554b9d4603d5e65f69c2b924fba66565f7f6c3c4 Partially-Implements: blueprint cyborg-resource-quota --- .gitignore | 1 + doc/specs/rocky/resource-quotas.rst | 204 ++++++++++++++++++++++++++++ 2 files changed, 205 insertions(+) create mode 100644 doc/specs/rocky/resource-quotas.rst diff --git a/.gitignore b/.gitignore index e021715d..1632627c 100644 --- a/.gitignore +++ b/.gitignore @@ -10,3 +10,4 @@ ChangeLog # Sample profile etc/cyborg/policy.json.sample +.idea/* diff --git a/doc/specs/rocky/resource-quotas.rst b/doc/specs/rocky/resource-quotas.rst new file mode 100644 index 00000000..b4fad8a9 --- /dev/null +++ b/doc/specs/rocky/resource-quotas.rst @@ -0,0 +1,204 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +================================ +Quota Usage for Cyborg Resources +================================ + +Launchpad blueprint: +https://blueprints.launchpad.net/openstack-cyborg/+spec/cyborg-resource-quota + +There are multiple ways to slice an OpenStack cloud. Imposing quota on these +various slices puts a limitation on the amount of resources that can be +consumed which helps to guarantee "fairness" or fair distribution of resources +at the creation time. If a particular project needs more resources, the +concept of quota gives the ability to increase the resource count on-demand, +given that the system constraints are not exceeded. + + +Problem description +=================== +At present in Cyborg we don't have the concept of Quota on acceleration +resources, so users can consume as many resources as they want. +Quotas are tied closely to physical resources and billable entities, hence from +Cyborg's perspective, it helps to limit the allocation and consumption +of a particular kind of resources at a certain value. + +In place of implementing quota like other services, we want to enable +the unified limit which is provided by Keystone to manage our quota limit[1]. +With unified limits, all limits will be set in Keystone and enforced by +oslo.limit. So we decided to implement quota usage part first. +Once the oslo.limit is ready for other services, Cyborg will invoke oslo.limit +to get the limit information and do limit check etc. + +This specs aims at the implementation of quota usage in Cyborg. As the +oslo.limit is not finished yet, we can directly set the value of limit +manually, and reserved the function calling oslo.limit with a "pass" inside. + + +Use cases +--------- +Alice is an admin. She would like to have a feature which will give her +details of Cyborg acceleration resource consumptions so that she can manage her +resources appropriately. + +She might run into following scenarios: + +* Ability to know current resource consumption. + +* Ability to prohibit overuse by a project. + +* Prevent situation where users in a project get starved because users in + other project consume all the resource. "Quota Management" would help to + gurantee "fairness". + +* Prevent DOS kind of attacks, abuse or error by users, which leads to an + excessive amount of resources allocation. + + +Proposed change +=============== +Proposed changes are introducing a Quota_Usage Table which primarily stores +the quota usage assigned for each resource in a project, and a Reservation +Table to store every modification of resource usage. + +When a new resource allocation request comes, the 'reserved' field in the Quota +usages table will be updated. This acceleration resource is being used to set +up VM. For example, the fpga quota hardlimit is 5 and 3 fgpas have +already been used, then two new fpga requests come in. Since we have 3 fpgas +already used, the 'used' field will be set to 3. Now the 'reserved' +field will be set to 2 untill the fpga attachment is successful. Once +the attachment is done this field will be reset to 0, and the 'used' +count will be updated from 3 to 5. So at this moment, hardlimit is 5, used +is 5 and in-progress is 0. So there is one more request comes in, this request +will be rejected since there is not enough quota available. + +In general, + +Resource quota available = Resource hard_limit - [ +(Resource reserved + Resources already allocated for project)] + +In this specs, we just focus on the update of quota usage and we will not check +if one user has already exceed his quota limit. The limit management will be +set in Keystone in the future and we just need to invoke the oslo.limit. + +Alternatives +------------ +At present there is no quota infrastructure in Cyborg. + +Adding Quota Management layer at the Orchestration layer could be an +alternative.However, our approach will give a finer view of resource +consumptions at the IaaS layer which can be used while provisioning Cyborg +resources. + +Data model impact +----------------- +New Quota usages and reservation table will be introduced to Cyborg database to +store quota consumption for each resource in a project. + +Quota usages table: + ++---------------+--------------+------+-----+---------+----------------+ +| Field | Type | Null | Key | Default | Extra | ++---------------+--------------+------+-----+---------+----------------+ +| created_at | datetime | YES | | NULL | | +| updated_at | datetime | YES | | NULL | | +| id | int(11) | NO | PRI | NULL | auto_increment | +| project_id | varchar(255) | YES | MUL | NULL | | +| resource | varchar(255) | NO | | NULL | | +| reserved | int(11) | NO | | NULL | | +| used | int(11) | NO | | NULL | | ++---------------+--------------+------+-----+---------+----------------+ + +Quota reservation table: + ++------------+--------------+------+-----+---------+----------------+ +| Field | Type | Null | Key | Default | Extra | ++------------+--------------+------+-----+---------+----------------+ +| created_at | datetime | YES | | NULL | | +| updated_at | datetime | YES | | NULL | | +| deleted_at | datetime | YES | | NULL | | +| deleted | tinyint(1) | YES | | NULL | | +| id | int(11) | NO | PRI | NULL | auto_increment | +| uuid | varchar(36) | NO | | NULL | | +| usage_id | int(11) | NO | MUL | NULL | | +| project_id | varchar(255) | YES | MUL | NULL | | +| resource | varchar(255) | YES | | NULL | | +| delta | int(11) | NO | | NULL | | +| expire | datetime | YES | | NULL | | ++------------+--------------+------+-----+---------+----------------+ + +We will also introduce QuotaEngine class which represents the set of +recognized quotas and DbQuotaDriver class which performs check to enforcement +of quotas and also allows to obtain quota information. + +REST API impact +--------------- +Not sure if we need to expose GET quota usage before oslo.limit settle down. + +Security impact +--------------- +None + +Notifications impact +-------------------- +None + +Other end user impact +--------------------- +None + +Performance Impact +------------------ +None + +Other deployer impact +--------------------- +None + +Developer impact +---------------- +None + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: +Xinran WANG + +Other contributors: +None + +Work Items +---------- + +* Introduce Quota usages and Reservation table in Cyborg databases. +* Update these two tables during allocation and deallocation of resources. +* Reserve the place of function which will invoke oslo.limit with a "pass" + inside. +* Add rollback mechanism when allocation fails. + +Dependencies +============ +None + +Testing +======= + +* Each commit will be accompanied with unit tests. +* Gate functional tests will also be covered. + +Documentation Impact +==================== +None + +References +========== + +[1] https://review.openstack.org/#/c/540803