19 Commits

Author SHA1 Message Date
melanie witt
bf9deb10c3 Add interfaces for getting limits without enforcing
We currently have a public Enforcer interface for getting limits and
calculating usage but it is not yet possible to retrieve only limits
without calculating usage.

While working on unified limits support in nova, we realized we need a
way to get limits only. In nova there are legacy APIs for showing quota
limits and initially we will provide compat by proxying to keystone to
get the limits.

This adds public interfaces for getting limits to Enforcer.

Related to blueprint unified-limits-nova

Change-Id: I22234e0bb6b3a1cecb29a6b99a3afcd02ffdbf5f
2022-01-21 19:35:33 +00:00
Dan Smith
7e4f36abdb Allow project_id=None for enforce/calculate
This allows a caller to pass None for the project_id if it only wants
it to check the registered limit for a given resource. This is useful
for non-project-scoped resourced where we just want to make sure some
global limit hasn't been exceeded. This would also be relevant for
resources that are created by system-scoped users, such as host
aggregates.

Change-Id: I5fea0143b6a96b5f79bc273961e3e284a260e25e
2022-01-21 19:35:26 +00:00
Dan Smith
a49f3a04d0 Make calculate_usage() work if limits are missing
The calculate_usage interface was added recently to allow consumers
to probe limits and usage without requiring the enforce behavior
workflow. If a limit was passed to it that was not registered in
keystone, get_project_limits() would raise a ProjectOverLimit
exception itself to abort the process immediately, providing the
"unregistered means zero" behavior. This works fine for the enforce
workflow, but not the calculate one.

This changes get_project_limits() to just return a zero limit for
a missing one, which will be considered by the enforce workflow in
the same way, keeping the existing behavior. It will merely be
reported by the calculate workflow, which is the desired change.

Change-Id: Iaab1f0d5eb0da9a667267537d86f6c70bc8db51d
2022-01-10 13:44:01 -08:00
melanie witt
43683f543e Add caching of limits in Enforcer
This adds caching of resource limits for an Enforcer in order to
improve performance when repeated limit checks are needed. The cache
lasts the lifetime of an Enforcer and is enabled by default. It can be
disabled by passing cache=False when instantiating an Enforcer.

One usage pattern for a caching Enforcer would be to create an Enforcer
per service request so that the caching lives only as long as the
request.

Change-Id: I8e43dceec76aecd2b2ae23a137e56519efe29777
2021-10-18 21:43:57 +00:00
Dan Smith
caa75c1bab Add a test fixture
This adds a fixture that can be used by consuming projects to
simulate a set of limits in keystone, without requiring actual
keystone. Currently, consumers have to mock oslo.limit internals (at
least) in order to do testing.

Change-Id: If72050e90ca8b03e26d128c7bbcef6bbea92b501
2021-06-09 08:04:01 -07:00
Dan Smith
1175b0f7c1 Add Enforcer.calculate_usage()
In multiple situations, it is necessary to be able to probe the limits
set for a project without actually enforcing. Examples:

1. Exposing a usage API where we want to not only report the current
   usage, but the limit as well. Otherwise clients have to do their
   own calls to keystone and correlation to get a single integer
   limit value, which we should be able to expose for them.
2. When checking quota as part of a long-running process of consuming
   an unbounded data stream, we need to be able to determine how much
   quota remains so that we can stop the transfer if we exceed the
   limit. Without this, we have to periodically call to keystone
   during the transfer, which is expensive and could fail.

This patch adds a calculate_usage() method to the Enforcer which
calculates the usage using the enforcement model and returns a
mapping of resource names to namedtuples that contain limit and usage
information.

Change-Id: Ic0632cc5ec52aefb85a04f879651963bfa54dcbe
2021-06-07 08:47:38 -07:00
Sean McGinnis
2c1ccf6263
Use unittest.mock instead of third party mock
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.

Change-Id: I0d2b84e2ed627481118151b36386014ed0930452
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-03-13 11:47:56 -05:00
John Garbutt
40ef2764a5 Add flat enforcer
Taking the Nova work as an example, looking to add a basic flat enforcer
that meets Nova's needs.

The user of the Enforce provides a callback. You can see an example of
the callback in the unit tests:

  def fake_callback(project_id, resources):
    return {
      "a": 5,
      "b": 10,
      "c": 0,
    }

In the code where you want to check if you increase the amount of
resources that are being consumed, you can make this call:

   enforcer = limit.Enforcer(fake_callback)
   enforcer.enforce(project_id, {"a": 5})

The enforce function looks up the limits that apply for the given
project_id, uses the callback to count the current usage. The proposed
usage is then calculated be adding the delta and the current usage
together. This is compared to any limits that apply.

If you later want to check if you have races that mean you are
over your limit, you can do this call:

   enforcer.enforce(project_id, {'a': 0})

Summary of key design points:

* single set of logic to enforce limits shared between all projects
  that adopt unified limits

* interface should work for both flat and strict-two-level enforcement

* it is assumed that oslo.limit will control which type of enforcement
  is being applied

* callback lists all resources that need counting
  for the given project_id, in Nova this helps limit
  the number of API calls made to placement

* allows to check if proposed additional usage means you are over
  your limit, and also double check if the current usages means
  you are over quota

* if the code is checking a resource where you do not have a
  registered limit, we default to a limit of zero, i.e. no
  resources can be created unless you set that registered limit
  There will be an appropriate warning logged to help the operator
  understand what needs to be setup in keystone.

This builds on various previous prototypes from:
Co-Authored-By: Lance Bragstad<lbragstad@gmail.com>
Co-Authored-By: wangxiyuan <wangxiyuan@huawei.com>

Change-Id: I294a922ea80af673291c991f07a4a203f25c289d
2019-11-25 18:30:28 +00:00
John Garbutt
a044cd9d70 Fetch unified limits from keystone
For a given resource find out the current limits for the project.
If there are no per project limits look up the registered limits
that act as a default for when there are no project limits.

Future patches will look at comparing these limits with the current and
proposed resource usage to enforce the limits.

Change-Id: Ia5ced4a46833b194f397ac936b99b0c9573b50d1
Co-Authored-By: wangxiyuan <wangxiyuan1007@gmail.com>
Co-Authored-By: Lance Bragstad <lbragstad@gmail.com>
2019-11-25 18:29:05 +00:00
Lance Bragstad
6e5b28a80d Pick between Flat and StrictTwoLevel enforcement
The enforcer needs to be able to determine which model to use and pass
information to it. This commit sets the model attribute on the
enforcer, adds a method to determine which model to use, and
defines a basic interface for enforcement models to use.

Change-Id: Id03d361b702c2ee0811f37ad23bb2b9a3171b1f7
Co-Authored-By: wangxiyuan <wangxiyuan1007@gmail.com>
Co-Authored-By: Lance Bragstad <lbragstad@gmail.com>
Co-Authored-By: John Garbutt <john.garbutt@stackhpc.com>
2019-11-22 16:58:48 +00:00
Lance Bragstad
c02c025a66 Add ksa connection logic
This commit pulls the ksa-specific logic from a previous attempt:

  Ibc35360dd4a952bbe63ee68ddafd5def0608949d

Subsequent patches will use this connection to query unified limit
information out of keystone.

Change-Id: Iae45df091e98904fd6a30db9923b141279fb2cc0
Co-Authored-By: wangxiyuan <wangxiyuan1007@gmail.com>
2019-11-22 16:58:36 +00:00
Lance Bragstad
996edbae12 Add skeleton enforce() method to Enforcer
The `enforce()` method is going to be the main entry point into
oslo.limit for services enforcing usage against unified limits.

Future patches will add methods for collecting information from
keystone, resolving callback functions for project resource usage, and
implement different enforcement models (flat, strict-two-level).

Change-Id: I3cca109213e6d3fad91160ebb632d15499690093
2019-06-20 14:18:38 +00:00
Lance Bragstad
8bf64bb278 Remove verification functionality
We want the initial implementation of oslo.limit to be as simple as
possible. Originally, we thought we were going to implement resource
verification, but that's something we can do later.

Resource verification is meant to protect against race conditions when
two clients claim the same amount of resources and the project is over
limit. Unfortunately, having services implement resource verification
requires them to use either a context manager or make multiple calls
to enforce limits and sprinkling those calls around existing business
logic. Since we don't know which approach we want to use to implement
this, let's just focus on enforcing limits and come back to resource
verification when we get more feedback from developers incorporating
this into their services.

Change-Id: I50f52189862bc3d109654e158cba381b156e70b6
2019-06-19 18:18:19 +00:00
Lance Bragstad
af184185fe Remove ProjectClaim object from oslo.limit
The concept of having project claims and tacking them on to an
Enforcer object is too heavy-handed for what we need right now.
Instead, we can just use a dictionary with resource names as the key
and their respective usage/request limits as values.

Future patches will remove the context manager functionality from
oslo.limit, too. We can revisit the concept of a context manager to
implement additional functionality for verification and race
conditions when we have a for sure need for it.

The goal of this patch is to make the interface as simple as possible
so that it is easier for projects to adopt it.

Note: we're able to make these changes because we haven't released
oslo.limit, yet. We can make breaking changes up to version 1.0, so we
can easily come back to this approach up to that point.

Change-Id: I3f04ffc6d27208500b8e6409c3c30e6cb3147169
2019-06-19 18:18:16 +00:00
wangxiyuan
2773ea9edd Add opts file
Added opts file

added [oslo_limit] group.

keystoneauth options and "endpoint_id" are in this group.

Change-Id: I31e8f7c9f97b7175a7580f720383228e6f355f9a
2018-09-06 14:38:39 +08:00
wangxiyuan
88684c2074 Fix CI
We enabled some basic jobs for oslo.limit recently. This patch
fixed the related CI error.

Change-Id: I1fd27724686ae12c0a279d7a7e45c2711b62bf45
2018-07-30 14:22:38 +08:00
Lance Bragstad
9359e9c1c7 Implement basic Enforcer context manager
This commit builds the foundation for a context manager that can be
used for usage enforcement.

Subsequent patches will incorporate keystoneauth and perform calls
to keystone to verify project limits and registered limits,
incorporating them into the usage check.

Change-Id: I7c78c1979444a2a83189e2035a4ab72890abc5ab
2018-07-02 20:53:13 +00:00
Lance Bragstad
9f7fd57753 Implement ProjectClaim objects
When calculating usage of a resource against a project, there is some
common information that we need to know as an enforcement library.
This can be referred to as a "claim", which consists of the name of
the resource being claimed, the project claiming the resources in the
request, and the quantity being claimed.

This commit adds some basic plumbing and tests that introduce this
object. Subsequent patches will introduce a context manager that
performs usage calculation using ProjecClaim objects.

Change-Id: Iacfa39535c58a7c6b49ba01c5ebf6bbb5f3fe8d1
2018-07-02 19:51:10 +00:00
wangxiyuan
69eb4440dc Init repo
This patch initialises the limit repo using oslo-cookiecutter
tools.

Change-Id: Iaf748223fee886d057f40384c080d4dcd7d3ab73
2018-05-17 18:06:06 +08:00