Browse Source

Address follow-on comments in strict-two-level spec

This change addresses concerns raised in the original review of the
strict two level enforcement model:

  Ibfb2ba2ffb0115fa7cf81d30bf9a025652d9ba42

bp strict-two-level-enforcement-model

Change-Id: I7190443de8be189eac06a6f01f99a5e5bfabbbc9
Lance Bragstad 10 months ago
parent
commit
3cdec964e2
1 changed files with 84 additions and 48 deletions
  1. 84
    48
      specs/keystone/rocky/strict-two-level-enforcement-model.rst

+ 84
- 48
specs/keystone/rocky/strict-two-level-enforcement-model.rst View File

@@ -100,13 +100,12 @@ respectively.
100 100
       A -> B;
101 101
       A -> C;
102 102
 
103
-      A [label="A (l=20, u=0)"];
103
+      A [label="A (l=20, u=4)"];
104 104
       B [label="B (u=0)"];
105 105
       C [label="C (u=0)"];
106 106
    }
107 107
 
108
-Technically, both ``B`` and ``C`` can use up to 10 ``cores`` each and consume
109
-the entire limit for the tree. Resulting in:
108
+Technically, both ``B`` and ``C`` can use up to 8 ``cores`` each, resulting in:
110 109
 
111 110
 .. blockdiag::
112 111
 
@@ -116,15 +115,16 @@ the entire limit for the tree. Resulting in:
116 115
       A -> B;
117 116
       A -> C;
118 117
 
119
-      A [label="A (l=20, u=0)"];
120
-      B [label="B (u=10)", textcolor = "#00af00"];
121
-      C [label="C (u=10)", textcolor = "#00af00"];
118
+      A [label="A (l=20, u=4)"];
119
+      B [label="B (u=8)", textcolor = "#00af00"];
120
+      C [label="C (u=8)", textcolor = "#00af00"];
122 121
    }
123 122
 
124
-If ``A`` attempts to claim two ``cores``, the usage check will fail because
125
-``oslo.limit`` will fetch the hierarchy from keystone and check the usage of
126
-each project in the hierarchy by using the callback provided by the service to
127
-see that both ``B`` and ``C`` have 10 ``cores`` each:
123
+If ``A`` attempts to claim two more ``cores``, the usage check will fail
124
+because ``oslo.limit`` will fetch the hierarchy from keystone and check the
125
+usage of each project in the hierarchy by using the callback provided by the
126
+service to see that total usage of ``A``, ``B``, and ``C`` is equal to the
127
+limit of the tree, set by ``A.limit``.
128 128
 
129 129
 .. blockdiag::
130 130
 
@@ -134,9 +134,9 @@ see that both ``B`` and ``C`` have 10 ``cores`` each:
134 134
       A -> B;
135 135
       A -> C;
136 136
 
137
-      A [label="A (l=20, u=2)", textcolor = "#FF0000"];
138
-      B [label="B (u=10)"];
139
-      C [label="C (u=10)"];
137
+      A [label="A (l=20, u=6)", textcolor = "#FF0000"];
138
+      B [label="B (u=8)"];
139
+      C [label="C (u=8)"];
140 140
    }
141 141
 
142 142
 Despite the usage of the tree being equal to the limit, we can still add
@@ -151,9 +151,9 @@ children to the tree:
151 151
       A -> C;
152 152
       A -> D;
153 153
 
154
-      A [label="A (l=20, u=0)"];
155
-      B [label="B (u=10)"];
156
-      C [label="C (u=10)"];
154
+      A [label="A (l=20, u=4)"];
155
+      B [label="B (u=8)"];
156
+      C [label="C (u=8)"];
157 157
       D [label="D (u=0)", textcolor = "#00af00"];
158 158
    }
159 159
 
@@ -169,9 +169,9 @@ tree prevents ``D`` from claiming any ``cores``:
169 169
       A -> C;
170 170
       A -> D;
171 171
 
172
-      A [label="A (l=20, u=0)"];
173
-      B [label="B (u=10)"];
174
-      C [label="C (u=10)"];
172
+      A [label="A (l=20, u=4)"];
173
+      B [label="B (u=8)"];
174
+      C [label="C (u=8)"];
175 175
       D [label="D (u=2)", textcolor = "#FF0000"];
176 176
    }
177 177
 
@@ -181,9 +181,9 @@ because it provides a very clear escalation path. When a request fails because
181 181
 the tree limit has been exceeded, a user has all the information they need to
182 182
 provide meaningful context in a support ticket (e.g. their project ID and the
183 183
 parent project ID). An administrator of project ``A`` should be able to
184
-reshuffle usage accordingly. A system administrator should be able to do the
185
-same thing. Providing this information in tree structures with more than a
186
-depth of two is much harder, but may be implemented with a separate model.
184
+reshuffle usage accordingly. A system administrator should be able to as well.
185
+Providing this information in tree structures with more than a depth of two is
186
+much harder, but may be implemented with a separate model.
187 187
 
188 188
 .. blockdiag::
189 189
 
@@ -194,9 +194,9 @@ depth of two is much harder, but may be implemented with a separate model.
194 194
       A -> C;
195 195
       C -> D;
196 196
 
197
-      A [label="A (l=20, u=0)"];
198
-      B [label="B (u=10)"];
199
-      C [label="C (u=10)"];
197
+      A [label="A (l=20, u=4)"];
198
+      B [label="B (u=8)"];
199
+      C [label="C (u=8)"];
200 200
       D [label="D (u=0)", textcolor = "#FF0000"];
201 201
    }
202 202
 
@@ -211,14 +211,15 @@ project-specific override for ``cores``:
211 211
       A -> B;
212 212
       A -> C;
213 213
 
214
-      A [label="A (l=20, u=0)"];
215
-      B [label="B (l=12, u=10)", textcolor = "#00af00"];
216
-      C [label="C (u=10)"];
214
+      A [label="A (l=20, u=4)"];
215
+      B [label="B (l=12, u=8)", textcolor = "#00af00"];
216
+      C [label="C (u=8)"];
217 217
    }
218 218
 
219 219
 Note that regardless of this update, any subsequent requests to claim more
220 220
 ``cores`` in the tree will be forbidden since the usage is equal to the limit
221
-of the ``A``. If ``cores`` are released from ``C``, ``B`` can claim them:
221
+of the ``A``. If ``cores`` are released from ``A`` and ``C``, ``B`` can claim
222
+them:
222 223
 
223 224
 .. blockdiag::
224 225
 
@@ -228,9 +229,9 @@ of the ``A``. If ``cores`` are released from ``C``, ``B`` can claim them:
228 229
       A -> B;
229 230
       A -> C;
230 231
 
231
-      A [label="A (l=20, u=0)"];
232
-      B [label="B (l=12, u=10)"];
233
-      C [label="C (u=8)", textcolor = "#00af00"];
232
+      A [label="A (l=20, u=2)", textcolor = "#00af00"];
233
+      B [label="B (l=12, u=8)"];
234
+      C [label="C (u=6)", textcolor = "#00af00"];
234 235
    }
235 236
 
236 237
 .. blockdiag::
@@ -241,9 +242,9 @@ of the ``A``. If ``cores`` are released from ``C``, ``B`` can claim them:
241 242
       A -> B;
242 243
       A -> C;
243 244
 
244
-      A [label="A (l=20, u=0)"];
245
+      A [label="A (l=20, u=2)"];
245 246
       B [label="B (l=12, u=12)", textcolor = "#00af00"];
246
-      C [label="C (u=8)"];
247
+      C [label="C (u=6)"];
247 248
    }
248 249
 
249 250
 While ``C`` is still under its default allocation of 10 ``cores``, it won't be
@@ -259,9 +260,9 @@ had:
259 260
       A -> B;
260 261
       A -> C;
261 262
 
262
-      A [label="A (l=20, u=0)"];
263
+      A [label="A (l=20, u=2)"];
263 264
       B [label="B (l=12, u=12)"];
264
-      C [label="C (u=10)", textcolor = "#FF0000"];
265
+      C [label="C (u=8)", textcolor = "#FF0000"];
265 266
    }
266 267
 
267 268
 Creating or updating a project with a limit that exceeds the limit of ``A`` is
@@ -362,12 +363,17 @@ to validate limits before writing them to the backend.
362 363
 
363 364
 If keystone is configured to use the ``strict-two-level`` enforcement model and
364 365
 current project structure within keystone violates the two-level project
365
-constraint, keystone should fail to start. To aid operators, we can develop a
366
-``keystone-manage`` command, to check the hierarchical structure of the
367
-projects in the deployment and warn operators if keystone is going to fail to
368
-start. This gives operators the ability to check and fix their project
369
-hierarchy before they deploy keystone with the new model. This clearly
370
-communicates a set project structure to operators at run time.
366
+constraint, keystone should fail to start. On start-up, keystone will scan the
367
+database to ensure that all projects don't exceed two levels of hierarchy and
368
+that ``keystone.conf [DEFAULT] max_project_tree_depth = 2``. If either
369
+condition fails, keystone will report an appropriate error message and refuse
370
+to start.
371
+
372
+To aid operators, we can develop a ``keystone-manage`` command, to check the
373
+hierarchical structure of the projects in the deployment and warn operators if
374
+keystone is going to fail to start. This gives operators the ability to check
375
+and fix their project hierarchy before they deploy keystone with the new model.
376
+This clearly communicates a set project structure to operators at run time.
371 377
 
372 378
 Proposed Library Changes & Consumption
373 379
 --------------------------------------
@@ -544,22 +550,52 @@ None
544 550
 Performance Impact
545 551
 ------------------
546 552
 
553
+Usage Caching
554
+^^^^^^^^^^^^^
555
+
547 556
 Performance of this model is expected to be sub-optimal in comparison to flat
548 557
 enforcement. The main factor contributing to expected performance loss is the
549 558
 calculation of usage for the tree. The ``oslo.limit`` library will need to
550 559
 calculate the usage for every project in the tree in order to provide an answer
551 560
 to the service regarding the request.
552 561
 
562
+One possible solution to mitigate performance concerns here would be to
563
+calculate usage for projects in parallel.
564
+
565
+Limit Caching
566
+^^^^^^^^^^^^^
567
+
553 568
 Other services will be required to make additional calls to keystone to
554 569
 retrieve limit information in order to do quota enforcement. This will add some
555
-overhead to the overall performance of the API call.
570
+overhead to the overall performance of the API call because it requires a
571
+round-trip to keystone.
556 572
 
557 573
 It is also worth noting that both Registered Limits and Project Limits are not
558
-expected to change frequently. This means the data is safe to cache for some
559
-period of time. Caching has already been implemented internally to keystone,
560
-similar to how keystone caches responses for other resources. Caching can also
561
-be done client-side to avoid making frequent calls to keystone for relatively
562
-static limit information.
574
+expected to change frequently. This means that the limit data could possibly be
575
+cached client-side in ``oslo.limit``. However, this introduces concerns about
576
+limit invalidation. Consider the following example:
577
+
578
+* client-side cache TTL is set to one hour for limit information from keystone
579
+* one minute later, an administrator decreases the limit for ``cores`` on a
580
+  particular project
581
+* two minutes later, a user makes a request to create an instance in the
582
+  project that should bring them over the limit just set by the administrator
583
+* due to client-side caching, the service considers the project within it's
584
+  limit and allows the instance to be created
585
+* the current usage is out of sync with the limit set by the administrator and
586
+  the service won't realize this until the TTL expires in another 57 minutes
587
+
588
+Client-side caching is going to be a very specific case that needs to be
589
+handled carefully because cache invalidation strategies are going to be
590
+distributed across services. One possible mitigation would be for client-side
591
+caching and keystone to share the same cache instance, making it easier to
592
+perform cache invalidation. Conversely, this raises the operational bar for
593
+administrators and requires assumptions about underlying infrastructure.
594
+
595
+Until we know we can make those types of assumptions or find an alternative
596
+solution for cache invalidation, client-side caching should be avoided to
597
+prevent situations like what was described above. We should error on the side
598
+of accuracy when retrieving limit information.
563 599
 
564 600
 Other Deployer Impact
565 601
 ---------------------

Loading…
Cancel
Save