Keystone scoring for 2018.01 guideline

Scoring Keystone for guideline 2018.01
Changes: Update notes on the working materials folder.

Items To Be Discussed:
identity-v3-list-projects capability was marked required 2017.09.
But the only TC available was flagged since it needs 2 users.
Q. Should we move the capability back to advisory until TC is fixed?
   or until other suitable TCs are added?

Change-Id: Ic84cbe834474e579345e22256f2e956ad2b4b897
Note: waiting for additional comments from PTL.

working_materials/keystone_capabilities_info.csv

@@ -1,7 +1,7 @@
 Capability,Program,Status,Method,Endpoint,Test available?,interop relevant?,PTL Comments,From Defcore Discussion,Scorer Comments,
-identity-v3-tokens-create,platform/compute/object,required,POST,/v3/auth/tokens,1,yes,The returned token value is in the X-Auth-Token header,,tempest.api.identity.v3.test_tokens{test_create_token}, This TC refers to API https://developer.openstack.org/api-ref/identity/v3/#password-authentication-with-unscoped-authorization. Should we add other test cases to tempest in order to validate API for: password-authentication-with-scoped-authorization and password-authentication-with-explicit-unscoped-authorization?,
-identity-v3-api-discovery,platform/compute,required,,,3,yes,,make required,"tempest.api.identity.v3.test_api_discovery{test_api_version_resources, test_api_media_types, test_api_version_statuses}",
-identity-v3-list-projects,platform/compute,advisory,GET,/v3/users/{user_id}/projects,1,yes,,,,
+identity-v3-tokens-create,platform/compute/object,required,POST,/v3/auth/tokens,1,yes,The returned token value is in the X-Auth-Token header,,tempest.api.identity.v3.test_tokens{test_create_token}, This TC refers to API https://developer.openstack.org/api-ref/identity/v3/#password-authentication-with-unscoped-authorization. Should we add other test cases to tempest in order to validate API for: password-authentication-with-scoped-authorization and password-authentication-with-explicit-unscoped-authorization?
+identity-v3-api-discovery,platform/compute,required,GET,/v3,3,yes,,make required,"tempest.api.identity.v3.test_api_discovery{test_api_version_resources, test_api_media_types, test_api_version_statuses}",
+identity-v3-list-projects,platform/compute,required,GET,/v3/users/{user_id}/projects,1,yes,,,Flagged since require 2 set of user credentials.,
 ,,,,,,,,,,
 identity-v3-create-ec2-credentials,,,POST,/v3/credentials,1,yes,,Should we make ec2 compatibility required? unclear,,
 identity-v3-list-ec2-credentials,,,GET,/v3/credentials,1,yes,,Should we make ec2 compatibility required? unclear,,
@@ -9,8 +9,7 @@ identity-v3-show-ec2-credentials,,,GET,/v3/credentials/{credential_id},1,yes,,Sh
 identity-v3-delete-ec2-credentials,,,DELETE,/v3/credentials/{credential_id},1,yes,,Should we make ec2 compatibility required? unclear,,
 identity-v3-update-ec2-credentials,,,PATCH,/v3/credentials/{credential_id},,,,Should we make ec2 compatibility required? unclear,,
 identity-v3-catalog,(make sure it works on all supported releases),,,,,,returned with the token,,,
-identity-v3-password-update,,,POST,/v3/users/{user_id}/password,1,yes,,"
-Untestable without changing user's password, security risk. Also password policies are very particular to different companies, making a test that would pass on all is near impossible.",tempest.api.identity.v3.test_users{test_update_own_password},
+identity-v3-password-update,,,POST,/v3/users/{user_id}/password,1,yes,,"Untestable without changing user's password, security risk. Also password policies are very particular to different companies, making a test that would pass on all is near impossible.",tempest.api.identity.v3.test_users{test_update_own_password},
 ,,,,,,,,,,
 identity-v3-list-groups,platform/compute,,GET,/v3/users/{user_id}/groups,0,yes,,,no test available for this feature,
 identity-v3-get-project,platform/compute,,GET,/v3/projects/{project_id},0,yes,,,admin required,
@@ -19,9 +18,9 @@ identity-v3-get-role,platform/compute,,GET,/v3/roles/{role_id},,no,,,admin requi
 identity-v3-list-domains,platform/compute,,GET,/v3/domains,,no,,,admin required,
 identity-v3-get-domain,platform/compute,,GET,/v3/domains/{domain_id},,no,,,admin required,
 ,,,,,,,,,,
-identity-v3-tokens-validate,platform/compute,,GET,/v3/auth/tokens,,yes,Token to be validated is passed in the X-Subject-Token header,,,"This sounds backwards to me, need to check with steve, shouldn't it be POST for validating and GET for getting a token?"
+identity-v3-tokens-validate,platform/compute,advisory,GET,/v3/auth/tokens,,yes,Token to be validated is passed in the X-Subject-Token header,,,
 identity-v3-revoke-token,platform/compute,,DELETE,/v3/auth/tokens,1,yes,Token to be revoked is passed in the X-Subject-Token header,keystone.keystone.tests.unit.test_revoke{test_revoke_by_user},,
-identity-v3-get-catalog,platform/compute/object,,GET,/v3/auth/catalog,0,yes,,,"couldn't find a test specific for this, there are some tests related in keystone.tests.unit.test_v3_auth.py",
+identity-v3-get-catalog,platform/compute/object,advisory,GET,/v3/auth/catalog,0,yes,,,"TC added in Tempest",
 identity-v3-get-auth-projects,platform/compute,,GET,/v3/auth/projects,0,yes,,,"equivalent as far as I can tell to identity-v3-list-projects. couldn't find a test specific for this, there are some tests related in keystone.tests.unit.test_v3_auth.py",
 ,,,,,,,,,,
 identity-v2-list-versions,,,GET,/,1,yes,,,Deprecated,

working_materials/scoring.txt

@@ -288,20 +288,19 @@ identity-v3-api-discovery:       [1,0,1] [1,1,1] [1,1,1] [1,1,1] [1] [94]*
 identity-v3-catalog:             [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]*
 identity-v3-list-projects:       [1,1,1] [1,1,1] [1,1,0] [0,1,0] [1] [74]*
 identity-v3-list-groups:         [1,1,1] [1,1,1] [1,1,0] [0,1,0] [1] [74]*
+identity-v3-tokens-create:       [1,1,1] [1,1,1] [1,1,1] [1,1,0] [1] [92]*
 identity-v3-tokens-validate:     [1,1,1] [1,1,1] [1,1,0] [0,1,0] [1] [74]*
 
 Notes:
-  * identity-v3-catalog is returned when the api for
-  identity-v3-tokens-create is called (GET /v3/auth/tokens). It is
-  important to consider it because end users may be relying on this
-  catalog for their apps (even though there are other API calls that
-  also show the catalog such as GET /v3/auth/catalog).
   * identity-v3-list-projects and identity-v3-list-groups didn't have usable
   tests in the past, but one was added for identity-v3-list-projects last year.
   Providers like Fog.io
   now actually use the /v3/users/{user_id}/[projects|groups] API's:
   https://git.io/vX9S6
   https://git.io/vX9SP
+  Capability became required 2017.09 but the only TC available was flagged
+  since it requires two sets of credentials. Capability needs additional TCs
+  or existing test should be changed to require only one set of credentials.
   * identity-v3-change-password was considered here but it's applicability is
   a bit hard to gauge: many systems using third-party authentication (such as
   an LDAP/AD server, an external oauth system, etc) require password changes

working_materials/tabulated_scores.csv

@@ -105,6 +105,7 @@ identity-v3-api-discovery,1,0,1,1,1,1,1,1,1,1,1,1,1,94*
 identity-v3-catalog,1,0,1,1,1,1,1,1,0,1,1,1,1,85*
 identity-v3-list-projects,1,1,1,1,1,1,1,1,0,0,1,0,1,74*
 identity-v3-list-groups,1,1,1,1,1,1,1,1,0,0,1,0,1,74*
+identity-v3-tokens-create,1,1,1,1,1,1,1,1,1,1,1,0,1,92*
 identity-v3-tokens-validate,1,1,1,1,1,1,1,1,0,0,1,0,1,74*
 objectstore-object-copy,1,1,1,1,1,1,1,1,1,1,1,1,1,100*
 objectstore-object-create,1,1,1,1,1,1,1,1,1,1,1,1,1,100*