diff --git a/doc/source/test_plans/keystone/plan.rst b/doc/source/test_plans/keystone/plan.rst index 8b6a7a9..7a10208 100644 --- a/doc/source/test_plans/keystone/plan.rst +++ b/doc/source/test_plans/keystone/plan.rst @@ -1,3 +1,5 @@ +.. _keystone_performance: + ============================ Keystone Performance testing ============================ diff --git a/doc/source/test_results/keystone/all-in-one/index.rst b/doc/source/test_results/keystone/all-in-one/index.rst index 1145402..1e8e2b1 100644 --- a/doc/source/test_results/keystone/all-in-one/index.rst +++ b/doc/source/test_results/keystone/all-in-one/index.rst @@ -4,8 +4,8 @@ Keystone DB / cache operations analysis Environment description ^^^^^^^^^^^^^^^^^^^^^^^ -The test plan (test case #1) is executed at the all-in-one virtual environment, -created with Oracle VM VirtualBox Manager. +The :ref:`keystone_performance` (test case #1) is executed at the all-in-one +virtual environment, created with Oracle VM VirtualBox Manager. Hardware ~~~~~~~~ @@ -60,26 +60,28 @@ Due to the collected profiling information, it can be clearly seen that Keystone was significantly changed during Mitaka timeframe, and lots of changes seem to be related to the following: * Federation support (introduced more complexity on the DB level) - * Moving to oslo.cache instead of local `dogpile.cache` usage in Liberty. + * Moving to oslo.cache instead of local `dogpile.cache` usage in Liberty and + introducing local context cache layer for per-request caching. Federation support introduced multiple SQL JOINs usage in Keystone and made -the database scheme a bit more complex. As for the caching layer usage, one -specific issue is clearly seen in Mitaka keystone operations caching. Although -all Liberty and Mitaka environments had identical caching layer configuration, -from HTML reports it can be clearly seen that on Liberty all possible methods -were successfully cached and while their calling cached copy was always used. -On Mitaka OpenStack for some reason cached copy was not used sometimes and the -DB requests were processed. +the database scheme a bit more complex. In further multinode research it's +planned to check how this is influencing operations DB operations performance +in case, for instance, if Galera cluster is used. + +As for the caching layer usage, one specific issue is clearly seen in Mitaka +Keystone operations caching. Although local context cache should reduce number +of calls to Memcache via storing already grabbed data for the specific API +request in local thread, this was not observed (the duplicated function calls +still used Memcache for cache purposes). The `Keystone bug`_ was filed to +investigate this behaviour. + +One more interesting moment is also related to the cache usage. If the cache +will be turned off in Keystone configuration explicitly, the profiling still +shows `data being fetched`_ from the Memcache. + +.. _Keystone bug: https://bugs.launchpad.net/keystone/+bug/1567403 +.. _data being fetched: https://bugs.launchpad.net/keystone/+bug/1567413 -One more interesting moment is related to the `keystone_authtoken` middleware -(and, more specifically, its cache) usage. Although all environments that took -part in the research had exactly the same `keystone_authtoken` middleware -configuration, described in the test plan and containing `memcache_servers` -parameter set up, Mitaka environments profiling shows that all OpenStack -services that used `keystone_authtoken` middleware did not use the external -Memcached cached token copy. All of them used Keystone API every time REST API -request was coming to the OpenStack services. This behaviour needs to be -investigated separately. Reports ^^^^^^^ diff --git a/doc/source/test_results/keystone/all-in-one/mitaka_fernet_cache_on.rst b/doc/source/test_results/keystone/all-in-one/mitaka_fernet_cache_on.rst index 78f53f9..1c3b1e6 100644 --- a/doc/source/test_results/keystone/all-in-one/mitaka_fernet_cache_on.rst +++ b/doc/source/test_results/keystone/all-in-one/mitaka_fernet_cache_on.rst @@ -23,21 +23,21 @@ Endpoint list request stats +--------------------------------------------------------------+-----------+ | **Metric** | **Value** | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries count | 398 | +| Total (*) Keystone DB queries count | 36 | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries time spent, ms | 4107 | +| Total (*) Keystone DB queries time spent, ms | 203 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries count | 152 | +| Infrastructure (SELECT 1) Keystone DB queries count | 15 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 1477 | +| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 82 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries count | 246 | +| Real Keystone DB queries count | 21 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries time spent, ms | 2630 | +| Real Keystone DB queries time spent, ms | 121 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries count | 246 | +| SELECT Keystone DB queries count | 21 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries time spent, ms | 2630 | +| SELECT Keystone DB queries time spent, ms | 121 | +--------------------------------------------------------------+-----------+ .. note:: (*) OSprofiler uses specific SQLalchemy cursor events to track @@ -46,50 +46,25 @@ Endpoint list request stats sure that connection to the database is still in place. -.. note:: Number of DB queries is much bigger for this request comparing with - similar Liberty environment. The difference is in - `keystone_authtoken` middleware (and, more specifically, its cache) - usage. Although all environments that took part in the research had - exactly the same `keystone_authtoken` middleware configuration, - described in the test plan and containing `memcache_servers` - parameter set up, Mitaka environments profiling shows that all - OpenStack services that used `keystone_authtoken` middleware did not - use the external Memcached cached token copy. All of them used - Keystone API every time REST API request was coming to the OpenStack - services. This behaviour needs to be investigated separately. - - **Keystone DB queries outliers** +------------------------------------------------------------------------------------------------------+--------------------+ | **DB query** | **Time spent, ms** | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT password.id AS password_id, password.local_user_id AS password_local_user_id, | 29 | -| password.password AS password_password | | -| FROM password | | -| WHERE %(param_1)s = password.local_user_id | | +| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 3 | +| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | +| anon_1_user_id | | +| FROM (SELECT user.id AS user_id | | +| FROM user | | +| WHERE user.id = %(param_1)s) AS anon_1 INNER JOIN local_user ON anon_1.user_id = local_user.user_id | | +| ORDER BY anon_1.user_id | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 18 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = false | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT revocation_event.id AS revocation_event_id, revocation_event.domain_id AS | 24 | -| revocation_event_domain_id, revocation_event.project_id AS revocation_event_project_id, | | -| revocation_event.user_id AS revocation_event_user_id, revocation_event.role_id AS | | -| revocation_event_role_id, revocation_event.trust_id AS revocation_event_trust_id, | | -| revocation_event.consumer_id AS revocation_event_consumer_id, revocation_event.access_token_id AS | | -| revocation_event_access_token_id, revocation_event.issued_before AS revocation_event_issued_before, | | -| revocation_event.expires_at AS revocation_event_expires_at, revocation_event.revoked_at AS | | -| revocation_event_revoked_at, revocation_event.audit_id AS revocation_event_audit_id, | | -| revocation_event.audit_chain_id AS revocation_event_audit_chain_id | | -| FROM revocation_event ORDER BY revocation_event.revoked_at | | +| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 8 | +| service.extra AS service_extra | | +| FROM service | | +| WHERE service.id = %(param_1)s | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ @@ -103,93 +78,7 @@ Endpoint list request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT `group`.id AS group_id, `group`.name AS group_name, `group`.domain_id AS group_domain_id, | 42 | -| `group`.description AS group_description, `group`.extra AS group_extra | | -| FROM `group` INNER JOIN user_group_membership ON `group`.id = user_group_membership.group_id | | -| WHERE user_group_membership.user_id = %(user_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 19 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 4 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 12 | -| service.extra AS service_extra | | -| FROM service | | -| WHERE service.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service_provider.id AS service_provider_id, service_provider.enabled AS | 24 | -| service_provider_enabled, service_provider.description AS service_provider_description, | | -| service_provider.auth_url AS service_provider_auth_url, service_provider.sp_url AS | | -| service_provider_sp_url, service_provider.relay_state_prefix AS service_provider_relay_state_prefix | | -| FROM service_provider | | -| WHERE service_provider.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 29 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user | | -| WHERE user.id = %(param_1)s) AS anon_1 INNER JOIN local_user ON anon_1.user_id = local_user.user_id | | -| ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS | 52 | -| project_domain_id, project.description AS project_description, project.enabled AS project_enabled, | | -| project.extra AS project_extra, project.parent_id AS project_parent_id, project.is_domain AS | | -| project_is_domain | | -| FROM project | | -| WHERE project.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT role.id AS role_id, role.name AS role_name, role.domain_id AS role_domain_id, role.extra AS | 24 | -| role_extra | | -| FROM role | | -| WHERE role.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 19 | -| service.extra AS service_extra, endpoint_1.id AS endpoint_1_id, endpoint_1.legacy_endpoint_id AS | | -| endpoint_1_legacy_endpoint_id, endpoint_1.interface AS endpoint_1_interface, endpoint_1.region_id AS | | -| endpoint_1_region_id, endpoint_1.service_id AS endpoint_1_service_id, endpoint_1.url AS | | -| endpoint_1_url, endpoint_1.enabled AS endpoint_1_enabled, endpoint_1.extra AS endpoint_1_extra | | -| FROM service LEFT OUTER JOIN endpoint AS endpoint_1 ON service.id = endpoint_1.service_id | | -| WHERE service.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 14 | -| user.default_project_id AS user_default_project_id | | -| FROM user | | -| WHERE user.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 29 | +| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 7 | | federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | | federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | | federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | @@ -200,79 +89,55 @@ Endpoint list request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT implied_role.prior_role_id AS implied_role_prior_role_id, implied_role.implied_role_id AS | 15 | -| implied_role_implied_role_id | | -| FROM implied_role | | -| WHERE implied_role.prior_role_id = %(prior_role_id_1)s | | +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 9 | +| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | +| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | +| endpoint_enabled, endpoint.extra AS endpoint_extra | | +| FROM endpoint | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ **Keystone DB queries with multi JOINs inside** -+------------------------------------------------------------------------------------------------------+--------------------+ -| **DB query** | **Time spent, ms** | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 4 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 5 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 5 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 9 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ ++--------------+--------------------+ +| **DB query** | **Time spent, ms** | ++--------------+--------------------+ **Keystone cached methods stats** -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| **Cached operations** | **args** | **kwargs** | **Times used** | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_role | , u'4ed859be7342465c945448395bac826f' | | 2 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_role | , u'4ed859be7342465c945448395bac826f' | | 9 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_domain | , u'default' | | 2 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_domain | , u'default' | | 9 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_project | , '267e50e2487c46cdb60c1ec82adce5f9' | | 27 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_project | , u'267e50e2487c46cdb60c1ec82adce5f9' | | 6 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| **Cache** | **Cached operations** | **args** | **kwargs** | **Times used** | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_domain | , u'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project | , '1f093e18f4ab4318bdb2876a03d81811' | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_role | , u'3481c11394a64189bafc60551e1ee25c' | | 3 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_roles_for_user_and_project | , u'b4a6b5f09b9c4b57a582b16537f4a976', | | 2 | +| | | u'1f093e18f4ab4318bdb2876a03d81811' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project_by_name | , u'admin', 'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_domain | , u'default' | | 4 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_user_by_name | , u'admin', 'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | _get_revoke_tree | , | | 9 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | _validate_token | , 'gAAAAABXBnVbEtemCkqIkBAri3f0P5C2s3z- | | 8 | +| | | ZzP5MCkHMCHVGuLdwaJsxuw9k731LHrIuKE9krW- | | | +| | | 2bU7ToROLD9oPTZA38kOU4jy5kqJWzBj7O5VpoQDSi_3VerUdQytO31d47N6v-dYmLtatUKPAHJmMGvRikjhBN4laTh8gay- | | | +| | | LEPId9yhdeA' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_roles_for_user_and_project | , 'b4a6b5f09b9c4b57a582b16537f4a976', | | 1 | +| | | '1f093e18f4ab4318bdb2876a03d81811' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project | , u'1f093e18f4ab4318bdb2876a03d81811' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_user | , 'b4a6b5f09b9c4b57a582b16537f4a976' | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ Server create request stats ~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -282,21 +147,21 @@ Server create request stats +--------------------------------------------------------------+-----------+ | **Metric** | **Value** | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries count | 2108 | +| Total (*) Keystone DB queries count | 22 | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries time spent, ms | 19003 | +| Total (*) Keystone DB queries time spent, ms | 213 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries count | 807 | +| Infrastructure (SELECT 1) Keystone DB queries count | 8 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 6813 | +| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 77 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries count | 1301 | +| Real Keystone DB queries count | 14 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries time spent, ms | 12190 | +| Real Keystone DB queries time spent, ms | 136 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries count | 1301 | +| SELECT Keystone DB queries count | 14 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries time spent, ms | 12190 | +| SELECT Keystone DB queries time spent, ms | 136 | +--------------------------------------------------------------+-----------+ .. note:: (*) OSprofiler uses specific SQLalchemy cursor events to track @@ -305,143 +170,12 @@ Server create request stats sure that connection to the database is still in place. -.. note:: Number of DB queries is much bigger for this request comparing with - similar Liberty environment. The difference is in - `keystone_authtoken` middleware (and, more specifically, its cache) - usage. Although all environments that took part in the research had - exactly the same `keystone_authtoken` middleware configuration, - described in the test plan and containing `memcache_servers` - parameter set up, Mitaka environments profiling shows that all - OpenStack services that used `keystone_authtoken` middleware did not - use the external Memcached cached token copy. All of them used - Keystone API every time REST API request was coming to the OpenStack - services. This behaviour needs to be investigated separately. - - **Keystone DB queries outliers** +------------------------------------------------------------------------------------------------------+--------------------+ | **DB query** | **Time spent, ms** | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 16 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT password.id AS password_id, password.local_user_id AS password_local_user_id, | 27 | -| password.password AS password_password | | -| FROM password | | -| WHERE %(param_1)s = password.local_user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 27 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = false | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT revocation_event.id AS revocation_event_id, revocation_event.domain_id AS | 18 | -| revocation_event_domain_id, revocation_event.project_id AS revocation_event_project_id, | | -| revocation_event.user_id AS revocation_event_user_id, revocation_event.role_id AS | | -| revocation_event_role_id, revocation_event.trust_id AS revocation_event_trust_id, | | -| revocation_event.consumer_id AS revocation_event_consumer_id, revocation_event.access_token_id AS | | -| revocation_event_access_token_id, revocation_event.issued_before AS revocation_event_issued_before, | | -| revocation_event.expires_at AS revocation_event_expires_at, revocation_event.revoked_at AS | | -| revocation_event_revoked_at, revocation_event.audit_id AS revocation_event_audit_id, | | -| revocation_event.audit_chain_id AS revocation_event_audit_chain_id | | -| FROM revocation_event ORDER BY revocation_event.revoked_at | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 12 | -| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | -| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | -| endpoint_enabled, endpoint.extra AS endpoint_extra, service_1.id AS service_1_id, service_1.type AS | | -| service_1_type, service_1.enabled AS service_1_enabled, service_1.extra AS service_1_extra | | -| FROM endpoint LEFT OUTER JOIN service AS service_1 ON service_1.id = endpoint.service_id | | -| WHERE endpoint.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT `group`.id AS group_id, `group`.name AS group_name, `group`.domain_id AS group_domain_id, | 26 | -| `group`.description AS group_description, `group`.extra AS group_extra | | -| FROM `group` INNER JOIN user_group_membership ON `group`.id = user_group_membership.group_id | | -| WHERE user_group_membership.user_id = %(user_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 26 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 4 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 12 | -| service.extra AS service_extra | | -| FROM service | | -| WHERE service.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service_provider.id AS service_provider_id, service_provider.enabled AS | 23 | -| service_provider_enabled, service_provider.description AS service_provider_description, | | -| service_provider.auth_url AS service_provider_auth_url, service_provider.sp_url AS | | -| service_provider_sp_url, service_provider.relay_state_prefix AS service_provider_relay_state_prefix | | -| FROM service_provider | | -| WHERE service_provider.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 26 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user | | -| WHERE user.id = %(param_1)s) AS anon_1 INNER JOIN local_user ON anon_1.user_id = local_user.user_id | | -| ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS | 196 | -| project_domain_id, project.description AS project_description, project.enabled AS project_enabled, | | -| project.extra AS project_extra, project.parent_id AS project_parent_id, project.is_domain AS | | -| project_is_domain | | -| FROM project | | -| WHERE project.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT role.id AS role_id, role.name AS role_name, role.domain_id AS role_domain_id, role.extra AS | 15 | -| role_extra | | -| FROM role | | -| WHERE role.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 50 | +| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 14 | | service.extra AS service_extra, endpoint_1.id AS endpoint_1_id, endpoint_1.legacy_endpoint_id AS | | | endpoint_1_legacy_endpoint_id, endpoint_1.interface AS endpoint_1_interface, endpoint_1.region_id AS | | | endpoint_1_region_id, endpoint_1.service_id AS endpoint_1_service_id, endpoint_1.url AS | | @@ -451,14 +185,51 @@ Server create request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 18 | -| user.default_project_id AS user_default_project_id | | -| FROM user | | -| WHERE user.id = %(param_1)s | | +| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 8 | +| service.extra AS service_extra | | +| FROM service | | +| WHERE service.id = %(param_1)s | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 18 | +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 23 | +| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | +| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | +| endpoint_enabled, endpoint.extra AS endpoint_extra, service_1.id AS service_1_id, service_1.type AS | | +| service_1_type, service_1.enabled AS service_1_enabled, service_1.extra AS service_1_extra | | +| FROM endpoint LEFT OUTER JOIN service AS service_1 ON service_1.id = endpoint.service_id | | +| WHERE endpoint.enabled = true | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 3 | +| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | +| anon_1_user_id | | +| FROM (SELECT user.id AS user_id | | +| FROM user | | +| WHERE user.id = %(param_1)s) AS anon_1 INNER JOIN local_user ON anon_1.user_id = local_user.user_id | | +| ORDER BY anon_1.user_id | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT service_provider.id AS service_provider_id, service_provider.enabled AS | 13 | +| service_provider_enabled, service_provider.description AS service_provider_description, | | +| service_provider.auth_url AS service_provider_auth_url, service_provider.sp_url AS | | +| service_provider_sp_url, service_provider.relay_state_prefix AS service_provider_relay_state_prefix | | +| FROM service_provider | | +| WHERE service_provider.enabled = true | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 9 | +| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | +| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | +| endpoint_enabled, endpoint.extra AS endpoint_extra | | +| FROM endpoint | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 7 | | federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | | federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | | federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | @@ -469,95 +240,46 @@ Server create request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT implied_role.prior_role_id AS implied_role_prior_role_id, implied_role.implied_role_id AS | 18 | -| implied_role_implied_role_id | | -| FROM implied_role | | -| WHERE implied_role.prior_role_id = %(prior_role_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ **Keystone DB queries with multi JOINs inside** -+------------------------------------------------------------------------------------------------------+--------------------+ -| **DB query** | **Time spent, ms** | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 16 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 6 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 5 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 9 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ ++--------------+--------------------+ +| **DB query** | **Time spent, ms** | ++--------------+--------------------+ **Keystone cached methods stats** -+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| **Cached operations** | **args** | **kwargs** | **Times used** | -+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_domain | , u'default' | | 2 | -+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| _get_revoke_tree | , | | 64 | -+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_project | , '2dc93f73479940d0b39ce2e4c52c108c' | | 106 | -+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_domain | , u'default' | | 160 | -+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_role | , u'c76d7947edb44079b7fca0b376674315' | | 36 | -+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_role | , u'3481c11394a64189bafc60551e1ee25c' | | 2 | -+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_project | , u'1f093e18f4ab4318bdb2876a03d81811' | | 6 | -+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_project | , '1f093e18f4ab4318bdb2876a03d81811' | | 108 | -+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_user | , 'a74255d40f634b9d8c83e7fb0959c082' | | 10 | -+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_roles_for_user_and_project | , 'a74255d40f634b9d8c83e7fb0959c082', | | 10 | -| | '2dc93f73479940d0b39ce2e4c52c108c' | | | -+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_roles_for_user_and_project | , 'b4a6b5f09b9c4b57a582b16537f4a976', | | 27 | -| | '1f093e18f4ab4318bdb2876a03d81811' | | | -+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_role | , u'3481c11394a64189bafc60551e1ee25c' | | 66 | -+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_user | , 'b4a6b5f09b9c4b57a582b16537f4a976' | | 27 | -+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| **Cache** | **Cached operations** | **args** | **kwargs** | **Times used** | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_domain | , u'default' | | 4 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | _validate_token | , | | 1 | +| | | 'gAAAAABXBmrRWacx2eWnMbEUxvtXGHLKT3SQfT56J-61d7WyHqAAHH7KN5jv1EFuCusKWBtTZ2KmoXhRN6-u0NdLlgBHYvSwho- | | | +| | | sOmmnD1IhHkHr0ZTml39hLZXhM0HmkAy3tSbq76aLvVYGDqE9BHjjVlU4W-P_gXwad6ZhdH9XkoOCkzmWJR4' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project | , '1f093e18f4ab4318bdb2876a03d81811' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_role | , u'3481c11394a64189bafc60551e1ee25c' | | 4 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_roles_for_user_and_project | , u'b4a6b5f09b9c4b57a582b16537f4a976', | | 2 | +| | | u'1f093e18f4ab4318bdb2876a03d81811' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project_by_name | , u'admin', 'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_domain | , u'default' | | 4 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_user_by_name | , u'admin', 'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | _get_revoke_tree | , | | 3 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_roles_for_user_and_project | , 'b4a6b5f09b9c4b57a582b16537f4a976', | | 2 | +| | | '1f093e18f4ab4318bdb2876a03d81811' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project | , u'1f093e18f4ab4318bdb2876a03d81811' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_user | , 'b4a6b5f09b9c4b57a582b16537f4a976' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ Service list request stats ~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -567,21 +289,21 @@ Service list request stats +--------------------------------------------------------------+-----------+ | **Metric** | **Value** | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries count | 110 | +| Total (*) Keystone DB queries count | 20 | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries time spent, ms | 1048 | +| Total (*) Keystone DB queries time spent, ms | 126 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries count | 40 | +| Infrastructure (SELECT 1) Keystone DB queries count | 7 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 389 | +| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 38 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries count | 70 | +| Real Keystone DB queries count | 13 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries time spent, ms | 659 | +| Real Keystone DB queries time spent, ms | 88 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries count | 70 | +| SELECT Keystone DB queries count | 13 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries time spent, ms | 659 | +| SELECT Keystone DB queries time spent, ms | 88 | +--------------------------------------------------------------+-----------+ .. note:: (*) OSprofiler uses specific SQLalchemy cursor events to track @@ -590,34 +312,11 @@ Service list request stats sure that connection to the database is still in place. -.. note:: Number of DB queries is much bigger for this request comparing with - similar Liberty environment. The difference is in - `keystone_authtoken` middleware (and, more specifically, its cache) - usage. Although all environments that took part in the research had - exactly the same `keystone_authtoken` middleware configuration, - described in the test plan and containing `memcache_servers` - parameter set up, Mitaka environments profiling shows that all - OpenStack services that used `keystone_authtoken` middleware did not - use the external Memcached cached token copy. All of them used - Keystone API every time REST API request was coming to the OpenStack - services. This behaviour needs to be investigated separately. - - **Keystone DB queries outliers** +------------------------------------------------------------------------------------------------------+--------------------+ | **DB query** | **Time spent, ms** | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 12 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ | SELECT password.id AS password_id, password.local_user_id AS password_local_user_id, | 4 | | password.password AS password_password | | | FROM password | | @@ -625,107 +324,6 @@ Service list request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 27 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = false | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT revocation_event.id AS revocation_event_id, revocation_event.domain_id AS | 18 | -| revocation_event_domain_id, revocation_event.project_id AS revocation_event_project_id, | | -| revocation_event.user_id AS revocation_event_user_id, revocation_event.role_id AS | | -| revocation_event_role_id, revocation_event.trust_id AS revocation_event_trust_id, | | -| revocation_event.consumer_id AS revocation_event_consumer_id, revocation_event.access_token_id AS | | -| revocation_event_access_token_id, revocation_event.issued_before AS revocation_event_issued_before, | | -| revocation_event.expires_at AS revocation_event_expires_at, revocation_event.revoked_at AS | | -| revocation_event_revoked_at, revocation_event.audit_id AS revocation_event_audit_id, | | -| revocation_event.audit_chain_id AS revocation_event_audit_chain_id | | -| FROM revocation_event ORDER BY revocation_event.revoked_at | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 13 | -| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | -| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | -| endpoint_enabled, endpoint.extra AS endpoint_extra, service_1.id AS service_1_id, service_1.type AS | | -| service_1_type, service_1.enabled AS service_1_enabled, service_1.extra AS service_1_extra | | -| FROM endpoint LEFT OUTER JOIN service AS service_1 ON service_1.id = endpoint.service_id | | -| WHERE endpoint.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT `group`.id AS group_id, `group`.name AS group_name, `group`.domain_id AS group_domain_id, | 5 | -| `group`.description AS group_description, `group`.extra AS group_extra | | -| FROM `group` INNER JOIN user_group_membership ON `group`.id = user_group_membership.group_id | | -| WHERE user_group_membership.user_id = %(user_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 17 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 4 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 12 | -| service.extra AS service_extra | | -| FROM service | | -| WHERE service.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service_provider.id AS service_provider_id, service_provider.enabled AS | 23 | -| service_provider_enabled, service_provider.description AS service_provider_description, | | -| service_provider.auth_url AS service_provider_auth_url, service_provider.sp_url AS | | -| service_provider_sp_url, service_provider.relay_state_prefix AS service_provider_relay_state_prefix | | -| FROM service_provider | | -| WHERE service_provider.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 5 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user | | -| WHERE user.id = %(param_1)s) AS anon_1 INNER JOIN local_user ON anon_1.user_id = local_user.user_id | | -| ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS | 4 | -| project_domain_id, project.description AS project_description, project.enabled AS project_enabled, | | -| project.extra AS project_extra, project.parent_id AS project_parent_id, project.is_domain AS | | -| project_is_domain | | -| FROM project | | -| WHERE project.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT role.id AS role_id, role.name AS role_name, role.domain_id AS role_domain_id, role.extra AS | 15 | -| role_extra | | -| FROM role | | -| WHERE role.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ | SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 14 | | service.extra AS service_extra, endpoint_1.id AS endpoint_1_id, endpoint_1.legacy_endpoint_id AS | | | endpoint_1_legacy_endpoint_id, endpoint_1.interface AS endpoint_1_interface, endpoint_1.region_id AS | | @@ -736,14 +334,51 @@ Service list request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 16 | -| user.default_project_id AS user_default_project_id | | -| FROM user | | -| WHERE user.id = %(param_1)s | | +| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 8 | +| service.extra AS service_extra | | +| FROM service | | +| WHERE service.id = %(param_1)s | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 13 | +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 23 | +| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | +| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | +| endpoint_enabled, endpoint.extra AS endpoint_extra, service_1.id AS service_1_id, service_1.type AS | | +| service_1_type, service_1.enabled AS service_1_enabled, service_1.extra AS service_1_extra | | +| FROM endpoint LEFT OUTER JOIN service AS service_1 ON service_1.id = endpoint.service_id | | +| WHERE endpoint.enabled = true | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 4 | +| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | +| anon_1_user_id | | +| FROM (SELECT user.id AS user_id | | +| FROM user | | +| WHERE user.id = %(param_1)s) AS anon_1 INNER JOIN local_user ON anon_1.user_id = local_user.user_id | | +| ORDER BY anon_1.user_id | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT service_provider.id AS service_provider_id, service_provider.enabled AS | 13 | +| service_provider_enabled, service_provider.description AS service_provider_description, | | +| service_provider.auth_url AS service_provider_auth_url, service_provider.sp_url AS | | +| service_provider_sp_url, service_provider.relay_state_prefix AS service_provider_relay_state_prefix | | +| FROM service_provider | | +| WHERE service_provider.enabled = true | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 9 | +| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | +| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | +| endpoint_enabled, endpoint.extra AS endpoint_extra | | +| FROM endpoint | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 4 | | federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | | federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | | federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | @@ -754,79 +389,42 @@ Service list request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT implied_role.prior_role_id AS implied_role_prior_role_id, implied_role.implied_role_id AS | 17 | -| implied_role_implied_role_id | | -| FROM implied_role | | -| WHERE implied_role.prior_role_id = %(prior_role_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ **Keystone DB queries with multi JOINs inside** -+------------------------------------------------------------------------------------------------------+--------------------+ -| **DB query** | **Time spent, ms** | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 11 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 12 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 7 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 10 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ ++--------------+--------------------+ +| **DB query** | **Time spent, ms** | ++--------------+--------------------+ **Keystone cached methods stats** -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| **Cached operations** | **args** | **kwargs** | **Times used** | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_role | , u'4ed859be7342465c945448395bac826f' | | 2 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_role | , u'4ed859be7342465c945448395bac826f' | | 1 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_domain | , u'default' | | 2 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_domain | , u'default' | | 1 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_project | , '267e50e2487c46cdb60c1ec82adce5f9' | | 3 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_project | , u'267e50e2487c46cdb60c1ec82adce5f9' | | 6 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| **Cache** | **Cached operations** | **args** | **kwargs** | **Times used** | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_domain | , u'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project | , '1f093e18f4ab4318bdb2876a03d81811' | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_role | , u'3481c11394a64189bafc60551e1ee25c' | | 3 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_roles_for_user_and_project | , u'b4a6b5f09b9c4b57a582b16537f4a976', | | 2 | +| | | u'1f093e18f4ab4318bdb2876a03d81811' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project_by_name | , u'admin', 'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_domain | , u'default' | | 4 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_user_by_name | , u'admin', 'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | _get_revoke_tree | , | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_roles_for_user_and_project | , 'b4a6b5f09b9c4b57a582b16537f4a976', | | 1 | +| | | '1f093e18f4ab4318bdb2876a03d81811' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project | , u'1f093e18f4ab4318bdb2876a03d81811' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_user | , 'b4a6b5f09b9c4b57a582b16537f4a976' | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ Token issue request stats ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -836,21 +434,21 @@ Token issue request stats +--------------------------------------------------------------+-----------+ | **Metric** | **Value** | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries count | 37 | +| Total (*) Keystone DB queries count | 7 | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries time spent, ms | 308 | +| Total (*) Keystone DB queries time spent, ms | 49 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries count | 13 | +| Infrastructure (SELECT 1) Keystone DB queries count | 2 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 120 | +| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 14 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries count | 24 | +| Real Keystone DB queries count | 5 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries time spent, ms | 188 | +| Real Keystone DB queries time spent, ms | 35 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries count | 24 | +| SELECT Keystone DB queries count | 5 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries time spent, ms | 188 | +| SELECT Keystone DB queries time spent, ms | 35 | +--------------------------------------------------------------+-----------+ .. note:: (*) OSprofiler uses specific SQLalchemy cursor events to track @@ -858,150 +456,18 @@ Token issue request stats DB requests "SELECT 1", processed by SQLalchemy itself to make sure that connection to the database is still in place. - -.. note:: Number of DB queries is much bigger for this request comparing with - similar Liberty environment. The difference is in - `keystone_authtoken` middleware (and, more specifically, its cache) - usage. Although all environments that took part in the research had - exactly the same `keystone_authtoken` middleware configuration, - described in the test plan and containing `memcache_servers` - parameter set up, Mitaka environments profiling shows that all - OpenStack services that used `keystone_authtoken` middleware did not - use the external Memcached cached token copy. All of them used - Keystone API every time REST API request was coming to the OpenStack - services. This behaviour needs to be investigated separately. - - **Keystone DB queries outliers** +------------------------------------------------------------------------------------------------------+--------------------+ | **DB query** | **Time spent, ms** | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 12 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT password.id AS password_id, password.local_user_id AS password_local_user_id, | 9 | +| SELECT password.id AS password_id, password.local_user_id AS password_local_user_id, | 4 | | password.password AS password_password | | | FROM password | | | WHERE %(param_1)s = password.local_user_id | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 27 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = false | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT revocation_event.id AS revocation_event_id, revocation_event.domain_id AS | 18 | -| revocation_event_domain_id, revocation_event.project_id AS revocation_event_project_id, | | -| revocation_event.user_id AS revocation_event_user_id, revocation_event.role_id AS | | -| revocation_event_role_id, revocation_event.trust_id AS revocation_event_trust_id, | | -| revocation_event.consumer_id AS revocation_event_consumer_id, revocation_event.access_token_id AS | | -| revocation_event_access_token_id, revocation_event.issued_before AS revocation_event_issued_before, | | -| revocation_event.expires_at AS revocation_event_expires_at, revocation_event.revoked_at AS | | -| revocation_event_revoked_at, revocation_event.audit_id AS revocation_event_audit_id, | | -| revocation_event.audit_chain_id AS revocation_event_audit_chain_id | | -| FROM revocation_event ORDER BY revocation_event.revoked_at | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 9 | -| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | -| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | -| endpoint_enabled, endpoint.extra AS endpoint_extra, service_1.id AS service_1_id, service_1.type AS | | -| service_1_type, service_1.enabled AS service_1_enabled, service_1.extra AS service_1_extra | | -| FROM endpoint LEFT OUTER JOIN service AS service_1 ON service_1.id = endpoint.service_id | | -| WHERE endpoint.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT `group`.id AS group_id, `group`.name AS group_name, `group`.domain_id AS group_domain_id, | 8 | -| `group`.description AS group_description, `group`.extra AS group_extra | | -| FROM `group` INNER JOIN user_group_membership ON `group`.id = user_group_membership.group_id | | -| WHERE user_group_membership.user_id = %(user_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 17 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 4 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 12 | -| service.extra AS service_extra | | -| FROM service | | -| WHERE service.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service_provider.id AS service_provider_id, service_provider.enabled AS | 23 | -| service_provider_enabled, service_provider.description AS service_provider_description, | | -| service_provider.auth_url AS service_provider_auth_url, service_provider.sp_url AS | | -| service_provider_sp_url, service_provider.relay_state_prefix AS service_provider_relay_state_prefix | | -| FROM service_provider | | -| WHERE service_provider.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 4 | -| user.default_project_id AS user_default_project_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 15 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user | | -| WHERE user.id = %(param_1)s) AS anon_1 INNER JOIN local_user ON anon_1.user_id = local_user.user_id | | -| ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS | 9 | -| project_domain_id, project.description AS project_description, project.enabled AS project_enabled, | | -| project.extra AS project_extra, project.parent_id AS project_parent_id, project.is_domain AS | | -| project_is_domain | | -| FROM project | | -| WHERE project.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT role.id AS role_id, role.name AS role_name, role.domain_id AS role_domain_id, role.extra AS | 8 | -| role_extra | | -| FROM role | | -| WHERE role.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ | SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 14 | | service.extra AS service_extra, endpoint_1.id AS endpoint_1_id, endpoint_1.legacy_endpoint_id AS | | | endpoint_1_legacy_endpoint_id, endpoint_1.interface AS endpoint_1_interface, endpoint_1.region_id AS | | @@ -1012,14 +478,51 @@ Token issue request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 16 | -| user.default_project_id AS user_default_project_id | | -| FROM user | | -| WHERE user.id = %(param_1)s | | +| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 8 | +| service.extra AS service_extra | | +| FROM service | | +| WHERE service.id = %(param_1)s | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 10 | +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 23 | +| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | +| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | +| endpoint_enabled, endpoint.extra AS endpoint_extra, service_1.id AS service_1_id, service_1.type AS | | +| service_1_type, service_1.enabled AS service_1_enabled, service_1.extra AS service_1_extra | | +| FROM endpoint LEFT OUTER JOIN service AS service_1 ON service_1.id = endpoint.service_id | | +| WHERE endpoint.enabled = true | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 4 | +| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | +| anon_1_user_id | | +| FROM (SELECT user.id AS user_id | | +| FROM user | | +| WHERE user.id = %(param_1)s) AS anon_1 INNER JOIN local_user ON anon_1.user_id = local_user.user_id | | +| ORDER BY anon_1.user_id | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT service_provider.id AS service_provider_id, service_provider.enabled AS | 13 | +| service_provider_enabled, service_provider.description AS service_provider_description, | | +| service_provider.auth_url AS service_provider_auth_url, service_provider.sp_url AS | | +| service_provider_sp_url, service_provider.relay_state_prefix AS service_provider_relay_state_prefix | | +| FROM service_provider | | +| WHERE service_provider.enabled = true | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 9 | +| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | +| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | +| endpoint_enabled, endpoint.extra AS endpoint_extra | | +| FROM endpoint | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 4 | | federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | | federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | | federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | @@ -1030,52 +533,31 @@ Token issue request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT implied_role.prior_role_id AS implied_role_prior_role_id, implied_role.implied_role_id AS | 19 | -| implied_role_implied_role_id | | -| FROM implied_role | | -| WHERE implied_role.prior_role_id = %(prior_role_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ **Keystone DB queries with multi JOINs inside** -+------------------------------------------------------------------------------------------------------+--------------------+ -| **DB query** | **Time spent, ms** | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 5 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 5 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ ++--------------+--------------------+ +| **DB query** | **Time spent, ms** | ++--------------+--------------------+ **Keystone cached methods stats** -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| **Cached operations** | **args** | **kwargs** | **Times used** | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_role | , u'4ed859be7342465c945448395bac826f' | | 1 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_project | , u'267e50e2487c46cdb60c1ec82adce5f9' | | 3 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_domain | , u'default' | | 1 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| **Cache** | **Cached operations** | **args** | **kwargs** | **Times used** | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_role | , u'3481c11394a64189bafc60551e1ee25c' | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project_by_name | , u'admin', 'default' | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_user_by_name | , u'admin', 'default' | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_roles_for_user_and_project | , u'b4a6b5f09b9c4b57a582b16537f4a976', | | 1 | +| | | u'1f093e18f4ab4318bdb2876a03d81811' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project | , u'1f093e18f4ab4318bdb2876a03d81811' | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_domain | , u'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ User list request stats ~~~~~~~~~~~~~~~~~~~~~~~ @@ -1085,21 +567,21 @@ User list request stats +--------------------------------------------------------------+-----------+ | **Metric** | **Value** | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries count | 120 | +| Total (*) Keystone DB queries count | 30 | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries time spent, ms | 906 | +| Total (*) Keystone DB queries time spent, ms | 306 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries count | 40 | +| Infrastructure (SELECT 1) Keystone DB queries count | 7 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 270 | +| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 70 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries count | 80 | +| Real Keystone DB queries count | 23 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries time spent, ms | 636 | +| Real Keystone DB queries time spent, ms | 236 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries count | 80 | +| SELECT Keystone DB queries count | 23 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries time spent, ms | 636 | +| SELECT Keystone DB queries time spent, ms | 236 | +--------------------------------------------------------------+-----------+ .. note:: (*) OSprofiler uses specific SQLalchemy cursor events to track @@ -1107,35 +589,11 @@ User list request stats DB requests "SELECT 1", processed by SQLalchemy itself to make sure that connection to the database is still in place. - -.. note:: Number of DB queries is much bigger for this request comparing with - similar Liberty environment. The difference is in - `keystone_authtoken` middleware (and, more specifically, its cache) - usage. Although all environments that took part in the research had - exactly the same `keystone_authtoken` middleware configuration, - described in the test plan and containing `memcache_servers` - parameter set up, Mitaka environments profiling shows that all - OpenStack services that used `keystone_authtoken` middleware did not - use the external Memcached cached token copy. All of them used - Keystone API every time REST API request was coming to the OpenStack - services. This behaviour needs to be investigated separately. - - **Keystone DB queries outliers** +------------------------------------------------------------------------------------------------------+--------------------+ | **DB query** | **Time spent, ms** | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 4 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ | SELECT password.id AS password_id, password.local_user_id AS password_local_user_id, | 14 | | password.password AS password_password | | | FROM password | | @@ -1143,29 +601,7 @@ User list request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 15 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = false | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT revocation_event.id AS revocation_event_id, revocation_event.domain_id AS | 15 | -| revocation_event_domain_id, revocation_event.project_id AS revocation_event_project_id, | | -| revocation_event.user_id AS revocation_event_user_id, revocation_event.role_id AS | | -| revocation_event_role_id, revocation_event.trust_id AS revocation_event_trust_id, | | -| revocation_event.consumer_id AS revocation_event_consumer_id, revocation_event.access_token_id AS | | -| revocation_event_access_token_id, revocation_event.issued_before AS revocation_event_issued_before, | | -| revocation_event.expires_at AS revocation_event_expires_at, revocation_event.revoked_at AS | | -| revocation_event_revoked_at, revocation_event.audit_id AS revocation_event_audit_id, | | -| revocation_event.audit_chain_id AS revocation_event_audit_chain_id | | -| FROM revocation_event ORDER BY revocation_event.revoked_at | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 9 | +| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 23 | | federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | | federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | | federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | @@ -1176,93 +612,14 @@ User list request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 9 | -| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | -| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | -| endpoint_enabled, endpoint.extra AS endpoint_extra, service_1.id AS service_1_id, service_1.type AS | | -| service_1_type, service_1.enabled AS service_1_enabled, service_1.extra AS service_1_extra | | -| FROM endpoint LEFT OUTER JOIN service AS service_1 ON service_1.id = endpoint.service_id | | -| WHERE endpoint.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT `group`.id AS group_id, `group`.name AS group_name, `group`.domain_id AS group_domain_id, | 11 | -| `group`.description AS group_description, `group`.extra AS group_extra | | -| FROM `group` INNER JOIN user_group_membership ON `group`.id = user_group_membership.group_id | | -| WHERE user_group_membership.user_id = %(user_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 4 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 5 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 12 | -| service.extra AS service_extra | | -| FROM service | | -| WHERE service.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service_provider.id AS service_provider_id, service_provider.enabled AS | 23 | -| service_provider_enabled, service_provider.description AS service_provider_description, | | -| service_provider.auth_url AS service_provider_auth_url, service_provider.sp_url AS | | -| service_provider_sp_url, service_provider.relay_state_prefix AS service_provider_relay_state_prefix | | -| FROM service_provider | | -| WHERE service_provider.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 4 | +| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 21 | | user.default_project_id AS user_default_project_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 5 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | | FROM user | | -| WHERE user.id = %(param_1)s) AS anon_1 INNER JOIN local_user ON anon_1.user_id = local_user.user_id | | -| ORDER BY anon_1.user_id | | +| WHERE user.id = %(param_1)s | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS | 12 | -| project_domain_id, project.description AS project_description, project.enabled AS project_enabled, | | -| project.extra AS project_extra, project.parent_id AS project_parent_id, project.is_domain AS | | -| project_is_domain | | -| FROM project | | -| WHERE project.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT role.id AS role_id, role.name AS role_name, role.domain_id AS role_domain_id, role.extra AS | 14 | -| role_extra | | -| FROM role | | -| WHERE role.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 11 | +| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 14 | | service.extra AS service_extra, endpoint_1.id AS endpoint_1_id, endpoint_1.legacy_endpoint_id AS | | | endpoint_1_legacy_endpoint_id, endpoint_1.interface AS endpoint_1_interface, endpoint_1.region_id AS | | | endpoint_1_region_id, endpoint_1.service_id AS endpoint_1_service_id, endpoint_1.url AS | | @@ -1272,10 +629,47 @@ User list request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 24 | -| user.default_project_id AS user_default_project_id | | +| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 8 | +| service.extra AS service_extra | | +| FROM service | | +| WHERE service.id = %(param_1)s | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 23 | +| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | +| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | +| endpoint_enabled, endpoint.extra AS endpoint_extra, service_1.id AS service_1_id, service_1.type AS | | +| service_1_type, service_1.enabled AS service_1_enabled, service_1.extra AS service_1_extra | | +| FROM endpoint LEFT OUTER JOIN service AS service_1 ON service_1.id = endpoint.service_id | | +| WHERE endpoint.enabled = true | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 4 | +| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | +| anon_1_user_id | | +| FROM (SELECT user.id AS user_id | | | FROM user | | -| WHERE user.id = %(param_1)s | | +| WHERE user.id = %(param_1)s) AS anon_1 INNER JOIN local_user ON anon_1.user_id = local_user.user_id | | +| ORDER BY anon_1.user_id | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT service_provider.id AS service_provider_id, service_provider.enabled AS | 13 | +| service_provider_enabled, service_provider.description AS service_provider_description, | | +| service_provider.auth_url AS service_provider_auth_url, service_provider.sp_url AS | | +| service_provider_sp_url, service_provider.relay_state_prefix AS service_provider_relay_state_prefix | | +| FROM service_provider | | +| WHERE service_provider.enabled = true | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 9 | +| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | +| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | +| endpoint_enabled, endpoint.extra AS endpoint_extra | | +| FROM endpoint | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ @@ -1290,76 +684,39 @@ User list request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT implied_role.prior_role_id AS implied_role_prior_role_id, implied_role.implied_role_id AS | 4 | -| implied_role_implied_role_id | | -| FROM implied_role | | -| WHERE implied_role.prior_role_id = %(prior_role_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ **Keystone DB queries with multi JOINs inside** -+------------------------------------------------------------------------------------------------------+--------------------+ -| **DB query** | **Time spent, ms** | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 5 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 4 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 7 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 7 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ ++--------------+--------------------+ +| **DB query** | **Time spent, ms** | ++--------------+--------------------+ **Keystone cached methods stats** -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| **Cached operations** | **args** | **kwargs** | **Times used** | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_role | , u'4ed859be7342465c945448395bac826f' | | 2 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_role | , u'4ed859be7342465c945448395bac826f' | | 1 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_domain | , u'default' | | 2 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_domain | , u'default' | | 1 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_project | , '267e50e2487c46cdb60c1ec82adce5f9' | | 3 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_project | , u'267e50e2487c46cdb60c1ec82adce5f9' | | 6 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| **Cache** | **Cached operations** | **args** | **kwargs** | **Times used** | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_domain | , u'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project | , '1f093e18f4ab4318bdb2876a03d81811' | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_role | , u'3481c11394a64189bafc60551e1ee25c' | | 3 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_roles_for_user_and_project | , u'b4a6b5f09b9c4b57a582b16537f4a976', | | 2 | +| | | u'1f093e18f4ab4318bdb2876a03d81811' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project_by_name | , u'admin', 'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_domain | , u'default' | | 4 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_user_by_name | , u'admin', 'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | _get_revoke_tree | , | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_roles_for_user_and_project | , 'b4a6b5f09b9c4b57a582b16537f4a976', | | 1 | +| | | '1f093e18f4ab4318bdb2876a03d81811' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project | , u'1f093e18f4ab4318bdb2876a03d81811' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_user | , 'b4a6b5f09b9c4b57a582b16537f4a976' | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ diff --git a/doc/source/test_results/keystone/all-in-one/mitaka_uuid_cache_on.rst b/doc/source/test_results/keystone/all-in-one/mitaka_uuid_cache_on.rst index b10e14c..f4449a1 100644 --- a/doc/source/test_results/keystone/all-in-one/mitaka_uuid_cache_on.rst +++ b/doc/source/test_results/keystone/all-in-one/mitaka_uuid_cache_on.rst @@ -23,25 +23,25 @@ Endpoint list request stats +--------------------------------------------------------------+-----------+ | **Metric** | **Value** | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries count | 132 | +| Total (*) Keystone DB queries count | 36 | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries time spent, ms | 1240 | +| Total (*) Keystone DB queries time spent, ms | 365 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries count | 55 | +| Infrastructure (SELECT 1) Keystone DB queries count | 15 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 506 | +| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 158 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries count | 77 | +| Real Keystone DB queries count | 21 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries time spent, ms | 734 | +| Real Keystone DB queries time spent, ms | 207 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries count | 75 | +| SELECT Keystone DB queries count | 19 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries time spent, ms | 722 | +| SELECT Keystone DB queries time spent, ms | 199 | +--------------------------------------------------------------+-----------+ | INSERT Keystone DB queries count | 2 | +--------------------------------------------------------------+-----------+ -| INSERT Keystone DB queries time spent, ms | 12 | +| INSERT Keystone DB queries time spent, ms | 8 | +--------------------------------------------------------------+-----------+ .. note:: (*) OSprofiler uses specific SQLalchemy cursor events to track @@ -49,112 +49,36 @@ Endpoint list request stats DB requests "SELECT 1", processed by SQLalchemy itself to make sure that connection to the database is still in place. - -.. note:: Number of DB queries is much bigger for this request comparing with - similar Liberty environment. The difference is in - `keystone_authtoken` middleware (and, more specifically, its cache) - usage. Although all environments that took part in the research had - exactly the same `keystone_authtoken` middleware configuration, - described in the test plan and containing `memcache_servers` - parameter set up, Mitaka environments profiling shows that all - OpenStack services that used `keystone_authtoken` middleware did not - use the external Memcached cached token copy. All of them used - Keystone API every time REST API request was coming to the OpenStack - services. This behaviour needs to be investigated separately. - - **Keystone DB queries outliers** +------------------------------------------------------------------------------------------------------+--------------------+ | **DB query** | **Time spent, ms** | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS | 2 | -| project_domain_id, project.description AS project_description, project.enabled AS project_enabled, | | -| project.extra AS project_extra, project.parent_id AS project_parent_id, project.is_domain AS | | -| project_is_domain | | -| FROM project | | -| WHERE project.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 4 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = false | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT token.id AS token_id, token.expires AS token_expires, token.extra AS token_extra, token.valid | 24 | -| AS token_valid, token.user_id AS token_user_id, token.trust_id AS token_trust_id | | -| FROM token | | -| WHERE token.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT revocation_event.id AS revocation_event_id, revocation_event.domain_id AS | 15 | -| revocation_event_domain_id, revocation_event.project_id AS revocation_event_project_id, | | -| revocation_event.user_id AS revocation_event_user_id, revocation_event.role_id AS | | -| revocation_event_role_id, revocation_event.trust_id AS revocation_event_trust_id, | | -| revocation_event.consumer_id AS revocation_event_consumer_id, revocation_event.access_token_id AS | | -| revocation_event_access_token_id, revocation_event.issued_before AS revocation_event_issued_before, | | -| revocation_event.expires_at AS revocation_event_expires_at, revocation_event.revoked_at AS | | -| revocation_event_revoked_at, revocation_event.audit_id AS revocation_event_audit_id, | | -| revocation_event.audit_chain_id AS revocation_event_audit_chain_id | | -| FROM revocation_event ORDER BY revocation_event.revoked_at | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 13 | +| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 11 | | local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | | anon_1_user_id | | | FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | +| FROM user | | +| WHERE user.id = %(param_1)s) AS anon_1 INNER JOIN local_user ON anon_1.user_id = local_user.user_id | | +| ORDER BY anon_1.user_id | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT `group`.id AS group_id, `group`.name AS group_name, `group`.domain_id AS group_domain_id, | 17 | -| `group`.description AS group_description, `group`.extra AS group_extra | | -| FROM `group` INNER JOIN user_group_membership ON `group`.id = user_group_membership.group_id | | -| WHERE user_group_membership.user_id = %(user_id_1)s | | +| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 17 | +| user.default_project_id AS user_default_project_id | | +| FROM user | | +| WHERE user.id = %(param_1)s | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 40 | +| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 4 | | service.extra AS service_extra | | | FROM service | | | WHERE service.id = %(param_1)s | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 12 | -| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | -| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | -| endpoint_enabled, endpoint.extra AS endpoint_extra, service_1.id AS service_1_id, service_1.type AS | | -| service_1_type, service_1.enabled AS service_1_enabled, service_1.extra AS service_1_extra | | -| FROM endpoint LEFT OUTER JOIN service AS service_1 ON service_1.id = endpoint.service_id | | -| WHERE endpoint.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 4 | -| user.default_project_id AS user_default_project_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT role.id AS role_id, role.name AS role_name, role.domain_id AS role_domain_id, role.extra AS | 4 | -| role_extra | | -| FROM role | | -| WHERE role.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 12 | +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 5 | | endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | | endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | | endpoint_enabled, endpoint.extra AS endpoint_extra | | @@ -165,63 +89,35 @@ Endpoint list request stats **Keystone DB queries with multi JOINs inside** -+------------------------------------------------------------------------------------------------------+--------------------+ -| **DB query** | **Time spent, ms** | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 7 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 8 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 11 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 13 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ ++--------------+--------------------+ +| **DB query** | **Time spent, ms** | ++--------------+--------------------+ **Keystone cached methods stats** -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| **Cached operations** | **args** | **kwargs** | **Times used** | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_project | , u'a465ecd9f7004c38b30792a07b363b2c' | | 6 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_role | , u'31c541984f48440ab8753d11d84e58ca' | | 2 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_domain | , u'default' | | 2 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| **Cache** | **Cached operations** | **args** | **kwargs** | **Times used** | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_roles_for_user_and_project | , u'31d31420a3e349268cd7875aa1825663', | | 2 | +| | | u'f1a058ad00364df29cb819a5478e8884' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | _get_token | , | | 1 | +| | | '4826bf1d6cc04e658316a0135d225f32' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project | , u'f1a058ad00364df29cb819a5478e8884' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_role | , u'8272f6d8be65464e94d3be094cf35e4c' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | _validate_token | , '4826bf1d6cc04e658316a0135d225f32' | | 8 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_user_by_name | , u'admin', 'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project_by_name | , u'admin', 'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | _get_revoke_tree | , | | 9 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_domain | , u'default' | | 4 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ Server create request stats ~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -231,25 +127,25 @@ Server create request stats +--------------------------------------------------------------+-----------+ | **Metric** | **Value** | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries count | 1140 | +| Total (*) Keystone DB queries count | 22 | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries time spent, ms | 9581 | +| Total (*) Keystone DB queries time spent, ms | 225 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries count | 451 | +| Infrastructure (SELECT 1) Keystone DB queries count | 8 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 3500 | +| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 115 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries count | 689 | +| Real Keystone DB queries count | 14 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries time spent, ms | 6081 | +| Real Keystone DB queries time spent, ms | 110 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries count | 687 | +| SELECT Keystone DB queries count | 12 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries time spent, ms | 6067 | +| SELECT Keystone DB queries time spent, ms | 95 | +--------------------------------------------------------------+-----------+ | INSERT Keystone DB queries count | 2 | +--------------------------------------------------------------+-----------+ -| INSERT Keystone DB queries time spent, ms | 14 | +| INSERT Keystone DB queries time spent, ms | 15 | +--------------------------------------------------------------+-----------+ .. note:: (*) OSprofiler uses specific SQLalchemy cursor events to track @@ -258,70 +154,12 @@ Server create request stats sure that connection to the database is still in place. -.. note:: Number of DB queries is much bigger for this request comparing with - similar Liberty environment. The difference is in - `keystone_authtoken` middleware (and, more specifically, its cache) - usage. Although all environments that took part in the research had - exactly the same `keystone_authtoken` middleware configuration, - described in the test plan and containing `memcache_servers` - parameter set up, Mitaka environments profiling shows that all - OpenStack services that used `keystone_authtoken` middleware did not - use the external Memcached cached token copy. All of them used - Keystone API every time REST API request was coming to the OpenStack - services. This behaviour needs to be investigated separately. - - **Keystone DB queries outliers** +------------------------------------------------------------------------------------------------------+--------------------+ | **DB query** | **Time spent, ms** | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT implied_role.prior_role_id AS implied_role_prior_role_id, implied_role.implied_role_id AS | 39 | -| implied_role_implied_role_id | | -| FROM implied_role | | -| WHERE implied_role.prior_role_id = %(prior_role_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS | 61 | -| project_domain_id, project.description AS project_description, project.enabled AS project_enabled, | | -| project.extra AS project_extra, project.parent_id AS project_parent_id, project.is_domain AS | | -| project_is_domain | | -| FROM project | | -| WHERE project.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 24 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = false | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT token.id AS token_id, token.expires AS token_expires, token.extra AS token_extra, token.valid | 27 | -| AS token_valid, token.user_id AS token_user_id, token.trust_id AS token_trust_id | | -| FROM token | | -| WHERE token.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT revocation_event.id AS revocation_event_id, revocation_event.domain_id AS | 12 | -| revocation_event_domain_id, revocation_event.project_id AS revocation_event_project_id, | | -| revocation_event.user_id AS revocation_event_user_id, revocation_event.role_id AS | | -| revocation_event_role_id, revocation_event.trust_id AS revocation_event_trust_id, | | -| revocation_event.consumer_id AS revocation_event_consumer_id, revocation_event.access_token_id AS | | -| revocation_event_access_token_id, revocation_event.issued_before AS revocation_event_issued_before, | | -| revocation_event.expires_at AS revocation_event_expires_at, revocation_event.revoked_at AS | | -| revocation_event_revoked_at, revocation_event.audit_id AS revocation_event_audit_id, | | -| revocation_event.audit_chain_id AS revocation_event_audit_chain_id | | -| FROM revocation_event ORDER BY revocation_event.revoked_at | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 40 | +| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 11 | | local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | | anon_1_user_id | | | FROM (SELECT user.id AS user_id | | @@ -331,67 +169,14 @@ Server create request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS | 8 | -| project_domain_id, project.description AS project_description, project.enabled AS project_enabled, | | -| project.extra AS project_extra, project.parent_id AS project_parent_id, project.is_domain AS | | -| project_is_domain | | -| FROM project | | -| WHERE project.name = %(name_1)s AND project.domain_id = %(domain_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 9 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT `group`.id AS group_id, `group`.name AS group_name, `group`.domain_id AS group_domain_id, | 18 | -| `group`.description AS group_description, `group`.extra AS group_extra | | -| FROM `group` INNER JOIN user_group_membership ON `group`.id = user_group_membership.group_id | | -| WHERE user_group_membership.user_id = %(user_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 63 | +| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 17 | | user.default_project_id AS user_default_project_id | | | FROM user | | | WHERE user.id = %(param_1)s | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 27 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 9 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 40 | -| service.extra AS service_extra | | -| FROM service | | -| WHERE service.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 16 | +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 24 | | endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | | endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | | endpoint_enabled, endpoint.extra AS endpoint_extra, service_1.id AS service_1_id, service_1.type AS | | @@ -401,37 +186,14 @@ Server create request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service_provider.id AS service_provider_id, service_provider.enabled AS | 22 | -| service_provider_enabled, service_provider.description AS service_provider_description, | | -| service_provider.auth_url AS service_provider_auth_url, service_provider.sp_url AS | | -| service_provider_sp_url, service_provider.relay_state_prefix AS service_provider_relay_state_prefix | | -| FROM service_provider | | -| WHERE service_provider.enabled = true | | +| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 4 | +| service.extra AS service_extra | | +| FROM service | | +| WHERE service.id = %(param_1)s | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 9 | -| user.default_project_id AS user_default_project_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT role.id AS role_id, role.name AS role_name, role.domain_id AS role_domain_id, role.extra AS | 14 | -| role_extra | | -| FROM role | | -| WHERE role.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT password.id AS password_id, password.local_user_id AS password_local_user_id, | 28 | -| password.password AS password_password | | -| FROM password | | -| WHERE %(param_1)s = password.local_user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 12 | +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 5 | | endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | | endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | | endpoint_enabled, endpoint.extra AS endpoint_extra | | @@ -439,180 +201,44 @@ Server create request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 25 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user | | -| WHERE user.id = %(param_1)s) AS anon_1 INNER JOIN federated_user ON anon_1.user_id = | | -| federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 33 | -| service.extra AS service_extra, endpoint_1.id AS endpoint_1_id, endpoint_1.legacy_endpoint_id AS | | -| endpoint_1_legacy_endpoint_id, endpoint_1.interface AS endpoint_1_interface, endpoint_1.region_id AS | | -| endpoint_1_region_id, endpoint_1.service_id AS endpoint_1_service_id, endpoint_1.url AS | | -| endpoint_1_url, endpoint_1.enabled AS endpoint_1_enabled, endpoint_1.extra AS endpoint_1_extra | | -| FROM service LEFT OUTER JOIN endpoint AS endpoint_1 ON service.id = endpoint_1.service_id | | -| WHERE service.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ **Keystone DB queries with multi JOINs inside** -+------------------------------------------------------------------------------------------------------+--------------------+ -| **DB query** | **Time spent, ms** | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 9 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 9 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 5 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 6 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ ++--------------+--------------------+ +| **DB query** | **Time spent, ms** | ++--------------+--------------------+ **Keystone cached methods stats** -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| **Cached operations** | **args** | **kwargs** | **Times used** | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| _get_token | , | | 6 | -| | 'c1613dbf55ac423fb5c0ae8833e4884b' | | | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| _get_revoke_tree | , | | 66 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_project | , u'a7e61cc2e4634e8c9179257bdb8dcb47' | | 81 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_role | , u'141ab89993aa4d9fa645540dad1a3e2a' | | 27 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| _get_token | , | | 16 | -| | 'c1613dbf55ac423fb5c0ae8833e4884b' | | | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_domain | , u'default' | | 27 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| _get_token | , | | 5 | -| | 'c1613dbf55ac423fb5c0ae8833e4884b' | | | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_project | , u'a7e61cc2e4634e8c9179257bdb8dcb47' | | 6 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| _get_token | , | | 5 | -| | 'e9d77e583df44f7ebf84de745882e722' | | | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| _get_token | , | | 1 | -| | 'e9d77e583df44f7ebf84de745882e722' | | | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| _validate_v3_token | , {u'tenant': {u'domain': {u'id': | | 6 | -| | u'default', u'name': u'Default'}, u'id': u'87dfc2348604428590293a3e6ec1c62c', u'name': u'service'}, | | | -| | 'user_id': u'bb2514e4901d4682a8ebdedc13b8df83', 'expires': datetime.datetime(2016, 3, 29, 21, 26, | | | -| | 35), u'token_data': {u'token': {u'methods': [u'password'], u'roles': [{u'id': | | | -| | u'bab7f95dd62e46d68d6aecca9d18c4fd', u'name': u'service'}], u'expires_at': | | | -| | u'2016-03-29T21:26:35.010736Z', u'project': {u'domain': {u'id': u'default', u'name': u'Default'}, | | | -| | u'id': u'87dfc2348604428590293a3e6ec1c62c', u'name': u'service'}, u'catalog': [{u'endpoints': | | | -| | [{u'url': u'http://10.0.2.15:8776/v1/87dfc2348604428590293a3e6ec1c62c', u'interface': u'public', | | | -| | u'region': u'RegionOne', u'region_id': u'RegionOne', u'id': u'56205029c0a1478cbd8c4fc075315746'}, | | | -| | {u'url': u'http://10.0.2.15:8776/v1/87dfc2348604428590293a3e6ec1c62c', u'interface': u'admin', | | | -| | u'region': u'RegionOne', u'region_id': u'RegionOne', u'id': u'80de6747b5f14681a14ebcadcd1bfa3c'}, | | | -| | {u'url': u'http://10.0.2.15:8776/v1/87dfc2348604428590293a3e6ec1c62c', u'interface': u'internal', | | | -| | u'region': u'RegionOne', u'region_id': u'RegionOne', u'id': u'cba6f65a146b4973ace876b38b3480b9'}], | | | -| | u'type': u'volume', u'id': u'4ebaed7acb634017a6913526311d98e9', u'name': u'cinder'}, {u'endpoints': | | | -| | [{u'url': u'http://10.0.2.15:8776/v2/87dfc2348604428590293a3e6ec1c62c', u'interface': u'public', | | | -| | u'region': u'RegionOne', u'region_id': u'RegionOne', u'id': u'853305e44f5d4236b081a46e1e9b83af'}, | | | -| | {u'url': u'http://10.0.2.15:8776/v2/87dfc2348604428590293a3e6ec1c62c', u'interface': u'admin', | | | -| | u'region': u'RegionOne', u'region_id': u'RegionOne', u'id': u'89fdb562d14e40e4ba01268ae79910c5'}, | | | -| | {u'url': u'http://10.0.2.15:8776/v2/87dfc2348604428590293a3e6ec1c62c', u'interface': u'internal', | | | -| | u'region': u'RegionOne', u'region_id': u'RegionOne', u'id': u'b27a27f962e44d69939a30ef3cb6b2a3'}], | | | -| | u'type': u'volumev2', u'id': u'4f1650e001bb49b1a995755b88812998', u'name': u'cinderv2'}, | | | -| | {u'endpoints': [{u'url': u'http://10.0.2.15:8774/v2/87dfc2348604428590293a3e6ec1c62c', u'interface': | | | -| | u'admin', u'region': u'RegionOne', u'region_id': u'RegionOne', u'id': | | | -| | u'605496c2c01549e09e3c3511d5d050c1'}, {u'url': | | | -| | u'http://10.0.2.15:8774/v2/87dfc2348604428590293a3e6ec1c62c', u'interface': u'internal', u'region': | | | -| | u'RegionOne', u'region_id': u'RegionOne', u'id': u'a359f6aaf80643dd90fd5598f4cf0614'}, {u'url': | | | -| | u'http://10.0.2.15:8774/v2/87dfc2348604428590293a3e6ec1c62c', u'interface': u'public', u'region': | | | -| | u'RegionOne', u'region_id': u'RegionOne', u'id': u'f552054d19ca4cfe981ca02ff4558595'}], u'type': | | | -| | u'compute_legacy', u'id': u'4fb62933662a4017ac2f8beb7239a7c5', u'name': u'nova_legacy'}, | | | -| | {u'endpoints': [{u'url': u'http://10.0.2.15:8774/v2.1/87dfc2348604428590293a3e6ec1c62c', | | | -| | u'interface': u'internal', u'region': u'RegionOne', u'region_id': u'RegionOne', u'id': | | | -| | u'21878ce4ff7d48a89f621675d89a0994'}, {u'url': | | | -| | u'http://10.0.2.15:8774/v2.1/87dfc2348604428590293a3e6ec1c62c', u'interface': u'public', u'region': | | | -| | u'RegionOne', u'region_id': u'RegionOne', u'id': u'645b3a5079ae4ed68c61890659b0af96'}, {u'url': | | | -| | u'http://10.0.2.15:8774/v2.1/87dfc2348604428590293a3e6ec1c62c', u'interface': u'admin', u'region': | | | -| | u'RegionOne', u'region_id': u'RegionOne', u'id': u'b20358e6f0574a4abbefd7d9fea42d27'}], u'type': | | | -| | u'compute', u'id': u'8c04194b75b846618a673b59a0a4bf3c', u'name': u'nova'}, {u'endpoints': [{u'url': | | | -| | u'http://10.0.2.15:9292', u'interface': u'admin', u'region': u'RegionOne', u'region_id': | | | -| | u'RegionOne', u'id': u'3f4f3c602e0c4b94aaef28d42f3fe1bc'}, {u'url': u'http://10.0.2.15:9292', | | | -| | u'interface': u'internal', u'region': u'RegionOne', u'region_id': u'RegionOne', u'id': | | | -| | u'610b5a73106b412a992ad987782b7cd0'}, {u'url': u'http://10.0.2.15:9292', u'interface': u'public', | | | -| | u'region': u'RegionOne', u'region_id': u'RegionOne', u'id': u'f39bfd16ee7e43b7bc82fd78eb20a9c4'}], | | | -| | u'type': u'image', u'id': u'a4d2690b9ad34c4ea4e2eafe12125496', u'name': u'glance'}, {u'endpoints': | | | -| | [{u'url': u'http://10.0.2.15:8777', u'interface': u'public', u'region': u'RegionOne', u'region_id': | | | -| | u'RegionOne', u'id': u'40f20ee570504169902791abf506a49c'}, {u'url': u'http://10.0.2.15:8777', | | | -| | u'interface': u'internal', u'region': u'RegionOne', u'region_id': u'RegionOne', u'id': | | | -| | u'49b66e88bdda4e648855b2c0e8b55227'}, {u'url': u'http://10.0.2.15:8777', u'interface': u'admin', | | | -| | u'region': u'RegionOne', u'region_id': u'RegionOne', u'id': u'5f2f9f9a58064a04a449a170d176bce4'}], | | | -| | u'type': u'metering', u'id': u'ac90aa229d7048adb18219359fbbd437', u'name': u'ceilometer'}, | | | -| | {u'endpoints': [{u'url': u'http://10.0.2.15:9696/', u'interface': u'public', u'region': | | | -| | u'RegionOne', u'region_id': u'RegionOne', u'id': u'12dbac67dfd842929a73cc2da267a585'}, {u'url': | | | -| | u'http://10.0.2.15:9696/', u'interface': u'admin', u'region': u'RegionOne', u'region_id': | | | -| | u'RegionOne', u'id': u'b6f1f5f59a3649658a1c02c7c9be0e6b'}, {u'url': u'http://10.0.2.15:9696/', | | | -| | u'interface': u'internal', u'region': u'RegionOne', u'region_id': u'RegionOne', u'id': | | | -| | u'd4f8473ff3bc4ba6ad754c5b9ea6fe11'}], u'type': u'network', u'id': | | | -| | u'e86e501b77344f35a4877f48f570f86a', u'name': u'neutron'}, {u'endpoints': [{u'url': | | | -| | u'http://10.0.2.15:5000/v2.0', u'interface': u'public', u'region': u'RegionOne', u'region_id': | | | -| | u'RegionOne', u'id': u'32eedd79382e42a693ae4ae36104e257'}, {u'url': u'http://10.0.2.15:5000/v2.0', | | | -| | u'interface': u'internal', u'region': u'RegionOne', u'region_id': u'RegionOne', u'id': | | | -| | u'6c623f00190348aea66a29e86e980449'}, {u'url': u'http://10.0.2.15:35357/v2.0', u'interface': | | | -| | u'admin', u'region': u'RegionOne', u'region_id': u'RegionOne', u'id': | | | -| | u'c4f3f0e4a90542e6af07805a9570a91d'}], u'type': u'identity', u'id': | | | -| | u'f0dc03fd61e84f338abd2b40371a767a', u'name': u'keystone'}], u'user': {u'domain': {u'id': | | | -| | u'default', u'name': u'Default'}, u'id': u'bb2514e4901d4682a8ebdedc13b8df83', u'name': u'neutron'}, | | | -| | u'audit_ids': [u'1rgcCrgFQJqlLiCoeniFUQ'], u'issued_at': u'2016-03-29T20:26:35.010777Z'}}, u'user': | | | -| | {u'domain': {u'id': u'default', u'name': u'Default'}, u'id': u'bb2514e4901d4682a8ebdedc13b8df83', | | | -| | u'name': u'neutron'}, u'key': u'e9d77e583df44f7ebf84de745882e722', u'token_version': u'v3.0', 'id': | | | -| | u'e9d77e583df44f7ebf84de745882e722', 'trust_id': None, u'metadata': {u'roles': | | | -| | [u'bab7f95dd62e46d68d6aecca9d18c4fd']}} | | | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_role | , u'141ab89993aa4d9fa645540dad1a3e2a' | | 2 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_domain | , u'default' | | 2 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| **Cache** | **Cached operations** | **args** | **kwargs** | **Times used** | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_roles_for_user_and_project | , u'31d31420a3e349268cd7875aa1825663', | | 3 | +| | | u'f1a058ad00364df29cb819a5478e8884' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project | , u'f1a058ad00364df29cb819a5478e8884' | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_domain | , u'default' | | 4 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project | , u'f1a058ad00364df29cb819a5478e8884' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_role | , u'8272f6d8be65464e94d3be094cf35e4c' | | 3 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | _get_token | , | | 2 | +| | | '428e793676454d04a9ea2fec753c35b1' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_user | , u'31d31420a3e349268cd7875aa1825663' | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_user_by_name | , u'admin', 'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_domain | , u'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project_by_name | , u'admin', 'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | _get_revoke_tree | , | | 3 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | _validate_token | , '18d75ba7355d4a8684a5d5658d005f1f' | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ Service list request stats ~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -622,25 +248,25 @@ Service list request stats +--------------------------------------------------------------+-----------+ | **Metric** | **Value** | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries count | 84 | +| Total (*) Keystone DB queries count | 20 | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries time spent, ms | 619 | +| Total (*) Keystone DB queries time spent, ms | 131 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries count | 31 | +| Infrastructure (SELECT 1) Keystone DB queries count | 7 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 207 | +| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 48 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries count | 53 | +| Real Keystone DB queries count | 13 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries time spent, ms | 412 | +| Real Keystone DB queries time spent, ms | 83 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries count | 51 | +| SELECT Keystone DB queries count | 11 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries time spent, ms | 393 | +| SELECT Keystone DB queries time spent, ms | 74 | +--------------------------------------------------------------+-----------+ | INSERT Keystone DB queries count | 2 | +--------------------------------------------------------------+-----------+ -| INSERT Keystone DB queries time spent, ms | 19 | +| INSERT Keystone DB queries time spent, ms | 9 | +--------------------------------------------------------------+-----------+ .. note:: (*) OSprofiler uses specific SQLalchemy cursor events to track @@ -648,188 +274,12 @@ Service list request stats DB requests "SELECT 1", processed by SQLalchemy itself to make sure that connection to the database is still in place. - -.. note:: Number of DB queries is much bigger for this request comparing with - similar Liberty environment. The difference is in - `keystone_authtoken` middleware (and, more specifically, its cache) - usage. Although all environments that took part in the research had - exactly the same `keystone_authtoken` middleware configuration, - described in the test plan and containing `memcache_servers` - parameter set up, Mitaka environments profiling shows that all - OpenStack services that used `keystone_authtoken` middleware did not - use the external Memcached cached token copy. All of them used - Keystone API every time REST API request was coming to the OpenStack - services. This behaviour needs to be investigated separately. - - **Keystone DB queries outliers** +------------------------------------------------------------------------------------------------------+--------------------+ | **DB query** | **Time spent, ms** | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 9 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 40 | -| service.extra AS service_extra | | -| FROM service | | -| WHERE service.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT implied_role.prior_role_id AS implied_role_prior_role_id, implied_role.implied_role_id AS | 10 | -| implied_role_implied_role_id | | -| FROM implied_role | | -| WHERE implied_role.prior_role_id = %(prior_role_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 5 | -| user.default_project_id AS user_default_project_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 9 | -| user.default_project_id AS user_default_project_id | | -| FROM user | | -| WHERE user.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT role.id AS role_id, role.name AS role_name, role.domain_id AS role_domain_id, role.extra AS | 16 | -| role_extra | | -| FROM role | | -| WHERE role.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 15 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user | | -| WHERE user.id = %(param_1)s) AS anon_1 INNER JOIN federated_user ON anon_1.user_id = | | -| federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT password.id AS password_id, password.local_user_id AS password_local_user_id, | 11 | -| password.password AS password_password | | -| FROM password | | -| WHERE %(param_1)s = password.local_user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 24 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = false | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 12 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT `group`.id AS group_id, `group`.name AS group_name, `group`.domain_id AS group_domain_id, | 18 | -| `group`.description AS group_description, `group`.extra AS group_extra | | -| FROM `group` INNER JOIN user_group_membership ON `group`.id = user_group_membership.group_id | | -| WHERE user_group_membership.user_id = %(user_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS | 12 | -| project_domain_id, project.description AS project_description, project.enabled AS project_enabled, | | -| project.extra AS project_extra, project.parent_id AS project_parent_id, project.is_domain AS | | -| project_is_domain | | -| FROM project | | -| WHERE project.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS | 9 | -| project_domain_id, project.description AS project_description, project.enabled AS project_enabled, | | -| project.extra AS project_extra, project.parent_id AS project_parent_id, project.is_domain AS | | -| project_is_domain | | -| FROM project | | -| WHERE project.name = %(name_1)s AND project.domain_id = %(domain_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT token.id AS token_id, token.expires AS token_expires, token.extra AS token_extra, token.valid | 27 | -| AS token_valid, token.user_id AS token_user_id, token.trust_id AS token_trust_id | | -| FROM token | | -| WHERE token.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT revocation_event.id AS revocation_event_id, revocation_event.domain_id AS | 4 | -| revocation_event_domain_id, revocation_event.project_id AS revocation_event_project_id, | | -| revocation_event.user_id AS revocation_event_user_id, revocation_event.role_id AS | | -| revocation_event_role_id, revocation_event.trust_id AS revocation_event_trust_id, | | -| revocation_event.consumer_id AS revocation_event_consumer_id, revocation_event.access_token_id AS | | -| revocation_event_access_token_id, revocation_event.issued_before AS revocation_event_issued_before, | | -| revocation_event.expires_at AS revocation_event_expires_at, revocation_event.revoked_at AS | | -| revocation_event_revoked_at, revocation_event.audit_id AS revocation_event_audit_id, | | -| revocation_event.audit_chain_id AS revocation_event_audit_chain_id | | -| FROM revocation_event ORDER BY revocation_event.revoked_at | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 33 | -| service.extra AS service_extra, endpoint_1.id AS endpoint_1_id, endpoint_1.legacy_endpoint_id AS | | -| endpoint_1_legacy_endpoint_id, endpoint_1.interface AS endpoint_1_interface, endpoint_1.region_id AS | | -| endpoint_1_region_id, endpoint_1.service_id AS endpoint_1_service_id, endpoint_1.url AS | | -| endpoint_1_url, endpoint_1.enabled AS endpoint_1_enabled, endpoint_1.extra AS endpoint_1_extra | | -| FROM service LEFT OUTER JOIN endpoint AS endpoint_1 ON service.id = endpoint_1.service_id | | -| WHERE service.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service_provider.id AS service_provider_id, service_provider.enabled AS | 22 | -| service_provider_enabled, service_provider.description AS service_provider_description, | | -| service_provider.auth_url AS service_provider_auth_url, service_provider.sp_url AS | | -| service_provider_sp_url, service_provider.relay_state_prefix AS service_provider_relay_state_prefix | | -| FROM service_provider | | -| WHERE service_provider.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 9 | -| service.extra AS service_extra | | -| FROM service | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 5 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 9 | +| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 11 | | local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | | anon_1_user_id | | | FROM (SELECT user.id AS user_id | | @@ -839,15 +289,14 @@ Service list request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 12 | -| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | -| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | -| endpoint_enabled, endpoint.extra AS endpoint_extra | | -| FROM endpoint | | +| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 17 | +| user.default_project_id AS user_default_project_id | | +| FROM user | | +| WHERE user.id = %(param_1)s | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 19 | +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 24 | | endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | | endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | | endpoint_enabled, endpoint.extra AS endpoint_extra, service_1.id AS service_1_id, service_1.type AS | | @@ -857,66 +306,51 @@ Service list request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 4 | +| service.extra AS service_extra | | +| FROM service | | +| WHERE service.id = %(param_1)s | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 5 | +| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | +| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | +| endpoint_enabled, endpoint.extra AS endpoint_extra | | +| FROM endpoint | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ **Keystone DB queries with multi JOINs inside** -+------------------------------------------------------------------------------------------------------+--------------------+ -| **DB query** | **Time spent, ms** | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 5 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 9 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 6 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 12 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ ++--------------+--------------------+ +| **DB query** | **Time spent, ms** | ++--------------+--------------------+ **Keystone cached methods stats** -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| **Cached operations** | **args** | **kwargs** | **Times used** | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_project | , u'a465ecd9f7004c38b30792a07b363b2c' | | 6 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_role | , u'31c541984f48440ab8753d11d84e58ca' | | 2 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_domain | , u'default' | | 2 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| **Cache** | **Cached operations** | **args** | **kwargs** | **Times used** | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_roles_for_user_and_project | , u'31d31420a3e349268cd7875aa1825663', | | 2 | +| | | u'f1a058ad00364df29cb819a5478e8884' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | _get_token | , | | 1 | +| | | '6982459b63f647ed9210bd0ce32f8e95' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project | , u'f1a058ad00364df29cb819a5478e8884' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_role | , u'8272f6d8be65464e94d3be094cf35e4c' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_user_by_name | , u'admin', 'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project_by_name | , u'admin', 'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | _get_revoke_tree | , | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_domain | , u'default' | | 4 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ Token issue request stats ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -926,21 +360,21 @@ Token issue request stats +--------------------------------------------------------------+-----------+ | **Metric** | **Value** | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries count | 39 | +| Total (*) Keystone DB queries count | 9 | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries time spent, ms | 448 | +| Total (*) Keystone DB queries time spent, ms | 114 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries count | 14 | +| Infrastructure (SELECT 1) Keystone DB queries count | 3 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 179 | +| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 41 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries count | 25 | +| Real Keystone DB queries count | 6 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries time spent, ms | 269 | +| Real Keystone DB queries time spent, ms | 73 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries count | 24 | +| SELECT Keystone DB queries count | 5 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries time spent, ms | 263 | +| SELECT Keystone DB queries time spent, ms | 67 | +--------------------------------------------------------------+-----------+ | INSERT Keystone DB queries count | 1 | +--------------------------------------------------------------+-----------+ @@ -952,188 +386,12 @@ Token issue request stats DB requests "SELECT 1", processed by SQLalchemy itself to make sure that connection to the database is still in place. - -.. note:: Number of DB queries is much bigger for this request comparing with - similar Liberty environment. The difference is in - `keystone_authtoken` middleware (and, more specifically, its cache) - usage. Although all environments that took part in the research had - exactly the same `keystone_authtoken` middleware configuration, - described in the test plan and containing `memcache_servers` - parameter set up, Mitaka environments profiling shows that all - OpenStack services that used `keystone_authtoken` middleware did not - use the external Memcached cached token copy. All of them used - Keystone API every time REST API request was coming to the OpenStack - services. This behaviour needs to be investigated separately. - - **Keystone DB queries outliers** +------------------------------------------------------------------------------------------------------+--------------------+ | **DB query** | **Time spent, ms** | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 20 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 40 | -| service.extra AS service_extra | | -| FROM service | | -| WHERE service.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT implied_role.prior_role_id AS implied_role_prior_role_id, implied_role.implied_role_id AS | 15 | -| implied_role_implied_role_id | | -| FROM implied_role | | -| WHERE implied_role.prior_role_id = %(prior_role_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 5 | -| user.default_project_id AS user_default_project_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 9 | -| user.default_project_id AS user_default_project_id | | -| FROM user | | -| WHERE user.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT role.id AS role_id, role.name AS role_name, role.domain_id AS role_domain_id, role.extra AS | 16 | -| role_extra | | -| FROM role | | -| WHERE role.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 15 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user | | -| WHERE user.id = %(param_1)s) AS anon_1 INNER JOIN federated_user ON anon_1.user_id = | | -| federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT password.id AS password_id, password.local_user_id AS password_local_user_id, | 11 | -| password.password AS password_password | | -| FROM password | | -| WHERE %(param_1)s = password.local_user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 24 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = false | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 12 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT `group`.id AS group_id, `group`.name AS group_name, `group`.domain_id AS group_domain_id, | 18 | -| `group`.description AS group_description, `group`.extra AS group_extra | | -| FROM `group` INNER JOIN user_group_membership ON `group`.id = user_group_membership.group_id | | -| WHERE user_group_membership.user_id = %(user_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS | 27 | -| project_domain_id, project.description AS project_description, project.enabled AS project_enabled, | | -| project.extra AS project_extra, project.parent_id AS project_parent_id, project.is_domain AS | | -| project_is_domain | | -| FROM project | | -| WHERE project.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS | 9 | -| project_domain_id, project.description AS project_description, project.enabled AS project_enabled, | | -| project.extra AS project_extra, project.parent_id AS project_parent_id, project.is_domain AS | | -| project_is_domain | | -| FROM project | | -| WHERE project.name = %(name_1)s AND project.domain_id = %(domain_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT token.id AS token_id, token.expires AS token_expires, token.extra AS token_extra, token.valid | 27 | -| AS token_valid, token.user_id AS token_user_id, token.trust_id AS token_trust_id | | -| FROM token | | -| WHERE token.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT revocation_event.id AS revocation_event_id, revocation_event.domain_id AS | 4 | -| revocation_event_domain_id, revocation_event.project_id AS revocation_event_project_id, | | -| revocation_event.user_id AS revocation_event_user_id, revocation_event.role_id AS | | -| revocation_event_role_id, revocation_event.trust_id AS revocation_event_trust_id, | | -| revocation_event.consumer_id AS revocation_event_consumer_id, revocation_event.access_token_id AS | | -| revocation_event_access_token_id, revocation_event.issued_before AS revocation_event_issued_before, | | -| revocation_event.expires_at AS revocation_event_expires_at, revocation_event.revoked_at AS | | -| revocation_event_revoked_at, revocation_event.audit_id AS revocation_event_audit_id, | | -| revocation_event.audit_chain_id AS revocation_event_audit_chain_id | | -| FROM revocation_event ORDER BY revocation_event.revoked_at | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 33 | -| service.extra AS service_extra, endpoint_1.id AS endpoint_1_id, endpoint_1.legacy_endpoint_id AS | | -| endpoint_1_legacy_endpoint_id, endpoint_1.interface AS endpoint_1_interface, endpoint_1.region_id AS | | -| endpoint_1_region_id, endpoint_1.service_id AS endpoint_1_service_id, endpoint_1.url AS | | -| endpoint_1_url, endpoint_1.enabled AS endpoint_1_enabled, endpoint_1.extra AS endpoint_1_extra | | -| FROM service LEFT OUTER JOIN endpoint AS endpoint_1 ON service.id = endpoint_1.service_id | | -| WHERE service.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service_provider.id AS service_provider_id, service_provider.enabled AS | 22 | -| service_provider_enabled, service_provider.description AS service_provider_description, | | -| service_provider.auth_url AS service_provider_auth_url, service_provider.sp_url AS | | -| service_provider_sp_url, service_provider.relay_state_prefix AS service_provider_relay_state_prefix | | -| FROM service_provider | | -| WHERE service_provider.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 9 | -| service.extra AS service_extra | | -| FROM service | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 5 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 9 | +| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 11 | | local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | | anon_1_user_id | | | FROM (SELECT user.id AS user_id | | @@ -1143,15 +401,14 @@ Token issue request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 12 | -| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | -| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | -| endpoint_enabled, endpoint.extra AS endpoint_extra | | -| FROM endpoint | | +| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 20 | +| user.default_project_id AS user_default_project_id | | +| FROM user | | +| WHERE user.id = %(param_1)s | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 19 | +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 24 | | endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | | endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | | endpoint_enabled, endpoint.extra AS endpoint_extra, service_1.id AS service_1_id, service_1.type AS | | @@ -1161,45 +418,46 @@ Token issue request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 4 | +| service.extra AS service_extra | | +| FROM service | | +| WHERE service.id = %(param_1)s | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 5 | +| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | +| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | +| endpoint_enabled, endpoint.extra AS endpoint_extra | | +| FROM endpoint | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ **Keystone DB queries with multi JOINs inside** -+------------------------------------------------------------------------------------------------------+--------------------+ -| **DB query** | **Time spent, ms** | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 5 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 10 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ ++--------------+--------------------+ +| **DB query** | **Time spent, ms** | ++--------------+--------------------+ **Keystone cached methods stats** -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| **Cached operations** | **args** | **kwargs** | **Times used** | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_project | , u'a465ecd9f7004c38b30792a07b363b2c' | | 3 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_role | , u'31c541984f48440ab8753d11d84e58ca' | | 1 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_domain | , u'default' | | 1 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| **Cache** | **Cached operations** | **args** | **kwargs** | **Times used** | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_roles_for_user_and_project | , u'31d31420a3e349268cd7875aa1825663', | | 1 | +| | | u'f1a058ad00364df29cb819a5478e8884' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project | , u'f1a058ad00364df29cb819a5478e8884' | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_role | , u'8272f6d8be65464e94d3be094cf35e4c' | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_user_by_name | , u'admin', 'default' | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project_by_name | , u'admin', 'default' | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_domain | , u'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ User list request stats ~~~~~~~~~~~~~~~~~~~~~~~ @@ -1209,25 +467,25 @@ User list request stats +--------------------------------------------------------------+-----------+ | **Metric** | **Value** | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries count | 94 | +| Total (*) Keystone DB queries count | 30 | +--------------------------------------------------------------+-----------+ -| Total (*) Keystone DB queries time spent, ms | 700 | +| Total (*) Keystone DB queries time spent, ms | 178 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries count | 31 | +| Infrastructure (SELECT 1) Keystone DB queries count | 7 | +--------------------------------------------------------------+-----------+ -| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 240 | +| Infrastructure (SELECT 1) Keystone DB queries time spent, ms | 33 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries count | 63 | +| Real Keystone DB queries count | 23 | +--------------------------------------------------------------+-----------+ -| Real Keystone DB queries time spent, ms | 460 | +| Real Keystone DB queries time spent, ms | 145 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries count | 61 | +| SELECT Keystone DB queries count | 21 | +--------------------------------------------------------------+-----------+ -| SELECT Keystone DB queries time spent, ms | 450 | +| SELECT Keystone DB queries time spent, ms | 136 | +--------------------------------------------------------------+-----------+ | INSERT Keystone DB queries count | 2 | +--------------------------------------------------------------+-----------+ -| INSERT Keystone DB queries time spent, ms | 10 | +| INSERT Keystone DB queries time spent, ms | 9 | +--------------------------------------------------------------+-----------+ .. note:: (*) OSprofiler uses specific SQLalchemy cursor events to track @@ -1235,52 +493,49 @@ User list request stats DB requests "SELECT 1", processed by SQLalchemy itself to make sure that connection to the database is still in place. - -.. note:: Number of DB queries is much bigger for this request comparing with - similar Liberty environment. The difference is in - `keystone_authtoken` middleware (and, more specifically, its cache) - usage. Although all environments that took part in the research had - exactly the same `keystone_authtoken` middleware configuration, - described in the test plan and containing `memcache_servers` - parameter set up, Mitaka environments profiling shows that all - OpenStack services that used `keystone_authtoken` middleware did not - use the external Memcached cached token copy. All of them used - Keystone API every time REST API request was coming to the OpenStack - services. This behaviour needs to be investigated separately. - - **Keystone DB queries outliers** +------------------------------------------------------------------------------------------------------+--------------------+ | **DB query** | **Time spent, ms** | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 20 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = true | | +| SELECT password.id AS password_id, password.local_user_id AS password_local_user_id, | 9 | +| password.password AS password_password | | +| FROM password | | +| WHERE %(param_1)s = password.local_user_id | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 40 | +| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 4 | +| user.default_project_id AS user_default_project_id | | +| FROM user LEFT OUTER JOIN local_user ON user.id = local_user.user_id | | +| WHERE local_user.domain_id = %(domain_id_1)s | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 4 | | service.extra AS service_extra | | | FROM service | | | WHERE service.id = %(param_1)s | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT implied_role.prior_role_id AS implied_role_prior_role_id, implied_role.implied_role_id AS | 15 | -| implied_role_implied_role_id | | -| FROM implied_role | | -| WHERE implied_role.prior_role_id = %(prior_role_id_1)s | | +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 12 | +| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | +| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | +| endpoint_enabled, endpoint.extra AS endpoint_extra, service_1.id AS service_1_id, service_1.type AS | | +| service_1_type, service_1.enabled AS service_1_enabled, service_1.extra AS service_1_extra | | +| FROM endpoint LEFT OUTER JOIN service AS service_1 ON service_1.id = endpoint.service_id | | +| WHERE endpoint.enabled = true | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, | 5 | -| user.default_project_id AS user_default_project_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s | | +| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 10 | +| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | +| anon_1_user_id | | +| FROM (SELECT user.id AS user_id | | +| FROM user | | +| WHERE user.id = %(param_1)s) AS anon_1 INNER JOIN local_user ON anon_1.user_id = local_user.user_id | | +| ORDER BY anon_1.user_id | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ @@ -1291,14 +546,25 @@ User list request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT role.id AS role_id, role.name AS role_name, role.domain_id AS role_domain_id, role.extra AS | 16 | -| role_extra | | -| FROM role | | -| WHERE role.id = %(param_1)s | | +| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 9 | +| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | +| anon_1_user_id | | +| FROM (SELECT user.id AS user_id | | +| FROM user LEFT OUTER JOIN local_user ON user.id = local_user.user_id | | +| WHERE local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN local_user ON anon_1.user_id = | | +| local_user.user_id ORDER BY anon_1.user_id | | | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 16 | +| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 5 | +| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | +| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | +| endpoint_enabled, endpoint.extra AS endpoint_extra | | +| FROM endpoint | | +| | | +| | | | ++------------------------------------------------------------------------------------------------------+--------------------+ +| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 4 | | federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | | federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | | federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | @@ -1309,209 +575,33 @@ User list request stats | | | | | | | +------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 11 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user LEFT OUTER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN federated_user ON anon_1.user_id | | -| = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT password.id AS password_id, password.local_user_id AS password_local_user_id, | 11 | -| password.password AS password_password | | -| FROM password | | -| WHERE %(param_1)s = password.local_user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, | 24 | -| assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, | | -| assignment.inherited AS assignment_inherited | | -| FROM assignment | | -| WHERE assignment.actor_id IN (%(actor_id_1)s) AND assignment.target_id IN (%(target_id_1)s) AND | | -| assignment.type IN (%(type_1)s) AND assignment.inherited = false | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 12 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT `group`.id AS group_id, `group`.name AS group_name, `group`.domain_id AS group_domain_id, | 18 | -| `group`.description AS group_description, `group`.extra AS group_extra | | -| FROM `group` INNER JOIN user_group_membership ON `group`.id = user_group_membership.group_id | | -| WHERE user_group_membership.user_id = %(user_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS | 13 | -| project_domain_id, project.description AS project_description, project.enabled AS project_enabled, | | -| project.extra AS project_extra, project.parent_id AS project_parent_id, project.is_domain AS | | -| project_is_domain | | -| FROM project | | -| WHERE project.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS | 9 | -| project_domain_id, project.description AS project_description, project.enabled AS project_enabled, | | -| project.extra AS project_extra, project.parent_id AS project_parent_id, project.is_domain AS | | -| project_is_domain | | -| FROM project | | -| WHERE project.name = %(name_1)s AND project.domain_id = %(domain_id_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT token.id AS token_id, token.expires AS token_expires, token.extra AS token_extra, token.valid | 27 | -| AS token_valid, token.user_id AS token_user_id, token.trust_id AS token_trust_id | | -| FROM token | | -| WHERE token.id = %(param_1)s | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT revocation_event.id AS revocation_event_id, revocation_event.domain_id AS | 4 | -| revocation_event_domain_id, revocation_event.project_id AS revocation_event_project_id, | | -| revocation_event.user_id AS revocation_event_user_id, revocation_event.role_id AS | | -| revocation_event_role_id, revocation_event.trust_id AS revocation_event_trust_id, | | -| revocation_event.consumer_id AS revocation_event_consumer_id, revocation_event.access_token_id AS | | -| revocation_event_access_token_id, revocation_event.issued_before AS revocation_event_issued_before, | | -| revocation_event.expires_at AS revocation_event_expires_at, revocation_event.revoked_at AS | | -| revocation_event_revoked_at, revocation_event.audit_id AS revocation_event_audit_id, | | -| revocation_event.audit_chain_id AS revocation_event_audit_chain_id | | -| FROM revocation_event ORDER BY revocation_event.revoked_at | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 33 | -| service.extra AS service_extra, endpoint_1.id AS endpoint_1_id, endpoint_1.legacy_endpoint_id AS | | -| endpoint_1_legacy_endpoint_id, endpoint_1.interface AS endpoint_1_interface, endpoint_1.region_id AS | | -| endpoint_1_region_id, endpoint_1.service_id AS endpoint_1_service_id, endpoint_1.url AS | | -| endpoint_1_url, endpoint_1.enabled AS endpoint_1_enabled, endpoint_1.extra AS endpoint_1_extra | | -| FROM service LEFT OUTER JOIN endpoint AS endpoint_1 ON service.id = endpoint_1.service_id | | -| WHERE service.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service_provider.id AS service_provider_id, service_provider.enabled AS | 22 | -| service_provider_enabled, service_provider.description AS service_provider_description, | | -| service_provider.auth_url AS service_provider_auth_url, service_provider.sp_url AS | | -| service_provider_sp_url, service_provider.relay_state_prefix AS service_provider_relay_state_prefix | | -| FROM service_provider | | -| WHERE service_provider.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, | 9 | -| service.extra AS service_extra | | -| FROM service | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 5 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 18 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user | | -| WHERE user.id = %(param_1)s) AS anon_1 INNER JOIN local_user ON anon_1.user_id = local_user.user_id | | -| ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 12 | -| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | -| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | -| endpoint_enabled, endpoint.extra AS endpoint_extra | | -| FROM endpoint | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, | 19 | -| endpoint.interface AS endpoint_interface, endpoint.region_id AS endpoint_region_id, | | -| endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS | | -| endpoint_enabled, endpoint.extra AS endpoint_extra, service_1.id AS service_1_id, service_1.type AS | | -| service_1_type, service_1.enabled AS service_1_enabled, service_1.extra AS service_1_extra | | -| FROM endpoint LEFT OUTER JOIN service AS service_1 ON service_1.id = endpoint.service_id | | -| WHERE endpoint.enabled = true | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ **Keystone DB queries with multi JOINs inside** -+------------------------------------------------------------------------------------------------------+--------------------+ -| **DB query** | **Time spent, ms** | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 6 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 7 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT federated_user.id AS federated_user_id, federated_user.user_id AS federated_user_user_id, | 9 | -| federated_user.idp_id AS federated_user_idp_id, federated_user.protocol_id AS | | -| federated_user_protocol_id, federated_user.unique_id AS federated_user_unique_id, | | -| federated_user.display_name AS federated_user_display_name, anon_1.user_id AS anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| federated_user ON anon_1.user_id = federated_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ -| SELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, | 5 | -| local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, anon_1.user_id AS | | -| anon_1_user_id | | -| FROM (SELECT user.id AS user_id | | -| FROM user INNER JOIN local_user ON user.id = local_user.user_id | | -| WHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s) AS anon_1 INNER JOIN | | -| local_user ON anon_1.user_id = local_user.user_id ORDER BY anon_1.user_id | | -| | | -| | | | -+------------------------------------------------------------------------------------------------------+--------------------+ ++--------------+--------------------+ +| **DB query** | **Time spent, ms** | ++--------------+--------------------+ **Keystone cached methods stats** -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| **Cached operations** | **args** | **kwargs** | **Times used** | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_project | , u'a465ecd9f7004c38b30792a07b363b2c' | | 6 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_role | , u'31c541984f48440ab8753d11d84e58ca' | | 2 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ -| get_domain | , u'default' | | 2 | -+-----------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| **Cache** | **Cached operations** | **args** | **kwargs** | **Times used** | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_roles_for_user_and_project | , u'31d31420a3e349268cd7875aa1825663', | | 2 | +| | | u'f1a058ad00364df29cb819a5478e8884' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_domain | , u'default' | | 4 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project | , u'f1a058ad00364df29cb819a5478e8884' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_role | , u'8272f6d8be65464e94d3be094cf35e4c' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_user_by_name | , u'admin', 'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | get_project_by_name | , u'admin', 'default' | | 2 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | _get_revoke_tree | , | | 1 | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ +| Memcache | _get_token | , | | 1 | +| | | '6c4b9b6e838c42ff83e6dcfed52f596e' | | | ++-----------+--------------------------------+------------------------------------------------------------------------------------------------------+------------+----------------+ diff --git a/doc/source/test_results/keystone/parse_json_res.py b/doc/source/test_results/keystone/parse_json_res.py index 9cc9a76..821514c 100644 --- a/doc/source/test_results/keystone/parse_json_res.py +++ b/doc/source/test_results/keystone/parse_json_res.py @@ -49,7 +49,8 @@ NODES_TEMPLATE = { } } }, - "Cached operations": {}, + "Memcache cached operations": {}, + "Context cached operations": {}, "Cached time spent": collections.OrderedDict(), } @@ -61,7 +62,7 @@ def define_node(node): if node["info"]["project"] != "keystone": return - if node["info"]["name"] not in ["db", "cache"]: + if node["info"]["name"] not in ["db", "cache", "memoize"]: return time_spent = node["info"]["finished"] - node["info"]["started"] @@ -70,21 +71,43 @@ def define_node(node): process_db_calls(node, time_spent) elif node["info"]["name"] == "cache": if not node["children"]: + process_cache_calls(node, time_spent, "cache") + else: + for child in node["children"]: + define_node(child) + elif node["info"]["name"] == "memoize": + if not node["children"] or len(node["children"]) == 1: process_cache_calls(node, time_spent) else: for child in node["children"]: define_node(child) -def process_cache_calls(node, time_spent): - cache_info = node["info"]["meta.raw_payload.cache-start"][ +def process_cache_calls(node, time_spent, prefix="memoize"): + cache_info = node["info"]["meta.raw_payload.%s-start" % prefix][ "info"]["fn_info"] - if not NODES["Cached operations"].get(cache_info): - NODES["Cached operations"][cache_info] = 0 - NODES["Cached operations"][cache_info] += 1 - if not NODES["Cached time spent"].get(time_spent): - NODES["Cached time spent"][time_spent] = [] - NODES["Cached time spent"][time_spent].append(cache_info) + + def add_info(pref): + if not NODES["%s cached operations" % pref].get(cache_info): + NODES["%s cached operations" % pref][cache_info] = 0 + NODES["%s cached operations" % pref][cache_info] += 1 + if not NODES["Cached time spent"].get(time_spent): + NODES["Cached time spent"][time_spent] = [] + NODES["Cached time spent"][time_spent].append(cache_info) + + # Liberty env, all cache is Memcached Cache + if not node["children"]: + add_info("Memcache") + return + + # Mitaka env with nested 2-layer cache + ctxt_cache_info = node["children"][0] + if ctxt_cache_info["children"]: + memcache_cache_info = ctxt_cache_info["children"][0] + if not memcache_cache_info["children"]: + add_info("Memcache") + else: + add_info("Context") def process_db_calls(node, time_spent): @@ -154,7 +177,7 @@ def process_selects(statement, time_spent): def define_nodes(data): for child in data["children"]: - if not child["children"]: + if not child["children"] or child["info"]["name"] == "memoize": define_node(child) else: define_nodes(child) @@ -255,24 +278,33 @@ def prepare_tables(nodes): "time_spent"][ts]: multi_join_queries.add_row([db_query_tmpl % query, ts]) - # prepare table with cache info - cache_table = prettytable.PrettyTable(["**Cached operations**", + # prepare table(s) with cache info + cache_table = prettytable.PrettyTable(["**Cache**", + "**Cached operations**", "**args**", "**kwargs**", "**Times used**"]) - cache_table.align["**Cached operations**"] = "l" - cache_table.align["**args**"] = "l" - cache_table.align["**kwargs**"] = "l" - cache_table.align["**Times used**"] = "l" - cache_table.max_width = 100 - cache_table.header = True - cache_table.hrules = prettytable.ALL - for operation, times in nodes["Cached operations"].iteritems(): - operation = operation[1:-1].split(", ") - cache_table.add_row([operation[0][1:-1], - ", ".join(operation[1:-1])[1:-1], - operation[-1][1:-1], times]) + for cache in ["Memcache cached operations", "Context cached operations"]: + name_map = { + "Memcache cached operations": "Memcache", + "Context cached operations": "Local context", + } + cache_table.align["**Cache**"] = "l" + cache_table.align["**Cached operations**"] = "l" + cache_table.align["**args**"] = "l" + cache_table.align["**kwargs**"] = "l" + cache_table.align["**Times used**"] = "l" + cache_table.max_width = 100 + cache_table.header = True + cache_table.hrules = prettytable.ALL + + for operation, times in nodes[cache].iteritems(): + operation = operation[1:-1].split(", ") + cache_table.add_row([name_map[cache], + operation[0][1:-1], + ", ".join(operation[1:-1])[1:-1], + operation[-1][1:-1], times]) return common_info_table, multi_join_queries, outliers_table, cache_table