StarlingX Distributed Cloud
Go to file
Gustavo Herzmann 2ac4be0d5a Add in-memory token caching for DC services
This commit introduces an in-memory, dictionary-based token caching
mechanism to reduce the number of token requests made to subclouds'
identity APIs.

The caching is implemented by subclassing the v3.Password
authentication class, which normally handles HTTP requests to the
identity API. The cache first checks if a valid, non-expired token
exists and returns it if found. If not, it proceeds with the actual
request and caches the new token for future use.

Tokens can be invalidated early when all fernet keys are rotated
(e.g., during the initial sync between subcloud and system controller).
The cache leverages Keystone's session reauthentication mechanism to
automatically invalidate cached tokens when necessary.

This commit also raises the open file descriptor limit for the DC
orchestrator service. With the use of sessions, TCP connections are
reused and are not closed immediately after each request.

Test Plan:
01. PASS - Deploy a subcloud and verify token caching behavior.
02. PASS - Deploy a subcloud with remote install, ensuring the token
    cache works.
03. PASS - Prestage a subcloud for install and software deployment,
    validating token caching during the process.
04. PASS - Run prestage orchestration and verify proper use of the
    token cache.
05. PASS - Manage a subcloud for the first time and verify that the
    initial sync functions as expected. Ensure fernet key rotation
    causes cached tokens to invalidate, and confirm reauthentication
    requests are made.
06. PASS - Unmanage a subcloud, rotate all fernet keys manually, then
    manage the subcloud again. Verify token invalidation and
    reauthentication function as expected.
07. PASS - Create a subcloud backup and ensure no token cache issues
    arise.
08. PASS - Restore a subcloud from backup and verify proper
    functionality of the token cache.
09. PASS - Deploy an N-1 subcloud and validate token caching for this
    subcloud.
10. PASS - Verify that audits correctly identify an N-1 subcloud
    without the USM patch as missing the USM service.
11. PASS - Apply the USM patch to the N-1 subcloud and verify that
    the audit detects the USM service and prestage orchestration for
    software deployment functions correctly.
12. PASS - Test DC orchestration audit and sync by creating a new
    OpenStack user, and verify the user is replicated to the subcloud.
13. PASS - Apply a patch to subclouds using software deployment
    orchestration, verifying token cache performance.
14. PASS - Test dcmanager API commands that send requests to
    subclouds (e.g., 'dcmanager subcloud show <subcloud> --details'),
    ensuring token cache is used.
15. PASS - Conduct a soak test of all DC services to verify token
    expiration, renewal, and cache behavior over extended use.
16. PASS - Monitor TCP connections to ensure they are properly
    closed after each use, preventing lingering open connections during
    token caching or HTTP request handling.
17. PASS - Run end-to-end geo-redundancy operation and verify that it
    completes successfully.
18. PASS - Run kube rootca update orchestration and verify that it
    completes successfully.
19. PASS - Verify that the number of POST token requests made by the DC
    audit to the subcloud per hour is equal to the number of DC audit
    workers on the system controller.
20. PASS - Monitor the number of open file descriptors to ensure it
    does not reach the new limit while executing a DC kube rootca
    update strategy with the maximum number of supported subclouds.
    Additionally, verify that all sessions are closed after the
    strategy is complete.

Closes-Bug: 2084490

Change-Id: Ie3c17f58c09ae08df8cd9f0c92f50ab0c556c263
Signed-off-by: Gustavo Herzmann <gustavo.herzmann@windriver.com>
2024-10-22 16:37:06 -03:00
api-ref/source Cleanup legacy load and upgrade functionality 2024-09-03 14:37:49 -03:00
distributedcloud Add in-memory token caching for DC services 2024-10-22 16:37:06 -03:00
doc Fix distributedcloud openstack-tox-docs 2023-08-29 11:45:59 -03:00
python/python3-redfish/debian python3-redfish: fix URL spelling 2023-11-14 17:15:16 -05:00
releasenotes Switch to newer openstackdocstheme and reno versions 2020-06-04 14:18:06 +02:00
tools/gate Zuul/Tox: Install custom python3-keystoneauth1 2024-08-02 16:46:44 +00:00
.git-blame-ignore-revs Remove run_black.py script 2024-07-30 21:13:38 +00:00
.gitignore Zuul/Tox: Install custom python3-keystoneauth1 2024-08-02 16:46:44 +00:00
.gitreview OpenDev Migration Patch 2019-04-19 19:52:44 +00:00
.mailmap StarlingX open source release updates 2018-08-07 11:51:21 -04:00
.yamllint Zuul/Tox: Install custom python3-keystoneauth1 2024-08-02 16:46:44 +00:00
.zuul.yaml Zuul/Tox: Install custom python3-keystoneauth1 2024-08-02 16:46:44 +00:00
bindep.txt Fix openstack-tox jobs for DC repo 2022-11-21 16:58:49 +00:00
CONTRIBUTING.rst StarlingX open source release updates 2018-08-07 11:51:21 -04:00
CONTRIBUTORS.wrs StarlingX open source release updates 2018-08-07 11:51:21 -04:00
debian_build_layer.cfg Add debian_build_layer.cfg file 2021-10-05 14:07:18 -04:00
debian_dev_docker_images.inc Port stx-keystone-api-proxy image to debian 2022-11-21 17:46:41 +00:00
debian_dev_wheels.inc Port stx-keystone-api-proxy image to debian 2022-11-21 17:46:41 +00:00
debian_iso_image.inc Add build package for dcagent 2024-07-24 19:37:25 +00:00
debian_pkg_dirs Decontainerizing rvmc.py and Modular Integration in DC Repo 2023-11-10 18:18:23 -05:00
debian_stable_docker_images.inc Port stx-keystone-api-proxy image to debian 2022-11-21 17:46:41 +00:00
debian_stable_wheels.inc Port stx-keystone-api-proxy image to debian 2022-11-21 17:46:41 +00:00
HACKING.rst Update setup.cfg to align with StarlingX project 2019-01-07 16:34:24 +00:00
README_DC Update location of openrc in readme 2019-03-15 10:56:03 -05:00
tox.ini Create stx-distcloud-tox-black job 2024-06-25 14:49:42 -03:00