From 1a4cc6c96dc9acd5e6bc467f85cb77dd9e32c8d4 Mon Sep 17 00:00:00 2001 From: Matt Riedemann Date: Wed, 19 Dec 2018 15:31:16 -0500 Subject: [PATCH] Document using service user tokens for long running operations A recent thread in the mailing list [1] reminded me that we don't have any documentation for the service user token feature added back in ocata under blueprint use-service-tokens. This change adds a troubleshooting entry for when using service user tokens would be useful, and links to it from two known trouble spots: live migration timeouts and creating images. [1] http://lists.openstack.org/pipermail/openstack-discuss/2018-December/001130.html Change-Id: I1dda889038ffe67d53ceb35049aa1f2a9da39ae8 Closes-Bug: #1809165 --- doc/source/admin/live-migration-usage.rst | 4 ++ .../admin/migrate-instance-with-snapshot.rst | 4 ++ doc/source/admin/support-compute.rst | 55 +++++++++++++++++++ 3 files changed, 63 insertions(+) diff --git a/doc/source/admin/live-migration-usage.rst b/doc/source/admin/live-migration-usage.rst index 9f8f7b5762aa..e1ebdb696e45 100644 --- a/doc/source/admin/live-migration-usage.rst +++ b/doc/source/admin/live-migration-usage.rst @@ -318,3 +318,7 @@ To make live-migration succeed, you have several options: - Post-copy may lead to an increased page fault rate during migration, which can slow the instance down. + +If live migrations routinely timeout or fail during cleanup operations due +to the user token timing out, consider configuring nova to use +:ref:`service user tokens `. diff --git a/doc/source/admin/migrate-instance-with-snapshot.rst b/doc/source/admin/migrate-instance-with-snapshot.rst index 06509003fc6f..65059679abb0 100644 --- a/doc/source/admin/migrate-instance-with-snapshot.rst +++ b/doc/source/admin/migrate-instance-with-snapshot.rst @@ -65,6 +65,10 @@ Create a snapshot of the instance $ openstack server image create --name myInstanceSnapshot myInstance + If snapshot operations routinely fail because the user token times out + while uploading a large disk image, consider configuring nova to use + :ref:`service user tokens `. + #. Use the :command:`openstack image list` command to check the status until the status is ``ACTIVE``: diff --git a/doc/source/admin/support-compute.rst b/doc/source/admin/support-compute.rst index 04b8eed98d4d..e4d82b89cfda 100644 --- a/doc/source/admin/support-compute.rst +++ b/doc/source/admin/support-compute.rst @@ -449,3 +449,58 @@ Ensure the ``compute`` endpoint in the identity service catalog is pointing at ``/v2.1`` instead of ``/v2``. The former route supports microversions, while the latter route is considered the legacy v2.0 compatibility-mode route which renders all requests as if they were made on the legacy v2.0 API. + +.. _user_token_timeout: + +User token times out during long-running operations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Problem +------- + +Long-running operations such as live migration or snapshot can sometimes +overrun the expiry of the user token. In such cases, post operations such +as cleaning up after a live migration can fail when the nova-compute service +needs to cleanup resources in other services, such as in the block-storage +(cinder) or networking (neutron) services. + +For example: + +.. code-block:: console + + 2018-12-17 13:47:29.591 16987 WARNING nova.virt.libvirt.migration [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live migration not completed after 2400 sec + 2018-12-17 13:47:30.097 16987 WARNING nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Migration operation was cancelled + 2018-12-17 13:47:30.299 16987 ERROR nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live Migration failure: operation aborted: migration job: canceled by client: libvirtError: operation aborted: migration job: canceled by client + 2018-12-17 13:47:30.685 16987 INFO nova.compute.manager [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Swapping old allocation on 3e32d595-bd1f-4136-a7f4-c6703d2fbe18 held by migration 17bec61d-544d-47e0-a1c1-37f9d7385286 for instance + 2018-12-17 13:47:32.450 16987 ERROR nova.volume.cinder [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] Delete attachment failed for attachment 58997d5b-24f0-4073-819e-97916fb1ee19. Error: The request you have made requires authentication. (HTTP 401) Code: 401: Unauthorized: The request you have made requires authentication. (HTTP 401) + +Solution +-------- + +Configure nova to use service user tokens to supplement the regular user token +used to initiate the operation. The identity service (keystone) will then +authenticate a request using the service user token if the user token has +already expired. + +To use, create a service user in the identity service similar as you would when +creating the ``nova`` service user. + +Then configure the :oslo.config:group:`service_user` section of the nova +configuration file, for example: + +.. code-block:: ini + + [service_user] + send_service_user_token = True + auth_type = password + project_domain_name = Default + project_name = service + user_domain_name = Default + password = secretservice + username = nova + auth_url = https://104.130.216.102/identity + ... + +And configure the other identity options as necessary for the service user, +much like you would configure nova to work with the image service (glance) +or networking service.