Document using service user tokens for long running operations
A recent thread in the mailing list [1] reminded me that we
don't have any documentation for the service user token feature
added back in ocata under blueprint use-service-tokens.
This change adds a troubleshooting entry for when using service
user tokens would be useful, and links to it from two known
trouble spots: live migration timeouts and creating images.
[1] http://lists.openstack.org/pipermail/openstack-discuss/2018-December/001130.html
Change-Id: I1dda889038ffe67d53ceb35049aa1f2a9da39ae8
Closes-Bug: #1809165
(cherry picked from commit 1a4cc6c96d)
This commit is contained in:
Matt Riedemann
parent
734df042a1
commit
3b18ab4b32
|
|
@ -312,3 +312,7 @@ To make live-migration succeed, you have several options:
|
|||
|
||||
- Post-copy may lead to an increased page fault rate during migration,
|
||||
which can slow the instance down.
|
||||
|
||||
If live migrations routinely timeout or fail during cleanup operations due
|
||||
to the user token timing out, consider configuring nova to use
|
||||
:ref:`service user tokens <user_token_timeout>`.
|
||||
|
|
|
|||
|
|
@ -65,6 +65,10 @@ Create a snapshot of the instance
|
|||
|
||||
$ openstack server image create --name myInstanceSnapshot myInstance
|
||||
|
||||
If snapshot operations routinely fail because the user token times out
|
||||
while uploading a large disk image, consider configuring nova to use
|
||||
:ref:`service user tokens <user_token_timeout>`.
|
||||
|
||||
#. Use the :command:`openstack image list` command to check the status
|
||||
until the status is ``ACTIVE``:
|
||||
|
||||
|
|
|
|||
|
|
@ -449,3 +449,58 @@ Ensure the ``compute`` endpoint in the identity service catalog is pointing
|
|||
at ``/v2.1`` instead of ``/v2``. The former route supports microversions,
|
||||
while the latter route is considered the legacy v2.0 compatibility-mode
|
||||
route which renders all requests as if they were made on the legacy v2.0 API.
|
||||
|
||||
.. _user_token_timeout:
|
||||
|
||||
User token times out during long-running operations
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Problem
|
||||
-------
|
||||
|
||||
Long-running operations such as live migration or snapshot can sometimes
|
||||
overrun the expiry of the user token. In such cases, post operations such
|
||||
as cleaning up after a live migration can fail when the nova-compute service
|
||||
needs to cleanup resources in other services, such as in the block-storage
|
||||
(cinder) or networking (neutron) services.
|
||||
|
||||
For example:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
2018-12-17 13:47:29.591 16987 WARNING nova.virt.libvirt.migration [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live migration not completed after 2400 sec
|
||||
2018-12-17 13:47:30.097 16987 WARNING nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Migration operation was cancelled
|
||||
2018-12-17 13:47:30.299 16987 ERROR nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live Migration failure: operation aborted: migration job: canceled by client: libvirtError: operation aborted: migration job: canceled by client
|
||||
2018-12-17 13:47:30.685 16987 INFO nova.compute.manager [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Swapping old allocation on 3e32d595-bd1f-4136-a7f4-c6703d2fbe18 held by migration 17bec61d-544d-47e0-a1c1-37f9d7385286 for instance
|
||||
2018-12-17 13:47:32.450 16987 ERROR nova.volume.cinder [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] Delete attachment failed for attachment 58997d5b-24f0-4073-819e-97916fb1ee19. Error: The request you have made requires authentication. (HTTP 401) Code: 401: Unauthorized: The request you have made requires authentication. (HTTP 401)
|
||||
|
||||
Solution
|
||||
--------
|
||||
|
||||
Configure nova to use service user tokens to supplement the regular user token
|
||||
used to initiate the operation. The identity service (keystone) will then
|
||||
authenticate a request using the service user token if the user token has
|
||||
already expired.
|
||||
|
||||
To use, create a service user in the identity service similar as you would when
|
||||
creating the ``nova`` service user.
|
||||
|
||||
Then configure the :oslo.config:group:`service_user` section of the nova
|
||||
configuration file, for example:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[service_user]
|
||||
send_service_user_token = True
|
||||
auth_type = password
|
||||
project_domain_name = Default
|
||||
project_name = service
|
||||
user_domain_name = Default
|
||||
password = secretservice
|
||||
username = nova
|
||||
auth_url = https://104.130.216.102/identity
|
||||
...
|
||||
|
||||
And configure the other identity options as necessary for the service user,
|
||||
much like you would configure nova to work with the image service (glance)
|
||||
or networking service.
|
||||
|
|
|
|||
Loading…
Reference in New Issue