openstack/nova
Code Issues Proposed changes

Document using service user tokens for long running operations

Browse Source 
A recent thread in the mailing list [1] reminded me that we
don't have any documentation for the service user token feature
added back in ocata under blueprint use-service-tokens.

This change adds a troubleshooting entry for when using service
user tokens would be useful, and links to it from two known
trouble spots: live migration timeouts and creating images.

[1] http://lists.openstack.org/pipermail/openstack-discuss/2018-December/001130.html

Change-Id: I1dda889038ffe67d53ceb35049aa1f2a9da39ae8
Closes-Bug: #1809165
This commit is contained in:
Matt Riedemann 2018-12-19 15:31:16 -05:00
parent 86da9c72c1
commit 1a4cc6c96d
3 changed files with 63 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
doc/source/admin/live-migration-usage.rst
View File

@ -318,3 +318,7 @@ To make live-migration succeed, you have several options:
- Post-copy may lead to an increased page fault rate during migration,
which can slow the instance down.
If live migrations routinely timeout or fail during cleanup operations due
to the user token timing out, consider configuring nova to use
:ref:`service user tokens <user_token_timeout>`.

4
doc/source/admin/migrate-instance-with-snapshot.rst
View File

@ -65,6 +65,10 @@ Create a snapshot of the instance
$ openstack server image create --name myInstanceSnapshot myInstance
If snapshot operations routinely fail because the user token times out
while uploading a large disk image, consider configuring nova to use
:ref:`service user tokens <user_token_timeout>`.
#. Use the :command:`openstack image list` command to check the status
until the status is ``ACTIVE``:

55
doc/source/admin/support-compute.rst
View File

@ -449,3 +449,58 @@ Ensure the ``compute`` endpoint in the identity service catalog is pointing
at ``/v2.1`` instead of ``/v2``. The former route supports microversions,
while the latter route is considered the legacy v2.0 compatibility-mode
route which renders all requests as if they were made on the legacy v2.0 API.
.. _user_token_timeout:
User token times out during long-running operations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Problem
-------
Long-running operations such as live migration or snapshot can sometimes
overrun the expiry of the user token. In such cases, post operations such
as cleaning up after a live migration can fail when the nova-compute service
needs to cleanup resources in other services, such as in the block-storage
(cinder) or networking (neutron) services.
For example:
.. code-block:: console
2018-12-17 13:47:29.591 16987 WARNING nova.virt.libvirt.migration [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live migration not completed after 2400 sec
2018-12-17 13:47:30.097 16987 WARNING nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Migration operation was cancelled
2018-12-17 13:47:30.299 16987 ERROR nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live Migration failure: operation aborted: migration job: canceled by client: libvirtError: operation aborted: migration job: canceled by client
2018-12-17 13:47:30.685 16987 INFO nova.compute.manager [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Swapping old allocation on 3e32d595-bd1f-4136-a7f4-c6703d2fbe18 held by migration 17bec61d-544d-47e0-a1c1-37f9d7385286 for instance
2018-12-17 13:47:32.450 16987 ERROR nova.volume.cinder [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] Delete attachment failed for attachment 58997d5b-24f0-4073-819e-97916fb1ee19. Error: The request you have made requires authentication. (HTTP 401) Code: 401: Unauthorized: The request you have made requires authentication. (HTTP 401)
Solution
--------
Configure nova to use service user tokens to supplement the regular user token
used to initiate the operation. The identity service (keystone) will then
authenticate a request using the service user token if the user token has
already expired.
To use, create a service user in the identity service similar as you would when
creating the ``nova`` service user.
Then configure the :oslo.config:group:`service_user` section of the nova
configuration file, for example:
.. code-block:: ini
[service_user]
send_service_user_token = True
auth_type = password
project_domain_name = Default
project_name = service
user_domain_name = Default
password = secretservice
username = nova
auth_url = https://104.130.216.102/identity
...
And configure the other identity options as necessary for the service user,
much like you would configure nova to work with the image service (glance)
or networking service.