766f01c604
This commit includes a documentation about how the agent operates and how to configure OpenStack to operate properly with it. Story: 2009299 Task: 43806 Signed-off-by: Heitor Matsui <HeitorVieira.Matsui@windriver.com> Change-Id: I5969b1a415daf97dbad02b09d8429f2d54efd533
95 lines
3.7 KiB
ReStructuredText
95 lines
3.7 KiB
ReStructuredText
|
|
=========
|
|
utilities
|
|
=========
|
|
|
|
This file serves as documentation for the components and features included
|
|
on the utilities repository.
|
|
|
|
PCI IRQ Affinity Agent
|
|
----------------------
|
|
|
|
While in OpenStack it is possible to enable instances to use PCI devices,
|
|
the interrupts generated by these devices may be handled by host CPUs that
|
|
are unrelated to the instance, and this can lead to a performance that is
|
|
lower than it could be if the device interrupts were handled by the instance
|
|
CPUs.
|
|
|
|
The agent only acts over instances with dedicated vCPUs. For instances using
|
|
shared vCPUs no action will be taken by the agent.
|
|
|
|
The expected outcome from the agent operation is achieving a higher
|
|
performance by assigning the instances core to handle the interrupts from
|
|
PCI devices used by these instances and avoid interrupts consuming excessive
|
|
cycles from the platform cores.
|
|
|
|
Agent operation
|
|
~~~~~~~~~~~~~~~
|
|
|
|
The agent operates by listening to RabbitMQ notifications from Nova. When
|
|
an instance is created or moved to the host, the agent checks for an specific
|
|
flavor spec (detailed below) and if it does then it queries libvirt to map
|
|
the instance vCPUs into pCPUs from the host.
|
|
|
|
Once the agent has the CPU mapping, it determines the IRQ for each PCI
|
|
device used by the instance, and then it loops over all PCI devices and
|
|
determines which host NUMA node is associated with the device, the pCPUs
|
|
that are associated with the NUMA node and finally set the CPU affinity
|
|
for the IRQs of the PCI device based on the pCPU list.
|
|
|
|
There is also a periodic audit that runs every minute and loops over the
|
|
existing IRQs, so that if there are new IRQs that weren't mapped before
|
|
the agent maps them, and if there are PCI devices that aren't associated
|
|
to an instance that they were before, their IRQ affinity is reset to the
|
|
default value.
|
|
|
|
Flavor spec
|
|
~~~~~~~~~~~
|
|
|
|
The PCI IRQ Affinity Agent uses a specific flavor spec for PCI interrupt
|
|
affining, that is used to determine which vCPUs assigned to the instance
|
|
must handle the interrupts from the PCI devices:
|
|
|
|
- ``hw:pci_irq_affinity_mask=<vcpus_cpulist>``
|
|
|
|
Where ``vcpus_cpulist`` can assume a comma-separated list of values that
|
|
can be expressed as:
|
|
|
|
- ``int``: the vCPU expressed by ``int`` will be assigned to handle the
|
|
interruptions from the PCI devices
|
|
- ``int1-int2``: the vCPUs between ``int1`` and ``int2`` (inclusive) will
|
|
be used to handle the interruptions from the PCI devices
|
|
- ``^int``: the vCPU expressed by ``int`` will not be assigned to handle the
|
|
interruptions from the PCI devices and shall be used to exclude a vCPU
|
|
that was included in a previous range
|
|
|
|
**NOTE**: ``int`` must be a value between ``0`` and ``flavor.vcpus - 1``
|
|
|
|
Example: ``hw_pci_irq_affinity_mask=1-4,^3,6`` means that vCPUs with indexes
|
|
``1,2,4 and 6`` from the vCPU list that Nova allocates to the instance will
|
|
be assigned to handle interruptions from the PCI devices.
|
|
|
|
Limitations
|
|
~~~~~~~~~~~
|
|
|
|
- No CPU affining is performed for instances using shared CPUs (i.e.,
|
|
when using flavor spec ``hw:cpu_policy=shared``)
|
|
- No CPU affining will be performed when invalid ranges are specified on
|
|
the flavor spec, the agent instead will log error messages indicating
|
|
the problem
|
|
|
|
Agent packaging
|
|
~~~~~~~~~~~~~~~
|
|
|
|
The agent code resides on the ``starlingx/utilities`` repo, along with the
|
|
spec and docker_image files that are used to build a CentOS image with the
|
|
agent wheel installed on it.
|
|
|
|
The agent is deployed by Armada along with the other OpenStack helm charts;
|
|
refer to `PCI IRQ Affinity Agent helm chart`_ on
|
|
``starlingx/openstack-armada-app`` repository.
|
|
|
|
.. _`PCI IRQ Affinity Agent helm chart`: https://opendev.org/starlingx
|
|
/openstack-armada-app/src/branch/master/stx-openstack-helm
|
|
/stx-openstack-helm/helm-charts/pci-irq-affinity-agent
|