Current versions of mypy run with no-implicit-optional
by default. This change gets Nova's mypy test environment
to pass again.
Change-Id: Ie50c8d364ad9c339355cc138b560ec4df14fe307
Fixed various small issues from the already merged (or being merged)
patches.
The logic behind the dropped FIXME and empty condition are handled in
the update_allocations() calls of the translator already.
The removal of excutils.save_and_reraise_exception from the report
client is safe as self._clear_provider_cache_for_tree does not raise.
blueprint: pci-device-tracking-in-placement
Change-Id: If87dedc6a14f7b116c4238e7534b67574428c01c
PCI devices which are allocated to instances can be removed from the
[pci]device_spec configuration or can be removed from the hypervisor
directly. The existing PciTracker code handle this cases by keeping the
PciDevice in the nova DB exists and allocated and issue a warning in the
logs during the compute service startup that nova is in an inconsistent
state. Similar behavior is now added to the PCI placement tracking code
so the PCI inventories and allocations in placement is kept in such
situation.
There is one case where we cannot simply accept the PCI device
reconfiguration by keeping the existing allocations and applying the new
config. It is when a PF that is configured and allocated is removed and
VFs from this PF is now configured in the [pci]device_spec. And vice
versa when VFs are removed and its parent PF is configured. In this case
keeping the existing inventory and allocations and adding the new inventory
to placement would result in placement model where a single PCI device
would provide both PF and VF inventories. This dependent device
configuration is not supported as it could lead to double consumption.
In such situation the compute service will refuse to start.
blueprint: pci-device-tracking-in-placement
Change-Id: Id130893de650cc2d38953cea7cf9f53af71ced93
During a normal update_available_resources run if the local provider
tree caches is invalid (i.e. due to the scheduler made an allocation
bumping the generation of the RPs) and the virt driver try to update the
inventory of an RP based on the cache Placement will report conflict,
the report client will invalidate the caches and the retry decorator
on ResourceTracker._update_to_placement will re-drive the top of the
fresh RP data.
However the same thing can happen during reshape as well but the retry
mechanism is missing in that code path so the stale caches can cause
reshape failures.
This patch adds specific error handling in the reshape code path to
implement the same retry mechanism as exists for inventory update.
blueprint: pci-device-tracking-in-placement
Change-Id: Ieb954a04e6aba827611765f7f401124a1fe298f3
We have many places where we implement singleton behavior for the
placement client. This unifies them into a single place and
implementation. Not only does this DRY things up, but may cause us
to initialize it fewer times and also allows for emitting a common
set of error messages about expected failures for better
troubleshooting.
Change-Id: Iab8a791f64323f996e1d6e6d5a7e7a7c34eb4fb3
Related-Bug: #1846820
A follow on patch will use this code to enforce the limits, this patch
provides integration with oslo.limit and a new internal nova API that is
able to enforce those limits.
The first part is providing a callback for oslo.limit to be able to count
the resources being used. We only count resources grouped by project_id.
For counting servers, we make use of the instance mappings list in the
api database, just as the existing quota code does. While we do check to
ensure the queued for delete migration has been completed, we simply
error out if that is not the case, rather than attempting to fallback to
any other counting system. We hope one day we can count this in
placement using consumer records, or similar.
For counting all other resource usage, they must refer to some usage
relating to a resource class being consumed in placement. This is similar
to how the count with placement variant of the existing placement code
works today. This is not restricted to RAM and VCPU, it is open to any
resource class that is known to placement.
The second part is the enforcement method, that keeps a similar
signature to the existing enforce_num_instnaces call that is use to
check quotas using the legacy quota system.
From the flavor we extract the current resource usage. This is
considered the simplest first step that helps us deliver Ironic limits
alongside all the existing RAM and VCPU limits. At a later date, we
would ideally get passed a more complete view of what resources are
being requested from placement.
NOTE: given the instance object doesn't exist when enforce is called, we
can't just pass the instance into here.
A [workarounds] option is also available for operators who need the
legacy quota usage behavior where VCPU = VCPU + PCPU.
blueprint unified-limits-nova
Change-Id: I272b59b7bc8975bfd602640789f80d2d5f7ee698
When a compute node goes down and all instances on the compute node
are evacuated, allocation records about these instance are still left
in the source compute node until nova-compute service is again started
on the node. However if a compute node is completely broken, it is not
possible to start the service again.
In this situation deleting nova-compute service for the compute node
doesn't delete its resource provider record, and even if a user tries
to delete the resource provider, the delete request is rejected because
allocations are still left on that node.
This change ensures that remaining allocations left by successful
evacuations are cleared when deleting a nova-compute service, to avoid
any resource provider record left even if a compute node can't be
recovered. Migration records are still left in 'done' status to trigger
clean-up tasks in case the compute node is recovered later.
Closes-Bug: #1829479
Change-Id: I3ce6f6275bfe09d43718c3a491b3991a804027bd
This adds a force kwarg to delete_allocation_for_instance which
defaults to True because that was found to be the most common use case
by a significant margin during implementation of this patch.
In most cases, this method is called when we want to delete the
allocations because they should be gone, e.g. server delete, failed
build, or shelve offload. The alternative in these cases is the caller
could trap the conflict error and retry but we might as well just force
the delete in that case (it's cleaner).
When force=True, it will DELETE the consumer allocations rather than
GET and PUT with an empty allocations dict and the consumer generation
which can result in a 409 conflict from Placement. For example, bug
1836754 shows that in one tempest test that creates a server and then
immediately deletes it, we can hit a very tight window where the method
GETs the allocations and before it PUTs the empty allocations to remove
them, something changes which results in a conflict and the server
delete fails with a 409 error.
It's worth noting that delete_allocation_for_instance used to just
DELETE the allocations before Stein [1] when we started taking consumer
generations into account. There was also a related mailing list thread
[2].
Closes-Bug: #1836754
[1] I77f34788dd7ab8fdf60d668a4f76452e03cf9888
[2] http://lists.openstack.org/pipermail/openstack-dev/2018-August/133374.html
Change-Id: Ife3c7a5a95c5d707983ab33fd2fbfc1cfb72f676
To implement the usage of same_subtree query parameter in the
allocation candidate request first the minimum requires placement
microversion needs to be bumped from 1.35 to 1.36. This patch makes such
bump and update the related nova upgrade check. Later patches will
modify the query generation to include the same_subtree param to the
request.
Change-Id: I5bfec9b9ec49e60c454d71f6fc645038504ef9ef
blueprint: qos-minimum-guaranteed-packet-rate
There is a race condition in nova-compute with the ironic virt driver
as nodes get rebalanced. It can lead to compute nodes being removed in
the DB and not repopulated. Ultimately this prevents these nodes from
being scheduled to.
The issue being addressed here is that if a compute node is deleted by a
host which thinks it is an orphan, then the resource provider for that
node might also be deleted. The compute host that owns the node might
not recreate the resource provider if it exists in the provider tree
cache.
This change fixes the issue by clearing resource providers from the
provider tree cache for which a compute node entry does not exist. Then,
when the available resource for the node is updated, the resource
providers are not found in the cache and get recreated in placement.
Change-Id: Ia53ff43e6964963cdf295604ba0fb7171389606e
Related-Bug: #1853009
Related-Bug: #1841481
There are already type hints in this file so now we enable checking
those hints. To make mypy happy some fields needed an explicit type
declaration.
Change-Id: Id1df976f067777d4115e3cb2c72d54764bb31c6a
This patch fixes minor comments from
Id4684093e8bdf3b61667490443e3d2f6ed65f4b3
Change-Id: I6e70e24d837b6f194972386a32ad757ce25dcee0
Blueprint: support-interface-attach-with-qos-ports
The SchedulerReportClient add_resources_to_instance_allocation and
remove_resources_from_instance_allocation are symmetric in nature. But
the remove_resources_from_instance_allocation took a differently
structured input than add_resources_to_instance_allocation. The
structure of the input of add_resources_to_instance_allocation is better
as it is the canonical form of the placement allocation request. This
patch therefore changes the input structure of
remove_resources_from_instance_allocation to match with
add_resources_to_instance_allocation.
Change-Id: I5ef4ef2aa8d10538323af0f6b831ba605aeeb383
blueprint: support-interface-attach-with-qos-ports
This patch add a new function to the SchedulerReportClient that allows
extending an existing instance allocation with extra resources.
Change-Id: I1bdc5e4971204fbfbf0dfcd232cabf2cfe02a966
Blueprint: support-interface-attach-with-qos-ports
The commit fd351903a1 removed the last
real user of get_allocations_for_consumer_by_provider() so this function
is now removed.
Change-Id: I7b883348bb610a435ee1b79e921da0e698037534
Replace six.text_type with str.
A subsequent patch will replace other six.text_type.
Change-Id: I23bb9e539d08f5c6202909054c2dd49b6c7a7a0e
Implements: blueprint six-removal
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
The COMPUTE_STATUS_DISABLED trait is supposed to be added to the compute
RP when the compute service is disabled, and the trait is supposed to be
removed when the service is enabled again. However adding and removing
traits is prone to generation conflict in placement. The original
implementation of blueprint pre-filter-disabled-computes noticed this
and prints a detailed warning message while the API operation succeeds.
We can ignore the conflict this way because the periodic
update_available_resource() call will re-sync the traits later.
Still this gives human noticeable time window where the trait and the
service state are not in sync.
Setting the compute service disable is the smaller problem as the
scheduler still uses the ComputeFilter that filters the computes based
on the service api. So during the enable->disable race window we only
lose scheduling performance as the placement filter is inefficient.
In case of setting the compute service to enabled the race is more
visible as the placement pre_filter will filter out the compute that
is enable by the admin until the re-sync happens. If the conflict would
only happen due to high load on the given compute the such delay could
be explained by the load itself. However conflict can happen simply due
to a new instance boot on the compute.
Fortunately the solution is easy and cheap. The current service state
handler code path has already queried placement about the existing traits
on the compute RP and therefore it receives the current RP generation as
well. But it does not really use this information but instead rely on
the potentially stale provide_tree cache.
This patch uses the much fresher RP generation known by the service state
handling code instead of the potentially stale provider_tree cache.
The change in the notification test is also due to the fixed behavior.
The test disables the compute. Until now this caused that the
FilterScheduler detected that there is no valid host. Now it is already
detect by the scheduler manager based on the empty placement response.
This causes now that that the FilterScheduler is not called and
therefore the select_destination.start notification is not sent.
Closes-Bug: #1886418
Change-Id: Ib3c455bf21f33923bb82e3f5c53035f6722480d3
When delete nova service by "nova service-delete <id>", this will delete the
compute node's resource provider in Placement. But if there are child resource
provider, Placement will throw exception(CannotDeleteParentResourceProvider)
and don't delete the resource provider. When we add the host service again,
Placement cannot insert new compute node resource provider, and we cannot use
the compute node any more. So we should delete sub resource provider in tree
when delete resource provider.
Modify unit test.
Change-Id: Ide8732be6c047ad1b141b89df676783b2fa2f25a
Closes-Bug: #1872385
Placement microversion 1.35 gives us the root_required queryparam to GET
/allocation_candidates, allowing us to filter out candidates where the
*root* provider has/lacks certain traits, independent of traits
specified in any of the individual request groups.
Use it.
And add affordance for specifying such traits to the RequestSpec.
Which allows us to fix up the couple of request filters that were
hacking traits into the RequestSpec.flavor.
Change-Id: I44f02044ce178e84c23d178e5a23a3aa1208e502
This commit cuts us over to using placement microversion 1.34 for GET
/allocation_candidates, thereby supporting string request group suffixes
(added in 1.33) when specified in flavor extra_specs.
The mappings (added in 1.34) are not used in code yet, but a future
patch will tie the group suffixes to the RequestGroup.requester_id so
that it can be correlated after GET /a_c. This will allow us to get rid
of map_requested_resources_to_providers, which was a hack to bridge the
gap until we had mappings from placement.
Change-Id: I52499ff6639c1a5815a8557b22dd33106dcc386b
The report client _ensure_resource_provider() call gathers RPs
associated to the compute RP via aggregates but only those that has the
MISC_SHARES_VIA_AGGREGATE trait, i.e. only the sharing RPs.
Change-Id: I3eaff83c9ea2e71843a73fb1d00e3b802c573e36
The only user of put_allocations of the report client is
heal_allocations. It is easy to remove the @safe_connect decorator
from put_allocations as heal_allocations already handles
PlacementAPIConnectFailure exception.
This patch changes heal_allocations to raise
PlacementAPIConnectFailure instead of relying on @safe_connect.
Change-Id: I6b3f61525883a019703cf564e693516b3b61febc
This patch enables counting both VCPU and PCPU when
`CONF.quota.count_usage_from_placement` enabled. Also add functional
test ensure the quota usage as expected.
Change-Id: I8377cbd1fb601d91deb7bb667a1ad5271c667399
Part of blueprint cpu-resources
This patch modifies Nova objects that will allow Destination object to
store list of forbidden aggregates that placement should ignore in
'GET /allocation_candidates' API at microversion 1.32, including the
code to generate the placement querystring syntax from them.
Change-Id: Ic2bcee40b41c97170a8603b27b935113f0633de7
Implements: blueprint placement-req-filter-forbidden-aggregates
This appears to have been unused since commit 73d7ef428. Remove both the
function and its tests.
Change-Id: I680f0bab766693b8cc6f0f912f0b7987a9debec9
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This swaps the SchedulerReportClient's raw KSA Adapter for the placement
Proxy (which is an Adapter subclass) from an OpenStack SDK Connection.
Part of blueprint openstacksdk-in-nova
Change-Id: I1641cbe06b7fd3f773a0a9779c94f3efb0c8dac8
Two different error conditions in
SchedulerReportClient.delete_allocation_for_instance printed the same
warning log, which could be confusing. Reword one of them to
disambiguate.
Change-Id: Id1c5493a7270f2d0594e918df8704f671416a221
This commit removes the @safe_connect decorator from
SchedulerReportClient._delete_provider and makes its callers deal with
resulting keystoneauth1.exceptions.ClientExceptionZ sanely:
- ServiceController.delete (via delete_resource_provider) logs an error
and continues (backward compatible behavior, best effort to delete
each provider).
- ComputeManager.update_available_resource (ditto) ditto (ditto).
- SchedulerReportClient.update_from_provider_tree raises the exception
through, per the contract described in the catch_all internal helper.
Change-Id: I8403a841f21a624a546ae5f26bb9ba19318ece6a
Release 3.15.0 of keystoneauth1 introduced the ability to pass
X-Openstack-Request-Id to request methods (get/put/etc) via a
global_request_id kwarg rather than having to put it in a headers dict.
This commit bumps the minimum ksa level to 3.15.0 and takes advantage of
the new kwarg to replace explicit header construction in
SchedulerReportClient (Placement) and neutronv2/api methods.
Also normalizes the way param lists were being passed from
SchedulerReportClient's REST primitives (get/put/post/delete) into the
Adapter equivalents. There was no reason for them to be different.
Change-Id: I2f6eb50f4cb428179ec788de8b7bd6ef9bbeeaf9
In the continuing saga to wipe @safe_connect from the annals of history,
and in preparation, for its use outside of SchedulerReportClient, this
commit does two things to _get_providers_in_tree:
- Removes @safe_connect from it. Callers now need to be aware that they
can get ClientExceptionZ from ksa. (The two existing callers were
vetted and needed no additional handling - it's way more appropriate
for them to raise ClientException than a mysterious NoneType error
somewhere down the line as they would have been doing previously.)
- Renames it to get_providers_in_tree.
Change-Id: I2b284d69d345d15287f04a7ca4cd422155768525
When no allocations exist for an instance the current heal code uses a
report client call that can only handle allocations from a single RP.
This call is now replaced with a more generic one so in a later patch
port allocations can be added to this code path too.
Related-Bug: #1819923
Change-Id: Ide343c1c922dac576b1944827dc24caefab59b74
This is used to optimize the SchedulerReportClient
delete_resource_provider method when deleting a compute
node and its related resource provider which happens
in both the API when a compute service is deleted and
in the compute service in the update_available_resource
method for orphan nodes (so the new InstanceList query
method needs to be remotable).
The actual DB query stuff in here is tested in the
nova.tests.functional.wsgi.test_services code.
Change-Id: Id033e8f95f853ccfc34bd350be6df11c1bfb1b7d
Related-Bug: #1811726
If cascade=True, we're getting all of the instances on the
compute node just to use the uuid, which will by default
join on the info_cache and security_groups for the instances.
This is a simple optimization to avoid those unnecessary joins.
A TODO is left to further optimize this with a new InstanceList
query method to just get the instance uuids on a given host/node
combo, but that requires an RPC change which we can't backport.
Change-Id: Ie121210456a240c257979d3269db115ddae2d23b
Related-Bug: #1811726
Related-Bug: #1756179
This makes _get_provider_by_name public to expose access for
callers outside of SchedulerReportClient. The @safe_connect
decorator is removed and the get() is wrapped in a try/except
to handle KSA ClientException errors and raise
PlacementAPIConnectFailure.
The two existing calls for adding/removing providers to/from
aggregates are updated to use the public method so they don't
need to individually do the safe_connect handling.
Change-Id: I504c374d3863a2a956d5c0156a43be2d2a2bc712
This counts instance mappings for counting quota usage for instances
and adds calls to placement for counting quota usage for cores and ram.
During an upgrade, if any un-migrated instance mappings are found (with
NULL user_id or NULL queued_for_delete fields), we will fall back to
the legacy counting method.
Counting quota usage from placement is opt-in via the
[quota]count_usage_from_placement configuration option because:
* Though beneficial for multi-cell deployments to be resilient to
down cells, the vast majority of deployments are single cell and
will not be able to realize a down cells resiliency benefit and may
prefer to keep legacy quota usage counting.
* Usage for resizes will reflect resources being held on both the
source and destination until the resize is confirmed or reverted.
Operators may not want to enable counting from placement based on
whether the behavior change is problematic for them.
* Placement does not yet support the ability to partition resource
providers from mulitple Nova deployments, so environments that are
sharing a single placement deployment would see usage that
aggregates all Nova deployments together. Such environments should
not enable counting from placement.
* Usage for unscheduled instances in ERROR state will not reflect
resource consumption for cores and ram because the instance has no
placement allocations.
* Usage for instances in SHELVED_OFFLOADED state will not reflect
resource consumption for cores and ram because the instance has no
placement allocations. Note that because of this, it will be possible for a
request to unshelve a server to be rejected if the user does not have
enough quota available to support the cores and ram needed by the server to
be unshelved.
Part of blueprint count-quota-usage-from-placement
Change-Id: Ie22b0acb5824a41da327abdcf9848d02fc9a92f5
This adds a method for requesting /usages from placement for the
purpose of counting quota usage for cores and ram. It is used in the
next patch in the series.
Part of blueprint count-quota-usage-from-placement
Change-Id: I35f98f88f8353602e1bfc135f35d1b7bc9ba42a4
This patch adds the translation of `RequestGroup.in_tree` to the
actual placement query and bumps microversion to enable it.
The release note for this change is added.
Change-Id: I8ec95d576417c32a57aa0298789dac6afb0cca02
Blueprint: use-placement-in-tree
Related-Bug: #1777591
Change a log level for overwriting allocation
to debug instead of warning
because the log message is output in the normal case
of reverting resize.
Change-Id: I4f2ee3f420b4c78385eeb56c7e8c343af894e75c
Closes-Bug: #1822917
Eric Harney