The NoDBTestCase base class does a lot of env setup that is not needed
for these tests. As we have issues with these tests (see bug 1823251)
this patch tries to decrease the complexity of them to help the
troubleshooting.
Change-Id: I46fafa470b9d378d96c0f364e9b487742bf58cb2
Related-Bug: #1823251
Seeing as these were missed for Rocky, we really ought to get them in
for Stein.
Change-Id: I29401773a0686e7be5d7d1cbb5ec82ffbb16fb4a
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Make use of the newer fixtures in oslo_db.sqlalchemy.test_fixtures
which is what was intended to supersede fixtures
in oslo_db.sqlalchemy.test_base.
Change-Id: Icb88c13336fffb499083197ed864b8e4e45dc241
Closes-Bug: #1797102
Add ``volume_type`` field to BlockDeviceMapping object to be used
in new instance block_device_mapping_v2.volume_type API extension.
Implements: blueprint boot-instance-specific-storage-backend
Change-Id: I66ff84585642ff8d40ecfefef3342349eb49f83d
This change adds a trusted_certs deferred-load column to
instance_extras, which stores a list of trusted x509 certificate
UUIDs for a given instance in the form of a JSON blob.
Change-Id: I3fd4e395b31ff1b69f35242d559f8caa17c05a6a
Implements: blueprint nova-validate-certificates
We need this for the subsequent request_filter, which will do queries
based only on metadata value.
Related to blueprint placement-req-filter
Change-Id: I1325a226823329384edd7197e5d5f2725dc16e8f
A database index is added on the ``(instance_uuid, updated_at)``
columns in the ``nova.instance_actions`` table to improve the
performance of filtering instance actions by the 'changes-since'
parameter.
Change-Id: Id79aa297c3248002a271e905dccd2c3ba27848c7
Implement: blueprint pagination-add-changes-since-for-instance-action-list
A database index will be added on migrations.updated_at to
improve the performance of filtering migrations by
‘changes-since’ parameter.
Change-Id: I6bf9e498597669c8641f78267d7da589d01ad786
Implement: blueprint add-pagination-and-change-since-for-migration-list
Later in the series, we add a query on the console_auth_tokens table
that filters on both token_hash and instance_uuid. This adds an index
for those columns.
Part of blueprint convert-consoles-to-objects
Change-Id: I0a4e7a59b9ac6480cd7e0df0a6b96c7e55dcc40d
Unfortunately this column was missed off when console_auth_tokens
was added to the database. This change adds it in.
The access_url is needed for a protocol check in the console proxies.
Console proxies run per cell and do not run on compute hosts and thus
do not have access to the access_url config options in the compute
hosts nova.conf files.
So, we need to store part of the access_url in the database. We cannot
store the entire access_url because it contains the unhashed token, but
we can keep the base part of the url and generate the full access_url
on demand. An access url base looks something like this:
http://127.0.0.1:6080/vnc_auto.html
and using it, we can generate the full access_url on demand for the
protocol check in the console proxies.
partially-implements: blueprint convert-consoles-to-objects
Change-Id: I0f672f5667d42b67d869ff9f467dbb64eb6c9ff9
This adds a uuid column to the block_device_mapping table,
and makes it semi-optional for graceful upgrade of existing rows
without a uuid.
Part of bp local-disk-serial-numbers
Co-Authored-By: Lee Yarwood <lyarwood@redhat.com>
Co-Authored-By: Matthew Booth <mbooth@redhat.com>
Partial-Bug: #1489581
Change-Id: Ibf0db6ad5b8367fc3267ac309516c08547d47e8c
This adds a uuid column to the migration table so that in a subsequent
patch we can start using them. Currently we only identify migrations
by internal database id, which is not only a problem for cellsv2, but
also not sufficient for us to use for making allocations that are owned
by a migration process.
Related to blueprint migration-allocations
Change-Id: I21dd6125094b13922b703caa34dffffab1789e1b
In future patches, we will be adding child resource provider records to
the placement API for certain PCI devices (e.g. SR-IOV physical
functions). Each resource provider record has a UUID identifier.
Unfortunately, the pci_devices table in the cell database does not have
a globally unique identifier field; it uses an autoincrementing integer
identifier for the pci_devices table. This means that code in the
scheduler report client would need to look up a resource provider
record for certain PCI devices by a unique name attribute, and the
scheduler report client would need to generate that unique name.
Looking up resource providers from the scheduler report client by
either UUID or name would result in an unreasonable increase in code
size for no real benefit. An easier solution is to simply add a uuid
identifier field to the pci_devices table and just look up resource
provider records via UUID.
Change-Id: I96ff82e50b16976f432b4dda65b9784cd5c0a16b
blueprint: nested-resource-providers
Since change Ie12acb76ec5affba536c3c45fbb6de35d64aea1b in Ocata
the FilterScheduler is calling the placement service to get a list
of resource providers that can service a request spec, and then
taking that list and querying the compute nodes table with a list
of uuids.
We should have a unique index on the uuid column if we're doing
queries like this, especially somewhere where performance matters
like the scheduler.
As a result of this change, several unit tests that were blindly
creating duplicate compute nodes with the same uuid have to be
fixed to use unique uuids.
Change-Id: I940979d85fc105c9733ae19c406e85debe8fdffb
This is an integer that defaults to zero. This can be used by API services
to efficiently filter out records that have already been mapped by a
HostMapping. This is an integer instead of a boolean in case we later need
to do some other sort of mapping of these records and need to distinguish
yet another level of mapped-ness.
Note this also removes some tests from TestNewtonBlocker which can no longer
work with the new ComputeNode model since they attempt to use it to create
compute nodes at a schema older than needed for this new field. The point of
the test was to verify, in Newton, that the blocker migration caught the thing
we needed to catch. It doesn't need to be in master anymore and certainly
is not worth the acrobatics that would be required to keep it working. (Note
that we haven't even had such tests for many of our blocker migrations)
Related to blueprint discover-hosts-faster
Change-Id: I902d75efb0bbe177680d7211c23235f42497e3fe
This change adds a uuid column to the Service model and versioned
object. This is a prerequisite to replacing id with uuid in the
services API.
The uuid is generated automatically for new services. For existing
records, it is generated on read from the database.
Change-Id: I85c39ab422b9687bf032cbe12d17a2babbda1c07
Partially-Implements: blueprint service-hyper-uuid-in-api
This wasn't walking the correct directory and it wasn't
including the API database migrations.
Change-Id: Idff0e8089faf4025a080a5842ff65eaad7b6fdb8
Closes-Bug: #1680240
This adds an attachment_id column to the block_device_mapping table as
required for Nova to begin interacting with cinder v3.
Implements: blueprint cinder-new-attach-apis
Change-Id: I5d6afbc22ee89b764440f3313229a29311883f52
Normally we reserve five slots for the API database instead of the default
ten. However, given all the stuff that merged in Ocata related to API-level
services, I'm going to reserve ten here just so we have space if we need
it.
Change-Id: I57c2edcf1fb80e24017cb1b4be00065aa20b342c
We should have an index on the instances.updated_at table because
that's what's used to filter instances when using the --changes-since
filter parameter with nova list.
Closes-Bug: #1642729
Change-Id: I4c723839f36efddd95cd24d745fb71c96c75d2c1
scheduled_at attribute column from instances table is no
longer used.
It has been removed from data model in liberty as a part of this commit
5e9df4baf9
This commit drops scheduled_at column from database schema.
Change-Id: Id7b397ff9bc5f0ab0b0af7167ce19ef14fbb1fa9
Closes-Bug: #1628200
Fixes unit tests related to incomparable types.
Fixes unit tests related to writing / reading bytes in / from files
or response bodies.
Fixes console websocketproxy unit tests.
Fixes consoleauth unit tests.
Fixes db unit tests.
Fixes image unit tests.
Fixes virt/block_devices unit tests.
Fixes virt/configdrive unit tests.
Fixes virt_drivers unit tests.
Fixes xenapi agent, vmops and volume_utils unit tests.
Fixes wsgi unit tests.
Enables some python3 unit tests.
Partially Implements: blueprint goal-python35
Change-Id: Ie98e968740d1015eae3278edeb93d4ba08155169
This adds a blocking schema migration to validate that all online
migrations required to be done in newton were finished before allowing
us to move on to ocata.
Change-Id: Iab714d9e752c334cc1cc14a0d524cc9cf5d115dc
internal_id attribute column from instances table is no
longer used. This commit removes internal_id from object
layer.
In the later release, this can be removed from schema
by adding migration script.
Change-Id: Ia285a16a8fab38a61959df847d30bce2d60ad081
Closes-Bug: #1441242
The db migration tests use a bunch of setup inside oslo_db that is
only exported as a base test class. This means that none of these do
the standard setup that the rest of the Nova tests do, including
things like logging capture.
Long term, the relationship between nova and oslo_db needs to be fixed
so that these are available as fixtures, not just base class. However,
in the short term we can setup logging correctly before initializing
the rest of the test class to keep these from scrambling the test
output.
Change-Id: I3c663d276cffa6146ac8f45bde7d70f7aece0c89
Adding a device_metadata column to store the virt_device_metadata object.
Partially implements blueprint virt-device-role-tagging
Co-authored-by: Artom Lifshitz <alifshit@redhat.com>
Change-Id: I035b188f706a8f371e9fbb94c605869ccaec462e
Console auth tokens will be saved in the database
instead of in memory in a console auth server.
Adding the table and models is the first step to
do this.
Co-Authored-By: Eli Qiao <qiaoliyong@gmail.com>
Change-Id: I42c25b465efb7e03769c9557f66feaa92cfa207c
partially-implements: blueprint convert-consoles-to-objects
This removes db2 support from tree completely. This is an oddball non
open database that made doing live data migrations difficult. It is
used by 0% of users in the OpenStack User Survey.
Supporting commercial software that doesn't have users at the cost of
delivering features and fixes to our community is the wrong
tradeoff. This corrects that.
It rewrites migrations, which we typically don't ever do, but it is
better if newton fresh environments fail early in db creation instead
of very deep in the data migration process.
Change-Id: Ifeb9929e4515e3483eb65d371126afd7672b92a4
Add the ability to sync the cell0 database using nova-manage.
The `db sync` command will be used to sync the cell0 database.
This ensures that operators will only have two db sync commands to
perform in the single cell case.
blueprint cells-cell0
Change-Id: I21ae13a6c029e8ac89484faa212434911160fd51
This adds a keypairs deferred-load column to instance_extra where we
can store the keypairs for a given instance instead of referring to
them by name. This is needed for the cell/api split.
Related to blueprint cells-keypairs-api-db
Change-Id: I5a3bf86e2fba21feacf6b59f6a96a0927f044e66
This patch series adds the ability for a user to specify a tag to be
applied to a network interface or a disk when booting an instance.
This tag is then exposed through the metadata API. For example, a user
boots a VM with two network interfaces. One is connected to a private
network, the other to the public Internet. There is currently no
direct way to tell which interface is which. Specifying tags allows
the user to distinguish between the two interfaces.
This patch contains the database migration to add a 'tag' column to
the virtual_interfaces and block_device_mapping tables as well as the
corresponding change to the VirtualInterface and BlockDeviceMapping
objects.
Implements: blueprint virt-device-role-tagging
Co-authored-by: Vladik Romanovsky <vromanso@redhat.com>
Change-Id: Ic8be3de4e970116772f9b6ce01c55e26b829e6cb
This migration enforces that all of our Mitaka online migrations have been
completed before we roll forward. This mirrors migration 291 for kilo that
we used to ensure correctness before migrating into Liberty.
Depends-On: Ibc02acc60b1023039f2613f8949b95d55169fd30
Change-Id: I9591f0b9227e30d4df2fada82fe236e474fc07b4
The process_sort_params method, used by instance_get_all_by_filters_sort,
defaults to ordering by the created_at and id columns.
Adding an index on the deleted and created_at columns significantly
reduces the time it takes to run the query on the instances table.
From the bug report:
Instead of 993,126 rows, it only grabbed 1490.
And drops from 10.61 seconds to 0.28 seconds on the data set in
the bug report.
The index was provided by Joseph bajin <josephbajin@gmail.com>
in the bug report.
Change-Id: I25ec8a0b8124d77926d938dc911464cc8b9d2945
Closes-Bug: #1553184
In testing it was discovered that a separate resource_pools
tables was not required. Instead the resource-providers table
itself could provide that and any given provider could provide
multiple resources classes, associated via the inventories
table.
Therefore, this adds 'name', 'generation', and 'can_host' columns
to resource_providers and changes an allocations index to add the
'used' column.
A 'resource_providers_aggregates' table is added to associate a
resource provider with an aggregate.
A unique constraint is added on the inventories table so that any
resource provider can have only one inventory for any given
resource class.
Change-Id: Ia28be42a75e0977faf9bbc91d1ad9d8661b99230
Partially-Implements: blueprint resource-providers
We forgot to provide the disk allocation ratio when moving RAM and CPU ratios.
This change is persisting now the disk ratio to the DB and adds the object
compat layer.
Note: Since we need to provide a compatibility path for upgrades and as we don't
want to ask the operators to change their config files before upgrading, we then
verify the allocation ratio value to match if defaulted and then return an
hardcoded value identitical to the behaviour we want. In order to know if the
compute related opt values were updated, we plan to change the ratios defaults
with 0.0 in a later change
Change-Id: I6d8b75dd085003d36728880f93cf6db86e6cf1a5
Partially-Implements: blueprint disk-allocation-ratio-to-rt
Add migration and models for resource_providers, allocations
and inventories tables which will track availability and
consumption of generic resources.
Note that this also adds the uuid column to the compute_nodes table,
which is required for the later modeling. It's squashed in here
because it's part of this line of work, required for later, and
because fewer migrations == bettersauce.
Co-Authored-By: Dan Smith <dansmith@redhat.com>
Partially-Implements: blueprint resource-providers
Change-Id: Id77ec737201881ecba5233439d2a7c173471fcfa
Balazs Gibizer