2486 Commits

Author SHA1 Message Date
Matt Riedemann
55868cbaf5 Remove aggregate uuid generation on load from DB
As the comments in the code suggest, we can remove this
code now. There was a blocking database migration which
enforced that the online data migrations to generate
uuids for aggregates has been completed:

4d0915568a1011ddee7317ddfb237be0803e7790

So this code is safe to remove now. As a result, we
also remove the online data migration routine which
relies on the object code.

Change-Id: I2050b5bdf906d2d2f8a87c79ca0804e0fc955755
2017-04-10 18:16:01 -04:00
Matt Riedemann
f56174657b Remove dead db api code
This was missed in change Ie069def25a807e4699f635d83abe243a044b43f7.

Change-Id: I24575db0609d6339ef3303887a25ccfbcf8d415a
2017-04-07 16:54:13 -04:00
Jenkins
f212aa15a0 Merge "Ensure reservation_expire actually expires reservations" 2017-04-07 20:08:12 +00:00
Matt Riedemann
a117e35b2d Ensure reservation_expire actually expires reservations
The existing unit test for this is a bit hard to follow
given the setup, so this change simply ensures that the
reservations created during the seutp are actually processed
during reservation_expire in the DB API.

Change-Id: I21fc6a441090b86f89b52c92976a81ba3c28f7d7
2017-03-31 15:50:07 -04:00
He Jie Xu
f307bc04db Rename the model object ResourceProviderTraits to ResourceProviderTrait
The name of model object should be singular. This patch corrects it.

Partial implement blueprint resource-provider-traits

Change-Id: I7600d7e3775d237e0593ec52fbb474e1dad079c1
2017-03-31 10:38:19 +08:00
Jenkins
cf9596ebce Merge "Remove the EC2 compatible API tags filter related codes" 2017-03-22 09:25:25 +00:00
Chris Friesen
de3388fc48 Add comment to instance_destroy()
The instance_actions and instance_actions_events tables are handled
differently than most when we soft-delete an instance.

It's not immediately obvious why this is, so let's add a comment
explaining what's going on.

Change-Id: Ie353650861c2911e4f55628c55f049fc1756e591
2017-03-03 14:26:48 -06:00
Jenkins
f032c0008d Merge "[placement] Add Traits related table to the api database" 2017-03-01 02:16:24 +00:00
He Jie Xu
befdeee0ed [placement] Add Traits related table to the api database
This patch adds DB table `traits` and `resource_provider_traits` into
the api database.

The traits table is used to represent the trait. The resource_provider_traits
table is used to represent the relationship between trait and resource provider.

Part of implementation blueprint shared-resources-pike

Change-Id: I5c34bdd1423beab53cc4af45e016d9a9bba5ffda
2017-02-28 17:43:54 -05:00
Dan Smith
494b317aad Ignore deleted services in minimum version calculation
When we go to detect the minimum version for a given service, we
should ignore any deleted services. Without this, we will return
the minimum version of all records, including those that have been
deleted with "nova service-delete". This patch filters deleted
services from the query.

Closes-Bug: #1668310
Change-Id: Ic96a5eb3728f97a3c35d2c5121e6fdcd4fd1c70b
2017-02-27 07:55:42 -08:00
Dan Smith
6b3a2e8f0c Reserve migration placeholders for Ocata backports
Normally we reserve five slots for the API database instead of the default
ten. However, given all the stuff that merged in Ocata related to API-level
services, I'm going to reserve ten here just so we have space if we need
it.

Change-Id: I57c2edcf1fb80e24017cb1b4be00065aa20b342c
2017-02-06 07:40:31 -08:00
Dan Smith
82a352af64 Only warn about hostmappings during ocata upgrade
The check and subsequent hard failure for HostMapping records in
API migration 30 is inconvenient at times during a new setup where
we have flavors in place but no hosts yet. Since we can now check
for and warn about missing HostMapping records in our upgrade
status command, this patch lowers the lack of host mappings check
from a failure to a warning. This migration was really just to make
sure you ran the simple setup command, and the cell mapping check
does that for us.

Change-Id: I8b757fa7c805ec6f4d578ecb6f33d3f1ceff29fc
2017-01-25 17:33:48 -08:00
Jenkins
9fae0bf71c Merge "[2/3]Replace six.iteritems() with .items()" 2017-01-21 06:55:52 +00:00
Dina Belova
aa8fbb5007 Integrate OSProfiler and Nova
* Add osprofiler wsgi middleware. This middleware is used for 2 things:
  1) It checks that person who want to trace is trusted and knows
     secret HMAC key.
  2) It starts tracing in case of proper trace headers
     and adds the first wsgi trace point with info about the HTTP request

* Add initialization of osprofiler on start of a service
  Currently that includes oslo.messaging notifier instance creation
  to send Ceilometer backend notifications.

oslo-spec: https://review.openstack.org/#/c/103825/
python-novaclient change: https://review.openstack.org/#/c/254699/
based on: https://review.openstack.org/#/c/105096/

Co-Authored-By: Boris Pavlovic <boris@pavlovic.me>
Co-Authored-By: Munoz, Obed N <obed.n.munoz@intel.com>
Co-Authored-By: Roman Podoliaka <rpodolyaka@mirantis.com>
Co-Authored-By: Tovin Seven <vinhnt@vn.fujitsu.com>

Implements: blueprint osprofiler-support-in-nova
Change-Id: I82d2badc8c1fcec27c3fce7c3c20e0f3b76414f1
2017-01-18 15:00:14 +07:00
Spencer Yu
f159d6da21 [2/3]Replace six.iteritems() with .items()
1.As mentioned in [1], we should avoid using
six.iteritems to achieve iterators. We can
use dict.items instead, as it will return
iterators in PY3 as well. And dict.items/keys
will more readable. 2.In py2, the performance
about list should be negligible, see the link [2].
[1] https://wiki.openstack.org/wiki/Python3
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html

The patch list:
    1. cells.
    2. compute api.
    3. image.
    4. network.
    5. objects.
    6. scheduler.
    7. virt.
    8. other resources.

Partial-Implements: blueprint replace-iteritems-with-items

Change-Id: Ic6e469eb80ee1774de1374bb36f38b5134b6b311
2017-01-09 09:11:00 +00:00
Michael Still
17b33495e4 Move quota options to a config group.
As suggested in John Garbutt in I7e1986c5f11356060cc9db12605b1322c39e79c0,
move the quota config options into a config group of their own.

Change-Id: Ie06a370868f01fb9a1cc246a77d7823fac83e70e
2017-01-04 22:57:14 -06:00
Jenkins
7a4248aff0 Merge "Only return latest instance fault for instances" 2017-01-04 21:37:14 +00:00
jeckxie
0a7b22216a [TrivialFix] Fix comment and function name typo error
Change-Id: I713dd471a76e82ee0a4471ea4e5f9dcb57227345
2016-12-22 15:20:23 +08:00
Jay Pipes
e24798a02e Only return latest instance fault for instances
This patch addresses slowness that can occur when doing a list servers
API operation when there are many thousands of records in the
instance_faults table.

Previously, in the Instance.fill_faults() method, we were getting all
instance fault records for a set of instances having one of a set of
supplied instance UUIDs and then iterating over those faults and
returning a dict of instance UUID to the first fault returned (which
happened to be the latest fault because of ordering the SQL query by
created_at).

This patch adds a new InstanceFaultList.get_latest_by_instance_uuids()
method that does some SQL-fu to only return the latest fault records for
each instance being inspected.

Closes-Bug: #1632247

Co-Authored-By: Roman Podoliaka <rpodolyaka@mirantis.com>
Change-Id: I8f2227b3969791ebb2d04d74a316b9d97a4b1571
2016-12-14 19:25:05 +02:00
Diana Clarke
2623fe2427 Simple tenant usage pagination
Add optional parameters 'limit' and 'marker' to the
os-simple-tenant-usage endpoints for pagaination.

  /os-simple-tenant-usage?limit={limit}&marker={instance_uuid}
  /os-simple-tenant-usage/{tenant}?limit={limit}&marker={instance_uuid}

The aggregate usage totals may no longer reflect all instances for a
tenant, but rather just the instances for a given page. API consumers
will need to stitch the aggregate data back together (add the totals)
if a tenant's instances span several pages.

Implements blueprint paginate-simple-tenant-usage
Change-Id: Ic8e9f869f1b855f968967bedbf77542f287f26c0
2016-12-14 03:33:50 +00:00
He Jie Xu
22635ce8ff Remove the EC2 compatible API tags filter related codes
There are codes for tags search in EC2 API in db layer.
Those code are added in the commit 24582abf82842cbb922e38c78e281eda56f981e2.

The EC2 compatible API already removed from Nova, and this parameter
didn't works for Nova REST API also. So those codes should be removed.

Change-Id: Ic7843e6b90f5c8ba204eb3183bbb21bea9e0f907
2016-12-13 21:11:29 +08:00
Jenkins
7d5b1b8555 Merge "Pass context as kwarg instead of positional arg to get_engine" 2016-12-13 03:04:30 +00:00
Jenkins
06b8c85489 Merge "Fix instructions for running simple_cell_setup" 2016-12-12 21:27:34 +00:00
Matt Riedemann
53a5b020ce Fix instructions for running simple_cell_setup
Change ff6b9998bb977421a5cbc94878ced8542d910c9e enforces in
a database migration that you've run the simple_cell_setup
command for cells v2 but the instructions in the error and
in the release note said to use 'nova-manage db' when it should
be 'nova-manage cell_v2'.

Change-Id: I8e71d1c7022d1000f26b7c16ed1c56f6e87ab8ac
Closes-Bug: #1649341
2016-12-12 12:52:27 -05:00
jichenjc
8f2538f8fb Refactor REGEX filters to eliminate 500 errors
You can currently create a 500 error on mysql by passing | as the name
filter because mysql assumes regex values are well crafted by the
application layer.

This puts in facilities to provide a safe regex filter per db engine.

It also refactors some of the inline code from _regex_instance_filter
into slightly more logical blocks, which will make it a little more
straight forward about where we need to do something smarter about
determining the dbtype in a cellsv2 world.

Change-Id: Ice2e21666905fdb76c001195e8fca21b427ea737
Closes-Bug: 1546396
2016-12-09 16:37:27 -05:00
melanie witt
96f8efe2d2 Pass context as kwarg instead of positional arg to get_engine
There are a couple of places in nova/db/api/sqlalchemy.py where the
context argument is passed as a positional arg instead of a kwarg,
causing it to be erroneously mapped to the use_slave kwarg:

  def get_engine(use_slave=False, context=None):

This corrects to calls to pass context=context instead.

Change-Id: I8fb7f04a54d9f7f645df8287cdda0ae665a22368
2016-12-06 16:35:22 +00:00
Dan Smith
4c6dbc4194 Require cellsv2 setup before migrating to Ocata
We have code going into Ocata that needs to be sure that cell and
host mappings are in place. Since this was required homework in
Newton, we can land a migration to intentionally fail if this was
not completed.

This is, however, a little difficult to require because a first-time
deployment will be initialized schema-wise with none of these records,
which is also sane. So, we look to see if any flavors are defined as
a sentinel to indicate that this is an upgrade of an existing
deployment instead of a first-time event. Not perfect, but since this
is really just a helper for the user, it seems like a reasonable
risk.

Depends-On: If1af9c478e8ea2420f2523a9bb8b70fafddc86b7
Change-Id: I72fb724dc13e1a5f4e97c58915b538ba761c582d
2016-12-05 19:51:36 -05:00
melanie witt
40ad8a2199 Use pick_context_manager throughout DB APIs
The pick_context_manager method will use a connection to a cell
database if one is present in the RequestContext, else it falls
back on the global main_context_manager in the DB API.

Currently, there are several places in our DB API code where
pick_context_manager isn't used because in a real scenario, each
cell is in a separate process where main_context_manager points
to its local database. This causes problems for testing though,
because we are unable to patch the DB API to simulate switching
between multiple 'main' databases in our functional tests because
of the global nature of main_context_manager.

This replaces all uses of main_context_manager with
pick_context_manager to:

  1. Make switching between multiple databases able to work in
     functional tests
  2. Fix any possible cases where pick_context_manager is not
     used for a DB API method that could be called from the
     API using target_cell

Change-Id: I31e3170e0953cefbf49bfc84b29edab514c90cb5
2016-11-18 17:15:51 +00:00
Chris Dent
320f60f28b [placement] add a placement_aggregates table to api_db
In order to facilitate a future extraction of the placement service
we want to record the association between a resource provider and an
arbitrary aggregate uuid in its own table.

A PlacementAggregate model is joined from ResourceProvider via
ResourceProviderAggregate. Note that this structure is used so we can
join on ids instead of strings (the uuids). A direct mapping between
ResourceProvider uuid and Aggregate uuid was mooted earlier in the year
but was determined to be suboptimal.

The name 'placement_aggregates' is used as the least problematic of
several choices after discussion amongst several parties.

The data will be used by the forthcoming get_ and set_aggregates
methods on the ResourceProvider object.

Change-Id: Id0355cb022f68e962af306ff04cf724d22b68d19
Partially-Implements: blueprint generic-resource-pools-ocata
2016-11-08 16:09:56 +00:00
Matt Riedemann
8efb8ca4be Make build_requests.instance MediumText
The build_requests.instance column is a serialized
instance object, and the instances.user_data column
is MediumText, so the build_requests.instance column
itself needs to be at least MediumText in size for MySQL.

Change-Id: I7d65df37c02750593037744543ad15e5bc64e913
Closes-Bug: #1635446
2016-10-26 06:27:07 -04:00
Mark Doffman
650ff9387c Add quota related tables to the api database.
Quotas are required to exist in the API database as we need to enforce
quotas across cells.

blueprint cells-quota-api-db

Change-Id: I52fd680eaa4880b06f7f8d4bd1bb74920e73195d
2016-10-12 07:40:00 +00:00
Jay Pipes
634b59c73a placement: add new resource_classes table
We will store custom resource classes in the new resource_classes table.
These custom resource classes represent non-standardized resource
classes. Followup patches add the plumbing to handle existing
standardized classes currently using the fields.ResourceClass field type
and to perform CRUD operations against a /resource-classes REST API
endpoint.

Change-Id: I60ea0dcb392c1b82fead4b859fc7ed6b32d4bda0
blueprint: custom-resource-classes
2016-10-11 12:30:44 -04:00
Sean Dague
12cfd5be6d Unwind circular import issue with api / utils
The DeleteFromSelect function was added to sqla.utils, which is mostly
the home of 2 shadow table functions that need to import api.py to get
constants related to shadow tables. We now exclusively use that
function in api.py for archiving code. That causes some oddities
around module import.

This moves DeleteFromSelect so we can get rid of these odd late
imports.

Change-Id: I59c3444c2258f59a09a9c885bd9490055e278998
2016-09-27 15:05:56 -04:00
Dan Smith
411113a2dd Archive instance-related rows when the parent instance is deleted
This is something I expect has been very broken for a long time. We
have rows in tables such as instance_extra, instance_faults, etc that
pertain to a single instance, and thus have a foreign key on their
instance_uuid column that points to the instance. If any of those
records exist, an instance can not be archived out of the main
instances table.

The archive routine currently "handles" this by skipping over said
instances, and eventually iterating over all the tables to pull out
any records that point to that instance, thus freeing up the instance
itself for archival. The problem is, this only happens if those extra
records are actually marked as deleted themselves. If we fail during
a cleanup routine and leave some of them not marked as deleted, but
where the instance they reference *is* marked as deleted, we will
never archive them.

This patch adds another phase of the archival process for any table
that has an "instance_uuid" column, which attempts to archive records
that point to these deleted instances. With this, using a very large
real world sample database, I was able to archive my way down to
zero deleted, un-archivable instances (from north of 100k).

Closes-Bug: #1622545
Change-Id: I77255c77780f0c2b99d59a9c20adecc85335bb18
2016-09-27 12:08:21 -07:00
hussainchachuliya
5c570e9f7f Replace uuid4() with generate_uuid() from oslo_utils
As of now, UUID is being generated using either uuid.uuid4()
or uuidutils.generate_uuid(). In order to maintain consistency,
we propose to use uuidutils.generate_uuid() from oslo_utils.

Change-Id: I69d4b979fc0e37bc351f9b5516dae899e7e7dba0
2016-09-23 14:39:09 +05:30
Dan Smith
85f42c04be Add placeholder DB migrations for Ocata
This adds placeholder migrations for potential backports of database
migrations to newton.

Change-Id: I3ead9c09fb033592f752b26cac7482cefaa5d12b
2016-09-22 07:42:57 -07:00
Dan Smith
ae221ed7da Remove PCI parent_addr online migration
This migration was added in mitaka and should have been done either in
mitaka or newton. Newton had some migrations that came after it which
were higher-visibility and should have forced everyone to have done
this by now.

Because of the nature of the migration, there isn't really an easy or
efficient way to validate that they have done this like we have done
for some other migrations.

Change-Id: Idb033e9e52b149372308eabb19c5774e10c56156
2016-09-22 07:42:57 -07:00
Takashi NATSUME
30b82fe0f0 Fix an error in archiving 'migrations' table
Add soft deleting 'migrations' table when the VM instance is deleted.
And add soft deleting 'migrations' table when archiving deleted rows
for the case to upgrade.

Change-Id: Ica35ce2628dfcf412eb097c2c61fdde8828e9d90
Closes-Bug: #1584702
2016-09-15 09:54:16 +09:00
Ken Burger
27bbeb5f5b Increase BDM column in build_requests table
When deploying with a very large number of volumes the
block_device_mappings column is not sufficient.  The column
needs to be increased to MEDIUMTEXT size to support this use case.

Change-Id: Ia34d06429c1f8f0a8259616bcba0c349c4c9aa33
Closes-Bug: #1621138
2016-09-08 20:22:20 -04:00
John Garbutt
84e5c16457 db: retry on deadlocks while adding an instance
We are hitting deadlocks in the gate when we are inserting the new
instance_extra row into the DB.

We should follow up this fix and look at way to avoid the deadlock
happening rather than retrying it. It currently doesn't happen too
often, so this should be enough to stop the problem while we work on a
better fix.

Closes-Bug: #1480305

Change-Id: Iba218bf28c7d1e6040c551fe836d6fa5e5e45f4d
2016-09-08 16:06:31 +00:00
Jenkins
63acdc77ce Merge "Reduce code complexity - api.py" 2016-08-30 12:03:09 +00:00
Alex Szarka
f2e3ecd47a Reduce code complexity - api.py
Reduce the code complexity in
nova/db/sqlalchemy/apy.py file.
The 351176 patch depends by this.

Change-Id: I39adb817b5e03b39a17df6f64152ce23ef3c9ed6
2016-08-26 18:55:33 +00:00
John Garbutt
a9d11231a3 Revert "Optional separate database for placement API"
This reverts commit 1b5f9f8203c90fe447d33c89f238104026052d1e.

On IRC I we decided that we agreed no migrations should be placement
specific, we should just use the API table migrations to generate the
schema for both DBs.

There is also a separate debate around the alias for the aggregates
table, but that is not really a reason to revert, its just things in
here that will need rework.

Change-Id: I275945aee9d9be8e35d6ddc05515df39d559457a
2016-08-25 13:51:16 -07:00
Jenkins
6a06cb083c Merge "Manage db sync command for cell0" 2016-08-25 04:26:07 +00:00
Mark Doffman
8707b1859f Manage db sync command for cell0
Add the ability to sync the cell0 database using nova-manage.
The `db sync` command will be used to sync the cell0 database.
This ensures that operators will only have two db sync commands to
perform in the single cell case.

blueprint cells-cell0

Change-Id: I21ae13a6c029e8ac89484faa212434911160fd51
2016-08-22 18:53:45 +03:00
Pavel Kholkin
ebd6193614 removed db_exc.DBDuplicateEntry in bw_usage_update
BandwidthUsage model has no UniqueConstraints.
In 'bw_usage_cache' table in nova db there is single primary
autoincrement key. So the duplicate entry problem is solved by
db itself and db_exc.DBDuplicateEntry could not be raised in Nova.

Ideally we should add UniqueConstraint to prevent multiple bw usage
records existing for the same date range and UUID. That fix for this
will mean we should be able to remove the .first() call and instead
use .one(). The current code that uses .first() is not correct
because there is no order_by() applied on the SQL query and
therefore the returned "first record" is indeterminate.

This workaround fix removed misleading note and exception and
added order_by() to ensure that the same record is updated every time.

Co-Authored-By: Sergey Nikitin <snikitin@mirantis.com>

Closes-bug: #1614561

Change-Id: I408bc3a3e5623965a619d8c7241e4e77c8bf44f5
2016-08-22 15:20:21 +03:00
Jenkins
e03ade34c5 Merge "List instances for secgroup without joining on rules" 2016-08-13 23:10:36 +00:00
Roman Podoliaka
bf31e145c9 Optional separate database for placement API
If 'connection' is set in the 'placement_database' conf group use
that as the connection URL for the placement database. Otherwise if
it is None, the default, then use the entire api_database conf group
to configure a database connection.

This works by:

* adding a 'placement sync' and 'placement version' command to
  nova-manage

* adding placement_migrations that that sync will run

* adding a placement_database config group with the relevant
  database configuration settings

* adding a placement_context_manager. If
  CONF.placement_database.connection is None this is the same as
  the api_context_manager, otherwise it is a new one from its own config

* adjust nova/tests/fixtures to be aware of a 'placement' database
  and the placement_context_manager

This version of this change differs from others by not requiring
separate placement commands for migration, instead using existing
tooling, which makes the size of the change a bit smaller and also
addresses problems with the test fixtures needing to be too aware of
what migration code to run. Now it runs the same code.

This functionality is being provided to allow deployers to choose
between establishing a new database now or requiring a migration
later. The default is migration later.

This is a modification of Id93cb93a0f4e8667c8e7848aa8cff1d994c2c364
and I3290e26d0a212911f8ef386418b9fa08c685380b.

Change-Id: Ice03144376c9868c064e4393d531510615fc6fc1
Co-Authored-By: Chris Dent <cdent@anticdent.org>
Partially-Implements: blueprint generic-resource-pools
2016-08-11 13:30:25 +00:00
Markus Zoeller
d806972b85 config options: improve help text of database (related) options (1/2)
This change enhances the help texts of the database related config
options. They are in the "DEFAULT" section of the "nova.conf" and
get used on the DB layer but describe a usage on the "compute" level.
They should probably live in another section than "DEFAULT".

bp centralize-config-options-newton

Change-Id: I5caeff8bcb5f0b63f5ec4e79ab5f6cde16ae018f
2016-08-11 14:23:30 +02:00
Paul Griffin
1359e025fe List instances for secgroup without joining on rules
Make db.security_group_get only join rules if specified in
the columns_to_join. This works around a performance issue
with lots of instances and security groups.

Co-Authored-By: Dan Smith <dansmith@redhat.com>
Change-Id: Ie3daed133419c41ed22646f9a790570ff47f0eec
Closes-Bug: #1552971
2016-08-10 16:30:55 -07:00