41 Commits

Author SHA1 Message Date
melanie witt
40ad8a2199 Use pick_context_manager throughout DB APIs
The pick_context_manager method will use a connection to a cell
database if one is present in the RequestContext, else it falls
back on the global main_context_manager in the DB API.

Currently, there are several places in our DB API code where
pick_context_manager isn't used because in a real scenario, each
cell is in a separate process where main_context_manager points
to its local database. This causes problems for testing though,
because we are unable to patch the DB API to simulate switching
between multiple 'main' databases in our functional tests because
of the global nature of main_context_manager.

This replaces all uses of main_context_manager with
pick_context_manager to:

  1. Make switching between multiple databases able to work in
     functional tests
  2. Fix any possible cases where pick_context_manager is not
     used for a DB API method that could be called from the
     API using target_cell

Change-Id: I31e3170e0953cefbf49bfc84b29edab514c90cb5
2016-11-18 17:15:51 +00:00
Dan Smith
465937e82e Address feedback on cell-aggregate-api-db patches
This fixes up just a few minor things pointed out in the
review of the prevous set of patches.

Change-Id: If19f36c3bdf1393e07a74c627d4b7054d16fbcbb
2016-08-01 11:46:59 -07:00
Mark Doffman
f0ea1c949d Add data migration methods for Aggregate.
Add data migration methods to migrate Aggregates to the API db.

blueprint cells-aggregate-api-db

Co-Authored-By: Dan Smith <dansmith@redhat.com>
Change-Id: Ia2b5ff8046195635b68eddfa7c83d60750ec9932
2016-08-01 11:25:53 -07:00
Mark Doffman
9cfd6fc760 Aggregate create and destroy work against API db
Make aggregate.create() and destroy() use the API rather than cell database.
Also block aggregate creation until main database empty. This makes
Aggregate.create() fail until the main database has had all of its aggreagtes
migrated. Since we want to avoid any overlap or clashes in integer ids we
need to enforce this.

Note that this includes a change to a notification sample, which encodes
the function and module of a sample exception (which happens to be during
an aggregate operation). Since the notifications are encoding internal
function names, which can and will change over time, this is an expected
change.

blueprint cells-aggregate-api-db

Co-Authored-By: Dan Smith <dansmith@redhat.com>
Change-Id: Ida70e3c05f93d6044ddef4fcbc1af999ac1b1944
2016-08-01 08:22:08 -07:00
Mark Doffman
2110b69656 Make Aggregate.save work with the API db
Aggregate.save now updates aggregates if they
are located in the api db.

blueprint cells-aggregate-api-db

Co-Authored-By: Dan Smith <dansmith@redhat.com>
Change-Id: I6066857a421bfe489ae70f0e22f338c434195bf9
2016-08-01 08:15:46 -07:00
Mark Doffman
88c45d17a9 Make Aggregate metadata functions work with API db
Adds the ability to update aggregate metadata where the aggregate is
contained in the API db. Modifications will be made to the 'aggregate_metadata'
table in the API db if that is where the aggregate is located or the cell
database otherwise.

blueprint cells-aggregate-api-db

Change-Id: I456beb753298f4aabbed6d20bea7cbc537842e50
2016-07-26 13:59:47 -05:00
Mark Doffman
5ac2105a86 Make Aggregate host operations work against API db
Adds the ability to add and delete hosts to Aggregates that are
contained in the API db. Modifications will be made to the
'aggregate_hosts' table in the API db if that is where the aggregate is
located, or the cell database otherwise.

blueprint cells-aggregate-api-db

Change-Id: Icb11e5cbcc743d1ae788c792cffba9de00fddc3f
2016-07-20 18:28:46 -05:00
bhagyashris
5b7bd79272 Fix invalid import order
Made corrections in import order as per OpenStack import standards [1].

[1] http://docs.openstack.org/developer/hacking/#import-order-template

TrivialFix

Change-Id: I58f20cda288d1332c3e7cd694a0413486b6302e8
2016-07-04 11:23:52 +05:30
Mark Doffman
6d7d9c7800 Make Aggregate.get_by_uuid use the API db.
Make Aggregate.get_by_uuid access the api db first only accessing the
cell db if not found.

blueprint cells-aggregate-api-db

Change-Id: Iaf89c041b6b6990f65e90c5086b13edea1af26f1
2016-06-12 14:51:09 -05:00
Chris Dent
5003a66a89 Add a get_by_uuid for aggregates
db.api.aggregate_get_by_uuid and objects.Aggregate.get_by_uuid
methods are added to make it possible to work with Aggregates by
their newly added uuid attribute.

Partially-Implements: blueprint generic-resource-pools
Change-Id: I7c46fd1ffebb6907c949cfa302131bfbfcd433de
2016-06-08 08:26:54 +00:00
Mark Doffman
92bcd0ae42 Make AggregateList.get_ return API & cell db items
Make the AggregateList get_all, get_by_hosts and get_by_metadata
functions return items from both the API and cell databases.

blueprint cells-aggregate-api-db

Change-Id: I0e1e11283e6ab2f7d2b976d126e914f9d7b4d8fb
2016-06-01 12:01:39 -05:00
Mark Doffman
5dad61b1f3 Make Aggregate.get operation favor the API db
This makes the Aggregate object use the API database first only falling
back to the cell database if it is not available.

blueprint cells-aggregate-api-db

Change-Id: Id3616cef1937744bf600763bae5de50f1d27b783
2016-06-01 12:01:39 -05:00
Dan Smith
9a0ed11827 Allocate uuids for aggregates as they are created or loaded
Note that this filters uuid from the aggregate view in the API because we would
need a microversion. That may be a thing in the future (for the 2.x api) but
not something we can do here.

Related to blueprint generic-resource-pools

Change-Id: I45006e546867d348563831986b91a317029a1173
2016-02-24 07:48:09 -08:00
Ryan Rossiter
6589a7866d Remove NovaObjectDictCompat from Aggregate
This change removes the NovaObjectDictCompat mixin from the Aggregate
versioned object. All occurrences of key access or get have been changed
to use getattr/setattr. Because none of the objects externally-facing
functionality has changed, the object version does not need to be
updated.

There was a fake aggregate in the xen api tests that contained two
empty dictionaries for a couple of keys in the metadata. These are actually strings in the
code, so they have been changed to empty strings.

Within the tests for the aggregate API, the fake dictionaries that were
returned by the AggregateAPI stubs were structured incorrectly. Within
the object, the availability zone is stored in the metadata dictionary,
not the top-level dictionary. Also, those dictionaries were changed to
Aggregate objects, so the stubs now return true objects (similar to how
the actual code works). The result of the calls are then compared by the
object primitives.

There was usage of .items() in _marshall_aggregate() within the api
controller for aggregates, so a unit test was added to run through that
function to make sure it is transforming the aggregate object correctly.
(props to Tempest for finding that one for me).

Partially-Implements: bp rm-object-dict-compat
Change-Id: Ibcb896b2eef673bce23491161ff394bcc87aa541
2016-01-08 20:32:18 +00:00
Dan Smith
2033bb5534 Remove obj_relationships from objects
In Liberty, we converted to using the manifest-based class
action and backport methods, which no longer rely on these
object relationship maps. They provide a full manifest of
client-side versions with each call, which is far more
robust than us trying to hand-edit these mappings (which
we get incorrect a lot).

Since we're now in Mitaka, we can drop compatibility with
clients that don't make the version-manifest calls. Those
have also been deprecated in oslo.versionedobjects.

Change-Id: Iea8b6348e3b01aaa335ec5408f07826f1e758d20
2015-10-30 00:22:23 +00:00
Ryan Rossiter
1700a2d59f Change List objects to use obj_relationships
List objects used to use child_versions, which was tested in a more
shallow way than test_relationships() tests obj_relationships. This
change moves over all list classes to use obj_relationships (and
therefore are now tested in test_relationships() with all other
objects).

Change-Id: I334dba243de45bf11ccb5ab55e2aa8390fe9d8c2
Closes-Bug: #1470154
2015-07-31 18:49:50 +00:00
Dan Smith
1e9fe44dc9 Begin the transition to an explicit object registry
In oslo.versionedobjects, the registry is opt-in instead of implicit.
This patch sets the stage for moving to that by defining a no-op
registry with a compatible register method so that we can go ahead and
decorate the object classes and then move the registry underneath them.

Related to blueprint use-oslo-objects

Change-Id: Iddc68a2d8a9b103621bc60a7426fe495edf80aad
2015-06-08 07:12:34 -07:00
Jenkins
75371fa4fc Merge "Fix logic for checking if az can be updated" 2015-04-28 21:31:39 +00:00
Dan Smith
b65106abd3 Remove context from remotable call signature
This is the final step in removing the context parameter from the
signature of a remotable method.

This includes a change of most of the object hashes, without version
bumps. That's because the hashing algorithm is just looking for changes
in things like a call signature, in order to signal that a version bump is
required. Since context is in the signature, removing it triggers the
alert. However, context is not _actually_ part of the on-the-wire API,
as it is stripped out on the caller side, and re-added on the callee side
(hence why we're making this change in the first place). If the test had
been uuber-pedantic to exclude context from consideration, then the hashes
would not be changing here, but alas.

In short, the hash changes are false alarms and do not mean we need
version bumps for all the things.

Related to blueprint kilo-objects

Change-Id: I89464c0ab7e6e0d84e677b9a69a86468727b6438
2015-03-18 06:53:38 -07:00
Dan Smith
88fc30cd82 Remove usage of remotable context parameter in agent, aggregate
This removes the use of the context parameter in remotable methods
in these objects. This is a precursor to actually removing the
context from the call, but we need to remove dependence on the
argument first. Remotable methods should be using the bundled object
context.

Related to blueprint kilo-objects

Change-Id: Iad5093622337077418a8ef94c9867dfd7a7fdf64
2015-03-17 07:38:33 -07:00
Hans Lindgren
3fcbe463a4 Fix logic for checking if az can be updated
The existing logic was overly complicated and missed out on those hosts
whose services were disabled.

This is a complete rewrite that makes use of a single aggregate query,
thereby bypassing a lot of the extra logic needed by the old code.

Fixes an issue in objects.AggregateList.get_by_metadata_key() where
filtering by an empty list of hosts will return metadata for all hosts.

Also removes a call to db.aggregate_metadata_get_by_metadata_key() to
avoid the need for special handling due to that method returning
metadata as sets of values instead of strings and also because the same
metadata is already fetched in another method.

Change-Id: I514e63ce863f2c77dcd47af3e3674019033a77de
Closes-Bug: #1419115
2015-03-02 23:19:31 +01:00
Jenkins
b6680b1a52 Merge "Aggregate.save() shouldn't return a value" 2015-01-05 15:26:17 +00:00
Daniel P. Berrange
6e6ea954ad objects: remove dict compat support from all XXXList() objects
The XXXList() classes only ever have a single 'objects' attribute
and this is never directly accessed as a dict key, instead all
callers use it in list context. Thus the dict compat support can
be removed from all these objects

Change-Id: I3f8d49d25b99dd0a498dfaad705c6332908cb72f
2014-12-12 14:06:12 +00:00
Daniel P. Berrange
c3d7bf19fb objects: allow creation of objects without dict item compat
The dict compat support in NovaObject is useful for incrementally
converting existing code over to use objects. Any brand new objects
though have no reason to support dictionary access, so there should
be a way to disable this compat mode.

This moves all the dict compat support methods out of NovaObject,
to a new class NovaObjectDictCompat. All existing objects are
updated to add this new class as a mix-in parent.

In future any completely new objects (ie ones which aren't being
used to converted existing code from dict instances) should avoid
inheriting from NovaObjectDictCompat. Existing objects should
also have this parent class removed once all callers are audited
and/or updated to ensure they do not require dict compat.

Change-Id: I03f93f0c40df6f5f7df9cefe28dff900c22294c9
2014-12-12 14:01:04 +00:00
Matthew Booth
b91b6145f2 Aggregate.save() shouldn't return a value
Unlike all object object save methods, Aggregate.save() returns an
aggregate object. This change removes the return to make it consistent
with other objects.

Change-Id: I2179bc90ae92dd3a4f739aae2802a417bd7e2d9d
2014-11-11 17:03:28 +00:00
melanie witt
41bef79201 add get_by_metadata_key to AggregateList object
This change adds the get_by_metadata_key function to the AggregateList object
and the aggregate_get_by_metadata_key function to the DB API.

Previously, the only DB API function available for finding aggregates matching
a metadata key returned a dictionary of aggregated metadata. The
aggregate_get_by_metadata_key function returns all rows matching the specified
metadata key, and is called by the existing aggregate_host_get_by_metadata_key
DB API function.

The get_by_metadata_key function returns a list of Aggregate objects matching
the specified metadata key.

Related to blueprint compute-manager-objects-juno

Change-Id: If50c9f613dd192ab44577f26a81dd5b40e3a7af7
2014-07-09 17:52:02 +00:00
Jenkins
665a4c7775 Merge "Remove duplicate code in Objects create() function" 2014-06-30 14:17:56 +00:00
Chris Behrens
dcb2808f52 Fix object code direct use of other object modules
The object code itself should not explicitly reference its own objects,
as objects could be subclassed.

Also: There were a few objects that had their own __init__()s that
didn't support accepting 'context' or kwargs. At some point here, we
should also require that context is passed when instantiating the
objects. So, I added the (currently optional) context argument on some
object instantiations near the places I was touching.

Partial-Blueprint: object-subclassing

Change-Id: Icf631c770e347b4bb32999c4ba2431d1db49e11c
2014-06-24 00:49:11 -07:00
tanlin
ba581eb24e Remove duplicate code in Objects create() function
Remove duplicate code updates.pop('id', None) in create() function.
Because if self.obj_attr_is_set('id'):
            raise exception.ObjectActionError(action='create',
            reason='already created')
already make sure object doesn't have 'id' values, which means that
'id' won't appear in updates.

Change-Id: I582ef8857b32c07ea8ba1eb5fc1e64516110f3ce
2014-06-16 14:20:53 +08:00
Jay Lau
652c8ca285 Payload meta_data is empty when remove metadata
When update metadata for a host aggregation, all of the values will
be in the variable of "updates", if remove metadata, the format would
be {"foo": None, "bar": None}; If adding metadata, the format would
be {"foo": "foo1", "bar": "bar1"}.

The current logic of update_metadata() is as following:
1) Get all metadata from variable "updates".
2) Traverse updates to see which values are new added and which needs
to be removed. For the new added values, put the metadata to
variable "to_add"; for the values which need to be removed, just
remove them from metadata.
3) Set the values of variable "to_add" as payload metadata.

The above logic will make the values which was removed from metadata
cannot be in payload, this caused the third party receiver cannot know
which metadata was removed.

The fix was always using the values from variable "updates" as the
metadata for payload as it include values removed and new added.

Change-Id: I7b78795e325f52cfb9c24eabd9ef42990666c02d
Closes-Bug: #1269684
2014-05-21 13:38:05 +00:00
Shane Wang
7cda04dd70 Prevent caller from specifying id during Aggregate.create()
The patch is to remove id when it was set before creating an aggregate
in objects, and check if the aggregate exists before it is created.

Related to blueprint icehouse-objects.

Change-Id: I0d7c5c5da9a93e52c7fee9eee7df80d2f460593e
2014-02-27 08:17:58 +00:00
Dan Smith
4b4182da55 Require List objects to be able to backlevel their contents
Right now, a client declares its supported version of a given object
automatically in the remoted calls it makes to conductor. However,
in the case of things like InstanceList.get_by_foo(), they are
reporting the version of their InstanceList object, not their
Instance object. Conductor fills a version-matching InstanceList
object with brand new Instance objects, which the client, of course,
barfs on.

There may be a better way to handle this going forward, but for now,
stop the bleeding by requiring a version bump to the corresponding
List object whenever the object type it contains takes a version
bump. This adds a test to validate that all the objects registered
have a suitable mapping for the current version in the tree.

Since this actually caused a breakage in the Instance object
recently, this also bumps the InstanceList version so that
conductors running this commit (or later) will properly send
version 1.9 Instance objects to Havana clients and version 1.10+
to newer ones.

Change-Id: I2668dead4784fbd0411d1b6372a9a8006eeb2e84
Related-Bug: #1258256
Closes-Bug: #1254902
2013-12-13 11:35:15 -08:00
Dan Smith
191d51f1ac Make Object FieldType take an object name instead of a class
The object infrastructure has a concept of an object name, which
is what we use to determine if something implements a particular
thing, and it is how we look up object classes when we
deserialize them. Since many objects will eventually reference
each other, we should be using their names to determine
compatibility, not their classes.

Related to blueprint structured-object-fields

Change-Id: I2edb8b8475662ad25e9f3319ce5e44317a2d0547
2013-10-21 14:26:25 -07:00
Dan Smith
a7a896590c Add missing key attribute to AggregateList.get_by_host()
This simply adds the ability to pass the key attribute to the
by-host lookup for aggregates.

Related to blueprint compute-manager-objects

Change-Id: I7651bc1470845b1e3586aa3616809bdf2ef2b4a2
2013-10-14 16:49:37 -07:00
Dan Smith
3122ceefa4 Migrate Aggregate object to Fields
Related to blueprint structured-object-fields

Change-Id: I0d1fab1e05ee748b4c8e7ec15e91aaff5eac1fda
2013-10-09 13:45:14 -07:00
Dan Smith
b963082431 Fix non-unicode string values on objects
Originally, all of the objects used str() to specify string type
attributes. This prevents the use of unicode values. A previous
patch errantly updated the utility function without bumping all
of the affected objects' versions. This does that, and also
fixes the issue for bare str() attributes.

Change-Id: Iad83589d399309b95c32813a1a660a12e571066b
Closes-bug: 1221346
2013-09-17 14:56:34 -07:00
Dan Smith
f407b79ae2 Clean up duplicated change-building code in objects
Almost every object thus far has done this:

  changes = {}
  for key in self.obj_what_changed():
      changes[key] = self[key]

to get a dict of updates to apply to the database. This patch adds
that as part of the base object and makes every place that does
the above to just use that.

Change-Id: I847f5d35181b0305668b107f86faa164e71c3375
2013-09-09 14:45:09 -07:00
Jenkins
5335937a04 Merge "Create mixin class for common DB fields" 2013-09-05 22:57:54 +00:00
Dan Smith
55f007be2c Generalize the _make_list() function for objects
Each object with a list duplicated the _make_list() method in its
own module. This removes that duplication and adds a generalized
helper in objects/base.py. The instance object still uses its own
because it has to do a bunch of other stuff in the loop for
efficiency.

Change-Id: Ic910f39087ebc167b2b930979f7951116caf8598
2013-09-04 14:27:11 -07:00
Chris Behrens
60507b90b9 Create mixin class for common DB fields
Creates a new DBObjectMixin class for DB-backed objects to use.  This
allows us to create non-DB-backed objects without having the common DB
fields automagically added to them.

Related to blueprint unified-object-model

Change-Id: Iaf2d8500505e9acdbffffb1d4bd3db0870ae82a3
2013-09-04 17:37:48 +00:00
Dan Smith
03b6412afb Add Aggregate object model
This adds an Aggregate object model implementation.

Related to blueprint compute-api-objects

Change-Id: Ic947718766181f8262fcf171b2bc0c329e0ae8f9
2013-09-03 13:03:38 -07:00