10848 Commits

Author SHA1 Message Date
Zuul
f776e49022 Merge "Add a doc describing a quick live environment" 2018-11-27 10:22:08 +00:00
Zuul
9b2fb83874 Merge "Documentation cleanup: front page" 2018-11-27 09:47:44 +00:00
Chris Dent
c28f87b8af Add a doc describing a quick live environment
Discussion in IRC suggested that
https://anticdent.org/quick-placement-development.html ought to do in
the contributor docs, so here is a version of it, translated from
Markdown to RST.

Change-Id: I7a014e96d2f4c01594d977c5a4af9894278dd511
2018-11-26 12:10:26 +00:00
Zuul
3ccbacfc01 Merge "Add assertions to verify placement-manage error output" 2018-11-26 06:22:41 +00:00
Zuul
19d9b2b2c5 Merge "manage: Do not use set_defaults on parent parsers with py2" 2018-11-23 13:59:44 +00:00
Chris Dent
176524643d Documentation cleanup: front page
This is the first in a series of changes to the content in
doc/source to make it more accurately reflect the current state of
placement, now that it is in its own repo.

There are small changes throughout the front page, mostly to make
it more clear and make useful stuff a bit more discoverable.

The two major changes are:

* Older (pre-stein) upgrade notes are found by linking back to
  nova's rocky documentation.
* The REST API history is moved to its own sub-page, otherwise it
  the page is overwhelming and noisy.

Close review is probably required. cdent typed this. That has
implications.

Change-Id: I66e0c7d18b253b0a5a8fdac65e30b5b3cef37db2
2018-11-23 08:44:14 +00:00
Chris Dent
da20d6101d Add assertions to verify placement-manage error output
The tests for placement-manage, in unit/cmd/test_manage were not
verifying help message output when partial or incorrect commands
were provided. This led to some confusion when trying to verify
a fix related to different behaviors of argparse.

Here, some verification is added. Because of differences between
python 2 and 3, we need to look at stderr and stdout, respectively, to
verify the output.

Change-Id: I337a25cfdc23bc56fbf15f128a94d2b1b1d1b664
Related-Bug: #1804420
2018-11-22 16:33:40 +00:00
Lee Yarwood
ce7b05a1f9 manage: Do not use set_defaults on parent parsers with py2
Doing so with cpython < 2.7.9 results in the following bug being hit:

argparse set_defaults on subcommands should override top level set_defaults
https://bugs.python.org/issue9351

As CentOS 7 continues to ship with cpython 2.7.5 this change simply
removes the parent parser call to set_defaults for py2 while leaving the
associated tests unchanged. This should ensure that the actual behaviour
of placement-manage is unchanged while allowing these tests to pass once
again on CentOS.

Change-Id: I87989ced31b1d39e30c992b34fe2ae6db06f7883
Closes-Bug: #1804420
2018-11-22 14:55:53 +00:00
Takashi NATSUME
b4d292586c Fix a bug tag for placement doc
The default bug tag for placement doc is defined
as 'docs' currently in doc/source/conf.py.

But it should be 'doc'(*1) instead of 'docs'.
So replace 'docs' with 'doc'.

*1: https://wiki.openstack.org/wiki/Nova/BugTriage#Tag_Owner_List

TrivialFix
Change-Id: I469ad687f655b96f7a0be11e1b92e30a31e3184d
2018-11-22 14:38:37 +09:00
Zuul
ddd993de1a Merge "Consider root id is None in the database case" 2018-11-21 04:17:33 +00:00
Zuul
6a715bec2e Merge "Add recreate test for bug 1799892" 2018-11-21 04:14:24 +00:00
Zuul
d43ebed4a5 Merge "Add placement-status upgrade check command" 2018-11-20 20:52:53 +00:00
Matt Riedemann
3ff9d0a5a0 Add placement-status upgrade check command
This adds the basic framework for the
placement-status upgrade check command which
is a community-wide goal for the Stein release.

A simple placeholder check is added which should
be replaced later when we have a real upgrade
check.

Change-Id: I9291386fe7fcbfc035c104ea9fdbe5eb875c4776
Story: 2003657
Task: 27518
2018-11-20 13:53:07 -05:00
Tetsuro Nakamura
b8972a3e45 Consider root id is None in the database case
There are cases where ``root_provider_id`` of a resource provider is
set to NULL just after it is upgraded to the Rocky release. In such
cases getting allocation candidates raises a Keyerror.

This patch fixes that bug for cases there is no sharing or nested
providers in play.

Change-Id: I9639d852078c95de506110f24d3f35e7cf5e361e
Closes-Bug:#1799892
2018-11-20 11:52:43 -05:00
Chris Dent
2c72b94719 Adapt placement fixtures for external use
This change makes it easier to use the PlacementFixture from external
systems (mostly nova) that would like to run functional tests with a
stubbed in placement service, including a database.

A 'set_config' boolean flag is added to the Database fixture to allow
the caller to indicate they have not themselves set a connection string
in the placement_database group and want the fixture to do so.

That fixture attempts to register the group and opt but allows that to
fail if it has already been done. This provides some resiliency in the
face of nova currently having the group and option but not having it
later.

Because both nova and placement use global config, we need to avoid
registering and processing all of placement's config options anywhere
during the use of the fixture (from nova), otherwise errors will
happen. Therefore, we cannot use migration.upgrade in the Database
fixture, as env.py currently pulls in and processes a global config.
Instead we use a create_schema method. Migrations continue to be
tested by the migration specific tests.

The necessary functionality of syncing traits and resources classes is
handled by the fixture itself, rather than things that call it, allowing
some cleanups in other fixtures. This centralizes database handling in
one place, which is probalby how it should have been all along, but as
with all evolving things, it meandered.

The PlacementFixture is updated to use the Database fixture
automatically, unless a db keyword arg is set to False.

Change-Id: I7d4099c1da425bb4ea115ac51c82f5748203f2d3
2018-11-20 15:00:58 +00:00
Chris Dent
7b5218d8e3 Add a placement-manage CLI
This change adds a rudimentary command line tool, 'placement-manage'
that can do three things:

* placement-manage --version: report the current version of the software
* placement-manage db sync: sync the db to the 'head' alembic version
* placement-manage db version: report the current version of the db

It is written following the examples set by both nova and ironic for
using oslo config driven command line parsing. It is not as full
featured as the nova version (with decorators for argument handling and
help text etc) because we don't need that yet. There's plenty of room
for improvement, but the intention here is to keep things simple until
we need them to be otherwise.

The provided unit tests cover that the arg parsing behaves as expected
and that the right commands are connected the right args. The actual
functionality of the 'db sync' command (a migration.upgrade) is tested
in runs that use devstack as well as the test_migrations functional
test.

Errors during the migration simply raise exceptions which will output
noisily to the console. At this stage this is desirable and suitable.

A new doc for the cli is added, with basic information for
placement-manage. Note that at this time the deployment documentation
has not been updated. That will be done in a later change.

Future changes may include (if we can't avoid it) tools for initiating
online migrations from the command line.

Needed-By: https://review.openstack.org/600162/
Change-Id: I1ff472106610150a587f5286f26a6bd7c1aa84f4
2018-11-20 15:00:32 +00:00
Zuul
f63b14c6a4 Merge "Remove build-openstack-api-ref jobs" 2018-11-19 23:34:18 +00:00
Zuul
41c328dbfc Merge "Remove sqlalchemy-migrate from requirements.txt" 2018-11-19 21:14:55 +00:00
Zuul
daaefd2c87 Merge "Fix comment in earlier patch" 2018-11-19 21:14:49 +00:00
Zuul
8aa66c4f2a Merge "Add a document for creating DB revisions" 2018-11-19 21:14:49 +00:00
Zuul
184483d562 Merge "Delete the old migrations" 2018-11-19 19:15:45 +00:00
Zuul
962dd108d9 Merge "Added alembic environment" 2018-11-19 19:06:13 +00:00
zhufl
ca89a614c3 Add missing ws seperator between words
This is to add missing ws seperator between words, usually
in log messages.

Change-Id: I2b7a752c19a37bba06cf13b6119939ce6c755835
2018-11-19 15:14:19 +08:00
Chris Dent
d4d0e52100 Remove sqlalchemy-migrate from requirements.txt
With the migrations now gone, we can safely remove sqlalchemy-migrate
from requirements.txt and lower-constraints.txt.

Change-Id: Id8d03d9532bcf4c4f7ad1c1039fa6a42990f8d5d
2018-11-15 05:12:20 -05:00
EdLeafe
70190fa438 Fix comment in earlier patch
An earlier patch [0] noticed an incorrect comment in the migration
script. This patch corrects it.

[0] https://review.openstack.org/#/c/614350/18/placement/db/sqlalchemy/alembic/versions/b4ed3a175331_initial.py@17

Change-Id: Idb23ebeaffb520f337323c30882059a23af0117b
2018-11-15 05:12:01 -05:00
EdLeafe
1b610b0d83 Add a document for creating DB revisions
This gives some basic documentation for creating a revision using
alembic. It then links to the alembic tutorial for more information.

Change-Id: I8647aaa55dca537d95c7c03753f2a24f364f0ddf
2018-11-15 05:11:32 -05:00
Takashi NATSUME
dbaf410349 Remove build-openstack-api-ref jobs
The api-ref-jobs has been added
since Ia4680f24d78af1260f2f0106a458b78a079c1287.
It includes build-openstack-api-ref jobs in check and gate.

So remove the build-openstack-api-ref job definitions
in .zuul.yaml because they are unnecessary.

Change-Id: I4b612fa87609240bd0a0f838163ee87c45a41a46
2018-11-14 15:46:45 +00:00
EdLeafe
b612f24b66 Delete the old migrations
As the first step to implementing alembic for our data migrations, we
need to get rid of the old migration files.

Change-Id: Ib6ee4895a3c5b18b027f1ac44905dff0e28cdcf8
2018-11-14 15:26:44 +00:00
EdLeafe
c8ab9ff906 Added alembic environment
Added alembic, and created an initial database revision.

For the sake of testing with real databases, tools/test-setup.sh is added
so that CI can start up a mysql and/or postgresql database on which to
test migrations. The OpportunisticDBMixin from oslo_db was originally
used for this, but because of the way placement configures the database
this proved hard to debug and get correct, so we've used something
that leverages the available oslo_db tools, but with more visibility.

Because these tests mix sqlite, mysql and postgresql settings in the
potentially the same process we need a way to insure that global
settings for databases do not leak into other tests. This is done with
a reset() on the placement db fixture, called by the msyql and
postgresql tests before and after they run. We also need careful
management of that cleanup when these tests are skipped (because db
server or database is not there).

Those tests will confirm that the models match the migrations so we also
need to remove model files that no longer matter.

Since we no longer need to distinguish among multiple database files, we
can simplify the naming of these files.

Co-Authored-By: Chris Dent <cdent@anticdent.org>
Change-Id: I51ed1e4e7dbb76a3eab23af7d0d106f716059112
2018-11-14 15:26:13 +00:00
Tetsuro Nakamura
917d7881ec Add recreate test for bug 1799892
There are cases where ``root_provider_id`` of a resource provider is
set to NULL just after it is upgraded to the Rocky release. In such
cases getting allocation candidates raises a Keyerror.

This patch recreate that bug by simulating the situation by
inserting the records to the database directly.

Change-Id: Iaed912314f3e8fef2f46453a6bf12011702ae1dd
Related-Bug:#1799892
2018-11-09 15:51:10 -05:00
Sean Mooney
6bb6c0bbdd Harden placement init under wsgi
- This change tries to address an edge case discovered
  when running placement under mod_wsgi where
  if the placement wsgi application is re-initialize the db_api
  configure method attempts to reconfigure a started transaction
  factory.

- This since oslo.db transaction factories do not support
  reconfiguration at runtime this result in an exception being
  raised preventing reloading of the Placement API without
  restarting apache to force mod_wsgi to recreate the
  python interpreter.

- This change introduces a run once decorator to allow annotating
  functions that should only be executed once for the lifetime of
  an interpreter.

- This change applies the run_once decorator to the db_api configure
  method, to suppress the attempt to reconfigure the current
  TransactionFactory on application reload.

Co-Authored-By: Balazs Gibizer <balazs.gibizer@ericsson.com>
Closes-Bug: #1799246
Related-Bug: #1784155
Change-Id: I704196711d30c1124e713ac31111a8ea6fa2f1ba
2018-11-08 12:49:14 +00:00
Zuul
ac0a732106 Merge "Make tox -ereleasenotes work" 2018-11-08 02:31:37 +00:00
Zuul
34b18b7c99 Merge "Add bandwidth related standard resource classes" 2018-11-07 13:42:02 +00:00
Chris Dent
565eab5ee5 Make tox -ereleasenotes work
One of the things Placement will need when it is properly publishing
itself is a working release notes job. Until this change there was
no source/conf.py or source/index.rst for releasenotes job to use.
This adds it, we can now create release notes output, with the
openstackdocs theme.

However: Because the placement repo has no branches, all the existing
release notes in releasenotes/notes are considered 'unreleased', which
makes for a lot of noise at the page generated by unreleased.rst.

What this probably means is that we will want to clear out all the
release notes that pre-date extraction and start anew. An experiment to
try removing some notes shows that the releasenotes will produce some
messages about missing release notes, but neither warn nor error, and
the output is good, so this should work out okay for us.

That will be easy to do now that we have a working releasenotes tox env.

Note: This change does not add any jobs to zuul, it just makes tox
work for releasenotes.

Change-Id: I61a7ce492395e0ae69dd0401a141ea4bb93516b9
2018-11-06 16:34:26 +00:00
Matt Riedemann
3f34f10cf6 Add bandwidth related standard resource classes
Introduce NET_BW_EGR_KILOBIT_PER_SEC and
NET_BW_IGR_KILOBIT_PER_SEC resource classes.

blueprint bandwidth-resource-provider

Co-Authored-By: Balazs Gibizer <balazs.gibizer@ericsson.com>

Change-Id: I996bf705b14b564106426a2e57299638fb178750
2018-11-05 17:56:19 -05:00
Chris Dent
5d8aa50074 Move ensure_consumer to a new placement.handlers.util
The driving reason for this change is to allow the placement.db_api
module to be able, in subsequent patches, to import generic utility
routines from placement.util without causing an cyclical import.

It is moved into the handlers package because the only caller is
in there too (in allocation.py).

Relevant unit test are moved, and the callers of ensure_consumer are
updated to reflect the new location.

Change-Id: Ifd1b10e5568b9d6f641e396c3c49101901daff31
2018-11-02 10:30:14 +00:00
Zuul
83e2ce65a0 Merge "Add a link to "Add Generation to Consumers" spec" 2018-10-30 16:29:14 +00:00
Zuul
2dd40b9bd0 Merge "fix wrong spelling of "explicit"" 2018-10-30 12:28:45 +00:00
wangqiangbj
888311992c fix wrong spelling of "explicit"
Change-Id: I6949ff5898adb31f27b3b307b31f81d0e55a9204
2018-10-30 19:16:32 +08:00
wanghui
1370c54b66 Fix the error package name in comment
Change-Id: I123dd422f44e9bd4f7a75f78e1f0e985b47caabd
2018-10-30 18:59:45 +08:00
Takashi NATSUME
117f44d215 Add a link to "Add Generation to Consumers" spec
The "handle allocation updates in a safe way" item and
the link to "Add Generation to Consumers" spec have been added
since I77f34788dd7ab8fdf60d668a4f76452e03cf9888 in the nova doc
(doc/source/user/placement.rst).

But it has not been added yet in the placement doc
(doc/source/index.rst). So add it.

Change-Id: I4c70da34d8907cab2373b1349bb2c6e9b8cdd35a
2018-10-30 18:09:35 +09:00
Takashi NATSUME
5490d4cbce Publish placement documents
Use publish-openstack-docs-pti job instead of
build-openstack-docs-pti job to publish placement documents
as https://docs.openstack.org/placement/latest/ .

Change-Id: I667387ec262680af899a628520c107fa0d4eec24
2018-10-30 00:05:56 +09:00
Tetsuro Nakamura
caef44bf25 De-nova-ify doc/source/index.rst
Some of the descriptions in the file ``doc/source/index.rst`` assume
placement still depends on nova, which isn't true, so this patch
updates it.

This patch also adds a upgrade note for Stein (19.0.0) in that file.

Change-Id: Ib98e370d6863cc8c679a07d04f053ba5a3723c43
2018-10-16 16:46:58 +09:00
Tetsuro Nakamura
8fdf74bace De-nova-ify doc/README.rst
The file ``doc/README.rst`` hasn't been changed since the placement
is extracted, so this patch de-nova-ify it.

Change-Id: I30bcfa987a87fd4672dcbcdf3f401295b2dad6f5
2018-10-16 10:58:23 +09:00
Tetsuro Nakamura
5f2dc9aa2d Clean up .gitignore file
When you create the placement document locally with `tox -e docs`,
some of the files are created under the `doc/source/_static` directory
and are tracked by git because the .gitignore file is not set
appropriately. This patch fixes it cleaning up old nova configurations
in the .gitignore file.

Change-Id: I1ecae90c9ba1cccdcd4befec3513ba2fdacb4cea
2018-10-16 00:14:36 +09:00
Zuul
2d974bac33 Merge "Fix genpolicy tox job" 2018-10-23 01:07:54 +00:00
Zuul
04847ffe6b Merge "Use unique consumer_id when doing online data migration" 2018-10-22 22:13:03 +00:00
Zuul
e877e08d29 Merge "Add recreate test for bug 1798163" 2018-10-22 22:12:29 +00:00
Chris Dent
8e35c6fe4e Fix genpolicy tox job
The etc/placement/policy-generator.conf file still referred
to nova. Now that it does not, policy generation happens as
expected. The name of the generated file has been changed
to policy.yaml.sample, as we no longer need to distinguish
from nova's.

Note that the readme still points to nova docs, because that
is where we currently have the docs. This will need to change
later.

Change-Id: I633431933666c15eaf8da7e91769d0b4335e6c8d
2018-10-22 15:23:54 +01:00
Yikun Jiang
e7fe0ab862 Add nova database migration script for postgresql
This is based on the mysql-migrate-db.sh script but
with the necessary changes for interacting with postgresql.

Long-term it might be possible to refactor both the mysql
and postgresql scripts into a generic script but the
potential complexity in doing that makes it a lower priority
than simply getting pg support for the nova_api/placement
table migration done in Stein.

Change-Id: I1cab3183067ab2e41663ca369509d753442e775c
2018-10-17 11:17:19 -04:00