385 Commits

Author SHA1 Message Date
Scott Hussey
eb996c27f4 (format): yapf reformat
- Run yapf to correct formatting
- Update one variable typo

Change-Id: Ic756ce4f9811cdf2726907c54299f4740fca9ce5
2018-07-27 20:16:25 -05:00
Scott Hussey
b53a3ff0ea Add support of MAAS node-results API
- Support extraction of log data from MAAS

Change-Id: I55fcbad5e06378f148c9d5de476e1eef8d898ed2
2018-07-27 20:16:25 -05:00
Scott Hussey
cff7420cff Support links for task status
- Some status changes in a task may have additional information that
  is referenced by a URI link. Support describing these links and
  returning them via API.
- Refactor alembic stuff to better handle table schema updates
- Add unit tests

Change-Id: Iae63a9716f2522578be0244925fc274a4338eac4
2018-07-27 20:16:24 -05:00
Scott Hussey
e27eaf94f5 (zuul) Consolidate tests
- Consolidate tests to minimize Zuul node consumption
- Remove WS linting as content-specific lints should catch
  this if it is valuable

Change-Id: I18065027644b8412d88219a40d490c12d558969b
2018-07-20 08:52:44 -05:00
Scott Hussey
4423ee985e Add additional repo options
- Some repos are built without components that MAAS
  assumes are there. This allows the configuration to
  describe that so MAAS can be updated to disregard
  those components

Change-Id: I4eafe9fb8278b944bcb993f10379966a01e94bf6
2018-07-14 14:23:34 -05:00
Aaron Sheffield
69a6f80031 Add test pods labels.
- Uses helm toolkit to add labels to test pods.

Change-Id: I60009c3c502c76d77f9edbf64d93118079f94253
2018-07-11 08:05:10 -05:00
Scott Hussey
06d6747b50 Make client HTTP connections resilient
- Make Keystone session use a timeout to prevent hangs
- Support retries
- Make the above configurable

Change-Id: I7123bd2fdcd329eae5b8b40f09168a1d599fa0f7
2018-06-26 14:41:24 -05:00
Scott Hussey
7e7739ce2c (fix) Guard against DB connection leaks
- Use connection contexts to ensure database connections
  are released back to the pool.
- Make database connection pool size configurable, defaults to 15

Change-Id: Id8011fbf45a1b3c87835ff5f47ebfa9334488319
2018-06-26 14:38:40 -05:00
Zuul
fec226fc66 Merge "Cleanup yaml files" 2018-06-22 21:35:34 +00:00
Zuul
342bb1d345 Merge "Unifying proxy variables for tox and docker build" 2018-06-22 17:17:54 +00:00
Roman Gorshunov
fa07ea4f4d Cleanup yaml files
Cleanup yaml files.

Change-Id: Iff474e33a12632ad74cfaaccc9152bb5066b90b1
2018-06-22 18:49:59 +02:00
Zuul
6af445ac4a Merge "Improvements for handling existing nodes" 2018-06-22 15:56:04 +00:00
Roman Gorshunov
7613f1c624 Unifying proxy variables for tox and docker build
This patch adds proxy-related variables to tox and 'docker build'
commands.

Change-Id: I29fee630bd4c98406797ffec93f8f80bf1a3a043
2018-06-22 14:45:51 +02:00
Scott Hussey
da60e2d127 Improvements for handling existing nodes
- Additional improvements in how Drydock handles nodes
  that are already known to MAAS.

Change-Id: I893348e0fd8a1907ebe35e33edf64e622dd79feb
2018-06-21 17:31:03 -05:00
Zuul
b93cf21fa2 Merge "(fix) Fix invalid kwarg" 2018-06-18 23:54:36 +00:00
Scott Hussey
a5cfdd25ca (fix) Fix invalid kwarg
Incorrect kwarg used.

Change-Id: Id65e37d1051d833d9a7c3a95efc73e8bfbbeab80
2018-06-18 16:30:33 -05:00
Zuul
f738bd4a1c Merge "Fix name of error CollectSubtaskTimeout" 2018-06-18 16:19:45 +00:00
Scott Hussey
396f1da5e0 (fix) Make noops considered a success
When a task for prepare_nodes encounters a node that
is already managed by the node provisioner, it skips
additional interactions to prevent disturbing existing
nodes. Add these nodes to the success list for the
task so upstream workflows can validate that a described
node was found and explicitly considered.

Change-Id: Ia387ab4d6fc0d9ad3838b9df54f46d6921d12a42
2018-06-18 09:20:57 -05:00
Bryan Strassner
2fc3e22dc2 Fix name of error CollectSubtaskTimeout
Was misnamed in the error.py as CollectSubaskTimeout (missing the 't' in
task)

Change-Id: Id57130955f7f68b535fda1616c8aac6b118ba1f3
2018-06-17 15:23:31 -05:00
Zuul
b4a31a79de Merge "[411429] maasdriver support of packages" 2018-06-14 14:56:47 +00:00
Mark Burnett
9dd1722368 Order hugepagesz and hugepages kernel parameters
These kernel parameters must be specified in order.

* Moves kernel parameter string from the maas driver to the model
* Forces hugepagesz to occur immediately before hugepages

Change-Id: Id0cc18b73bde17f9bfac980b0eefe37699e86eff
2018-06-13 12:41:08 -05:00
Scott Hussey
fda50be35e [411429] maasdriver support of packages
- Support pkg_list bootactions in the MAAS driver by using
  cloud-init user_data on deployment
- Add site definition caching to ease load on Deckhand

Change-Id: I2c8c7dfdd23992fae42fa32edab308f801d05867
2018-06-12 09:58:02 -05:00
Scott Hussey
e35712a573 [411430] Validate bootaction pkg_list
- Add a validator for bootactions to warn if a node doesn't
  have at least one
- Add a validator for bootactions to error if a package
  version specifier is invalid
- Unit tests for the validation

Change-Id: I61d8aa3831791af0484498e6fe9f7c1c83dbf540
2018-06-12 09:58:02 -05:00
Scott Hussey
1b0797440b [411428] Bootaction pkg_list support
- Support a list of debian packages as a bootaction asset
- Add unit testing for parsing the additional bootaction information
- Add __eq__ and __hash__ for DocumentReference to allow checking
  equality and list presence

Change-Id: I0ca42baf7aae6dc2e52efd5b311d0632e069dd79
2018-06-09 19:53:43 +00:00
huang.zhiping
cbd96b13fe fix tox python3 overrides
We want to default to running all tox environments under python 3, so
set the basepython value in each environment.

We do not want to specify a minor version number, because we do not
want to have to update the file every time we upgrade python.

We do not want to set the override once in testenv, because that
breaks the more specific versions used in default environments like
py35 and py36.

Change-Id: I0f865164b2e02577d5e57fc1fef836584dc627ea
2018-06-09 02:12:08 +00:00
Zuul
7f3a580c59 Merge "(zuul) Enable docker image jobs" 2018-06-08 22:02:40 +00:00
Mark Burnett
714528a549 Update Formatting with YAPF
Change-Id: I351f55b5ffbbae8c15d529777bfac058054c43c9
2018-06-08 13:24:26 +00:00
Scott Hussey
a577164443 (zuul) Enable docker image jobs
- Check/gate jobs to build docker image
- Post job to publish image to quay.io

Change-Id: I44b813a5bd38965208affcee12709e097d3d1665
2018-06-08 05:15:05 -05:00
Scott Hussey
6dad448ca6 [411390] Configure repo in MAAS
- Add a new action for ConfigureNodeProvisioner to configure a node
  provisioner with site-wide configuration
- Add a maasdriver action to configure repositories

Change-Id: I8e216a269b300159b7cc26c3a4542e8b61496dc7
2018-06-08 02:45:14 +00:00
Scott Hussey
cc77125953 [411387] Schema update to support repo
- Support one or more repo specifications for a site
- Add object model for repository
- Add testing for repository parsing

Update freeze job with make target

- Update the requirements freeze job
  to have a mark target that rebuilds the tox
  virtualenv each run
- Update Dockerfile to create a valid /etc/protocols file

Change-Id: I9d09b7dd7226827995e23756ff968b36eaa4d16c
2018-06-07 23:08:20 +00:00
Mark Burnett
786271af5c Fix: use only node's HW profile for nic logical names
Change-Id: Ib0d4c2e341ffb8c7a367d1a48cc7e99d60f70ef6
2018-06-05 12:03:38 -05:00
Zuul
def51688e4 Merge "Update docs for developer overview" 2018-06-01 20:47:09 +00:00
Scott Hussey
20873ad4f9 Update docs for developer overview
- New diagrams and documents for developer overview
- Update conf.py for docs to work w/ readthedocs.io
- Add policy and config gen to `make docs`
- Update zuul-linter to support checked in images
- Last fix to document publishing

Change-Id: I4faa1b87032ae5b0e786aa0fd998f809124b7987
2018-06-01 14:50:05 -05:00
Felipe Monteiro
e1463c6b79 [trivial] Rename tox jobs for zuul
This patchset renames the tox.ini jobs:

* coverage => cover
* lint => pep8
* unit => py35

to comply with OpenStack standards [0]. This is needed so that [1]
works seamlessly (it will enable py35 and cover jobs for Drydock).

[0] e.g. 04469a5181/tox.ini (L119)
[1] I174d5df008f6e000da1a3878afe75919312ea7aa

Change-Id: I9ead2156cde9b1bbbec8374845653825158dec19
2018-05-31 17:00:32 +01:00
Scott Hussey
c85e6ad668 Fix playbook to provide logs
Change-Id: I5d670634c8be5e5f577f5b5a5b69e5f6c681c351
2018-05-31 09:12:03 -05:00
Scott Hussey
fec2ecfb80 More logs on doc publish
Change-Id: I36e162f4e5535bb20ecb616fcdcd5b604daf87a0
2018-05-30 16:58:52 -05:00
Scott Hussey
e594ca7935 [Fix] Fix post-merge doc publish job
- Add secret attachment to the job

Change-Id: I21b7e83abc3a457e88e888be3609bb78cec23cb2
2018-05-30 15:42:56 -05:00
Scott Hussey
5ee84cc622 Log on doc publish job
- Allow logs on publish job to understand failure. Credentials
  will be promptly changed.

Change-Id: I3d9f2fb260f93269db4b26ec3a27b77a577da8bb
2018-05-30 15:07:07 -05:00
Scott Hussey
708fedf70e Documentation gating
- Add gate for doc build per change and on merge
- Add automation for document publishing

Change-Id: If82ce4c97da5f2c47ba5de050b8f3982bd08d30b
2018-05-29 15:48:36 -05:00
Scott Hussey
53f6ccbfb7 Clean up gates
- Remove OSH-based gate as it is not relevant at this point
- Add entries for unit testing, linting and security scans
- Some trivial changes to trigger all gates

Change-Id: Icb2731573b45f3824cde8cb44ef64e9d04c017bc
2018-05-25 16:36:45 -05:00
Scott Hussey
d107e65a98 Libvirt support in maasdriver
- Add validations that OOB configs for nodes are valid for the
  oob type defined
- Add documentation for using Drydock/MAAS to deploy libvirt VMs
- Add logic to update the MAAS node power parameters to allow power
  control of libvirt VMs

Change-Id: Ia7d5fbd1659636d46cf1790fe3fc66ca6c6fee89
2018-05-25 11:13:53 -05:00
Scott Hussey
dbad75775b Libvirt OOB driver
- Create a driver to support OOB actions via libvirt API
- Update Makefile with external dependency target
- Update Makefile and tooling to support new chart pipeline
   - Add 'drydock' make target for chart building
   - Add step to install helm binary

Change-Id: I8a3984d8fd70f99a82a954b7a869eab8e30145b4
2018-05-25 11:13:53 -05:00
Scott Hussey
cd9770a979 Zuul: Initial Airship-Drydock checks
This PS adds the skeleton for a set of zuul checks and gates for
Airship, using the framework from OpenStack-Helm.

[Fix] Zuul lint fix

- Fix whitespace linting errors
- Update docuemtation errors
- Remove dead tests

Change-Id: Id1ed3aa2321d518ed15fcbb3b698b49709bed287
Signed-off-by: Pete Birley <pete@port.direct>
2018-05-24 16:59:12 -05:00
Aaron Sheffield
ae6a990940 Updating requirements-lock.txt
- Ran tox -e freeze to update requirements-lock.txt.  Removes pyinotify.

Change-Id: I87c62bc6ec4b0b746e60eef70e69762cb37b5c9e
2018-05-17 15:37:28 -05:00
Felipe Monteiro
e7c2ed57d6 Update .gitreview for openstack infra
Change-Id: Ie0486066f738d52a1fa5df94ef78c7d14b3764bc
2018-05-17 19:23:48 +01:00
Aaron Sheffield
506e06623a Added ValidateDesign to API Client
- Added ValidateDesign to API client.
- Added ValidteDesign to CI client.
- Added a couple unit tests.

Change-Id: I1def14b3703eba12b10a063bf09a3120dcf54b8d
2018-05-14 10:54:26 -05:00
Aaron Sheffield
8274d56c71 Handle no baremetal_nodes in process_node_filter
- Added a try when getting baremetal_nodes from site_design.
   If baremetal_nodes isn't there it will return an empty set.

Change-Id: I413892274d938ffd23102c73a569b83a59025206
2018-05-10 09:45:16 -05:00
Aaron Sheffield
8af92eaf29 Change POST nodefilter Input
- Changed the expected input from site_design to design_ref.
- Converts the design_ref json/dict into a SiteDesign object.

Change-Id: Ie1f6738fec64749cde9d2843d9373175006794c0
2018-05-08 10:32:26 -05:00
Aaron Sheffield
d052664f74 Add Drydock API Client Options
- Added builddata, subtaskerrors, and layers as options to get_task.
- Added get_nodes_for_filter to call POST /nodefilter.
- Updated endpoint for POST /nodes to POST /nodefilter.
- Made node_filter optional in POST /nodefilter.

Change-Id: I456a6e9991d03af3d375c448f5cbf07a21e91f1d
2018-05-08 14:11:15 +00:00
Scott Hussey
3b41868802 [373577] Fix hostnames with underscores
Hostnames with underscores caused a deployment failure.

Update to use a double underscore as a delimiter and provide
a design validator to check that hostnames do not contain it.

Closes #78

Change-Id: Ib148aed5cffe7fd8bc08441eaef8a45af6601bdd
2018-05-03 15:42:33 +00:00