During the recent Airship Working Committee meeting, the committee
addressed feedback from the Airship confirmation review [0]. One such
item was concerned with copyright footers mistakenly claiming rights to
all Airship documentation.
This change updates the footer to attribute documentation to the
Spyglass authors.
[0] https://etherpad.openstack.org/p/airship-wc-meeting-2019-12-09
Change-Id: If9db77b1bc6a0abc710898f5f9c0d2ff7fc25bdf
Signed-off-by: Drew Walters <andrew.walters@att.com>
Updates CLI commands to match current code
Fixes title formatting on README
Pins six back to 1.12.0 for OpenSUSE incompatibility
Updates package dependencies list
Change-Id: Iedf6a438fbe441315dc1124364eea38056090065
Upgrades yapf to newest version, 0.28.0, and runs the formatter to
update all existing code.
Adds entry to .gitignore to ignore pyenv installations.
Change-Id: I11512a8a522cc530165461cc8f52f7ff010dd092
Pipenv validates that the Pipfile.lock that it uses for installing
packages is not out of date before following through on execution. If it
is not, it will throw an error. This change adds an env that tells
Pipenv to ignore the Pipfile and only use Pipfile.lock so gates do not
fail due to out of date packages.
Change-Id: I2fa541b31e14561c7cefc330e7ef2cc0b0e4abc6
When using pipenv, the targeted version must be used when attempting to
do pipenv check. This change updates the safety dependency check to use
openstack-tox-py36 so the correct python version will be used.
Change-Id: I4b9f71cfaaa74724ecee7b40ea6e9362e836d261
This change addresses some of the TODOs made in the Spyglass engine.
There will be additional follow-up patchsets that will address issues
with the rules engine and intermediary validation.
Change-Id: Iba70a51d291659bf827e46fc9070a898303082d1
Zuul gates broke for Spyglass due to an update made to either Zuul or
Openstack's Zuul jobs (I wasn't able to pin it down). The update caused
Zuul to attempt to install Spyglass's requirements.txt file instead of
referring to the method given in the tox configuration that utilizes
Pipenv. To bypass this issue, the requirements.txt will be removed and
Spyglass will be fully moved over to using Pipenv.
Removes requirements.txt
Updates Pipfile and Pipfile.lock for new package releases
Updates Dockerfile to install requirements from Pipfile.lock
Change-Id: I347ef6db18a44e1b88c811aa071ba8c60bccbd53
From recently merged document updates in [0] there is a desire to
standardize the Airship project python codebase. This is the effort
to do so for the Spyglass project.
[0] https://review.opendev.org/#/c/671291/
Co-Authored-By: Ian Pittwood <pittwoodian@gmail.com>
Change-Id: I2b0f2491d83675d7742b168ecb9e1dfb9a11e719
This change adds a couple tests for init in Spyglass' site processor
class, bringing total test coverage up to the goal of 90%.
Change-Id: I81c1dfdf2a9d65589d961451401802621058a9f2
The pyup.io database that is used to check dependencies for
vulnerabilities in Spyglass is currently down. When first implemented,
this check was placed in the tox pep8 job. This change moves the
vulnerability check into its own Zuul job so it can be disabled in
instances such as this where the database is down.
Specifies basepython to python3 for safety and bandit jobs.
Change-Id: I4e13c5341a9e62095587ec1820be79a621380f09
This change makes some mild restructuring and naming changes to the
data-extractor plugin base. Data extraction will now be separated into
two main workflows: loading raw data and source specific parsing into
model objects. Naming of methods was changed to help more accurately
reflect their function in these workflows.
Change-Id: Ia3bc892994ff96cce5c1672fadf35ef2d1c4164b
Increases test coverage of the Spyglass parser engine from 0% to 82%,
bringing overall test coverage to 86.4%. This change moves minimum
coverage to 84%.
Change-Id: I2de496b8d7f4c4252be22c713605fae6fd565b66
Plugins currently use a couple configuration methods to set up field
data for the class. This seems superfluous. This change moves these
configuration steps into the init method so plugin classes can ingest
data as kwargs.
Here is an example of how the change will be implemented in plugins:
https://review.opendev.org/#/c/670171/
Change-Id: Ib26636f1eb4146902ee801af5bcce53d137be2ad
This change took place in Pegleg, adding to Spyglass for project
consistency.
Pegleg change: https://review.opendev.org/#/c/670421/
Change-Id: I1c3610ce6041393c94252629194295cc28eb129e
This change moves all the shared CLI options between Spyglass and its
plugins back into the main Spyglass CLI file. The plugins will then
reference the main CLI file for these options instead of redefining
them in a future change.
Change-Id: I9a6fe7a7d84fed71c372beea9cf7d74a2f6430b1
Sphinx 2.1.0 has a bug [0] that causes whitespace to be excluded in
outputted docs. This change updates Sphinx to peg any version >2.1.0.
Safety dependency vulnerability checks now will also cover the doc
requirements.txt.
[0] https://github.com/sphinx-doc/sphinx/issues/6440
Change-Id: I35f1acf4385821969ffa7c9807cf209a59c1d73e
Spyglass has three different files that define exceptions. This change
merges all of those definitions into a single file for simplicity.
Before this change can be merged, spyglass-plugin-xls needs to move the
exceptions needed for its functions.
Related Change: https://review.opendev.org/#/c/667243/
Depends-On: I81c5ca2d9083aece3641bc8b5405dfd44baec810
Change-Id: Ibf34771653247850903e529beb17dbd60ba35fbf
Voting was previously turned off due to instability with OpenSUSE
mirrors. This appears to have been resolved over the last week so
this patch re-adds voting.
Change-Id: I293d298996468fee43db410ea54911aef7852a9a
Pipenv is a tool that brings better package dependency management to
python. It can automatically create and manage virtualenv as well as
managing package dependencies using Pipfile and Pipfile.lock. Adding
this dependency manager into Airship projects will decrease package
version conflicts between projects and help increase security through
hash validation of packages and vulnerability scans.
Changes:
- Imports requirements.txt type files into Pipfile
- Pipenv dependency management in tox
- Switches Safety package for "pipenv check", an implementation of
Safety
- Unpins or loosens pins on all development packages
- Soft pins package dependencies to encourage adoption of bug fixes and
prevent small version mismatches
Pipenv Docs: https://docs.pipenv.org/en/latest/
Helpful Pipenv Guide: https://realpython.com/pipenv-guide/
Change-Id: I2c9cec8acf6b6c6157f807b010def873c349d3ae
Spyglass uses additional configuration files to add in any information
not included by a data source. This configuration file should be usable
independent of the data source used. This change adds the
site_config.yaml example file back into the examples folder after it was
initially removed in the project separation of spyglass-plugin-xls.
Change-Id: I9be76c89076327b8ff928aa3da6c91cca0397cc2
This is a follow-up change to [0] which further implements data objects
to be used in the generation of documents in Jinja2. The following
additions and changes are made:
- Adds helper functions to data objects to filter hosts and networks for
ease of use in Jinja2 templates
- Adds SiteDocumentData factory function to convert intermediary yaml
dictionaries into a SiteDocumentData object with all associated objects
- Updates Jinja2 templates to use data objects
- Cleans up overly complex looping in Jinja2
- Adds tests for new code in models.py
[0] https://review.opendev.org/#/c/662092/
Change-Id: I66ebfeaf5d6ca76b6dee5a2285a74bad8b06b720
This change implements data object models from [0] in data extraction
and parsing. The change results in minor modifications to the outputted
intermediary, which can be seen between these two example intermeidary
files [1].
This fully implements the data objects from models.py in data extraction
and parsing. A follow-up change will implement use of the data objects
in Jinja2. Temporarily, all objects will be converted to dictionaries
for generating documents from templates.
[0] https://review.opendev.org/#/c/658917/
[1] https://www.diffchecker.com/NnjjJrb2
Change-Id: Ifd867787aab541be5dabecf9f6026faa2ec7049e
There's been issues in the past week with openSUSE image builds timing
out. This may be due to issues with openSUSE's infrastructure [0]. For
the time being, we should disable the openSUSE docker build until we are
able to consistently build without failures.
[0] https://status.opensuse.org/
Change-Id: I6f3cb9867898a3371cedc58835633ac6eb9ad99b
Adds unit tests for all data objects created in [0]. The changes in [0]
were merged in short succession with [1], causing the test coverage gate
to fail with all the newly introduced lines of code. This change adds
tests for all of the newly added code and increases test coverage
requirement to 60% (currently at 65.93%).
[0] https://review.opendev.org/#/c/658917/
[1] https://review.opendev.org/#/c/663729/
Change-Id: I96931e3e415af80ca5ab9202c2bda0344a9901f0
Spyglass recently had a change merged ([0]) that included tests. Now
that there are tests in Spyglass, gates can be enabled in Zuul and for
the Makefile.
Change-Id: I2da39f3601d4263aa674019205721d5f0ac5f227
This change switches Spyglass from storing data in large multi-level
dictionaries to using several data objects. The intent of this change is
to make management of Spyglass data easier for developers to understand.
Implementation of the data objects will be handled in a separate change
to keep code review more manageable.
Change-Id: I101ad1ccbd95822965b8da8b6a644522eb2908e7
This change verifies that manifests generated by Jinja2 do not contain
undefined data. If Spyglass attempts to generate a manifest file that
references undefined data, all previously created manifests will be
deleted and an error will be thrown. Users may bypass this function by
using the "--force" CLI option which will change all undefined data
errors to warnings instead.
Adds undefined data validation to Jinja2 manifest generation.
Adds "--force" option to bypass undefined data validation.
Adds tests for site_processor.py and enables tox testing/coverage.
Change-Id: Iff000eb173995156fbc6b44e621c59ba4dffae35
Pins all Spyglass dependencies explicitly for consistency.
Removes unused packages.
Related to https://review.opendev.org/#/c/662069/
Change-Id: I5ca302e7315a8606510c6e83092188291f661760
This patch:
1. Adds an OpenSUSE Dockerfile, as the other Airship projects have done
2. Adds OpenSUSE checks, gate, and publish jobs to .zuul.yaml
Change-Id: I210f0233387c28b607fe86d8589a095e892d7fef
