From 161528365d0aa93bc6410c4c768af8df48fe0f8d Mon Sep 17 00:00:00 2001 From: Ryan Schroder Date: Thu, 21 Nov 2019 10:47:55 -0600 Subject: [PATCH] Spyglass Docs Update Changed paths for all examples, removed references to Tugboat and Formation Change-Id: Ibeb80a1c239169bd4f4ecb124e86671891bef99a --- .zuul.yaml | 2 + README.rst | 33 ++++++----- doc/source/cli.rst | 89 ++++++++++++++++------------- doc/source/developer_quickstart.rst | 34 ++++++----- doc/source/getting_started.rst | 14 +++-- doc/source/index.rst | 12 ++++ 6 files changed, 105 insertions(+), 79 deletions(-) diff --git a/.zuul.yaml b/.zuul.yaml index e15e2e3..209ae3c 100644 --- a/.zuul.yaml +++ b/.zuul.yaml @@ -84,6 +84,7 @@ timeout: 1800 run: tools/gate/playbooks/docker-image-build.yaml nodeset: spyglass-single-node + voting: false vars: publish: false distro: opensuse_15 @@ -132,6 +133,7 @@ nodeset: spyglass-single-node secrets: - airship_spyglass_quay_creds + voting: false vars: publish: true distro: opensuse_15 diff --git a/README.rst b/README.rst index 48f5c97..697807b 100644 --- a/README.rst +++ b/README.rst @@ -6,27 +6,26 @@ Spyglass is the data extractor tool which can interface with different input data sources to generate site manifest YAML files. The data sources will provide all the configuration data needed for a site deployment. These site manifest YAML files generated -by spyglass will be saved in a Git repository, from where Pegleg +by Spyglass will be saved in a Git repository, from where Pegleg can access and aggregate them. This aggregated file can then be -fed to shipyard for site deployment / updates. +fed to Shipyard for site deployment/updates. -Spyglass follows plugin model to support multiple input data sources. -Current supported plugins are formation-plugin and Tugboat. Formation -is a rest API based service which will be the source of information -related to hardware, networking, site data. Formation plugin will -interact with Formation API to gather necessary configuration. -Similarly Tugboat accepts engineering spec which is in the form of -spreadsheet and an index file to read spreadsheet as inputs and -generates the site level manifests. -As an optional step it can generate an intermediary yaml which contain -all the information that will be rendered to generate Airship site -manifests. This optional step will help the deployment engineer to -modify any data if required. +Spyglass follows a plugin model to support multiple input data sources. +The currently supported plugin is the Spyglass Excel plugin +(`spyglass-plugin-xls`_). + +The Spyglass Excel plugin accepts an engineering spec in the form of a +spreadsheet and an index file to read the spreadsheet as inputs and +generates site level manifests. As an optional step, it can generate an +intermediary YAML which contains all the information that will be rendered to +generate Airship site manifests. This optional step will help the deployment +engineer modify any data if required. Getting Started --------------- -For more detailed installation and setup information, please refer to the -`Getting Started`_ guide. +For more detailed information, please refer to the `Index`_ to easily navigate +the Getting Started, Developer Quickstart, and Command Line Interface guides. -.. _`Getting Started`: ./doc/source/getting_started.rst +.. _spyglass-plugin-xls: https://opendev.org/airship/spyglass-plugin-xls +.. _`Index`: ./doc/source/index.rst diff --git a/doc/source/cli.rst b/doc/source/cli.rst index d65f295..f475709 100644 --- a/doc/source/cli.rst +++ b/doc/source/cli.rst @@ -28,19 +28,19 @@ The Spyglass CLI is used in conjunction with the script ``tools/spyglass.sh``. CLI Options =========== -**-v / --verbose** (Optional). False by default. +**-v / \\-\\-verbose** (Optional). False by default. Enable debug logging. Excel Plugin ************ -Commands available under the excel plugin package. +Commands available under the Excel plugin package. Generate Intermediary --------------------- -Generates an intermediary file from passed excel data. +Generates an intermediary file from passed Excel data. .. code-block:: bash @@ -52,42 +52,42 @@ Generates an intermediary file from passed excel data. Options ^^^^^^^ -**-d / --intermediary-dir** (Optional). +**-d / \\-\\-intermediary-dir** (Optional). Path where the intermediary file will be created. Must be a writeable directory. -**-x / --excel-file** (Required for "tugboat" plugin). +**-x / \\-\\-excel-file** (Required for Excel plugin). -Path to the engineering excel file. Multiple files can be included, provided +Path to the engineering Excel file. Multiple files can be included, provided they follow the same specification. Must be readable file(s) in a Microsoft Excel supported format (.xls, .xslx, etc...). -**-e / --excel-spec** (Required for "tugboat" plugin). +**-e / \\-\\-excel-spec** (Required for Excel plugin). Path to the specification YAML that defines the content of the provided -engineering excel files. Must be a readable file in YAML format. +engineering Excel files. Must be a readable file in YAML format. -**-c / --site-configuration** (Optional). +**-c / \\-\\-site-configuration** (Optional). Path to site specific configuration YAML. Must be a readable file. -**--intermediary-schema** (Optional). +**\\-\\-intermediary-schema** (Optional). Path to the intermediary schema to be used for validation. -**--no-validation** (Optional). +**\\-\\-no-validation** (Optional). Skips validation on generated intermediary data. -**-s / --site-name** (Optional). +**-s / \\-\\-site-name** (Optional). Name of the site for which the intermediary is generated. Generate Manifests ------------------ -Generates manifests from intermediary file created from passed excel data. +Generates manifests from intermediary file created from passed Excel data. Intermediary data is always generated, but will not be saved unless specified. .. code-block:: bash @@ -99,48 +99,48 @@ Intermediary data is always generated, but will not be saved unless specified. Options ^^^^^^^ -**-i / --generate-intermediary** (Optional). False by default. +**-i / \\-\\-generate-intermediary** (Optional). False by default. Saves the intermediary file used to make the manifests created by the command. -**-d / --intermediary-dir** (Optional). +**-d / \\-\\-intermediary-dir** (Optional). Path where the intermediary file will be created. Must be a writeable directory. -**-x / --excel-file** (Required for "tugboat" plugin). +**-x / \\-\\-excel-file** (Required for Excel plugin). -Path to the engineering excel file. Multiple files can be included, provided +Path to the engineering Excel file. Multiple files can be included, provided they follow the same specification. Must be readable file(s) in a Microsoft Excel supported format (.xls, .xslx, etc...). -**-e / --excel-spec** (Required for "tugboat" plugin). +**-e / \\-\\-excel-spec** (Required for Excel plugin). Path to the specification YAML that defines the content of the provided -engineering excel files. Must be a readable file in YAML format. +engineering Excel files. Must be a readable file in YAML format. -**-c / --site-configuration** (Optional). +**-c / \\-\\-site-configuration** (Optional). Path to site specific configuration YAML. Must be a readable file. -**--intermediary-schema** (Optional). +**\\-\\-intermediary-schema** (Optional). Path to the intermediary schema to be used for validation. -**--no-validation** (Optional). +**\\-\\-no-validation** (Optional). Skips validation on generated intermediary data. -**-s / --site-name** (Optional). +**-s / \\-\\-site-name** (Optional). Name of the site for which the intermediary is generated. -**-t / --template-dir** (Required). +**-t / \\-\\-template-dir** (Required). Path to the Jinja2 template files that will be used to generate manifest files. Must be a readable directory with Jinja2 files using the .j2 extension. -**-m / --manifest-dir** (Optional). +**-m / \\-\\-manifest-dir** (Optional). Path where generated manifest files should be written. Must be a writeable directory. @@ -169,17 +169,17 @@ manifests. Options ^^^^^^^ -**-t / --template-dir** (Required). +**-t / \\-\\-template-dir** (Required). Path to the Jinja2 template files that will be used to generate manifest files. Must be a readable directory with Jinja2 files using the .j2 extension. -**-m / --manifest-dir** (Optional). +**-m / \\-\\-manifest-dir** (Optional). Path where generated manifest files should be written. Must be a writeable directory. -**--force** (Optional). +**\\-\\-force** (Optional). Forces manifests to be written, regardless of undefined data. @@ -195,11 +195,11 @@ Validates pegleg documents against their schema. Options ^^^^^^^ -**-d / --document-path** +**-d / \\-\\-document-path** Path to the document(s) to validate. -**-p / --schema-path** +**-p / \\-\\-schema-path** Path to a schema or directory of schema files used to validate documents in document path. @@ -220,27 +220,33 @@ Generating intermediary and manifests .. code-block:: bash - spyglass excel documents -i -x SiteDesignSpec_v0.1.xlsx \ - -e excel_spec_upstream.yaml -c site_config.yaml \ - -s airship-seaworthy -t + spyglass excel documents -i \ + -x ../spyglass-plugin-xls/spyglass_plugin_xls/examples/SiteDesignSpec_v0.1.xlsx \ + -e ../spyglass-plugin-xls/spyglass_plugin_xls/examples/excel_spec.yaml \ + -c spyglass/examples/site_config.yaml \ + -s airship-seaworthy -t spyglass/examples/templates/ Generating intermediary without manifests ----------------------------------------- .. code-block:: bash - spyglass excel intermediary -x SiteDesignSpec_v0.1.xlsx \ - -e excel_spec_upstream.yaml -c site_config.yaml \ - -s airship-seaworthy + spyglass excel intermediary \ + -x ../spyglass-plugin-xls/spyglass_plugin_xls/examples/SiteDesignSpec_v0.1.xlsx \ + -e ../spyglass-plugin-xls/spyglass_plugin_xls/examples/excel_spec.yaml \ + -c spyglass/examples/site_config.yaml \ + -s airship-seaworthy Generating manifests without intermediary ----------------------------------------- .. code-block:: bash - spyglass excel documents -x SiteDesignSpec_v0.1.xlsx \ - -e excel_spec_upstream.yaml -c site_config.yaml \ - -s airship-seaworthy -t + spyglass excel documents \ + -x ../spyglass-plugin-xls/spyglass_plugin_xls/examples/SiteDesignSpec_v0.1.xlsx \ + -e ../spyglass-plugin-xls/spyglass_plugin_xls/examples/excel_spec.yaml \ + -c spyglass/examples/site_config.yaml \ + -s airship-seaworthy -t spyglass/examples/templates/ Generating manifests using intermediary *************************************** @@ -249,8 +255,9 @@ Generating manifests using intermediary spyglass mi -t -Where sample 'excel_spec_upstream.yaml', 'SiteDesignSpec_v0.1.xlsx' -'site_config.yaml' and J2 templates can be found under 'spyglass/examples' +Where sample `excel_spec.yaml` and `SiteDesignSpec_v0.1.xlsx` can be found in +spyglass-plugin-xls in the `spyglass_plugin_xls/examples` folder. The Jinja2 +templates and `site_config.yaml` can be found in the `spyglass/examples` folder. Validate Documents diff --git a/doc/source/developer_quickstart.rst b/doc/source/developer_quickstart.rst index 5f60c26..ab47e68 100644 --- a/doc/source/developer_quickstart.rst +++ b/doc/source/developer_quickstart.rst @@ -18,52 +18,56 @@ Developer Quickstart Guide ========================== -To run your first spyglass job, follow these steps from inside the -airship-spyglass directory. +1. Clone the Spyglass directory. (Perform the following steps from inside the +spyglass directory) -1. Install external dependencies if not already installed. + .. code-block:: console + + git clone https://opendev.org/airship/spyglass.git + +2. Install external dependencies if not already installed. .. code-block:: console sudo apt install -y python3-pip sudo apt install -y tox -2. Install Pipenv. +3. Install Pipenv. - .. code-block:: console + .. code-block:: console - pip3 install pipenv + pip3 install pipenv -2. Set up an environment with Pipenv +4. Set up an environment with Pipenv .. code-block:: console pipenv install -3. Enter the Pipenv environment. +5. Enter the Pipenv environment. .. code-block:: console pipenv shell -4. Install spyglass in the tox environment. +6. Install spyglass in the tox environment. .. code-block:: console pip3 install . -5. Run spyglass on the example files to generate an intermediate document. +7. Run spyglass on the example files to generate an intermediate document. .. code-block:: console mkdir intermediate spyglass excel documents -s airship-seaworthy -d intermediate -i \ - --excel-spec spyglass/examples/excel_spec.yaml \ - --excel-file spyglass/examples/SiteDesignSpec_v0.1.xlsx \ - --site-configuration spyglass/examples/site_config.yaml \ - --template-dir spyglass/examples/templates/ + --excel-spec ../spyglass-plugin-xls/spyglass_plugin_xls/examples/excel_spec.yaml \ + --excel-file ../spyglass-plugin-xls/spyglass_plugin_xls/examples/SiteDesignSpec_v0.1.xlsx \ + --site-configuration spyglass/examples/site_config.yaml \ + --template-dir spyglass/examples/templates/ -6. Run spyglass on the intermediate document to generate manifests. +8. Run spyglass on the intermediate document to generate manifests. .. code-block:: console diff --git a/doc/source/getting_started.rst b/doc/source/getting_started.rst index ad8ec32..9de9223 100644 --- a/doc/source/getting_started.rst +++ b/doc/source/getting_started.rst @@ -25,10 +25,10 @@ Spyglass is a data extraction tool which can interface with different input data sources to generate site manifest YAML files. The data sources will provide all the configuration data needed for a site deployment. These site manifest YAML files generated -by spyglass will be saved in a Git repository, from where Pegleg +by Spyglass will be saved in a Git repository, from where Pegleg can access and aggregate them. This aggregated file can then be fed to Shipyard for site deployment / updates. -Reference: https://airshipit.readthedocs.io/projects/specs/en/latest/specs/approved/data_config_generator.html +Reference: `airship-specs`_ Architecture ------------ @@ -39,7 +39,7 @@ Architecture | | | +-------+ | | | +------>| |Generic| | +-----------+ | | | |Object | | - |Tugboat(Xl)| I | | | +-------+ | + |Excel | I | | | +-------+ | |Plugin | N | | | | | +-----------+ T | | | | | | E | | | +------+ | @@ -65,7 +65,7 @@ Architecture Supported Features ------------------ -1. Spyglass XLS Plugin: https://opendev.org/airship/spyglass-plugin-xls +1. Spyglass Excel Plugin: https://opendev.org/airship/spyglass-plugin-xls Future Work ----------- @@ -76,7 +76,7 @@ managed by considering a mapping of j2 templates with schemas and site type. List of Generated Site Manifests: --------------------------------- -The spyglass uses the plugin data source to generate the following site +The Spyglass uses the plugin data source to generate the following site manifests: - site-definition.yaml @@ -126,8 +126,10 @@ Before using Spyglass you must: git clone https://opendev.org/airship/spyglass.git -2. Install the required packages in spyglass: +2. Install the required packages in Spyglass: .. code-block:: console pip3 install pipenv && pipenv install + +.. _airship-specs: https://airshipit.readthedocs.io/projects/specs/en/latest/specs/1.x/approved/data_config_generator.html \ No newline at end of file diff --git a/doc/source/index.rst b/doc/source/index.rst index e802f35..923e89b 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -28,9 +28,21 @@ by spyglass will be saved in a Git repository, from where Pegleg can access and aggregate them. This aggregated file can then be fed to Shipyard for site deployment / updates. +Spyglass follows a plugin model to support multiple input data sources. +The currently supported plugin is the Spyglass Excel plugin +(`spyglass-plugin-xls`_). + +The Spyglass Excel plugin accepts an engineering spec in the form of a +spreadsheet and an index file to read the spreadsheet as inputs and +generates site level manifests. As an optional step, it can generate an +intermediary YAML which contains all the information that will be rendered to +generate Airship site manifests. This optional step will help the deployment +engineer modify any data if required. + .. toctree:: :maxdepth: 2 getting_started developer_quickstart cli +.. _spyglass-plugin-xls: https://opendev.org/airship/spyglass-plugin-xls \ No newline at end of file