airship/spyglass
Code Issues Proposed changes

Merge "Revise CLI documentation"

Browse Source
This commit is contained in:
Zuul 2019-05-28 19:47:40 +00:00 committed by Gerrit Code Review
commit b698288dfe
6 changed files with 330 additions and 139 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

232
doc/source/cli.rst Normal file
View File

@ -0,0 +1,232 @@
..
Copyright 2019 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
============
Spyglass CLI
============
The Spyglass CLI is used in conjunction with the script ``tools/spyglass.sh``.
.. note::
The default workspace for the ``spyglass.sh`` script is ``/workspace``. The
examples below require that this workspace be used.
CLI Options
===========
**-v / --verbose** (Optional). False by default.
Enable debug logging.
Generate Intermediary
---------------------
Generates an intermediary file from passed excel data.
.. code-block:: bash
./spyglass.sh i -p <plugin_type> -x <engineering_excel_file> \
-e <excel_spec> -c <additional_site_config> -s <site_name>
Options
^^^^^^^
**-p / --plugin-type** "tugboat" by default.
The plugin to use to open engineering data. Two plugins are available by
default: "tugboat" and "formation". Tugboat can be used for reading Excel data.
Formation can be used to read data from remote sources.
**-f / --formation-target** (Required for "formation" plugin).
Target remote for the formation plugin. Accepts a url and a username and
password to access the url.
::
-f <remote_url> <username> <password>
**-d / --intermediary-dir** (Optional).
Path where the intermediary file will be created. Must be a writeable
directory.
**-x / --excel-file** (Required for "tugboat" plugin).
Path to the engineering excel file. Multiple files can be included, provided
they follow the same specification. Must be readable file(s) in a Microsoft
Excel supported format (.xls, .xslx, etc...).
**-e / --excel-spec** (Required for "tugboat" plugin).
Path to the specification YAML that defines the content of the provided
engineering excel files. Must be a readable file in YAML format.
**-c / --site-configuration** (Optional).
Path to site specific configuration YAML. Must be a readable file.
**-s / --site-name** (Optional).
Name of the site for which the intermediary is generated.
Generate Manifests
------------------
Generates manifests from intermediary file created from passed excel data.
Intermediary data is always generated, but will not be saved unless specified.
.. code-block:: bash
./spyglass.sh m -t <plugin_type> -x <engineering_excel_file> \
-e <excel_spec> -c <additional_site_config> \
-s <site_name> -t <j2_template_directory>
Options
^^^^^^^
**-i / --save-intermediary** (Optional). False by default.
Saves the intermediary file used to make the manifests created by the command.
**-p / --plugin-type** "tugboat" by default.
The plugin to use to open engineering data. Two plugins are available by
default: "tugboat" and "formation". Tugboat can be used for reading Excel data.
Formation can be used to read data from remote sources.
**-f / --formation-target** (Required for "formation" plugin).
Target remote for the formation plugin. Requires a url, a username, and a
password to access the url.
::
-f <remote_url> <username> <password>
**-d / --intermediary-dir** (Optional).
Path where the intermediary file will be created. Must be a writeable
directory.
**-x / --excel-file** (Required for "tugboat" plugin).
Path to the engineering excel file. Multiple files can be included, provided
they follow the same specification. Must be readable file(s) in a Microsoft
Excel supported format (.xls, .xslx, etc...).
**-e / --excel-spec** (Required for "tugboat" plugin).
Path to the specification YAML that defines the content of the provided
engineering excel files. Must be a readable file in YAML format.
**-c / --site-configuration** (Optional).
Path to site specific configuration YAML. Must be a readable file.
**-s / --site-name** (Optional).
Name of the site for which the intermediary is generated.
**-t / --template-dir** (Required).
Path to the Jinja2 template files that will be used to generate manifest files.
Must be a readable directory with Jinja2 files using the .j2 extension.
**-m / --manifest-dir** (Optional).
Path where generated manifest files should be written. Must be a writeable
directory.
Generate Manifests from Intermediary
------------------------------------
Generates manifests using an existing intermediary file. This is a shortcut to
skip intermediary generation if it has already been completed.
.. code-block:: bash
./spyglass.sh mi <intermediary_file> -t <j2_template_directory>
Arguments
^^^^^^^^^
**INTERMEDIARY_FILE** (Required).
Path to an existing intermediary YAML file that can be used to generate
manifests.
Options
^^^^^^^
**-t / --template-dir** (Required).
Path to the Jinja2 template files that will be used to generate manifest files.
Must be a readable directory with Jinja2 files using the .j2 extension.
**-m / --manifest-dir** (Optional).
Path where generated manifest files should be written. Must be a writeable directory.
Examples
========
Running Spyglass with Excel Plugin
----------------------------------
.. code-block:: bash
spyglass m -i -p tugboat -x <Excel File> -e <Excel Spec> -c <Site Config> \
-s <Region> -t <j2 template dir>
Generating intermediary and manifests
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: bash
spyglass m -i -p tugboat -x SiteDesignSpec_v0.1.xlsx \
-e excel_spec_upstream.yaml -c site_config.yaml \
-s airship-seaworthy -t <j2 template dir>
Generating intermediary without manifests
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: bash
spyglass i -p tugboat -x SiteDesignSpec_v0.1.xlsx \
-e excel_spec_upstream.yaml -c site_config.yaml \
-s airship-seaworthy
Generating manifests without intermediary
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: bash
spyglass m -p tugboat -x SiteDesignSpec_v0.1.xlsx \
-e excel_spec_upstream.yaml -c site_config.yaml \
-s airship-seaworthy --template_dir=<j2 template dir>
Generating manifests using intermediary
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: bash
spyglass mi <intermediary.yaml> -t <j2 template dir>
Where sample 'excel_spec_upstream.yaml', 'SiteDesignSpec_v0.1.xlsx'
'site_config.yaml' and J2 templates can be found under 'spyglass/examples'
folder.

23
doc/source/conf.py
View File

@ -1,3 +1,16 @@
# Copyright 2019 AT&T Intellectual Property. All other rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# -*- coding: utf-8 -*-
#
# shipyard documentation build configuration file, created by
@ -51,18 +64,18 @@ source_suffix = '.rst'
master_doc = 'index'
# General information about the project.
project = 'tugboat'
project = 'spyglass'
copyright = '2018 AT&T Intellectual Property.'
author = 'Tugboat Authors'
author = 'Spyglass Authors'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = '0.1.0'
version = '0.0.1'
# The full version, including alpha/beta/rc tags.
release = '0.1.0'
release = '0.0.1'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
@ -104,7 +117,7 @@ html_static_path = []
# -- Options for HTMLHelp output ------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = 'ucpintdoc'
htmlhelp_basename = 'spyglassdoc'
# -- Options for LaTeX output ---------------------------------------------

31
doc/source/developer_quickstart.rst
View File

@ -1,5 +1,5 @@
..
Copyright 2018 AT&T Intellectual Property.
Copyright 2019 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
@ -25,43 +25,42 @@ airship-spyglass directory.
.. code-block:: console
sudo apt install -y python3-pip
sudo apt install -y tox
sudo apt install -y python3-pip
sudo apt install -y tox
2. Set up an environment with tox.
.. code-block:: console
tox -e py36 --notest
tox -e py36 --notest
3. Enter the tox environment.
.. code-block:: console
source .tox/py36/bin/activate
source .tox/py36/bin/activate
4. Install spyglass in the tox environment.
.. code-block:: console
pip install -e .
pip install -e .
5. Run spyglass on the example files to generate an intermediate document.
.. code-block:: console
mkdir intermediate
spyglass -g -s airship-seaworthy -t tugboat \
-idir intermediate \
--excel_spec spyglass/examples/excel_spec.yaml \
--excel spyglass/examples/SiteDesignSpec_v0.1.xlsx \
--additional_config spyglass/examples/site_config.yaml \
--template_dir spyglass/examples/templates/
mkdir intermediate
spyglass m -s airship-seaworthy -p tugboat -d intermediate \
--excel-spec spyglass/examples/excel_spec.yaml \
--excel-file spyglass/examples/SiteDesignSpec_v0.1.xlsx \
--site-configuration spyglass/examples/site_config.yaml \
--template-dir spyglass/examples/templates/
6. Run spyglass on the intermediate document to generate manifests.
.. code-block:: console
mkdir manifest_dir
spyglass -m -i intermediate/airship-seaworthy_intermediary.yaml \
-mdir manifest_dir/ -tdir spyglass/examples/templates/
mkdir manifest_dir
spyglass mi intermediate/airship-seaworthy_intermediary.yaml \
-m manifest_dir/ -t spyglass/examples/templates/

76
doc/source/getting_started.rst
View File

@ -1,5 +1,5 @@
..
Copyright 2018 AT&T Intellectual Property.
Copyright 2019 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
@ -28,7 +28,7 @@ for a site deployment. These site manifest YAML files generated
by spyglass will be saved in a Git repository, from where Pegleg
can access and aggregate them. This aggregated file can then be
fed to Shipyard for site deployment / updates.
Reference: https://review.opendev.org/#/c/605227
Reference: https://airshipit.readthedocs.io/projects/specs/en/latest/specs/approved/data_config_generator.html
Architecture
------------
@ -75,9 +75,9 @@ Supported Features
Future Work
-----------
1) Schema based manifest generation instead of Jinja2 templates. It shall
be possible to cleanly transition to this schema based generation keeping a unique
mapping between schema and generated manifests. Currently this is managed by
considering a mapping of j2 templates with schemas and site type.
be possible to cleanly transition to this schema based generation keeping a
unique mapping between schema and generated manifests. Currently this is
managed by considering a mapping of j2 templates with schemas and site type.
List of Generated Site Manifests:
---------------------------------
@ -129,72 +129,10 @@ Before using Spyglass you must:
.. code-block:: console
git clone https://opendev.org/airship/spyglass.git
git clone https://opendev.org/airship/spyglass.git
2. Install the required packages in spyglass:
.. code-block:: console
pip3 install -r airship-spyglass/requirements.txt
CLI Options
-----------
Usage: spyglass [OPTIONS]
Options:
-s, --site TEXT Specify the site for which manifests to be
generated
-t, --type TEXT Specify the plugin type formation or tugboat
-f, --formation_url TEXT Specify the formation url
-u, --formation_user TEXT Specify the formation user id
-p, --formation_password TEXT Specify the formation user password
-i, --intermediary PATH Intermediary file path generate manifests,
use -m also with this option
-d, --additional_config PATH Site specific configuraton details
-g, --generate_intermediary Dump intermediary file from passed excel and
excel spec
-idir, --intermediary_dir PATH The path where intermediary file needs to be
generated
-m, --generate_manifests Generate manifests from the generated
intermediary file
-mdir, --manifest_dir PATH The path where manifest files needs to be
generated
-x, --excel PATH Path to engineering excel file, to be passed
with generate_intermediary
-e, --excel_spec PATH Path to excel spec, to be passed with
generate_intermediary
-l, --loglevel INTEGER Loglevel NOTSET:0 ,DEBUG:10, INFO:20,
WARNING:30, ERROR:40, CRITICAL:50 [default:
20]
--help Show this message and exit.
--------
Examples
--------
1. Running Spyglass with Remote Data Source Plugin
spyglass -mg --type formation -f <URL> -u <user_id> -p <password> -d <site_config> -s <sitetype> --template_dir=<j2 template dir>
2. Running Spyglass with Excel Plugin
spyglass -mg --type tugboat -x <Excel File> -e <Excel Spec> -d <Site Config> -s <Region> --template_dir=<j2 template dir>
for example:
2.1 Generating intermediary and manifests
spyglass -mg -t tugboat -x SiteDesignSpec_v1.1.xlsx -e excel_spec_upstream.yaml -d site_config.yaml -s airship-seaworthy --template_dir=<j2 template dir>
2.2 Generating intermediary without manifests
spyglass -g -t tugboat -x SiteDesignSpec_v1.1.xlsx -e excel_spec_upstream.yaml -d site_config.yaml -s airship-seaworthy
2.3 Generating manifests without intermediary
spyglass -m -t tugboat -x SiteDesignSpec_v1.1.xlsx -e excel_spec_upstream.yaml -d site_config.yaml -s airship-seaworthy --template_dir=<j2 template dir>
2.4 Generating manifests using intermediary
spyglass -mi <intermediary.yaml> --template_dir=<j2 template dir>
Where sample 'excel_spec_upstream.yaml', 'SiteDesignSpec_v0.1.xlsx'
'site_config.yaml' and J2 templates can be found under 'spyglass/examples'
folder
pip3 install -r airship-spyglass/requirements.txt

3
doc/source/index.rst
View File

@ -1,5 +1,5 @@
..
Copyright 2018 AT&T Intellectual Property.
Copyright 2019 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
@ -33,4 +33,5 @@ fed to Shipyard for site deployment / updates.
getting_started
developer_quickstart
cli
tugboat

104
doc/source/tugboat.rst
View File

@ -1,5 +1,5 @@
..
Copyright 2018 AT&T Intellectual Property.
Copyright 2019 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
@ -37,51 +37,56 @@ need to be filled by the Deployment Engineer.
Below is the definition for each key in the Excel spec
::
ipmi_sheet_name - name of the sheet from where IPMI and host profile information is to be read
start_row - row number from where the IPMI and host profile information starts
end_row - row number from where the IPMI and host profile information ends
hostname_col - column number where the hostnames are to be read from
ipmi_address_col - column number from where the ipmi addresses are to be read
host_profile_col - column number from where the host profiles are to be read
ipmi_gateway_col - column number from where the ipmi gateways are to be read
private_ip_sheet - name of the sheet which has the private IP information
net_type_col - column number from where the network type is to be read
vlan_col - column number from where the network vlan is to be read
vlan_start_row - row number from where the vlan information starts
vlan_end_row - row number from where the vlan information ends
net_start_row - row number from where the network information starts
net_end_row - row number from where the network information ends
net_col - column number where the IP ranges for network is to be read
net_vlan_col - column number where the vlan information is present in the pod wise network section
public_ip_sheet - name of the sheet which has the public IP information
oam_vlan_col - column number from where the OAM vlan information is to be read from
oam_ip_row - row number from where the OAM network information is to be read from
oam_ip_col - column number from where the OAM network information is to be read from
oob_net_row - row number which has the OOB network subnet ranges
oob_net_start_col - column number from where the OOB network ranges start
oob_net_end_col - column number from where the OOB network ranges end
ingress_ip_row - row number from where the Ingress network information is to be read from
dns_ntp_ldap_sheet - name of the sheet which has the DNS, NTP and LDAP information
login_domain_row - row number which has the ldap login domain
ldap_col - column number which has the all ldap related information
global_group - row number which has the ldap group information
ldap_search_url_row - row number which has the ldap url
ntp_row - row number which has the ntp information
ntp_col - column number which has the ntp information
dns_row - row number which has the dns information
dns_col - column number which has the dns information
domain_row - row number which has the domain information
domain_col - column number which has the domain information
location_sheet - name of the sheet which has the location information
column - column number which has all the information
corridor_row - row number which has the corridor information
site_name_row - row number which has the site name
state_name_row - row number which has the state name
country_name_row - row number which has the country name
clli_name_row - row number which has CLLI information
* ipmi_sheet_name - name of the sheet from where IPMI and host profile
information is to be read
* start_row - row number from where the IPMI and host profile information
starts
* end_row - row number from where the IPMI and host profile information ends
* hostname_col - column number where the hostnames are to be read from
* ipmi_address_col - column number from where the ipmi addresses are to be read
* host_profile_col - column number from where the host profiles are to be read
* ipmi_gateway_col - column number from where the ipmi gateways are to be read
* private_ip_sheet - name of the sheet which has the private IP information
* net_type_col - column number from where the network type is to be read
* vlan_col - column number from where the network vlan is to be read
* vlan_start_row - row number from where the vlan information starts
* vlan_end_row - row number from where the vlan information ends
* net_start_row - row number from where the network information starts
* net_end_row - row number from where the network information ends
* net_col - column number where the IP ranges for network is to be read
* net_vlan_col - column number where the vlan information is present in the
pod wise network section
* public_ip_sheet - name of the sheet which has the public IP information
* oam_vlan_col - column number from where the OAM vlan information is to be
read from
* oam_ip_row - row number from where the OAM network information is to be read
from
* oam_ip_col - column number from where the OAM network information is to be
read from
* oob_net_row - row number which has the OOB network subnet ranges
* oob_net_start_col - column number from where the OOB network ranges start
* oob_net_end_col - column number from where the OOB network ranges end
* ingress_ip_row - row number from where the Ingress network information is to
be read from
* dns_ntp_ldap_sheet - name of the sheet which has the DNS, NTP and LDAP
information
* login_domain_row - row number which has the ldap login domain
* ldap_col - column number which has the all ldap related information
* global_group - row number which has the ldap group information
* ldap_search_url_row - row number which has the ldap url
* ntp_row - row number which has the ntp information
* ntp_col - column number which has the ntp information
* dns_row - row number which has the dns information
* dns_col - column number which has the dns information
* domain_row - row number which has the domain information
* domain_col - column number which has the domain information
* location_sheet - name of the sheet which has the location information
* column - column number which has all the information
* corridor_row - row number which has the corridor information
* site_name_row - row number which has the site name
* state_name_row - row number which has the state name
* country_name_row - row number which has the country name
* clli_name_row - row number which has CLLI information
Example: Tugboat Plugin Usage
-----------------------------
@ -95,6 +100,9 @@ Example: Tugboat Plugin Usage
e) Site name: airship-seaworthy
2. Spyglass CLI Command:
::
spyglass -mg -t tugboat -x SiteDesignSpec_v0.1.xlsx -e excel_spec_upstream.yaml -d site_config.yaml -s airship-seaworthy --template_dir=<relative path to '../examples/templates'
.. code-block:: bash
spyglass m -i -p tugboat -x SiteDesignSpec_v0.1.xlsx \
-e excel_spec_upstream.yaml -c site_config.yaml \
-s airship-seaworthy -t <relative path to J2 templates dir>