airship
/
specs
Code Issues Proposed changes

Specs for pegleg site manifest generation tool

Browse Source 
Change-Id: I8ff411a18f56cc744e22004139f604430eb81708
changes/27/605227/18
purnendu ghosh 5 years ago committed by Purnendu Ghosh
parent 2bbe9a4cf8
commit 8302c3f966
1 changed files with 409 additions and 0 deletions
Show Stats Download Patch File Download Diff File

409
specs/approved/data_config_generator.rst
Unescape View File

@ -0,0 +1,409 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
========
Spyglass
========
Spyglass is a data extraction tool which can interface with
different input data sources to generate site manifest YAML files.
The data sources will provide all the configuration data needed
for a site deployment. These site manifest YAML files generated
by spyglass will be saved in a Git repository, from where Pegleg
can access and aggregate them. This aggregated file can then be
fed to Shipyard for site deployment / updates.
Problem description
===================
During the deployment of Airship Genesis node via Pegleg, it expects that
the deployment engineer provides all the information pertained to Genesis,
Controller & Compute nodes such as PXE IPs, VLANs pertaining to Storage
network, Kubernetes network, Storage disks, Host profiles, etc. as
manifests/YAMLs that are easily understandable by Pegleg.
Currently there exists multiple data sources and these inputs are processed
manually by deployment engineers. Considering the fact that there are
multiple sites for which we need to generate such data, the current process
is cumbersome, error-prone and time-intensive.
The solution to this problem is to automate the overall process so that
the resultant work-flow has standardized operations to handle multiple data
sources and generate site YAMLs considering site type and version.
Impacted components
===================
None.
Proposed change
===============
Proposal here is to develop a standalone stateless automation utility to
extract relevant information from a given site data source and process
it against site specific templates to generate site manifests which can
be consumed by Pegleg. The data sources could be different engineering packages
or extracted from remote external sources. One example of a remote data source
can be an API endpoint.
The application shall perform the automation in two stages. In the first stage
it shall generate a standardized intermediary YAML object after parsing extracted
information from the data source. In the second stage the intermediary YAML shall be
processed by a site processor using site specific templates to generate
site manifests.
Overall Architecture
====================
::
+-----------+ +-------------+
| | | +-------+ |
| | +------>| |Generic| |
+-----------+ | | | |Object | |
|Tugboat(Xl)| I | | | +-------+ |
|Plugin | N | | | | |
+-----------+ T | | | | |
| E | | | +------+ |
+------------+ R | | | |Parser| +------> Intermediary YAML
|Remote Data | F |---+ | +------+ |
|SourcePlugin| A | | | |
+------------+ C | | |(Intermediary YAML)
| E | | | |
| | | | |
| H | | v |
| A | | +---------+|(templates) +------------+
| N | | |Site |+<--------------|Repository |
| D | | |Processor||-------------->|Adapter |
| L | | +---------+|(Generated +------------+
| E | | ^ | Site Manifests)
| R | | +---|-----+|
| | | | J2 ||
| | | |Templates||
| | | +---------+|
+-----------+ +-------------+
--
1)Interface handler: Acts as an interface to support multiple plugins like Excel,
Remote Data Source, etc. The interface would define abstract APIs which would be overridden
by different plugins. A plugin would implement these APIs based on the type of data source
to collect raw site data and convert them to a generic object for further processing.
For example: Consider the APIs connect_data_source() and get_host_profile(). For Excel plugin
the connect_data_source API would implement file-open methods and the get_host_profile would
extract host profile related information from the Excel file.
In the case of a remote data source (for example an API endpoint), the API "connect_data_source"
shall authenticate (if required) and establish a connection to the remote site and the
"get_host_profile" API shall implement the logic to extract appropriate details over the established
connection. In order to support future plugins, one needs to override these interface handler
APIs and develop logic to extract site data from the corresponding data source.
2)Parser: It processes the information obtained from generic YAML object to create an
intermediary YAML using the following inputs:
a) Global Design Rules: Common rules for generating manifest for any kind of site.
These rule are used for every plugin. for example: IPs to skip before considering allocation to host.
b) Site Config Rules: These are settings specific to a particular site.
For example http_proxy, bgp asn number, etc. It can be referred by all plugins. Sometimes these
site specific information can also be received from plugin data sources. In such cases the
information from plugin data sources would be used instead of the ones specified in site config rules.
3)Intermediary YAML: It holds the complete site information after getting it from interface
handler plugin and after application of site specific rules. It maintains a common format agnostic
of the corresponding data source used. So it act as a primary input to Site Processor for generating
site manifests.
4)Tugboat(Excel Parser) Plugin: It uses the interface handler APIs to open and parse the Excel file to
extract site details and create an in memory generic YAML object. This generic object is further processed
using site specific config rules and global rules to generate an intermediary YAML. The name "Tugboat"
here is used to identify "Excel Parser". For Excel parser the plugin shall use a Site specification file
which defines the various location(s) of the site information items in file. The location is specified by
mentioning rows and columns of the spreadsheet cell containing the specific site data.
5)Remote Data Source Plugin: It uses the interface handler APIs to connect to the data source and extract
site specific information and then construct a generic in memory YAML object. This object is then parsed
to generate an intermediary YAML. There may be situations wherein the information extracted from API
endpoints are incomplete. In such scenarios, the missing information can be supplied from Site Config Rules.
6)Site Processor: The site processor consumes the intermediary YAML and generates site manifests
based on corresponding site templates that are written in python Jinja2.
For example, for template file "baremetal.yaml.j2", the site processor will generate "baremetal.yaml"
with the information obtained from intermediary YAML and also by following the syntax present in the
corresponding template file.
7)Site Templates(J2 templates): These define the manifest file formats for various entities like
baremetal, network, host-profiles, etc. The site processor applies these templates to an intermediary
YAML and generates the corresponding site manifests.
For example: calico-ip-rules.yaml.j2 will generate calico-ip-rules.yaml when processed by the
site processor.
8)Repository Adapter: This helps in importing site specific templates from a repository and also
push generated site manifest YAMLs. The aim of the repository adapter shall be to abstract the
specific repository operations and maintain an uniform interface irrespective of the type of
repository used. It shall be possible to add newer repositories in the future without any change
to this interface. The access to this repository can be regulated by credentials if required and
those will be passed as parameters to the site specific config file.
9)Sample data flow: for example generating OAM network information from site manifests.
- Raw rack information from plugin:
vlan_network_data:
oam:
subnet: 12.0.0.64/26
vlan: '1321'
- Rules to define gateway, ip ranges from subnet:
rule_ip_alloc_offset:
name: ip_alloc_offset
ip_alloc_offset:
default: 10
gateway: 1
The above rule specify the ip offset to considered to define ip address for gateway, reserved
and static ip ranges from the subnet pool.
So ip range for 12.0.0.64/26 is : 12.0.0.65 ~ 12.0.0.126
The rule "ip_alloc_offset" now helps to define additional information as follows:
- gateway: 12.0.0.65 (the first offset as defined by the field 'gateway')
- reserved ip ranges: 12.0.0.65 ~ 12.0.0.76 (the range is defined by adding
"default" to start ip range)
- static ip ranges: 12.0.0.77 ~ 12.0.0.126 (it follows the rule that we need
to skip first 10 ip addresses as defined by "default")
- Intermediary YAML file information generated after applying the above rules
to the raw rack information:
::
network:
vlan_network_data:
oam:
network: 12.0.0.64/26
gateway: 12.0.0.65 --------+
reserved_start: 12.0.0.65 |
reserved_end: 12.0.0.76 |
routes: +--> Newly derived information
- 0.0.0.0/0 |
static_start: 12.0.0.77 |
static_end: 12.0.0.126 ----+
vlan: '1321'
--
- J2 templates for specifying oam network data: It represents the format in
which the site manifests will be generated with values obtained from
Intermediary YAML
::
---
schema: 'drydock/Network/v1'
metadata:
schema: 'metadata/Document/v1'
name: oam
layeringDefinition:
abstract: false
layer: 'site'
parentSelector:
network_role: oam
topology: cruiser
actions:
- method: merge
path: .
storagePolicy: cleartext
data:
cidr: {{ data['network']['vlan_network_data']['oam']['network'] }}}
routes:
- subnet: {{ data['network']['vlan_network_data']['oam']['routes'] }}
gateway: {{ data['network']['vlan_network_data']['oam']['gateway'] }}
metric: 100
ranges:
- type: reserved
start: {{ data['network']['vlan_network_data']['oam']['reserved_start'] }}
end: {{ data['network']['vlan_network_data']['oam']['reserved_end'] }}
- type: static
start: {{ data['network']['vlan_network_data']['oam']['static_start'] }}
end: {{ data['network']['vlan_network_data']['oam']['static_end'] }}
...
--
- OAM Network information in site manifests after applying intermediary YAML to J2
templates.:
::
---
schema: 'drydock/Network/v1'
metadata:
schema: 'metadata/Document/v1'
name: oam
layeringDefinition:
abstract: false
layer: 'site'
parentSelector:
network_role: oam
topology: cruiser
actions:
- method: merge
path: .
storagePolicy: cleartext
data:
cidr: 12.0.0.64/26
routes:
- subnet: 0.0.0.0/0
gateway: 12.0.0.65
metric: 100
ranges:
- type: reserved
start: 12.0.0.65
end: 12.0.0.76
- type: static
start: 12.0.0.77
end: 12.0.0.126
...
--
Security impact
---------------
The impact would be limited to the use of credentials for accessing the data source, templates and
also for uploading generated manifest files.
Performance impact
------------------
None.
Alternatives
------------
No existing utilities available to transform site information automatically.
Implementation
==============
The following high-level implementation tasks are identified:
a) Interface Handler
b) Plugins (Excel and a sample Remote data source plugin)
c) Parser
d) Site Processor
e) Repository Adapter
Usage
=====
The tool will support Excel and remote data source plugin from the beginning.
The section below lists the required input files for each of the aforementioned
plugins.
* Preparation: The preparation steps differ based on selected data source.
A. Excel Based Data Source.
- Gather the following input files:
1) Excel based site Engineering package. This file contains detail specification
covering IPMI, Public IPs, Private IPs, VLAN, Site Details, etc.
2) Excel Specification to aid parsing of the above Excel file. It contains
details about specific rows and columns in various sheet which contain the
necessary information to build site manifests.
3) Site specific configuration file containing additional configuration like
proxy, bgp information, interface names, etc.
4) Intermediary YAML file. In this cases Site Engineering Package and Excel
specification are not required.
B. Remote Data Source
- Gather the following input information:
1) End point configuration file containing credentials to enable its access.
Each end-point type shall have their access governed by their respective plugins
and associated configuration file.
2) Site specific configuration file containing additional configuration like
proxy, bgp information, interface names, etc. These will be used if information
extracted from remote site is insufficient.
* Program execution
1) CLI Options:
-g, --generate_intermediary Dump intermediary file from passed Excel and
Excel spec.
-m, --generate_manifests Generate manifests from the generated
intermediary file.
-x, --excel PATH Path to engineering Excel file, to be passed
with generate_intermediary. The -s option is
mandatory with this option. Multiple engineering
files can be used. For example: -x file1.xls -x file2.xls
-s, --exel_spec PATH Path to Excel spec, to be passed with
generate_intermediary. The -x option is
mandatory along with this option.
-i, --intermediary PATH Path to intermediary file,to be passed
with generate_manifests. The -g and -x options
are not required with this option.
-d, --site_config PATH Path to the site specific YAML file [required]
-l, --loglevel INTEGER Loglevel NOTSET:0 ,DEBUG:10, INFO:20,
WARNING:30, ERROR:40, CRITICAL:50 [default:20]
-e, --end_point_config File containing end-point configurations like user-name
password, certificates, URL, etc.
--help Show this message and exit.
2) Example:
2-1) Using Excel spec as input data source:
Generate Intermediary: spyglass -g -x <DesignSpec> -s <excel spec> -d <site-config>
Generate Manifest & Intermediary: spyglass -mg -x <DesignSpec> -s <excel spec> -d <site-config>
Generate Manifest with Intermediary: spyglass -m -i <intermediary>
2-1) Using external data source as input:
Generate Manifest and Intermediary : spyglass -m -g -e<end_point_config> -d <site-config>
Generate Manifest : spyglass -m -e<end_point_config> -d <site-config>
Note: The end_point_config shall include attributes of the external data source that are
necessary for its access. Each external data source type shall have its own plugin to configure
its corresponding credentials.
* Program output:
a) Site Manifests: As an initial release, the program shall output manifest files for
"airship-seaworthy" site. For example: baremetal, deployment, networks, pki, etc.
Reference:https://github.com/openstack/airship-treasuremap/tree/master/site/airship-seaworthy
b) Intermediary YAML: Containing aggregated site information generated from data sources that is
used to generate the above site manifests.
Future Work
============
1) Schema based manifest generation instead of Jinja2 templates. It shall
be possible to cleanly transition to this schema based generation keeping a unique
mapping between schema and generated manifests. Currently this is managed by
considering a mapping of j2 templates with schemas and site type.
2) UI editor for intermediary YAML
Alternatives
============
1) Schema based manifest generation instead of Jinja2 templates.
2) Develop the data source plugins as an extension to Pegleg.
Dependencies
============
1) Availability of a repository to store Jinja2 templates.
2) Availability of a repository to store generated manifests.
References
==========
None
Write Preview
Loading…
Cancel
Save