Browse Source
This change removes plugins from Spyglass and places them in separate repositories. Formation, a proprietary plugin, will be removed by this change and Tugboat will become its own OpenDev maintained repo, spyglass-plugin-xls. By creating more streamlined plugin management, end users should be able to more easily create their own plugins for different data sources. Related change https://review.opendev.org/#/c/659116/ Depends-On: Ib2f75878b1a29e835cb8e2323aebe9d431c479e7 Change-Id: Ie0eb2e5aefe6bb764e1aa608e53371adaabb9a17changes/55/653555/18
Ian Pittwood
3 years ago
16 changed files with 17 additions and 1702 deletions
@ -1,108 +0,0 @@
|
||||
.. |
||||
Copyright 2019 AT&T Intellectual Property. |
||||
All Rights Reserved. |
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may |
||||
not use this file except in compliance with the License. You may obtain |
||||
a copy of the License at |
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
Unless required by applicable law or agreed to in writing, software |
||||
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT |
||||
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the |
||||
License for the specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
.. _tugboatinfo: |
||||
|
||||
======= |
||||
Tugboat |
||||
======= |
||||
|
||||
What is Tugboat? |
||||
---------------- |
||||
|
||||
Tugboat is a Spyglass plugin to generate airship-seaworthy site manifest files |
||||
from an excel based engineering spec. The plugin is configured with an Excel |
||||
sheet and its corresponding excel specification as inputs. Spyglass uses this |
||||
plugin to construct an intermediary yaml which is processed further using J2 |
||||
templates to generate site manifests. |
||||
|
||||
Excel specification |
||||
------------------- |
||||
Excel Spec is like an index to the Excel sheet to look for the data to be |
||||
collected by the tool. Excel Spec Sample specifies all the details that |
||||
need to be filled by the Deployment Engineer. |
||||
|
||||
Below is the definition for each key in the Excel spec |
||||
|
||||
* ipmi_sheet_name - name of the sheet from where IPMI and host profile |
||||
information is to be read |
||||
* start_row - row number from where the IPMI and host profile information |
||||
starts |
||||
* end_row - row number from where the IPMI and host profile information ends |
||||
* hostname_col - column number where the hostnames are to be read from |
||||
* ipmi_address_col - column number from where the ipmi addresses are to be read |
||||
* host_profile_col - column number from where the host profiles are to be read |
||||
* ipmi_gateway_col - column number from where the ipmi gateways are to be read |
||||
* private_ip_sheet - name of the sheet which has the private IP information |
||||
* net_type_col - column number from where the network type is to be read |
||||
* vlan_col - column number from where the network vlan is to be read |
||||
* vlan_start_row - row number from where the vlan information starts |
||||
* vlan_end_row - row number from where the vlan information ends |
||||
* net_start_row - row number from where the network information starts |
||||
* net_end_row - row number from where the network information ends |
||||
* net_col - column number where the IP ranges for network is to be read |
||||
* net_vlan_col - column number where the vlan information is present in the |
||||
pod wise network section |
||||
* public_ip_sheet - name of the sheet which has the public IP information |
||||
* oam_vlan_col - column number from where the OAM vlan information is to be |
||||
read from |
||||
* oam_ip_row - row number from where the OAM network information is to be read |
||||
from |
||||
* oam_ip_col - column number from where the OAM network information is to be |
||||
read from |
||||
* oob_net_row - row number which has the OOB network subnet ranges |
||||
* oob_net_start_col - column number from where the OOB network ranges start |
||||
* oob_net_end_col - column number from where the OOB network ranges end |
||||
* ingress_ip_row - row number from where the Ingress network information is to |
||||
be read from |
||||
* dns_ntp_ldap_sheet - name of the sheet which has the DNS, NTP and LDAP |
||||
information |
||||
* login_domain_row - row number which has the ldap login domain |
||||
* ldap_col - column number which has the all ldap related information |
||||
* global_group - row number which has the ldap group information |
||||
* ldap_search_url_row - row number which has the ldap url |
||||
* ntp_row - row number which has the ntp information |
||||
* ntp_col - column number which has the ntp information |
||||
* dns_row - row number which has the dns information |
||||
* dns_col - column number which has the dns information |
||||
* domain_row - row number which has the domain information |
||||
* domain_col - column number which has the domain information |
||||
* location_sheet - name of the sheet which has the location information |
||||
* column - column number which has all the information |
||||
* corridor_row - row number which has the corridor information |
||||
* site_name_row - row number which has the site name |
||||
* state_name_row - row number which has the state name |
||||
* country_name_row - row number which has the country name |
||||
* clli_name_row - row number which has CLLI information |
||||
|
||||
Example: Tugboat Plugin Usage |
||||
----------------------------- |
||||
|
||||
1. Required Input(Refer to 'spyglass/examples' folder to get these inputs) |
||||
|
||||
a) Excel File: SiteDesignSpec_v0.1.xlsx |
||||
b) Excel Spec: excel_spec_upstream.yaml |
||||
c) Site Config: site_config.yaml |
||||
d) Template_dir: '../examples/templates' |
||||
e) Site name: airship-seaworthy |
||||
|
||||
2. Spyglass CLI Command: |
||||
|
||||
.. code-block:: bash |
||||
|
||||
spyglass m -i -p tugboat -x SiteDesignSpec_v0.1.xlsx \ |
||||
-e excel_spec_upstream.yaml -c site_config.yaml \ |
||||
-s airship-seaworthy -t <relative path to J2 templates dir> |
||||
@ -1,7 +1,10 @@
|
||||
click==7.0 |
||||
click-plugins==1.1.1 |
||||
jinja2==2.10 |
||||
jsonschema==3.0.1 |
||||
openpyxl==2.5.4 |
||||
netaddr==0.7.19 |
||||
pyyaml==5.1 |
||||
requests==2.21.0 |
||||
|
||||
git+https://opendev.org/airship/spyglass-plugin-xls.git#egg=spyglass-plugin-xls |
||||
|
||||
@ -1,508 +0,0 @@
|
||||
# Copyright 2018 AT&T Intellectual Property. All other rights reserved. |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
import logging |
||||
import pprint |
||||
import re |
||||
|
||||
import formation_client |
||||
import requests |
||||
import urllib3 |
||||
|
||||
from spyglass.data_extractor.base import BaseDataSourcePlugin |
||||
import spyglass.data_extractor.custom_exceptions as exceptions |
||||
|
||||
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) |
||||
|
||||
LOG = logging.getLogger(__name__) |
||||
|
||||
|
||||
class FormationPlugin(BaseDataSourcePlugin): |
||||
|
||||
def __init__(self, region): |
||||
# Save site name is valid |
||||
if not region: |
||||
LOG.error("Site: None! Spyglass exited!") |
||||
LOG.info("Check spyglass --help for details") |
||||
exit() |
||||
super().__init__(region) |
||||
|
||||
self.source_type = "rest" |
||||
self.source_name = "formation" |
||||
|
||||
# Configuration parameters |
||||
self.formation_api_url = None |
||||
self.user = None |
||||
self.password = None |
||||
self.token = None |
||||
|
||||
# Formation objects |
||||
self.client_config = None |
||||
self.formation_api_client = None |
||||
|
||||
# Site related data |
||||
self.region_zone_map = {} |
||||
self.site_name_id_mapping = {} |
||||
self.zone_name_id_mapping = {} |
||||
self.region_name_id_mapping = {} |
||||
self.rack_name_id_mapping = {} |
||||
self.device_name_id_mapping = {} |
||||
LOG.info("Initiated data extractor plugin:{}".format(self.source_name)) |
||||
|
||||
def set_config_opts(self, conf): |
||||
"""Sets the config params passed by CLI""" |
||||
|
||||
LOG.info("Plugin params passed:\n{}".format(pprint.pformat(conf))) |
||||
self._validate_config_options(conf) |
||||
self.formation_api_url = conf["url"] |
||||
self.user = conf["user"] |
||||
self.password = conf["password"] |
||||
self.token = conf.get("token", None) |
||||
|
||||
self._get_formation_client() |
||||
self._update_site_and_zone(self.region) |
||||
|
||||
def get_plugin_conf(self, kwargs): |
||||
"""Validates the plugin param and return if success""" |
||||
|
||||
if not kwargs["formation_url"]: |
||||
LOG.error("formation_url not specified! Spyglass exited!") |
||||
exit() |
||||
url = kwargs["formation_url"] |
||||
|
||||
if not kwargs["formation_user"]: |
||||
LOG.error("formation_user not specified! Spyglass exited!") |
||||
exit() |
||||
user = kwargs["formation_user"] |
||||
|
||||
if not kwargs["formation_password"]: |
||||
LOG.error("formation_password not specified! Spyglass exited!") |
||||
exit() |
||||
password = kwargs['formation_password'] |
||||
|
||||
plugin_conf = {"url": url, "user": user, "password": password} |
||||
return plugin_conf |
||||
|
||||
def _validate_config_options(self, conf): |
||||
"""Validate the CLI params passed |
||||
|
||||
The method checks for missing parameters and terminates |
||||
Spyglass execution if found so. |
||||
""" |
||||
|
||||
missing_params = [] |
||||
for key in conf.keys(): |
||||
if conf[key] is None: |
||||
missing_params.append(key) |
||||
if len(missing_params) != 0: |
||||
LOG.error("Missing Plugin Params{}:".format(missing_params)) |
||||
exit() |
||||
|
||||
# Implement helper classes |
||||
|
||||
def _generate_token(self): |
||||
"""Generate token for Formation |
||||
|
||||
Formation API does not provide separate resource to generate |
||||
token. This is a workaround to call directly Formation API |
||||
to get token instead of using Formation client. |
||||
""" |
||||
|
||||
# Create formation client config object |
||||
self.client_config = formation_client.Configuration() |
||||
self.client_config.host = self.formation_api_url |
||||
self.client_config.username = self.user |
||||
self.client_config.password = self.password |
||||
self.client_config.verify_ssl = False |
||||
|
||||
# Assumes token is never expired in the execution of this tool |
||||
if self.token: |
||||
return self.token |
||||
|
||||
url = self.formation_api_url + "/zones" |
||||
try: |
||||
token_response = requests.get( |
||||
url, |
||||
auth=(self.user, self.password), |
||||
verify=self.client_config.verify_ssl, |
||||
) |
||||
except requests.exceptions.ConnectionError: |
||||
raise exceptions.FormationConnectionError( |
||||
"Incorrect URL: {}".format(url)) |
||||
|
||||
if token_response.status_code == 200: |
||||
self.token = token_response.json().get("X-Subject-Token", None) |
||||
else: |
||||
raise exceptions.TokenGenerationError( |
||||
"Unable to generate token because {}".format( |
||||
token_response.reason)) |
||||
|
||||
return self.token |
||||
|
||||
def _get_formation_client(self): |
||||
"""Create formation client object |
||||
|
||||
Formation uses X-Auth-Token for authentication and should be in |
||||
format "user|token". |
||||
Generate the token and add it formation config object. |
||||
""" |
||||
|
||||
token = self._generate_token() |
||||
self.client_config.api_key = {"X-Auth-Token": self.user + "|" + token} |
||||
self.formation_api_client = \ |
||||
formation_client.ApiClient(self.client_config) |
||||
|
||||
def _update_site_and_zone(self, region): |
||||
"""Get Zone name and Site name from region""" |
||||
|
||||
zone = self._get_zone_by_region_name(region) |
||||
site = self._get_site_by_zone_name(zone) |
||||
|
||||
# zone = region[:-1] |
||||
# site = zone[:-1] |
||||
|
||||
self.region_zone_map[region] = {} |
||||
self.region_zone_map[region]["zone"] = zone |
||||
self.region_zone_map[region]["site"] = site |
||||
|
||||
def _get_zone_by_region_name(self, region_name): |
||||
zone_api = formation_client.ZonesApi(self.formation_api_client) |
||||
zones = zone_api.zones_get() |
||||
|
||||
# Walk through each zone and get regions |
||||
# Return when region name matches |
||||
for zone in zones: |
||||
self.zone_name_id_mapping[zone.name] = zone.id |
||||
zone_regions = self.get_regions(zone.name) |
||||
if region_name in zone_regions: |
||||
return zone.name |
||||
|
||||
return None |
||||
|
||||
def _get_site_by_zone_name(self, zone_name): |
||||
site_api = formation_client.SitesApi(self.formation_api_client) |
||||
sites = site_api.sites_get() |
||||
|
||||
# Walk through each site and get zones |
||||
# Return when site name matches |
||||
for site in sites: |
||||
self.site_name_id_mapping[site.name] = site.id |
||||
site_zones = self.get_zones(site.name) |
||||
if zone_name in site_zones: |
||||
return site.name |
||||
|
||||
return None |
||||
|
||||
def _get_site_id_by_name(self, site_name): |
||||
if site_name in self.site_name_id_mapping: |
||||
return self.site_name_id_mapping.get(site_name) |
||||
|
||||
site_api = formation_client.SitesApi(self.formation_api_client) |
||||
sites = site_api.sites_get() |
||||
for site in sites: |
||||
self.site_name_id_mapping[site.name] = site.id |
||||
if site.name == site_name: |
||||
return site.id |
||||
|
||||
def _get_zone_id_by_name(self, zone_name): |
||||
if zone_name in self.zone_name_id_mapping: |
||||
return self.zone_name_id_mapping.get(zone_name) |
||||
|
||||
zone_api = formation_client.ZonesApi(self.formation_api_client) |
||||
zones = zone_api.zones_get() |
||||
for zone in zones: |
||||
if zone.name == zone_name: |
||||
self.zone_name_id_mapping[zone.name] = zone.id |
||||
return zone.id |
||||
|
||||
def _get_region_id_by_name(self, region_name): |
||||
if region_name in self.region_name_id_mapping: |
||||
return self.region_name_id_mapping.get(region_name) |
||||
|
||||
for zone in self.zone_name_id_mapping: |
||||
self.get_regions(zone) |
||||
|
||||
return self.region_name_id_mapping.get(region_name, None) |
||||
|
||||
def _get_rack_id_by_name(self, rack_name): |
||||
if rack_name in self.rack_name_id_mapping: |
||||
return self.rack_name_id_mapping.get(rack_name) |
||||
|
||||
for zone in self.zone_name_id_mapping: |
||||
self.get_racks(zone) |
||||
|
||||
return self.rack_name_id_mapping.get(rack_name, None) |
||||
|
||||
def _get_device_id_by_name(self, device_name): |
||||
if device_name in self.device_name_id_mapping: |
||||
return self.device_name_id_mapping.get(device_name) |
||||
|
||||
self.get_hosts(self.zone) |
||||
|
||||
return self.device_name_id_mapping.get(device_name, None) |
||||
|
||||
def _get_racks(self, zone, rack_type="compute"): |
||||
zone_id = self._get_zone_id_by_name(zone) |
||||
rack_api = formation_client.RacksApi(self.formation_api_client) |
||||
racks = rack_api.zones_zone_id_racks_get(zone_id) |
||||
|
||||
racks_list = [] |
||||
for rack in racks: |
||||
rack_name = rack.name |
||||
self.rack_name_id_mapping[rack_name] = rack.id |
||||
if rack.rack_type.name == rack_type: |
||||
racks_list.append(rack_name) |
||||
|
||||
return racks_list |
||||
|
||||
# Functions that will be used internally within this plugin |
||||
|
||||
def get_zones(self, site=None): |
||||
zone_api = formation_client.ZonesApi(self.formation_api_client) |
||||
|
||||
if site is None: |
||||
zones = zone_api.zones_get() |
||||
else: |
||||
site_id = self._get_site_id_by_name(site) |
||||
zones = zone_api.sites_site_id_zones_get(site_id) |
||||
|
||||
zones_list = [] |
||||
for zone in zones: |
||||
zone_name = zone.name |
||||
self.zone_name_id_mapping[zone_name] = zone.id |
||||
zones_list.append(zone_name) |
||||
|
||||
return zones_list |
||||
|
||||
def get_regions(self, zone): |
||||
zone_id = self._get_zone_id_by_name(zone) |
||||
region_api = formation_client.RegionApi(self.formation_api_client) |
||||
regions = region_api.zones_zone_id_regions_get(zone_id) |
||||
regions_list = [] |
||||
for region in regions: |
||||
region_name = region.name |
||||
self.region_name_id_mapping[region_name] = region.id |
||||
regions_list.append(region_name) |
||||
|
||||
return regions_list |
||||
|
||||
# Implement Abstract functions |
||||
|
||||
def get_racks(self, region): |
||||
zone = self.region_zone_map[region]["zone"] |
||||
return self._get_racks(zone, rack_type="compute") |
||||
|
||||
def get_hosts(self, region, rack=None): |
||||
zone = self.region_zone_map[region]["zone"] |
||||
zone_id = self._get_zone_id_by_name(zone) |
||||
device_api = formation_client.DevicesApi(self.formation_api_client) |
||||
control_hosts = device_api.zones_zone_id_control_nodes_get(zone_id) |
||||
compute_hosts = device_api.zones_zone_id_devices_get( |
||||
zone_id, type="KVM") |
||||
|
||||
hosts_list = [] |
||||
for host in control_hosts: |
||||
self.device_name_id_mapping[host.aic_standard_name] = host.id |
||||
hosts_list.append( |
||||
{ |
||||
"name": host.aic_standard_name, |
||||
"type": "controller", |
||||
"rack_name": host.rack_name, |
||||
"host_profile": host.host_profile_name, |
||||
}) |
||||
|
||||
for host in compute_hosts: |
||||
self.device_name_id_mapping[host.aic_standard_name] = host.id |
||||
hosts_list.append( |
||||
{ |
||||
"name": host.aic_standard_name, |
||||
"type": "compute", |
||||
"rack_name": host.rack_name, |
||||
"host_profile": host.host_profile_name, |
||||
}) |
||||
""" |
||||
for host in itertools.chain(control_hosts, compute_hosts): |
||||
self.device_name_id_mapping[host.aic_standard_name] = host.id |
||||
hosts_list.append({ |
||||
'name': host.aic_standard_name, |
||||
'type': host.categories[0], |
||||
'rack_name': host.rack_name, |
||||
'host_profile': host.host_profile_name |
||||
}) |
||||
""" |
||||
|
||||
return hosts_list |
||||
|
||||
def get_networks(self, region): |
||||
zone = self.region_zone_map[region]["zone"] |
||||
zone_id = self._get_zone_id_by_name(zone) |
||||
region_id = self._get_region_id_by_name(region) |
||||
vlan_api = formation_client.VlansApi(self.formation_api_client) |
||||
vlans = vlan_api.zones_zone_id_regions_region_id_vlans_get( |
||||
zone_id, region_id) |
||||
|
||||
# Case when vlans list is empty from |
||||
# zones_zone_id_regions_region_id_vlans_get |
||||
if len(vlans) == 0: |
||||
# get device-id from the first host and get the network details |
||||
hosts = self.get_hosts(self.region) |
||||
host = hosts[0]["name"] |
||||
device_id = self._get_device_id_by_name(host) |
||||
vlans = \ |
||||
vlan_api.zones_zone_id_devices_device_id_vlans_get(zone_id, |
||||
device_id) |
||||
|
||||
LOG.debug("Extracted region network information\n{}".format(vlans)) |
||||
vlans_list = [] |
||||
for vlan_ in vlans: |
||||
if len(vlan_.vlan.ipv4) != 0: |
||||
tmp_vlan = { |
||||
"name": self._get_network_name_from_vlan_name( |
||||
vlan_.vlan.name), |
||||
"vlan": vlan_.vlan.vlan_id, |
||||
"subnet": vlan_.vlan.subnet_range, |
||||
"gateway": vlan_.ipv4_gateway, |
||||
"subnet_level": vlan_.vlan.subnet_level |
||||
} |
||||
vlans_list.append(tmp_vlan) |
||||
|
||||
return vlans_list |
||||
|
||||
def get_ips(self, region, host=None): |
||||
zone = self.region_zone_map[region]["zone"] |
||||
zone_id = self._get_zone_id_by_name(zone) |
||||
|
||||
if host: |
||||
hosts = [host] |
||||
else: |
||||
hosts = [] |
||||
hosts_dict = self.get_hosts(zone) |
||||
for host in hosts_dict: |
||||
hosts.append(host["name"]) |
||||
|
||||
vlan_api = formation_client.VlansApi(self.formation_api_client) |
||||
ip_ = {} |
||||
|
||||
for host in hosts: |
||||
device_id = self._get_device_id_by_name(host) |
||||
vlans = \ |
||||
vlan_api.zones_zone_id_devices_device_id_vlans_get(zone_id, |
||||
device_id) |
||||
LOG.debug("Received VLAN Network Information\n{}".format(vlans)) |
||||
ip_[host] = {} |
||||
for vlan_ in vlans: |
||||
# TODO(pg710r) We need to handle the case when incoming ipv4 |
||||
# list is empty |
||||
if len(vlan_.vlan.ipv4) != 0: |
||||
name = self._get_network_name_from_vlan_name( |
||||
vlan_.vlan.name) |
||||
ipv4 = vlan_.vlan.ipv4[0].ip |
||||
LOG.debug( |
||||
"vlan:{},name:{},ip:{},vlan_name:{}".format( |
||||
vlan_.vlan.vlan_id, name, ipv4, vlan_.vlan.name)) |
||||
# TODD(pg710r) This code needs to extended to support ipv4 |
||||
# and ipv6 |
||||
# ip_[host][name] = {'ipv4': ipv4} |
||||
ip_[host][name] = ipv4 |
||||
|
||||
return ip_ |
||||
|
||||
def _get_network_name_from_vlan_name(self, vlan_name): |
||||
"""Network names are ksn, oam, oob, overlay, storage, pxe |
||||
|
||||
The following mapping rules apply: |
||||
vlan_name contains "ksn" the network name is "calico" |
||||
vlan_name contains "storage" the network name is "storage" |
||||
vlan_name contains "server" the network name is "oam" |
||||
vlan_name contains "ovs" the network name is "overlay" |
||||
vlan_name contains "ILO" the network name is "oob" |
||||
""" |
||||
|
||||
network_names = { |
||||
"ksn": "calico", |
||||
"storage": "storage", |
||||
"server": "oam", |
||||
"ovs": "overlay", |
||||
"ILO": "oob", |
||||
"pxe": "pxe", |
||||
} |
||||
|
||||
for name in network_names: |
||||
# Make a pattern that would ignore case. |
||||
# if name is 'ksn' pattern name is '(?i)(ksn)' |
||||
name_pattern = "(?i)({})".format(name) |
||||
if re.search(name_pattern, vlan_name): |
||||
return network_names[name] |
||||
# Return empty string is vlan_name is not matched with network_names |
||||
return "" |
||||
|
||||
def get_dns_servers(self, region): |
||||
try: |
||||
zone = self.region_zone_map[region]["zone"] |
||||
zone_id = self._get_zone_id_by_name(zone) |
||||
zone_api = formation_client.ZonesApi(self.formation_api_client) |
||||
zone_ = zone_api.zones_zone_id_get(zone_id) |
||||
except formation_client.rest.ApiException as e: |
||||
raise exceptions.ApiClientError(e.msg) |
||||
|
||||
if not zone_.ipv4_dns: |
||||
LOG.warning("No dns server") |
||||
return [] |
||||
|
||||
dns_list = [] |
||||
for dns in zone_.ipv4_dns: |
||||
dns_list.append(dns.ip) |
||||
|
||||
return dns_list |
||||
|
||||
def get_ntp_servers(self, region): |
||||
return [] |
||||
|
||||
def get_ldap_information(self, region): |
||||
return {} |
||||
|
||||
def get_location_information(self, region): |
||||
"""Get location information for a zone and return""" |
||||
|
||||
site = self.region_zone_map[region]["site"] |
||||
site_id = self._get_site_id_by_name(site) |
||||
site_api = formation_client.SitesApi(self.formation_api_client) |
||||
site_info = site_api.sites_site_id_get(site_id) |
||||
|
||||
try: |
||||
return { |
||||
# 'corridor': site_info.corridor, |
||||
"name": site_info.city, |
||||
"state": site_info.state, |
||||
"country": site_info.country, |
||||
"physical_location_id": site_info.clli, |
||||
} |
||||
except AttributeError as e: |
||||
raise exceptions.MissingAttributeError( |
||||
"Missing {} information in {}".format(e, site_info.city)) |
||||
|
||||
def get_domain_name(self, region): |
||||
try: |
||||
zone = self.region_zone_map[region]["zone"] |
||||
zone_id = self._get_zone_id_by_name(zone) |
||||
zone_api = formation_client.ZonesApi(self.formation_api_client) |
||||
zone_ = zone_api.zones_zone_id_get(zone_id) |
||||
except formation_client.rest.ApiException as e: |
||||
raise exceptions.ApiClientError(e.msg) |
||||
|
||||
if not zone_.dns: |
||||
LOG.warning("Got None while running get domain name") |
||||
return None |
||||
|
||||
return zone_.dns |
||||
@ -1,38 +0,0 @@
|
||||
# Copyright 2018 AT&T Intellectual Property. All other rights reserved. |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
|
||||
class BaseError(Exception): |
||||
pass |
||||
|
||||
|
||||
class NotEnoughIp(BaseError): |
||||
|
||||
def __init__(self, cidr, total_nodes): |
||||
self.cidr = cidr |
||||
self.total_nodes = total_nodes |
||||
|
||||
def display_error(self): |
||||
print("{} can not handle {} nodes".format(self.cidr, self.total_nodes)) |
||||
|
||||
|
||||
class NoSpecMatched(BaseError): |
||||
|
||||
def __init__(self, excel_specs): |
||||
self.specs = excel_specs |
||||
|
||||
def display_error(self): |
||||
print( |
||||
"No spec matched. Following are the available specs:\n".format( |
||||
self.specs)) |
||||
@ -1,417 +0,0 @@
|
||||
# Copyright 2018 AT&T Intellectual Property. All other rights reserved. |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
import logging |
||||
import pprint |
||||
import re |
||||
import sys |
||||
|
||||
from openpyxl import load_workbook |
||||
from openpyxl import Workbook |
||||
import yaml |
||||
|
||||
from spyglass.data_extractor.custom_exceptions import NoSpecMatched |
||||
|
||||
LOG = logging.getLogger(__name__) |
||||
|
||||
|
||||
class ExcelParser(object): |
||||
"""Parse data from excel into a dict""" |
||||
|
||||
def __init__(self, file_name, excel_specs): |
||||
self.file_name = file_name |
||||
with open(excel_specs, "r") as f: |
||||
spec_raw_data = f.read() |
||||
self.excel_specs = yaml.safe_load(spec_raw_data) |
||||
# A combined design spec, returns a workbook object after combining |
||||
# all the inputs excel specs |
||||
combined_design_spec = self.combine_excel_design_specs(file_name) |
||||
self.wb_combined = combined_design_spec |
||||
self.filenames = file_name |
||||
self.spec = "xl_spec" |
||||
|
||||
@staticmethod |
||||
def sanitize(string): |
||||
"""Remove extra spaces and convert string to lower case""" |
||||
|
||||
return string.replace(" ", "").lower() |
||||
|
||||
def compare(self, string1, string2): |
||||
"""Compare the strings""" |
||||
|
||||
return bool(re.search(self.sanitize(string1), self.sanitize(string2))) |
||||
|
||||
def validate_sheet(self, spec, sheet): |
||||
"""Check if the sheet is correct or not""" |
||||
|
||||
ws = self.wb_combined[sheet] |
||||
header_row = self.excel_specs["specs"][spec]["header_row"] |
||||
ipmi_header = self.excel_specs["specs"][spec]["ipmi_address_header"] |
||||
ipmi_column = self.excel_specs["specs"][spec]["ipmi_address_col"] |
||||
header_value = ws.cell(row=header_row, column=ipmi_column).value |
||||
return bool(self.compare(ipmi_header, header_value)) |
||||
|
||||
def find_correct_spec(self): |
||||
"""Find the correct spec""" |
||||
|
||||
for spec in self.excel_specs["specs"]: |
||||
sheet_name = self.excel_specs["specs"][spec]["ipmi_sheet_name"] |
||||
for sheet in self.wb_combined.sheetnames: |
||||
if self.compare(sheet_name, sheet): |
||||
self.excel_specs["specs"][spec]["ipmi_sheet_name"] = sheet |
||||
if self.validate_sheet(spec, sheet): |
||||
return spec |
||||
raise NoSpecMatched(self.excel_specs) |
||||
|
||||
def get_ipmi_data(self): |
||||
"""Read IPMI data from the sheet""" |
||||
|
||||
ipmi_data = {} |
||||
hosts = [] |
||||
spec_ = self.excel_specs["specs"][self.spec] |
||||
provided_sheetname = spec_["ipmi_sheet_name"] |
||||
workbook_object, extracted_sheetname = \ |
||||
self.get_xl_obj_and_sheetname(provided_sheetname) |
||||
if workbook_object is not None: |
||||
ws = workbook_object[extracted_sheetname] |
||||
else: |
||||
ws = self.wb_combined[provided_sheetname] |
||||
row = spec_["start_row"] |
||||
end_row = spec_["end_row"] |
||||
hostname_col = spec_["hostname_col"] |
||||
ipmi_address_col = spec_["ipmi_address_col"] |
||||
host_profile_col = spec_["host_profile_col"] |
||||
ipmi_gateway_col = spec_["ipmi_gateway_col"] |
||||
previous_server_gateway = None |
||||
while row <= end_row: |
||||
hostname = \ |
||||
self.sanitize(ws.cell(row=row, column=hostname_col).value) |
||||
hosts.append(hostname) |
||||
ipmi_address = ws.cell(row=row, column=ipmi_address_col).value |
||||
if "/" in ipmi_address: |
||||
ipmi_address = ipmi_address.split("/")[0] |
||||
ipmi_gateway = ws.cell(row=row, column=ipmi_gateway_col).value |
||||
if ipmi_gateway: |
||||
previous_server_gateway = ipmi_gateway |
||||
else: |
||||
ipmi_gateway = previous_server_gateway |
||||
host_profile = ws.cell(row=row, column=host_profile_col).value |
||||
try: |
||||
if host_profile is None: |
||||
raise RuntimeError( |
||||
"No value read from " |
||||
"{} sheet:{} row:{}, col:{}".format( |
||||
self.file_name, self.spec, row, host_profile_col)) |
||||
except RuntimeError as rerror: |
||||
LOG.critical(rerror) |
||||
sys.exit("Tugboat exited!!") |
||||
ipmi_data[hostname] = { |
||||
"ipmi_address": ipmi_address, |
||||
"ipmi_gateway": ipmi_gateway, |
||||
"host_profile": host_profile, |
||||
"type": type, # FIXME (Ian Pittwood): shadows type built-in |
||||
} |
||||
row += 1 |
||||
LOG.debug( |
||||
"ipmi data extracted from excel:\n{}".format( |
||||
pprint.pformat(ipmi_data))) |
||||
LOG.debug( |
||||
"host data extracted from excel:\n{}".format( |
||||
pprint.pformat(hosts))) |
||||
return [ipmi_data, hosts] |
||||
|
||||
def get_private_vlan_data(self, ws): |
||||
"""Get private vlan data from private IP sheet""" |
||||
|
||||
vlan_data = {} |
||||
row = self.excel_specs["specs"][self.spec]["vlan_start_row"] |
||||
end_row = self.excel_specs["specs"][self.spec]["vlan_end_row"] |
||||
type_col = self.excel_specs["specs"][self.spec]["net_type_col"] |
||||
vlan_col = self.excel_specs["specs"][self.spec]["vlan_col"] |
||||
while row <= end_row: |
||||
cell_value = ws.cell(row=row, column=type_col).value |
||||
if cell_value: |
||||
vlan = ws.cell(row=row, column=vlan_col).value |
||||
if vlan: |
||||
vlan = vlan.lower() |
||||
vlan_data[vlan] = cell_value |
||||
row += 1 |
||||
LOG.debug( |
||||
"vlan data extracted from excel:\n%s" % pprint.pformat(vlan_data)) |
||||
return vlan_data |
||||
|
||||
def get_private_network_data(self): |
||||
"""Read network data from the private ip sheet""" |
||||
|
||||
spec_ = self.excel_specs["specs"][self.spec] |
||||
provided_sheetname = spec_["private_ip_sheet"] |
||||
workbook_object, extracted_sheetname = \ |
||||
self.get_xl_obj_and_sheetname(provided_sheetname) |
||||
if workbook_object is not None: |
||||
ws = workbook_object[extracted_sheetname] |
||||
else: |
||||
ws = self.wb_combined[provided_sheetname] |
||||
vlan_data = self.get_private_vlan_data(ws) |
||||
network_data = {} |
||||
row = spec_["net_start_row"] |
||||
end_row = spec_["net_end_row"] |
||||
col = spec_["net_col"] |
||||
vlan_col = spec_["net_vlan_col"] |
||||
old_vlan = "" |
||||
while row <= end_row: |
||||
vlan = ws.cell(row=row, column=vlan_col).value |
||||
if vlan: |
||||
vlan = vlan.lower() |
||||
network = ws.cell(row=row, column=col).value |
||||
if vlan and network: |
||||
net_type = vlan_data[vlan] |
||||
if "vlan" not in network_data: |
||||
network_data[net_type] = {"vlan": vlan, "subnet": []} |
||||
elif not vlan and network: |
||||
# If vlan is not present then assign old vlan to vlan as vlan |
||||
# value is spread over several rows |
||||
vlan = old_vlan |
||||
else: |
||||
row += 1 |
||||
continue |
||||
network_data[vlan_data[vlan]]["subnet"].append(network) |
||||
old_vlan = vlan |
||||
row += 1 |
||||
for network in network_data: |
||||
network_data[network]["is_common"] = True |
||||
""" |
||||
if len(network_data[network]['subnet']) > 1: |
||||
network_data[network]['is_common'] = False |
||||
else: |
||||
network_data[network]['is_common'] = True |
||||
LOG.debug("private network data extracted from excel:\n%s" |
||||
% pprint.pformat(network_data)) |
||||
""" |
||||
return network_data |
||||
|
||||
def get_public_network_data(self): |
||||
"""Read public network data from public ip data""" |
||||
|
||||
spec_ = self.excel_specs["specs"][self.spec] |
||||
provided_sheetname = spec_["public_ip_sheet"] |
||||
workbook_object, extracted_sheetname = self.get_xl_obj_and_sheetname( |
||||
provided_sheetname) |
||||
if workbook_object is not None: |
||||
ws = workbook_object[extracted_sheetname] |
||||
else: |
||||
ws = self.wb_combined[provided_sheetname] |
||||
oam_row = spec_["oam_ip_row"] |
||||
oam_col = spec_["oam_ip_col"] |
||||
oam_vlan_col = spec_["oam_vlan_col"] |
||||
ingress_row = spec_["ingress_ip_row"] |
||||
oob_row = spec_["oob_net_row"] |
||||
col = spec_["oob_net_start_col"] |
||||
end_col = spec_["oob_net_end_col"] |
||||
network_data = { |
||||
"oam": { |
||||
"subnet": [ws.cell(row=oam_row, column=oam_col).value], |
||||
"vlan": ws.cell(row=oam_row, column=oam_vlan_col).value, |
||||
}, |
||||
"ingress": ws.cell(row=ingress_row, column=oam_col).value, |
||||
"oob": { |
||||
"subnet": [], |
||||
} |
||||
} |
||||
while col <= end_col: |
||||
cell_value = ws.cell(row=oob_row, column=col).value |
||||
if cell_value: |
||||
network_data["oob"]["subnet"].append(self.sanitize(cell_value)) |
||||
col += 1 |
||||
LOG.debug( |
||||
"public network data extracted from excel:\n%s" % |
||||
pprint.pformat(network_data)) |
||||
return network_data |
||||
|
||||
def get_site_info(self): |
||||
"""Read location, dns, ntp and ldap data""" |
||||
|
||||
spec_ = self.excel_specs["specs"][self.spec] |
||||
provided_sheetname = spec_["dns_ntp_ldap_sheet"] |
||||
workbook_object, extracted_sheetname = \ |
||||
self.get_xl_obj_and_sheetname(provided_sheetname) |
||||
if workbook_object is not None: |
||||
ws = workbook_object[extracted_sheetname] |
||||
else: |
||||
ws = self.wb_combined[provided_sheetname] |
||||
dns_row = spec_["dns_row"] |
||||
dns_col = spec_["dns_col"] |
||||
ntp_row = spec_["ntp_row"] |
||||
ntp_col = spec_["ntp_col"] |
||||
domain_row = spec_["domain_row"] |
||||
domain_col = spec_["domain_col"] |
||||
login_domain_row = spec_["login_domain_row"] |
||||
ldap_col = spec_["ldap_col"] |
||||
global_group = spec_["global_group"] |
||||
ldap_search_url_row = spec_["ldap_search_url_row"] |
||||
dns_servers = ws.cell(row=dns_row, column=dns_col).value |
||||
ntp_servers = ws.cell(row=ntp_row, column=ntp_col).value |
||||
try: |
||||
if dns_servers is None: |
||||
raise RuntimeError( |
||||
"No value for dns_server from: " |
||||
"{} Sheet:'{}' Row:{} Col:{}".format( |
||||
self.file_name, provided_sheetname, dns_row, dns_col)) |
||||
if ntp_servers is None: |
||||
raise RuntimeError( |
||||
"No value for ntp_server from: " |
||||
"{} Sheet:'{}' Row:{} Col:{}".format( |
||||
self.file_name, provided_sheetname, ntp_row, ntp_col)) |
||||
except RuntimeError as rerror: |
||||
LOG.critical(rerror) |
||||
sys.exit("Tugboat exited!!") |
||||
|
||||
dns_servers = dns_servers.replace("\n", " ") |
||||
ntp_servers = ntp_servers.replace("\n", " ") |
||||
if "," in dns_servers: |
||||
dns_servers = dns_servers.split(",") |
||||
else: |
||||
dns_servers = dns_servers.split() |
||||
if "," in ntp_servers: |
||||
ntp_servers = ntp_servers.split(",") |
||||
else: |
||||
ntp_servers = ntp_servers.split() |
||||
site_info = { |
||||
"location": self.get_location_data(), |
||||
"dns": dns_servers, |
||||
"ntp": ntp_servers, |
||||
"domain": ws.cell(row=domain_row, column=domain_col).value, |
||||
"ldap": { |
||||
"subdomain": ws.cell(row=login_domain_row, |
||||
column=ldap_col).value, |
||||
"common_name": ws.cell(row=global_group, |
||||
column=ldap_col).value, |
||||
"url": ws.cell(row=ldap_search_url_row, column=ldap_col).value, |
||||
}, |
||||
} |
||||
LOG.debug( |
||||
"Site Info extracted from\ |
||||
excel:\n%s", |
||||
pprint.pformat(site_info), |
||||
) |
||||
return site_info |
||||
|
||||
def get_location_data(self): |
||||
"""Read location data from the site and zone sheet""" |
||||
|
||||
spec_ = self.excel_specs["specs"][self.spec] |
||||
provided_sheetname = spec_["location_sheet"] |
||||
workbook_object, extracted_sheetname = \ |
||||
self.get_xl_obj_and_sheetname(provided_sheetname) |
||||
if workbook_object is not None: |
||||
ws = workbook_object[extracted_sheetname] |
||||
else: |
||||
ws = self.wb_combined[provided_sheetname] |
||||
corridor_row = spec_["corridor_row"] |
||||
column = spec_["column"] |
||||
site_name_row = spec_["site_name_row"] |
||||
state_name_row = spec_["state_name_row"] |
||||
country_name_row = spec_["country_name_row"] |
||||
clli_name_row = spec_["clli_name_row"] |
||||
return { |
||||
"corridor": ws.cell(row=corridor_row, column=column).value, |
||||
"name": ws.cell(row=site_name_row, column=column).value, |
||||
"state": ws.cell(row=state_name_row, column=column).value, |
||||
"country": ws.cell(row=country_name_row, column=column).value, |
||||
"physical_location": ws.cell(row=clli_name_row, |
||||
column=column).value, |
||||
} |
||||
|
||||
def validate_sheet_names_with_spec(self): |
||||
"""Checks is sheet name in spec file matches with excel file""" |
||||
|
||||
spec = list(self.excel_specs["specs"].keys())[0] |
||||
spec_item = self.excel_specs["specs"][spec] |
||||
sheet_name_list = [] |
||||
ipmi_header_sheet_name = spec_item["ipmi_sheet_name"] |
||||
sheet_name_list.append(ipmi_header_sheet_name) |
||||
private_ip_sheet_name = spec_item["private_ip_sheet"] |
||||
sheet_name_list.append(private_ip_sheet_name) |
||||
public_ip_sheet_name = spec_item["public_ip_sheet"] |
||||
sheet_name_list.append(public_ip_sheet_name) |
||||
dns_ntp_ldap_sheet_name = spec_item["dns_ntp_ldap_sheet"] |
||||
sheet_name_list.append(dns_ntp_ldap_sheet_name) |
||||
location_sheet_name = spec_item["location_sheet"] |
||||
sheet_name_list.append(location_sheet_name) |
||||
try: |
||||
for sheetname in sheet_name_list: |
||||
workbook_object, extracted_sheetname = \ |
||||
self.get_xl_obj_and_sheetname(sheetname) |
||||
if workbook_object is not None: |
||||
wb = workbook_object |
||||
sheetname = extracted_sheetname |
||||
else: |
||||
wb = self.wb_combined |
||||
|
||||
if sheetname not in wb.sheetnames: |
||||
raise RuntimeError( |
||||
"SheetName '{}' not found ".format(sheetname)) |
||||
except RuntimeError as rerror: |
||||
LOG.critical(rerror) |
||||
sys.exit("Tugboat exited!!") |
||||
|
||||
LOG.info("Sheet names in excel spec validated") |
||||
|
||||
def get_data(self): |
||||
"""Create a dict with combined data""" |
||||
|
||||
self.validate_sheet_names_with_spec() |
||||
ipmi_data = self.get_ipmi_data() |
||||
network_data = self.get_private_network_data() |
||||
public_network_data = self.get_public_network_data() |
||||
site_info_data = self.get_site_info() |
||||
data = { |
||||
"ipmi_data": ipmi_data, |
||||
"network_data": { |
||||
"private": network_data, |
||||
"public": public_network_data, |
||||
}, |
||||
"site_info": site_info_data, |
||||
} |
||||
LOG.debug( |
||||
"Location data extracted from excel:\n%s" % pprint.pformat(data)) |
||||
return data |
||||
|
||||
def combine_excel_design_specs(self, filenames): |
||||
"""Combines multiple excel file to a single design spec""" |
||||
|
||||
design_spec = Workbook() |
||||
for exel_file in filenames: |
||||
loaded_workbook = load_workbook(exel_file, data_only=True) |
||||
for names in loaded_workbook.sheetnames: |
||||
design_spec_worksheet = design_spec.create_sheet(names) |
||||
loaded_workbook_ws = loaded_workbook[names] |
||||
for row in loaded_workbook_ws: |
||||
for cell in row: |
||||
design_spec_worksheet[cell.coordinate].value = \ |
||||
cell.value |
||||
return design_spec |
||||
|
||||
def get_xl_obj_and_sheetname(self, sheetname): |
||||
"""The logic confirms if the sheetname is specified for example as: |
||||
|
||||