starlingx/app-gen-tool
Code Issues Proposed changes

Convert repository into python package & refactor.

Browse Source 
Application files and folders moved into package format.
Added a setup.py which creates the package version, name
and also a console script so the software can be run as
a Command Line program after installation.

Software is also refactored and code split up into files.
This makes software more maintainable.

Follow up changes may be required, until this repo is
added to PiPY, the package must be installed via direct
url to git repoistory.  This package uses template files
which have to be deliverately installed into the target
system.

Updated README to reflect changes.

Test Plan:
PASS: Ability to install via "pip install -e ." locally.
PASS: Ability to generate app via examples/ folder inputs
      using locall install.
PASS: Test ability for package to work when installed
      via url.

Story: 2010937
Task: 48915
Change-Id: If286dc02db68c9f2f91295559b5ebc77cd6091e8
Signed-off-by: Joshua Reed <joshua.reed@windriver.com>
This commit is contained in:
Joshua Reed 2023-11-01 14:03:45 -07:00
parent 26c30e6cfd
commit 056f47449e
29 changed files with 513 additions and 316 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
.gitignore vendored
View File

@ -1,3 +1,11 @@
# Virtual Environment
venv/
env/
env/
example/output
example/adminer
# Package / cache items
.eggs
.idea
__pycache__
*.egg-info/

40
.vscode/launch.json vendored Normal file
View File

@ -0,0 +1,40 @@
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Python: Generator",
"type": "python",
"request": "launch",
"program": "${workspaceFolder}/app_gen_tool/cmd/generator.py",
"cwd": "${workspaceFolder}",
"console": "integratedTerminal",
"justMyCode": true,
"args": [
"--input=${workspaceFolder}/example/app-test-adminer-1.yaml",
"--output=${workspaceFolder}/example/output",
"--type=fluxcd",
"--overwrite"
]
},
{
// First pip install this repo
// starlingx-app-generator --input=./TEST/app-test-adminer-1.yaml.yaml --output=./TEMP/output --overwrite
"name": "Python: Installed Generator",
"type": "python",
"request": "launch",
"program": "${workspaceFolder}/venv/bin/starlingx-app-generator",
"cwd": "${workspaceFolder}",
"console": "integratedTerminal",
"justMyCode": true,
"args": [
"--input=${workspaceFolder}/example/app-test-adminer-1.yaml",
"--output=${workspaceFolder}/example/output",
"--type=fluxcd",
"--overwrite"
]
}
]
}

129
README.md
View File

@ -43,11 +43,12 @@ ways to the Kubernetes cluster(s) that StarlingX manages:
```TODO Elaborate on the vantages of deploying an app as a StarlingX app```
## Tools requirements
## Software Requirements
- Helm version 2+
- Python version 3.8+
- `pyyaml` version 6.0+
- Python packages tracked in ./requirements.txt
- For Testing, Python packages tracked in ./test-requirements.txt
## Prerequisites
@ -56,43 +57,81 @@ application to be deployed as a StarlingX App it needs to be designed so it can
run on [Kubernetes](https://kubernetes.io/).
Additionally, it needs to provide a [Helm Chart](https://helm.sh/)
which will be managed via [Flux](https://fluxcd.io/) by StarlingX itself.
which will be managed via [FluxCD](https://fluxcd.io/) by StarlingX itself.
## Installation
### Option 1 - Install in same folder
```shell
git clone https://opendev.org/starlingx/app-gen-tool.git
cd ./app-gen-tool/
python -m venv venv
source ./venv/bin/activate
pip install -e .
```
### Option 2 - Install From Url
```shell
python -m venv venv
source ./venv/bin/activate
pip install git+https://opendev.org/starlingx/app-gen-tool.git
```
## Generate the StarlingX Application package
Clone the app-generator repository.
```shell
git clone https://opendev.org/starlingx/tools.git
cd tools/app-gen-tool/
```
This is what you'll find in the `app-gen-tool` folder of the repository:
This is what you'll find in the `app-gen-tool` repository:
```shell
.
├── app-gen.py
├── app_manifest.yaml
├── bin
│   └── fetch_chart_info.sh
├── LICENSE
├── README.md
├── template
│   ├── armada-chartgroup.template
│   ├── armada-chart.template
│   └── armada-manifest.template
├── templates_flux
│   ├── base
│   │   ├── helmrepository.template
│   │   ├── kustomization.template
│   │   └── namespace.template
│   ├── fluxcd-manifest
│   │   ├── helmrelease.template
├── app_gen_tool
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-39.pyc
│   │   ├── application.cpython-39.pyc
│   │   ├── common.cpython-39.pyc
│   │   ├── constants.cpython-39.pyc
│   │   └── generator.cpython-39.pyc
│   ├── application.py
│   ├── cmd
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   │   ├── __init__.cpython-39.pyc
│   │   │   └── generator.cpython-39.pyc
│   │   └── generator.py
│   ├── common.py
│   ├── constants.py
│   ├── generator.py
│   ├── template_armada
│   │   ├── armada-chart.template
│   │   ├── armada-chartgroup.template
│   │   └── armada-manifest.template
│   ├── templates_flux
│   │   ├── base
│   │   │   ├── helmrepository.template
│   │   │   ├── kustomization.template
│   │   │   └── namespace.template
│   │   ├── fluxcd-manifest
│   │   │   ├── helmrelease.template
│   │   │   └── kustomization.template
│   │   └── kustomization.template
│   └── kustomization.template
└── templates_plugins
├── common.template
├── helm.template
├── kustomize.template
└── lifecycle.template
│   └── templates_plugins
│   ├── common.template
│   ├── helm.template
│   ├── kustomize.template
│   └── lifecycle.template
├── app_manifest.yaml
├── example
│   ├── adminer-0.2.1.tgz
│   ├── app-test-adminer-1.yaml
├── requirements.txt
├── scripts
│   └── fetch_chart_info.sh
├── setup.py
├── test-requirements.txt
├── tox.ini
```
@ -140,7 +179,7 @@ Note that the minimum required fields that will need to be filled in order
for the StarlingX App Generator to work properly will depend on the intended
type of packaging.
>_NOTE_: The two other sections bellow ([Metadata file configuration](#metadata-file-configuration)
>_NOTE_: The two other sections bellow ([Metadata file configuration](#metadata-file-configuration)
and [App Setup Configuration](#app-setup-configuration)) will only be necessary
if you intend to package your application utilizing FluxCD.
### Metadata File Configuration
@ -175,17 +214,33 @@ more advanced use cases you may want to refer to [the documentation](https://set
## Run the StarlingX App Generator
One must install via pip install method described in the Installation section.
To get command line options:
```shell
python3 app-gen.py -i app_manifest.yaml -t <armada/flux/both>
starlingx-app-generator -h
```
Recommend reviewing the '-h' output for a full list of options.
Here is an example.
```shell
starlingx-app-generator -i app_manifest.yaml -t [armada/fluxcd/both] -o ./output
```
With the command above, the StarlingX App Generator will create a set of files
and package everything in the chosed StarlingX format.
and package everything in the chosen StarlingX format, and then finally place
the file in an output folder.
## Supported Packaging Methods
The following sections explain in high-level the most important parts of the
package.
### Flux Packaging
#### FluxCD Manifest
The generator will first create the FluxCD Manifest following the structure below:
@ -302,7 +357,7 @@ In order to allow such customization, the generator provides additional
functions to modify specific files in the package.
```shell
python3 app-gen.py -i app_manifest.yaml -t <armada/flux/both> [-o ./output] [--overwrite] [--no-package]|[--package-only]
starlingx-app-generator -i app_manifest.yaml -t <armada/flux/both> [-o ./output] [--overwrite] [--no-package]|[--package-only]
```
Where:
@ -366,3 +421,5 @@ have been created as they should. Particularly, the `setup.cfg` may need careful
attention if the modifications on the plugin file should be reflected in it.
### Armada Manifest
#TODO

0
app_gen_tool/__init__.py Normal file
View File

292
app-gen.py → app_gen_tool/application.py
View File

@ -1,44 +1,26 @@
import yaml
import os
import sys, getopt, getpass
import subprocess
import hashlib
import tarfile
import re
import shutil
import subprocess
import sys
import tarfile
import yaml
from urllib import request
## Variables for armada packaging
ARMADA_CHART_TEMPLATE = 'template_armada/armada-chart.template'
ARMADA_CHARTGROUP_TEMPLATE = 'template_armada/armada-chartgroup.template'
ARMADA_MANIFEST_TEMPLATE = 'template_armada/armada-manifest.template'
BIN_FETCH_CHART_INFO = 'bin/fetch_chart_info.sh'
from app_gen_tool import constants
from app_gen_tool.common import to_camel_case
## Variables for FluxCD packaging
FLUX_KUSTOMIZATION_TEMPLATE = 'templates_flux/kustomization.template'
FLUX_BASE_TEMPLATES = 'templates_flux/base/'
FLUX_MANIFEST_TEMPLATE = 'templates_flux/fluxcd-manifest'
FLUX_COMMON_TEMPLATE = 'templates_plugins/common.template'
FLUX_HELM_TEMPLATE = 'templates_plugins/helm.template'
FLUX_KUSTOMIZE_TEMPLATE = 'templates_plugins/kustomize.template'
FLUX_LIFECYCLE_TEMPLATE = 'templates_plugins/lifecycle.template'
class Application():
TEMP_USER_DIR = '/tmp/' + getpass.getuser() + '/'
# Temp app work dir to hold git repo and upstream tarball
# TEMP_APP_DIR = TEMP_USER_DIR/appName
TEMP_APP_DIR = ''
APP_GEN_PY_PATH = os.path.split(os.path.realpath(__file__))[0]
def to_camel_case(s):
return s[0].lower() + s.title().replace('_','')[1:] if s else s
class Application:
def __init__(self, app_data):
def __init__(self, app_data: dict, app_type: str):
# Initialize application config
self._app = {}
self._app_type = app_type
self._app = app_data['appManifestFile-config']
self._temp_app_dir = constants.TEMP_USER_DIR + self.get_app_name() + '/'
self.APP_NAME = self._app['appName']
self.APP_NAME_WITH_UNDERSCORE = self._app['appName'].replace('-', '_')
@ -58,14 +40,16 @@ class Application:
self._chart = app_data['appManifestFile-config']['chart']
for i in range(len(self._chart)):
self._chart[i]['namespace'] = self._app['namespace']
self._chart[i]['releasePrefix'] = self._app['manifest']['releasePrefix']
self._listcharts['chart_names'].append(self._chart[i]['name'])
if self._app_type == "armada":
self._chart[i]['releasePrefix'] = self._app['manifest']['releasePrefix']
# Initialize Armada manifest
self._manifest = app_data['appManifestFile-config']['manifest']
self._manifest['chart_groups'] = []
for i in range(len(self._chartgroup)):
self._manifest['chart_groups'].append(self._chartgroup[i]['name'])
if self._app_type == "armada":
self._manifest = app_data['appManifestFile-config']['manifest']
self._manifest['chart_groups'] = []
for i in range(len(self._chartgroup)):
self._manifest['chart_groups'].append(self._chartgroup[i]['name'])
# Initialize setup data
self.plugin_setup = app_data['setupFile-config']
@ -242,7 +226,7 @@ class Application:
def _fetch_info_from_chart(self, chart_idx):
a_chart = self._chart[chart_idx]
bin_fetch_script = APP_GEN_PY_PATH + '/' + BIN_FETCH_CHART_INFO
bin_fetch_script = constants.APP_GEN_PY_PATH + '/' + constants.BIN_FETCH_CHART_INFO
# waitLabelKey
# search for the key of label which indicates '.Release.Name'
# within deployment, statefulset, daemonset yaml file
@ -285,13 +269,13 @@ class Application:
if chart['_pathType'] == 'git':
gitname = ''
# download git
if not os.path.exists(TEMP_APP_DIR):
os.makedirs(TEMP_APP_DIR)
if not os.path.exists(self._temp_app_dir):
os.makedirs(self._temp_app_dir)
# if the git folder exists, check git name and use that folder
# otherwise git clone from upstream
if not os.path.exists(TEMP_APP_DIR + chart['_gitname']):
if not os.path.exists(self._temp_app_dir + chart['_gitname']):
saved_pwd = os.getcwd()
os.chdir(TEMP_APP_DIR)
os.chdir(self._temp_app_dir)
cmd = ['git', 'clone', chart['path']]
subproc = subprocess.run(cmd, env=os.environ.copy(), \
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
@ -305,7 +289,7 @@ class Application:
else:
# git pull to ensure folder up-to-date
saved_pwd = os.getcwd()
os.chdir(TEMP_APP_DIR + chart['_gitname'])
os.chdir(self._temp_app_dir + chart['_gitname'])
cmd = ['git', 'pull']
subproc = subprocess.run(cmd, env=os.environ.copy(), \
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
@ -316,15 +300,15 @@ class Application:
os.chdir(saved_pwd)
return False
os.chdir(saved_pwd)
path = TEMP_APP_DIR + chart['_gitname'] + '/' + chart['subpath']
path = self._temp_app_dir + chart['_gitname'] + '/' + chart['subpath']
elif chart['_pathType'] == 'tarball':
if not os.path.exists(TEMP_APP_DIR):
os.makedirs(TEMP_APP_DIR)
if not os.path.exists(self._temp_app_dir):
os.makedirs(self._temp_app_dir)
try:
# check whether it's a url or local tarball
if not os.path.exists(chart['path']):
# download tarball
tarpath = TEMP_APP_DIR + chart['_tarname'] + '.tgz'
tarpath = self._temp_app_dir + chart['_tarname'] + '.tgz'
if not os.path.exists(tarpath):
res = request.urlopen(chart['path'])
with open(tarpath, 'wb') as f:
@ -339,12 +323,12 @@ class Application:
chart['_tarArcname'] = chart_files[0].split('/')[0]
if not os.path.exists(chart['_tarArcname']):
for chart_file in chart_files:
chart_tar.extract(chart_file, TEMP_APP_DIR)
chart_tar.extract(chart_file, self._temp_app_dir)
chart_tar.close()
except Exception as e:
print('Error: %s' % e)
return False
path = TEMP_APP_DIR + chart['_tarArcname'] + '/' + chart['subpath']
path = self._temp_app_dir + chart['_tarArcname'] + '/' + chart['subpath']
elif chart['_pathType'] == 'dir':
path = chart['path']
@ -367,9 +351,9 @@ class Application:
os.remove(manifest_file)
# update schema path to abspath
chart_template = APP_GEN_PY_PATH + '/' + ARMADA_CHART_TEMPLATE
chartgroup_template = APP_GEN_PY_PATH + '/' + ARMADA_CHARTGROUP_TEMPLATE
manifest_template = APP_GEN_PY_PATH + '/' + ARMADA_MANIFEST_TEMPLATE
chart_template = constants.APP_GEN_PY_PATH + '/' + constants.ARMADA_CHART_TEMPLATE
chartgroup_template = constants.APP_GEN_PY_PATH + '/' + constants.ARMADA_CHARTGROUP_TEMPLATE
manifest_template = constants.APP_GEN_PY_PATH + '/' + constants.ARMADA_MANIFEST_TEMPLATE
# generate chart schema
try:
@ -441,14 +425,14 @@ class Application:
flux_dir = self._app['outputManifestDir']
# update schema path to abspath
kustomization_template = APP_GEN_PY_PATH + '/' + FLUX_KUSTOMIZATION_TEMPLATE
kustomization_template = constants.APP_GEN_PY_PATH + '/' + constants.FLUX_KUSTOMIZATION_TEMPLATE
base_helmrepo_template = APP_GEN_PY_PATH + '/' + FLUX_BASE_TEMPLATES + '/helmrepository.template'
base_kustom_template = APP_GEN_PY_PATH + '/' + FLUX_BASE_TEMPLATES + '/kustomization.template'
base_namespace_template = APP_GEN_PY_PATH + '/' + FLUX_BASE_TEMPLATES + '/namespace.template'
base_helmrepo_template = constants.APP_GEN_PY_PATH + '/' + constants.FLUX_BASE_TEMPLATES + '/helmrepository.template'
base_kustom_template = constants.APP_GEN_PY_PATH + '/' + constants.FLUX_BASE_TEMPLATES + '/kustomization.template'
base_namespace_template = constants.APP_GEN_PY_PATH + '/' + constants.FLUX_BASE_TEMPLATES + '/namespace.template'
manifest_helmrelease_template = APP_GEN_PY_PATH + '/' + FLUX_MANIFEST_TEMPLATE + '/helmrelease.template'
manifest_kustomization_template = APP_GEN_PY_PATH + '/' + FLUX_MANIFEST_TEMPLATE + '/kustomization.template'
manifest_helmrelease_template = constants.APP_GEN_PY_PATH + '/' + constants.FLUX_MANIFEST_TEMPLATE + '/helmrelease.template'
manifest_kustomization_template = constants.APP_GEN_PY_PATH + '/' + constants.FLUX_MANIFEST_TEMPLATE + '/kustomization.template'
manifest = self._app
chartgroup = self._listcharts
@ -575,10 +559,10 @@ class Application:
plugin_dir = self._app['outputPluginDir']
common_template = APP_GEN_PY_PATH + '/' + FLUX_COMMON_TEMPLATE
helm_template = APP_GEN_PY_PATH + '/' + FLUX_HELM_TEMPLATE
kustomize_template = APP_GEN_PY_PATH + '/' + FLUX_KUSTOMIZE_TEMPLATE
lifecycle_template = APP_GEN_PY_PATH + '/' + FLUX_LIFECYCLE_TEMPLATE
common_template = constants.APP_GEN_PY_PATH + '/' + constants.FLUX_COMMON_TEMPLATE
helm_template = constants.APP_GEN_PY_PATH + '/' + constants.FLUX_HELM_TEMPLATE
kustomize_template = constants.APP_GEN_PY_PATH + '/' + constants.FLUX_KUSTOMIZE_TEMPLATE
lifecycle_template = constants.APP_GEN_PY_PATH + '/' + constants.FLUX_LIFECYCLE_TEMPLATE
appname = 'k8sapp_' + self.APP_NAME_WITH_UNDERSCORE
namespace = self._app['namespace']
@ -1010,8 +994,7 @@ class Application:
# 6 - Package helm-charts
for chart in self._chart:
ret = self._gen_helm_chart_tarball(
chart, self._app['outputFluxChartDir'])
ret = self._gen_helm_chart_tarball(chart, self._app['outputFluxChartDir'])
if ret:
print('Helm chart %s tarball generated!' % chart['name'])
print('')
@ -1032,7 +1015,7 @@ class Application:
ret = self._gen_checksum_and_app_tarball(self._app['outputFluxCDDir'])
if ret:
print('Checksum generated!')
print('FluxCD App tarball generated at %s/%s' % (self._app['outputDir'], ret))
print('FluxCD App tarball generated at %s/%s' % (self._app['outputFluxCDDir'], ret))
print('')
else:
print('Checksum and App tarball generation failed!')
@ -1045,186 +1028,3 @@ class Application:
print(self._manifest)
print(self._chartgroup)
print(self._chart)
def parse_yaml(yaml_in):
yaml_data=''
try:
with open(yaml_in) as f:
yaml_data = yaml.safe_load(f)
except FileNotFoundError:
print('Error: %s no found' % yaml_in )
except Exception as e:
print('Error: Invalid yaml file')
return yaml_data
def check_manifest(manifest_data):
# TODO: check more mandatory key/values in manifest yaml
# check app values
if 'appName' not in manifest_data['appManifestFile-config']:
print('Error: \'appName\' is missing.')
return False
if 'namespace' not in manifest_data['appManifestFile-config']:
print('Error: \'namespace\' is missing.')
return False
if 'appVersion' not in manifest_data['appManifestFile-config']:
print('Error: \'appVersion\' is missing.')
return False
# # check manifest values
# if 'manifest' not in manifest_data['appManifestFile-config']:
# print('Error: \'manifest\'is missing.')
# return False
# if 'releasePrefix' not in manifest_data['manifest']:
# print('Error: Manifest attribute \'releasePrefix\' is missing.')
# return False
# check chartGroup values
if 'chartGroup' not in manifest_data['appManifestFile-config']:
print('Error: \'chartGroup\' is missing.')
return False
# check chart values
if 'chart' not in manifest_data['appManifestFile-config']:
print('Error: \'chart\' is missing.')
return False
for chart in manifest_data['appManifestFile-config']['chart']:
# check chart name
if 'name' not in chart:
print('Error: Chart attribute \'name\' is missing.')
return False
# check chart version
if 'version' not in chart:
print('Error: Chart attribute \'version\' is missing.')
return False
# check chart path, supporting: dir, git, tarball
if 'path' not in chart:
print('Error: Chart attribute \'path\' is missing in chart %s.' % chart['name'])
return False
else:
# TODO: To support branches/tags in git repo
if chart['path'].endswith('.git'):
if 'subpath' not in chart:
print('Error: Chart attribute \'subpath\' is missing in chart %s.' % chart['name'])
return False
chart['_pathType'] = 'git'
gitname = re.search('[^/]+(?=\.git$)',chart['path']).group()
if gitname:
chart['_gitname'] = gitname
else:
print('Error: Invalid \'path\' in chart %s.' % chart['name'])
print(' only \'local dir\', \'.git\', \'.tar.gz\', \'.tgz\' are supported')
return False
elif chart['path'].endswith('.tar.gz') or chart['path'].endswith('.tgz'):
if 'subpath' not in chart:
print('Error: Chart attribute \'subpath\' is missing in chart %s.' % chart['name'])
return False
chart['_pathType'] = 'tarball'
tarname = re.search('[^/]+(?=\.tgz)|[^/]+(?=\.tar\.gz)',chart['path']).group()
if tarname:
chart['_tarname'] = tarname
else:
print('Error: Invalid \'path\' in chart %s.' % chart['name'])
print(' only \'local dir\', \'.git\', \'.tar.gz\', \'.tgz\' are supported')
return False
else:
if not os.path.isdir(chart['path']):
print('Error: Invalid \'path\' in chart %s.' % chart['name'])
print(' only \'local dir\', \'.git\', \'.tar.gz\', \'.tgz\' are supported')
return False
chart['_pathType'] = 'dir'
return True
def generate_app(file_in, out_folder, package_type, overwrite, no_package, package_only):
global TEMP_APP_DIR
app_data = parse_yaml(file_in)
if not app_data:
print('Parse yaml error')
return
if not check_manifest(app_data):
print('Application manifest is not valid')
return
app = Application(app_data)
TEMP_APP_DIR = TEMP_USER_DIR + app.get_app_name() + '/'
app_out = out_folder + '/' + app.get_app_name()
if not os.path.exists(app_out):
os.makedirs(app_out)
elif overwrite:
shutil.rmtree(app_out)
elif package_only:
pass
else:
print('Output folder %s exists, please remove it or use --overwrite.' % app_out)
sys.exit()
if package_type == 'armada' or package_type == 'both':
app.gen_armada_app(app_out, no_package, package_only)
if package_type == 'flux' or package_type == 'both':
app.gen_flux_app(app_out, no_package, package_only)
def main(argv):
input_file = ''
output_folder = '.'
package_type = ''
overwrite = False
package_only = False
no_package = False
try:
options, args = getopt.getopt(argv, 'hi:o:t:', \
['help', 'input=', 'output=', 'type=', 'overwrite', 'no-package', 'package-only'])
except getopt.GetoptError:
print('Error: Invalid argument')
sys.exit(1)
for option, value in options:
if option in ('-h', '--help'):
print('StarlingX User Application Generator')
print('')
print('Usage:')
print(' python app-gen.py [Option]')
print('')
print('Options:')
print(' -i, --input yaml_file generate app from yaml_file')
print(' -o, --output folder generate app to output folder')
print(' -t, --type package select Armada,Flux or Both packaging')
print(' --overwrite overwrite the output dir')
print(' --no-package does not create app tarball')
print(' --package-only only creates tarball from dir')
print(' -h, --help this help')
if option in ('--overwrite'):
overwrite = True
if option in ('-i', '--input'):
input_file = value
if option in ('-o', '--output'):
output_folder = value
if option in ('-t', '--type'):
package_type = value.lower()
if option in ('--no-package'):
no_package = True
if option in ('--package-only'):
package_only = True
if not package_type:
print('Error: Select type of packaging (armada/flux/both)')
sys.exit(1)
if not os.path.isfile(os.path.abspath(input_file)):
print('Error: input file not found')
sys.exit(1)
if input_file:
generate_app(os.path.abspath(input_file), os.path.abspath(output_folder), package_type, overwrite, no_package, package_only)
if __name__ == '__main__':
main(sys.argv[1:])

0
app_gen_tool/cmd/__init__.py Normal file
View File

11
app_gen_tool/cmd/generator.py Normal file
View File

@ -0,0 +1,11 @@
#!/usr/bin/env python
import sys
from app_gen_tool.generator import main as gen_main
def main():
gen_main(sys.argv[1:])
# Entry point here kept to allow for debug/testing.
if __name__ == "__main__":
main()

2
app_gen_tool/common.py Normal file
View File

@ -0,0 +1,2 @@
def to_camel_case(s):
return s[0].lower() + s.title().replace('_','')[1:] if s else s

21
app_gen_tool/constants.py Normal file
View File

@ -0,0 +1,21 @@
## Variables for armada packaging
import getpass
import os
ARMADA_CHART_TEMPLATE = 'template_armada/armada-chart.template'
ARMADA_CHARTGROUP_TEMPLATE = 'template_armada/armada-chartgroup.template'
ARMADA_MANIFEST_TEMPLATE = 'template_armada/armada-manifest.template'
BIN_FETCH_CHART_INFO = 'scripts/fetch_chart_info.sh'
## Variables for FluxCD packaging
FLUX_KUSTOMIZATION_TEMPLATE = 'templates_flux/kustomization.template'
FLUX_BASE_TEMPLATES = 'templates_flux/base/'
FLUX_MANIFEST_TEMPLATE = 'templates_flux/fluxcd-manifest'
FLUX_COMMON_TEMPLATE = 'templates_plugins/common.template'
FLUX_HELM_TEMPLATE = 'templates_plugins/helm.template'
FLUX_KUSTOMIZE_TEMPLATE = 'templates_plugins/kustomize.template'
FLUX_LIFECYCLE_TEMPLATE = 'templates_plugins/lifecycle.template'
TEMP_USER_DIR = '/tmp/' + getpass.getuser() + '/'
APP_GEN_PY_PATH = os.path.split(os.path.realpath(__file__))[0]

188
app_gen_tool/generator.py Normal file
View File

@ -0,0 +1,188 @@
import getopt
import os
import sys
import re
import shutil
import yaml
from app_gen_tool.application import Application
def parse_yaml(yaml_in):
yaml_data=''
try:
with open(yaml_in) as f:
yaml_data = yaml.safe_load(f)
except FileNotFoundError:
print('Error: %s no found' % yaml_in )
except Exception as e:
print('Error: Invalid yaml file')
return yaml_data
def check_manifest(manifest_data):
# TODO: check more mandatory key/values in manifest yaml
# check app values
if 'appName' not in manifest_data['appManifestFile-config']:
print('Error: \'appName\' is missing.')
return False
if 'namespace' not in manifest_data['appManifestFile-config']:
print('Error: \'namespace\' is missing.')
return False
if 'appVersion' not in manifest_data['appManifestFile-config']:
print('Error: \'appVersion\' is missing.')
return False
# # check manifest values
# if 'manifest' not in manifest_data['appManifestFile-config']:
# print('Error: \'manifest\'is missing.')
# return False
# if 'releasePrefix' not in manifest_data['manifest']:
# print('Error: Manifest attribute \'releasePrefix\' is missing.')
# return False
# check chartGroup values
if 'chartGroup' not in manifest_data['appManifestFile-config']:
print('Error: \'chartGroup\' is missing.')
return False
# check chart values
if 'chart' not in manifest_data['appManifestFile-config']:
print('Error: \'chart\' is missing.')
return False
for chart in manifest_data['appManifestFile-config']['chart']:
# check chart name
if 'name' not in chart:
print('Error: Chart attribute \'name\' is missing.')
return False
# check chart version
if 'version' not in chart:
print('Error: Chart attribute \'version\' is missing.')
return False
# check chart path, supporting: dir, git, tarball
if 'path' not in chart:
print('Error: Chart attribute \'path\' is missing in chart %s.' % chart['name'])
return False
else:
# TODO: To support branches/tags in git repo
if chart['path'].endswith('.git'):
if 'subpath' not in chart:
print('Error: Chart attribute \'subpath\' is missing in chart %s.' % chart['name'])
return False
chart['_pathType'] = 'git'
gitname = re.search('[^/]+(?=\.git$)',chart['path']).group()
if gitname:
chart['_gitname'] = gitname
else:
print('Error: Invalid \'path\' in chart %s.' % chart['name'])
print(' only \'local dir\', \'.git\', \'.tar.gz\', \'.tgz\' are supported')
return False
elif chart['path'].endswith('.tar.gz') or chart['path'].endswith('.tgz'):
if 'subpath' not in chart:
print('Error: Chart attribute \'subpath\' is missing in chart %s.' % chart['name'])
return False
chart['_pathType'] = 'tarball'
tarname = re.search('[^/]+(?=\.tgz)|[^/]+(?=\.tar\.gz)',chart['path']).group()
if tarname:
chart['_tarname'] = tarname
else:
print('Error: Invalid \'path\' in chart %s.' % chart['name'])
print(' only \'local dir\', \'.git\', \'.tar.gz\', \'.tgz\' are supported')
return False
else:
if not os.path.isdir(chart['path']):
print('Error: Invalid \'path\' in chart %s.' % chart['name'])
print(' only \'local dir\', \'.git\', \'.tar.gz\', \'.tgz\' are supported')
return False
chart['_pathType'] = 'dir'
return True
def generate_app(file_in, out_folder, package_type, overwrite, no_package, package_only):
app_data = parse_yaml(file_in)
if not app_data:
print('Parse yaml error')
return
if not check_manifest(app_data):
print('Application manifest is not valid')
return
app = Application(app_data, package_type)
app_out = out_folder + '/' + app.get_app_name()
if not os.path.exists(app_out):
os.makedirs(app_out)
elif overwrite:
shutil.rmtree(app_out)
elif package_only:
pass
else:
print('Output folder %s exists, please remove it or use --overwrite.' % app_out)
sys.exit()
if package_type == 'armada' or package_type == 'both':
app.gen_armada_app(app_out, no_package, package_only)
if package_type == 'fluxcd' or package_type == 'both':
app.gen_flux_app(app_out, no_package, package_only)
def main(argv):
input_file = ''
output_folder = '.'
package_type = ''
overwrite = False
package_only = False
no_package = False
try:
options, args = getopt.getopt(argv, 'hi:o:t:', \
['help', 'input=', 'output=', 'type=', 'overwrite', 'no-package', 'package-only'])
except getopt.GetoptError:
print('Error: Invalid argument')
sys.exit(1)
for option, value in options:
if option in ('-h', '--help'):
print('StarlingX User Application Generator')
print('')
print('Usage:')
print(' python app-gen.py [Option]')
print('')
print('Options:')
print(' -i, --input yaml_file generate app from yaml_file')
print(' -o, --output folder generate app to output folder')
print(' -t, --type package select Armada,Flux or Both packaging')
print(' --overwrite overwrite the output dir')
print(' --no-package does not create app tarball')
print(' --package-only only creates tarball from dir')
print(' -h, --help this help')
if option in ('--overwrite'):
overwrite = True
if option in ('-i', '--input', '--input='):
input_file = value
if option in ('-o', '--output', '--output='):
output_folder = value
if option in ('-t', '--type', '--type='):
package_type = value.lower()
if option in ('--no-package'):
no_package = True
if option in ('--package-only'):
package_only = True
if not package_type:
print('Error: Select type of packaging (armada/fluxcd/both)')
sys.exit(1)
if not os.path.isfile(os.path.abspath(input_file)):
print('Error: input file not found')
sys.exit(1)
if input_file:
generate_app(os.path.abspath(input_file), os.path.abspath(output_folder), package_type, overwrite, no_package, package_only)

0
template_armada/armada-chart.template → app_gen_tool/templates_armada/armada-chart.template
View File

0
template_armada/armada-chartgroup.template → app_gen_tool/templates_armada/armada-chartgroup.template
View File

0
template_armada/armada-manifest.template → app_gen_tool/templates_armada/armada-manifest.template
View File

0
templates_flux/base/helmrepository.template → app_gen_tool/templates_flux/base/helmrepository.template
View File

0
templates_flux/base/kustomization.template → app_gen_tool/templates_flux/base/kustomization.template
View File

0
templates_flux/base/namespace.template → app_gen_tool/templates_flux/base/namespace.template
View File

0
templates_flux/fluxcd-manifest/helmrelease.template → app_gen_tool/templates_flux/fluxcd-manifest/helmrelease.template
View File

0
templates_flux/fluxcd-manifest/kustomization.template → app_gen_tool/templates_flux/fluxcd-manifest/kustomization.template
View File

0
templates_flux/kustomization.template → app_gen_tool/templates_flux/kustomization.template
View File

0
templates_plugins/common.template → app_gen_tool/templates_plugins/common.template
View File

0
templates_plugins/helm.template → app_gen_tool/templates_plugins/helm.template
View File

0
templates_plugins/kustomize.template → app_gen_tool/templates_plugins/kustomize.template
View File

0
templates_plugins/lifecycle.template → app_gen_tool/templates_plugins/lifecycle.template
View File

32
example.yaml
View File

@ -1,32 +0,0 @@
---
appName: stx-app
namespace: stx-app
version: 1.0-1
chart:
- name: chart1
path: /path/to/chart1
wait: 600
values:
test_key: test_value
- name: chart2
path: https://git/of/chart2.git
- name: chart3
path: https://tarball/of/chart3-sha.tgz
chartGroup:
- name: chartgroup1
description: "This is the first chartgroup"
sequenced: true
chart_group:
- chart1
- chart2
- name: chartgroup2
description: "This is the second chartgroup"
sequenced: false
chart_group:
- chart3
manifest:
name: stx-app-manifest
releasePrefix: myprefix
chart_groups:
- chartgroup1
- chartgroup2

BIN
example/adminer-0.2.1.tgz Normal file
View File

Binary file not shown.

46
example/app-test-adminer-1.yaml Normal file
View File

@ -0,0 +1,46 @@
---
## App Manifest Configuration
appManifestFile-config:
appName: app-adminer
appVersion: 1.0-1
namespace: default
chart:
- name: adminer
version: 0.2.1
path: ./example/adminer
chartGroup:
- name: adminer
chart_names:
- adminer
#################################################
## App Metadata Configuration
# for further details about possible configurations on this file, please
# visit the link: https://wiki.openstack.org/wiki/StarlingX/Containers/StarlingXAppsInternals#metadata.yaml
metadataFile-config:
# the following configurations are optional
# uncomment and configure properly the ones you need for your application metadata
upgrades:
auto_update: true
supported_k8s_version:
minimum: 1.21.8
maximum: 1.26.1
k8s_upgrades:
auto_update: true
timing: pre
maintain_user_overrides: true
#################################################
## App Setup Configuration
# if you wish to see a setup.cfg example, please see the link
# https://opendev.org/starlingx/app-dell-storage/src/branch/master/python3-k8sapp-dell-storage/k8sapp_dell_storage/setup.cfg
setupFile-config:
metadata:
author: StarlingX
author_email: starlingx-discuss@lists.starlingx.io
url: https://www.starlingx.io/
classifier:
- "Environment :: OpenStack"

3
requirements.txt
View File

@ -1 +1,2 @@
PyYAML
pyyaml==6.0.1
wheel==0.41.2

0
bin/fetch_chart_info.sh → scripts/fetch_chart_info.sh
View File

55
setup.py Normal file
View File

@ -0,0 +1,55 @@
import os
import pathlib
import pkg_resources
import sysconfig
from glob import glob
from setuptools import setup, find_packages
PACKAGE_NAME = 'starlingx-app-generator'
PACKAGE_DIRECTORY = 'app_gen_tool'
SITE_PACKAGES_DIR = sysconfig.get_paths()["purelib"]
with pathlib.Path('requirements.txt').open() as requirements_txt:
install_requires = [
str(requirement)
for requirement
in pkg_resources.parse_requirements(requirements_txt)
]
def _get_list_of_files(directory):
final_list = []
file_list = glob(directory + "/**/*", recursive=True)
for file in file_list:
# Want to exclude folders
if os.path.isfile(file):
final_list.append(file)
return final_list
setup(
name=PACKAGE_NAME,
version='0.1.0',
description='Tool to generate StarlingX applications from templates using python.',
install_requires=install_requires,
entry_points={
'console_scripts': [
f'{PACKAGE_NAME} = {PACKAGE_DIRECTORY}.cmd.generator:main',
],
},
packages=find_packages(),
include_package_data=True,
data_files=[
(
f'{SITE_PACKAGES_DIR}/{PACKAGE_DIRECTORY}/templates_flux',
_get_list_of_files(f'{PACKAGE_DIRECTORY}/templates_flux')
),
(
f'{SITE_PACKAGES_DIR}/{PACKAGE_DIRECTORY}/templates_plugins',
_get_list_of_files(f'{PACKAGE_DIRECTORY}/templates_plugins')
),
(
f'{SITE_PACKAGES_DIR}/{PACKAGE_DIRECTORY}/templates_armada',
_get_list_of_files(f'{PACKAGE_DIRECTORY}/templates_armada')
),
]
)