Adds the parser and related files.
- Adding parser module. - Adds custom exceptions and tests cases. - Adds tools folder. - Updated the documentation. - Readme usage section now imports usage.rst. - Improvements to tox.ini - Adds config folder.
This commit is contained in:
parent
e279e39d75
commit
c6d99ea667
33
README.rst
33
README.rst
@ -26,8 +26,8 @@ These are the major goals which are accomplished by the parser:
|
||||
Training-Labs
|
||||
-------------
|
||||
|
||||
[Training-labs](https://git.openstack.org/openstack/training-labs) is part
|
||||
of OpeNStack Documentation team and provides an unique tool to deploy core
|
||||
`Training-labs <https://git.openstack.org/openstack/training-labs>`_ is part
|
||||
of OpenStack Documentation team and provides an unique tool to deploy core
|
||||
OpenStack services. Training labs closely follows installation guides for
|
||||
the OpenStack deployment steps.
|
||||
|
||||
@ -35,7 +35,7 @@ the OpenStack deployment steps.
|
||||
Installation Guides (OpenStack Installation Tutorial)
|
||||
-----------------------------------------------------
|
||||
|
||||
[Installation guides](https://docs.openstack.org) provides step by step
|
||||
`Installation guides <https://docs.openstack.org>`_ provides step by step
|
||||
instructions to deploy OpenStack on a multi-node cluster.
|
||||
|
||||
|
||||
@ -47,38 +47,15 @@ More Details
|
||||
and training-labs repository.
|
||||
- The generated output (parsed files) should then be triggered via.
|
||||
training-labs to deploy the OpenStack cluster.
|
||||
- Additionally, this project should showcase and allow the workflow in the
|
||||
- Additionally, this project should showcase and allow the work-flow in the
|
||||
OpenStack CI for installation guides and cross-project installation-guides.
|
||||
|
||||
|
||||
Usage
|
||||
-----
|
||||
|
||||
- To run the parser please clone the [openstack-manuals](git://git.openstack.org/openstack/openstack-manuals)
|
||||
repository and update the configuration file.
|
||||
- Additionally, if you wish to deploy OpenStack cluster, also clone the [training-labs](git://git.openstack.org/openstack/training-labs)
|
||||
repository.
|
||||
- Run the parser:
|
||||
|
||||
$ python parser.py
|
||||
|
||||
- Check the generated scripts (location in the configuration file), copy them
|
||||
to training-labs: labs/osbash/scripts/ folder.
|
||||
- Run training labs:
|
||||
|
||||
$ PROVIDER=kvm ./st.py -b cluster
|
||||
|
||||
- Sit back, relax and see the cluster deploy.
|
||||
|
||||
**Note:** This project is in its nascent state, especially the OpenStack
|
||||
cluster deployment part may break at many places.
|
||||
|
||||
|
||||
Roadmap
|
||||
-------
|
||||
|
||||
- Create glue-code scripts to automate setting up of various repositories
|
||||
required to easily carry the workflow.
|
||||
required to easily carry the work-flow.
|
||||
- Setup the non-voting jobs to deploy the cluster. This cluster should be
|
||||
a two node KVM/VirtualBox cluster which runs in the OpenStack CI.
|
||||
- Update the Bash templates (Jinja templates) to allow nicer Bash scripts
|
||||
|
@ -2,10 +2,11 @@
|
||||
Installation
|
||||
============
|
||||
|
||||
At the command line::
|
||||
- Clone the rst2bash repository.
|
||||
|
||||
- Clone the repository
|
||||
.. code-block:: bash
|
||||
|
||||
$ git clone git://git.openstack.org/openstack/rst2bash
|
||||
$ git clone git://git.openstack.org/openstack/rst2bash
|
||||
$ cd rst2bash
|
||||
|
||||
- Run the parser.
|
||||
- Check the usage section for more details.
|
||||
|
@ -2,19 +2,27 @@
|
||||
Usage
|
||||
=====
|
||||
|
||||
- To run the parser please clone the [openstack-manuals](git://git.openstack.org/openstack/openstack-manuals)
|
||||
repository and update the configuration file.
|
||||
- Additionally, if you wish to deploy OpenStack cluster, also clone the [training-labs](git://git.openstack.org/openstack/training-labs)
|
||||
repository.
|
||||
- Run the parser:
|
||||
- Run the parser, it will clone openstack-manuals repository, training-labs
|
||||
repository and parse the files
|
||||
|
||||
$ python parser.py
|
||||
.. code-block:: bash
|
||||
|
||||
$ ./tools/runparser.sh
|
||||
|
||||
|
||||
Make sure to run it from the root of the directory.
|
||||
- Check the generated scripts (location in the configuration file
|
||||
`rst2bash/conf`), copy them to training-labs:
|
||||
`labs/osbash/scripts/` folder.
|
||||
|
||||
- Check the generated scripts (location in the configuration file), copy them
|
||||
to training-labs: labs/osbash/scripts/ folder.
|
||||
to training-labs: `labs/osbash/scripts/` folder. Default configuration
|
||||
specifies the output location at `build/scripts/`.
|
||||
- Run training labs:
|
||||
|
||||
$ PROVIDER=kvm ./st.py -b cluster
|
||||
.. code-block:: bash
|
||||
|
||||
$ PROVIDER=kvm ./build/training-labs/labs/st.py -b cluster
|
||||
|
||||
- Sit back, relax and see the cluster deploy.
|
||||
|
||||
|
65
rst2bash/config/parser_config.yaml
Normal file
65
rst2bash/config/parser_config.yaml
Normal file
@ -0,0 +1,65 @@
|
||||
description: Provides input (RST) and output (BASH) based on the configuration below.
|
||||
rst_path: build/openstack-manuals/doc/install-guide/source
|
||||
bash_path:
|
||||
ubuntu: build/scripts/ubuntu
|
||||
rdo: build/scripts/rdo
|
||||
obs: build/scripts/obs
|
||||
debian: build/scripts/debian
|
||||
rst_files:
|
||||
- keystone-install.rst
|
||||
# - keystone-openrc.rst
|
||||
# - keystone.rst
|
||||
- keystone-users.rst
|
||||
- keystone-verify.rst
|
||||
- launch-instance-cinder.rst
|
||||
- launch-instance-networks-provider.rst
|
||||
- launch-instance-networks-selfservice.rst
|
||||
- launch-instance-provider.rst
|
||||
- launch-instance.rst
|
||||
- launch-instance-selfservice.rst
|
||||
- neutron-compute-install-option1.rst
|
||||
- neutron-compute-install-option2.rst
|
||||
- neutron-compute-install.rst
|
||||
- neutron-concepts.rst
|
||||
- neutron-controller-install-option1.rst
|
||||
- neutron-controller-install-option2.rst
|
||||
- neutron-controller-install.rst
|
||||
- neutron-next-steps.rst
|
||||
# - neutron.rst
|
||||
- neutron-verify-option1.rst
|
||||
- neutron-verify-option2.rst
|
||||
- neutron-verify.rst
|
||||
- nova-compute-install.rst
|
||||
- nova-controller-install.rst
|
||||
# - nova.rst
|
||||
- nova-verify.rst
|
||||
# - overview.rst
|
||||
- horizon-verify.rst
|
||||
- additional-services.rst
|
||||
- cinder-backup-install.rst
|
||||
- cinder-controller-install.rst
|
||||
- cinder-next-steps.rst
|
||||
# - cinder.rst
|
||||
- cinder-storage-install.rst
|
||||
- cinder-verify.rst
|
||||
- environment-memcached.rst
|
||||
- environment-messaging.rst
|
||||
- environment-networking-compute.rst
|
||||
- environment-networking-controller.rst
|
||||
- environment-networking.rst
|
||||
- environment-networking-storage-cinder.rst
|
||||
- environment-networking-verify.rst
|
||||
- environment-ntp-controller.rst
|
||||
- environment-ntp-other.rst
|
||||
- environment-ntp.rst
|
||||
- environment-ntp-verify.rst
|
||||
- environment-packages.rst
|
||||
# - environment.rst
|
||||
- environment-security.rst
|
||||
- environment-sql-database.rst
|
||||
- glance-install.rst
|
||||
# - glance.rst
|
||||
- glance-verify.rst
|
||||
- horizon-install.rst
|
||||
- horizon-next-steps.rst
|
||||
# - horizon.rst
|
41
rst2bash/exceptions.py
Normal file
41
rst2bash/exceptions.py
Normal file
@ -0,0 +1,41 @@
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
class Rst2BashException(Exception):
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class MissingTagsException(Rst2BashException):
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class NestedDistroBlocksException(Rst2BashException):
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class PathNotFoundException(Rst2BashException):
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class NoCodeBlocksException(Rst2BashException):
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class InvalidOperatorException(Rst2BashException):
|
||||
|
||||
pass
|
549
rst2bash/parser.py
Executable file
549
rst2bash/parser.py
Executable file
@ -0,0 +1,549 @@
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
from collections import defaultdict
|
||||
import os
|
||||
import re
|
||||
import yaml
|
||||
|
||||
|
||||
class BlockIndex(object):
|
||||
"""Creates indices which describes the location of blocks in rst file.
|
||||
|
||||
These indices describe the start and end location of the strings in the rst
|
||||
file. Different indices used to parse the file are:
|
||||
|
||||
AllBlocks: Contains sequential index values for all required blocks.
|
||||
CodeBlocks: Contains index values for blocks containing code.
|
||||
PathBlocks: Contains index values for blocks containing path.
|
||||
DistroBlocks: Contains index values for blocks containing OS.
|
||||
|
||||
These indices should provide the location to extract given blocks from the
|
||||
rst files. This class additionally provides various functionalities to
|
||||
easily carry out different tasks like iteration and more.
|
||||
"""
|
||||
|
||||
def __init__(self, startIndex=tuple(), endIndex=tuple()):
|
||||
|
||||
self.startIndex = tuple(startIndex)
|
||||
self.endIndex = tuple(endIndex)
|
||||
|
||||
def get_start_block(self, index):
|
||||
'''Returns the value of the start index.'''
|
||||
|
||||
return self.startIndex[index]
|
||||
|
||||
def get_end_block(self, index):
|
||||
'''Returns the value of the end index.'''
|
||||
|
||||
return self.endIndex[index]
|
||||
|
||||
def get_block(self, index):
|
||||
'''Returns the value of the block from the start and the end index.'''
|
||||
|
||||
return (self.get_start_index(index), self.getEndindex(index))
|
||||
|
||||
def get_start_index(self, block):
|
||||
'''Returns the index of the given block from the start index.'''
|
||||
|
||||
if self._block_exists(block, self.startIndex):
|
||||
return self.startIndex.index(block)
|
||||
|
||||
return False
|
||||
|
||||
def get_end_index(self, block):
|
||||
'''Returns the index of the given block from the end index.'''
|
||||
|
||||
if self._block_exists(block, self.endIndex):
|
||||
return self.endIndex.index(block)
|
||||
|
||||
return False
|
||||
|
||||
def get_index(self, block):
|
||||
'''Returns the index of the given block from both the indices.'''
|
||||
|
||||
return (self.get_start_block(block), self.get_end_index(block))
|
||||
|
||||
def _block_exists(self, block, index):
|
||||
"""Returns true or false if the block exists."""
|
||||
|
||||
return block in index
|
||||
|
||||
def get_startindex_generator(self):
|
||||
"""Returns a generator of startIndex."""
|
||||
|
||||
return self._generator(self.startIndex)
|
||||
|
||||
def get_endindex_generator(self):
|
||||
"""Returns a generator of endIndex."""
|
||||
|
||||
return self._generator(self.endIndex)
|
||||
|
||||
def _generator(self, index):
|
||||
"""Create a generator of the given index."""
|
||||
for i in index:
|
||||
yield i
|
||||
|
||||
|
||||
class CodeBlock(object):
|
||||
"""CodeBlock acts as a custom data-structure.
|
||||
|
||||
CodeBlock defines a rst block which contains a one or more lines of code or
|
||||
configuration files. Additionally CodeBlocks also organizes metadata about
|
||||
the rst block which could be as simple as the prompt/user or the path of
|
||||
the configuration file.
|
||||
|
||||
CodeBlock at the end of the day should contain the following keys and
|
||||
values extracted and parsed from the rst files.
|
||||
|
||||
commands = { <key: value_type, possible values, description>
|
||||
|
||||
distro: <str>, [ubuntu|rdo|obs|debian] or [all],
|
||||
This tag specifies the distro which could be a combination of
|
||||
different distros or all distros.
|
||||
action: <str>, [console|config|inject],
|
||||
This could either be a bash command, configuration or file
|
||||
inject.
|
||||
type: <str>, [ini|conf|apache|...],
|
||||
Describes the content of the command. It is fetched from the
|
||||
rst .. code-block|.. distro tag.
|
||||
path: <str>|<os.path>,
|
||||
If it is a config or inejct, the path of the given file. For
|
||||
commands, the path to run the command at.
|
||||
command: <list>, [<str>,<str>],
|
||||
Describes the command itself. This command along with the
|
||||
metadata provides easily BASHable datastructure.
|
||||
output_file: <dict>, {distro: path},
|
||||
Describes the absolute path of the given bash file where the
|
||||
command should be written.
|
||||
}
|
||||
|
||||
This class provides the datastructure along with methods to consume various
|
||||
actions required to fill the datastructure and traverse through it.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
|
||||
self.command = {}
|
||||
|
||||
def append(self, **kwargs):
|
||||
"""Add or update values to the datastructure."""
|
||||
|
||||
self.command.update(kwargs)
|
||||
|
||||
def __dict__(self):
|
||||
|
||||
return self.command
|
||||
|
||||
def generate_code(self):
|
||||
"""Generate BASH command with it's metadata.
|
||||
|
||||
This method should sensibly traverse through the command dictonary and
|
||||
generate and return the BASH code. Also return the distribution name.
|
||||
"""
|
||||
|
||||
command_wrapper = ''
|
||||
bashcodelines = ''
|
||||
bashCommands = defaultdict(list)
|
||||
newline = '\n'
|
||||
action = self.command['action']
|
||||
|
||||
path = self.command['path']
|
||||
if path:
|
||||
path = '{0}conf={1}{0}'.format(newline, path)
|
||||
bashcodelines = path
|
||||
|
||||
if 'config' in action:
|
||||
command_wrapper = 'iniset_sudo '
|
||||
elif 'inject' in action:
|
||||
command_wrapper = '{0}cat<< INJECT | sudo tee -a $conf{0}'
|
||||
command_wrapper = command_wrapper.format(newline)
|
||||
|
||||
for codeline in self.command['command']:
|
||||
bashcodelines += command_wrapper + codeline + newline
|
||||
|
||||
for distro in self.get_distro():
|
||||
bashCommands[distro].append(bashcodelines)
|
||||
|
||||
return bashCommands
|
||||
|
||||
def get_distro(self):
|
||||
"""Return the distribution."""
|
||||
|
||||
return self.command['distro']
|
||||
|
||||
|
||||
class ParseBlocks(object):
|
||||
"""Convert RST block to BASH code.
|
||||
|
||||
Logic to convert a given RST block into BASH. This class should extract
|
||||
given code from the RST block and consume the CodeBlocks datastructure to
|
||||
preserve the metadata along with the code.
|
||||
|
||||
ParseBlocks has three logical sections:
|
||||
|
||||
- Metadata extraction and code type detection.
|
||||
- Parsing Bash/Config/Inject content.
|
||||
- Assembling all the information in CodeBlocks format.
|
||||
"""
|
||||
|
||||
def extract_code(self, codeBlock, cmdType, distro, path):
|
||||
"""Parse the rst block into command and extract metadata info.
|
||||
|
||||
This method extracts all the metadata surrounding the given line of
|
||||
code and also detects the type of the code/config/inject before
|
||||
invoking the respective methods.
|
||||
"""
|
||||
|
||||
command = CodeBlock()
|
||||
|
||||
# Simple helper function.
|
||||
def getdistro(distro):
|
||||
distro = distro.replace('.. only::', '').split('or')
|
||||
return [d.strip() for d in distro]
|
||||
distro = getdistro(distro) if distro else ["ubuntu", "obs", "rdo"]
|
||||
|
||||
if path:
|
||||
path = path.replace('.. path', '').strip()
|
||||
|
||||
command.append(distro=distro, path=path)
|
||||
|
||||
if 'console' in cmdType:
|
||||
action = 'console'
|
||||
codeBlock = self._parse_code(codeBlock)
|
||||
elif 'apache' in cmdType:
|
||||
action = 'inject'
|
||||
codeBlock = self._parse_inject(codeBlock)
|
||||
elif 'ini' in cmdType or 'conf' in cmdType:
|
||||
action = 'config'
|
||||
codeBlock = self._parse_config(codeBlock)
|
||||
else:
|
||||
raise # TODO(dbite): Raise custom exception here.
|
||||
|
||||
command.append(action=action, command=codeBlock)
|
||||
|
||||
return command
|
||||
|
||||
def _parse_inject(self, rstBlock):
|
||||
"""Parse inject lines.
|
||||
|
||||
These lines are usually configuration lines which are copy pasted or
|
||||
appended at the end of a file. Appending newlines with EOL for better
|
||||
visual appearance and easier BASH syntax generation.
|
||||
"""
|
||||
|
||||
return [rstBlock + "\nEOL\n"]
|
||||
|
||||
def _parse_config(self, rstBlock):
|
||||
"""Parse configuration files.
|
||||
|
||||
Configuration file modifications, which mostly involves setting or
|
||||
resetting given variables and parameters. This method:
|
||||
|
||||
- Detects the configuration sections ``[section]``.
|
||||
- Parses the following lines under this section iteratively.
|
||||
- Go to step one if more lines.
|
||||
- Generate a list of configuration lines along with it's section
|
||||
in training-labs friendly format.
|
||||
- Also some syntax niceness sprinkled on top.
|
||||
"""
|
||||
|
||||
operator = ''
|
||||
|
||||
# Only works for a specific sequence of configuration options.
|
||||
parsedConfig = list()
|
||||
|
||||
for line in rstBlock.split('\n'):
|
||||
line = line.strip()
|
||||
if re.search('\[[a-zA-Z_]+\]', line):
|
||||
operator = line[1:-1]
|
||||
elif re.search('=', line) and not re.search('^#', line):
|
||||
line = operator + " " + line.replace("=", " ") + "\n"
|
||||
parsedConfig.append(line.strip())
|
||||
|
||||
return parsedConfig
|
||||
|
||||
@staticmethod
|
||||
def _get_bash_operator(operator):
|
||||
"""Helper function to convert the operator to its equivalent syntax.
|
||||
|
||||
# --> root --> sudo ...
|
||||
$ --> noroot --> ...
|
||||
> --> mysql --> mysql_exec ...
|
||||
"""
|
||||
|
||||
if "#" in operator:
|
||||
operator = "sudo "
|
||||
elif "$" in operator:
|
||||
operator = ""
|
||||
elif ">" in operator:
|
||||
operator = "mysql_exec "
|
||||
else:
|
||||
raise # TODO(dbite): Create custom exceptions!
|
||||
|
||||
return operator
|
||||
|
||||
def _parse_code(self, rstBlock):
|
||||
"""Parse code lines.
|
||||
|
||||
Code-blocks containing bash code (console|mysql) are sent here. These
|
||||
are bash code or mysql etc. which are to be formatted into proper bash
|
||||
format.
|
||||
|
||||
- Detects type of code, replace `mysql>` with `>` if detected.
|
||||
- Replace line continuation `asdb \` with equivalent HTML codes
|
||||
for `\` and `\n` to properly parse multi-line commands.
|
||||
- Iterate through all the code lines which are easily detected using
|
||||
the operator syntax.
|
||||
- Replace the HTML codes to it's respective ASCII/UNICODE equivalent.
|
||||
"""
|
||||
|
||||
parsedCmds = list()
|
||||
|
||||
if "mysql>" in rstBlock:
|
||||
rstBlock = rstBlock.replace("mysql>", ">")
|
||||
|
||||
# Substitute HTML codes for '\' and '\n'
|
||||
rstBlock = rstBlock.replace("\\\n", "


")
|
||||
|
||||
for index in re.finditer("[#\$>].*", rstBlock):
|
||||
|
||||
cmd = rstBlock[index.start():index.end()].replace("


",
|
||||
"\\\n")
|
||||
operator = self._get_bash_operator(cmd[0])
|
||||
parsedCmds.append(operator + cmd[1:].strip())
|
||||
|
||||
return parsedCmds
|
||||
|
||||
|
||||
class ExtractBlocks(object):
|
||||
"""Creates required indices form the rst code."""
|
||||
|
||||
def __init__(self, rstFile, bashPath):
|
||||
|
||||
self.rstFile = self.get_file_contents(rstFile)
|
||||
self.blocks = None # Lookup table.
|
||||
self.allBlocksIterator = None
|
||||
self.parseblocks = ParseBlocks()
|
||||
self.bashCode = list()
|
||||
bashFileName = os.path.basename(rstFile).replace('.rst', '.sh')
|
||||
self.bashPath = {distro: os.path.join(path, bashFileName)
|
||||
for distro, path in bashPath.iteritems()}
|
||||
|
||||
def __del__(self):
|
||||
"""Proper handling of the file pointer."""
|
||||
|
||||
self.filePointer.close()
|
||||
|
||||
def _get_indices(self, regexStr):
|
||||
"""Helper function to return a tuple containing indices.
|
||||
|
||||
The indices returned contains the location of the given blocks matched
|
||||
by the regex string. Returns the (start, end) index for the same.
|
||||
"""
|
||||
|
||||
searchBlocks = re.compile(regexStr, re.VERBOSE)
|
||||
indices = [index.span()
|
||||
for index in searchBlocks.finditer(self.rstFile)]
|
||||
|
||||
return indices
|
||||
|
||||
def get_file_contents(self, filePath):
|
||||
"""Return the contents of the given file."""
|
||||
|
||||
self.filePointer = open(filePath, 'r')
|
||||
|
||||
return self.filePointer.read()
|
||||
|
||||
def get_indice_blocks(self):
|
||||
"""Should fetch regex strings from the right location."""
|
||||
|
||||
# Regex string for extracting particular bits from RST file.
|
||||
# For some reason I want to keep the generic RegEX strings.
|
||||
searchAllBlocks = '''\.\.\s # Look for '.. '
|
||||
(code-block::|only::|path) # Look for required blocks
|
||||
[a-z\s/].*
|
||||
'''
|
||||
searchDistroBlocksStart = '''\.\.\sonly::
|
||||
[\sa-z].* # For matching all distros.
|
||||
'''
|
||||
searchDistroBlocksEnd = '''\.\.\sendonly\\n''' # Match end blocks.
|
||||
|
||||
searchCodeBlocksStart = '''\.\.\scode-block:: # Look for code block
|
||||
\s # Include whitespace
|
||||
(?!end) # Exclude code-block:: end
|
||||
(?:[a-z])* # Include everything else.
|
||||
'''
|
||||
searchCodeBlocksEnd = '''\.\.\send\\n''' # Look for .. end
|
||||
searchPath = '''\.\.\spath\s.*''' # Look for .. path
|
||||
|
||||
allBlocks = BlockIndex(self._get_indices(searchAllBlocks))
|
||||
distroBlocks = BlockIndex(self._get_indices(searchDistroBlocksStart),
|
||||
self._get_indices(searchDistroBlocksEnd))
|
||||
codeBlocks = BlockIndex(self._get_indices(searchCodeBlocksStart),
|
||||
self._get_indices(searchCodeBlocksEnd))
|
||||
pathBlocks = BlockIndex(self._get_indices(searchPath))
|
||||
|
||||
# Point to the blocks from a dictionary to create sensible index.
|
||||
self.blocks = {'distroBlock': distroBlocks,
|
||||
'codeBlock': codeBlocks,
|
||||
'pathBlock': pathBlocks,
|
||||
'allBlock': allBlocks
|
||||
}
|
||||
|
||||
def extract_codeblocks(self):
|
||||
"""Initialize the generator object and start the initial parsing."""
|
||||
|
||||
# Generate all blocks iterator
|
||||
self.allBlocksIterator = \
|
||||
self.blocks['allBlock'].get_startindex_generator()
|
||||
|
||||
self._extractblocks()
|
||||
|
||||
# Helper function for quick lookup from the blocks lookup table.
|
||||
def _block_lookup(self, allblock):
|
||||
"""Block Lookup Helper Function.
|
||||
|
||||
Look for the block in blocks and return the name and index of the
|
||||
location of the block.
|
||||
"""
|
||||
|
||||
for blockName in 'codeBlock', 'distroBlock', 'pathBlock':
|
||||
blockIndex = self.blocks[blockName].get_start_index(allblock)
|
||||
if blockIndex is not False:
|
||||
return blockName, blockIndex
|
||||
else:
|
||||
# TODO(dbite): Raise custom exception.
|
||||
raise
|
||||
|
||||
# Helper function for recursive-generator pattern.
|
||||
def _extractblocks(self, distro=None, path=None, distroEnd=None):
|
||||
"""Recursive function to sequentially parse the RST file.
|
||||
|
||||
This method deals with traversing through the given RST file by using
|
||||
the indices generated using regex. These indices indicate the location
|
||||
of different chunks of blocks and also the distribution for the same.
|
||||
|
||||
AllBlocks provides the location of all the blocks and is used to
|
||||
recurse and give the next block location. This block can either be
|
||||
CodeBlock, PathBlock or DistroBlock. The lookup table provides the
|
||||
information about which block a given index points to and fetches
|
||||
the equivalent end index. This allows further calls to ParseBlocks
|
||||
class to process the extracted chunk of code in the correct way.
|
||||
|
||||
Using recursion is more efficient as compared to iteration. It
|
||||
simplifies the implementation logic, performance and efficiency. This
|
||||
also allows the parsing to be accomplished with minimal variables and
|
||||
eliminates need for keeping track, toggle flags and complicated code
|
||||
which is hard to debug and understand.
|
||||
"""
|
||||
|
||||
try:
|
||||
blockName, blockIndex = self._block_lookup(
|
||||
self.allBlocksIterator.next())
|
||||
except StopIteration:
|
||||
return
|
||||
|
||||
block = self.blocks[blockName]
|
||||
|
||||
if distroEnd < block.get_start_block(blockIndex)[0]:
|
||||
distro = None
|
||||
|
||||
if 'codeBlock' in blockName:
|
||||
# Extract Code Block
|
||||
# Use path & distro variables.
|
||||
indexStart = block.get_start_block(blockIndex)
|
||||
indexEnd = block.get_end_block(blockIndex)
|
||||
codeBlock = self.rstFile[indexStart[1]:indexEnd[0]].strip()
|
||||
cmdType = self.rstFile[indexStart[0]:indexStart[1]]
|
||||
self.bashCode.append(
|
||||
self.parseblocks.extract_code(codeBlock,
|
||||
cmdType,
|
||||
distro,
|
||||
path))
|
||||
self._extractblocks(distro=distro, distroEnd=distroEnd)
|
||||
|
||||
elif 'pathBlock' in blockName:
|
||||
# Get path & recurse, the next one should be CodeBlock.
|
||||
pathIndex = block.get_start_block(blockIndex)
|
||||
path = self.rstFile[pathIndex[0]:pathIndex[1]]
|
||||
self._extractblocks(distro=distro, path=path, distroEnd=distroEnd)
|
||||
|
||||
elif 'distroBlock' in blockName:
|
||||
# Get distro & recurse
|
||||
distroStart = block.get_start_block(blockIndex)
|
||||
distro = self.rstFile[distroStart[0]:distroStart[1]]
|
||||
distroEnd = block.get_end_block(blockIndex)[1]
|
||||
self._extractblocks(distro=distro, distroEnd=distroEnd)
|
||||
|
||||
return
|
||||
|
||||
def get_bash_code(self):
|
||||
"""Returns bashCode which is a list containing <CodeBlock>'s."""
|
||||
|
||||
return self.bashCode
|
||||
|
||||
@staticmethod
|
||||
def write_to_file(path, value):
|
||||
"""Static method to write given content to the file."""
|
||||
|
||||
with open(path, 'w') as fp:
|
||||
fp.write(value)
|
||||
|
||||
def write_bash_code(self):
|
||||
"""Writes bash code to file."""
|
||||
|
||||
commands = defaultdict(str)
|
||||
|
||||
newline = "\n"
|
||||
|
||||
for code in self.bashCode:
|
||||
codeLines = code.generate_code()
|
||||
for distro, codeLine in codeLines.iteritems():
|
||||
commands[distro] += newline.join(codeLine)
|
||||
|
||||
for distro, command in commands.iteritems():
|
||||
ExtractBlocks.write_to_file(self.bashPath[distro], command)
|
||||
|
||||
return True
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
with open("rst2bash/config/parser_config.yaml", 'r') as ymlfile:
|
||||
cfg = yaml.load(ymlfile)
|
||||
|
||||
cwd = os.getcwd()
|
||||
rst_path = os.path.join(cwd, cfg['rst_path'])
|
||||
rst_files = cfg['rst_files']
|
||||
bash_path = {distro: os.path.join(cwd, path)
|
||||
for distro, path in cfg['bash_path'].iteritems()}
|
||||
|
||||
for path_value in bash_path.itervalues():
|
||||
|
||||
if not os.path.exists(path_value):
|
||||
os.mkdir(path_value)
|
||||
|
||||
for rst_file in rst_files:
|
||||
|
||||
rst_file_path = os.path.join(rst_path, rst_file)
|
||||
print("Parsing: %s\n") % (rst_file)
|
||||
code_blocks = ExtractBlocks(rst_file_path, bash_path)
|
||||
code_blocks.get_indice_blocks()
|
||||
try:
|
||||
code_blocks.extract_codeblocks()
|
||||
except Exception:
|
||||
pass
|
||||
bashCode = code_blocks.get_bash_code()
|
||||
if not code_blocks.write_bash_code():
|
||||
raise Exception("Could not write to bash")
|
@ -48,4 +48,4 @@ output_file = rst2bash/locale/rst2bash.pot
|
||||
[build_releasenotes]
|
||||
all_files = 1
|
||||
build-dir = releasenotes/build
|
||||
source-dir = releasenotes/source
|
||||
source-dir = releasenotes/source
|
||||
|
3
tools/cluster
Executable file
3
tools/cluster
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/bash
|
||||
|
||||
echo 'WIP: To be implemented.'
|
14
tools/runparser.sh
Executable file
14
tools/runparser.sh
Executable file
@ -0,0 +1,14 @@
|
||||
#!/bin/bash -
|
||||
|
||||
mkdir -p build/scripts
|
||||
git clone \
|
||||
--depth 10 \
|
||||
git://git.openstack.org/openstack/openstack-manuals \
|
||||
build/openstack-manuals
|
||||
git clone \
|
||||
git://git.openstack.org/openstack/training-labs \
|
||||
build/training-labs
|
||||
|
||||
python2.7 rst2bash/parser.py
|
||||
|
||||
echo "Please find the generated BASH scripts in the build/scripts folder!".
|
8
tox.ini
8
tox.ini
@ -1,15 +1,17 @@
|
||||
[tox]
|
||||
minversion = 2.0
|
||||
envlist = py34,py27,pypy,pep8
|
||||
envlist = py27,pep8
|
||||
skipsdist = True
|
||||
|
||||
[testenv]
|
||||
usedevelop = True
|
||||
install_command = pip install -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt} {opts} {packages}
|
||||
install_command = pip install {opts} {packages}
|
||||
setenv =
|
||||
VIRTUAL_ENV={envdir}
|
||||
PYTHONWARNINGS=default::DeprecationWarning
|
||||
deps = -r{toxinidir}/test-requirements.txt
|
||||
deps =
|
||||
-r{toxinidir}/requirements.txt
|
||||
-r{toxinidir}/test-requirements.txt
|
||||
commands = python setup.py test --slowest --testr-args='{posargs}'
|
||||
|
||||
[testenv:pep8]
|
||||
|
Loading…
Reference in New Issue
Block a user