Adds the parser and related files.

- Adding parser module.
- Adds custom exceptions and tests cases.
- Adds tools folder.
- Updated the documentation.
- Readme usage section now imports usage.rst.
- Improvements to tox.ini
- Adds config folder.

README.rst

@@ -26,8 +26,8 @@ These are the major goals which are accomplished by the parser:
 Training-Labs
 -------------
 
-[Training-labs](https://git.openstack.org/openstack/training-labs) is part
+`Training-labs <https://git.openstack.org/openstack/training-labs>`_ is part
-of OpeNStack Documentation team and provides an unique tool to deploy core
+of OpenStack Documentation team and provides an unique tool to deploy core
 OpenStack services. Training labs closely follows installation guides for
 the OpenStack deployment steps.
 
@@ -35,7 +35,7 @@ the OpenStack deployment steps.
 Installation Guides (OpenStack Installation Tutorial)
 -----------------------------------------------------
 
-[Installation guides](https://docs.openstack.org) provides step by step
+`Installation guides <https://docs.openstack.org>`_ provides step by step
 instructions to deploy OpenStack on a multi-node cluster.
 
 
@@ -47,38 +47,15 @@ More Details
   and training-labs repository.
 - The generated output (parsed files) should then be triggered via.
   training-labs to deploy the OpenStack cluster.
-- Additionally, this project should showcase and allow the workflow in the
+- Additionally, this project should showcase and allow the work-flow in the
   OpenStack CI for installation guides and cross-project installation-guides.
 
 
-Usage
------
-
-- To run the parser please clone the [openstack-manuals](git://git.openstack.org/openstack/openstack-manuals)
-  repository and update the configuration file.
-- Additionally, if you wish to deploy OpenStack cluster, also clone the [training-labs](git://git.openstack.org/openstack/training-labs)
-  repository.
-- Run the parser:
-
-    $ python parser.py
-
-- Check the generated scripts (location in the configuration file), copy them
-  to training-labs: labs/osbash/scripts/ folder.
-- Run training labs:
-
-    $ PROVIDER=kvm ./st.py -b cluster
-
-- Sit back, relax and see the cluster deploy.
-
-**Note:** This project is in its nascent state, especially the OpenStack
-          cluster deployment part may break at many places.
-
-
 Roadmap
 -------
 
 - Create glue-code scripts to automate setting up of various repositories
-  required to easily carry the workflow.
+  required to easily carry the work-flow.
 - Setup the non-voting jobs to deploy the cluster. This cluster should be
   a two node KVM/VirtualBox cluster which runs in the OpenStack CI.
 - Update the Bash templates (Jinja templates) to allow nicer Bash scripts

doc/source/installation.rst

@@ -2,10 +2,11 @@
 Installation
 ============
 
-At the command line::
+- Clone the rst2bash repository.
 
-- Clone the repository
+  .. code-block:: bash
 
-    $ git clone git://git.openstack.org/openstack/rst2bash
+     $ git clone git://git.openstack.org/openstack/rst2bash
+     $ cd rst2bash
 
-- Run the parser.
+- Check the usage section for more details.

doc/source/usage.rst

@@ -2,19 +2,27 @@
 Usage
 =====
 
-- To run the parser please clone the [openstack-manuals](git://git.openstack.org/openstack/openstack-manuals)
+- Run the parser, it will clone openstack-manuals repository, training-labs
-  repository and update the configuration file.
+  repository and parse the files
-- Additionally, if you wish to deploy OpenStack cluster, also clone the [training-labs](git://git.openstack.org/openstack/training-labs)
-  repository.
-- Run the parser:
 
-    $ python parser.py
+    .. code-block:: bash
+
+       $ ./tools/runparser.sh
+
+
+  Make sure to run it from the root of the directory.
+- Check the generated scripts (location in the configuration file
+  `rst2bash/conf`), copy them to training-labs:
+  `labs/osbash/scripts/` folder.
 
 - Check the generated scripts (location in the configuration file), copy them
-  to training-labs: labs/osbash/scripts/ folder.
+  to training-labs: `labs/osbash/scripts/` folder. Default configuration
+  specifies the output location at `build/scripts/`.
 - Run training labs:
 
-    $ PROVIDER=kvm ./st.py -b cluster
+    .. code-block:: bash
+
+       $ PROVIDER=kvm ./build/training-labs/labs/st.py -b cluster
 
 - Sit back, relax and see the cluster deploy.
 

rst2bash/config/parser_config.yaml

@@ -0,0 +1,65 @@
+description: Provides input (RST) and output (BASH) based on the configuration below.
+rst_path: build/openstack-manuals/doc/install-guide/source
+bash_path:
+    ubuntu: build/scripts/ubuntu
+    rdo: build/scripts/rdo
+    obs: build/scripts/obs
+    debian: build/scripts/debian
+rst_files:
+    - keystone-install.rst
+    # - keystone-openrc.rst
+    # - keystone.rst
+    - keystone-users.rst
+    - keystone-verify.rst
+    - launch-instance-cinder.rst
+    - launch-instance-networks-provider.rst
+    - launch-instance-networks-selfservice.rst
+    - launch-instance-provider.rst
+    - launch-instance.rst
+    - launch-instance-selfservice.rst
+    - neutron-compute-install-option1.rst
+    - neutron-compute-install-option2.rst
+    - neutron-compute-install.rst
+    - neutron-concepts.rst
+    - neutron-controller-install-option1.rst
+    - neutron-controller-install-option2.rst
+    - neutron-controller-install.rst
+    - neutron-next-steps.rst
+    # - neutron.rst
+    - neutron-verify-option1.rst
+    - neutron-verify-option2.rst
+    - neutron-verify.rst
+    - nova-compute-install.rst
+    - nova-controller-install.rst
+    # - nova.rst
+    - nova-verify.rst
+    # - overview.rst
+    - horizon-verify.rst
+    - additional-services.rst
+    - cinder-backup-install.rst
+    - cinder-controller-install.rst
+    - cinder-next-steps.rst
+    # - cinder.rst
+    - cinder-storage-install.rst
+    - cinder-verify.rst
+    - environment-memcached.rst
+    - environment-messaging.rst
+    - environment-networking-compute.rst
+    - environment-networking-controller.rst
+    - environment-networking.rst
+    - environment-networking-storage-cinder.rst
+    - environment-networking-verify.rst
+    - environment-ntp-controller.rst
+    - environment-ntp-other.rst
+    - environment-ntp.rst
+    - environment-ntp-verify.rst
+    - environment-packages.rst
+    # - environment.rst
+    - environment-security.rst
+    - environment-sql-database.rst
+    - glance-install.rst
+    # - glance.rst
+    - glance-verify.rst
+    - horizon-install.rst
+    - horizon-next-steps.rst
+    # - horizon.rst

rst2bash/exceptions.py

@@ -0,0 +1,41 @@
+#  Licensed under the Apache License, Version 2.0 (the "License"); you may
+#  not use this file except in compliance with the License. You may obtain
+#  a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#  WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#  License for the specific language governing permissions and limitations
+#  under the License.
+
+
+class Rst2BashException(Exception):
+
+    pass
+
+
+class MissingTagsException(Rst2BashException):
+
+    pass
+
+
+class NestedDistroBlocksException(Rst2BashException):
+
+    pass
+
+
+class PathNotFoundException(Rst2BashException):
+
+    pass
+
+
+class NoCodeBlocksException(Rst2BashException):
+
+    pass
+
+
+class InvalidOperatorException(Rst2BashException):
+
+    pass

rst2bash/parser.py

@@ -0,0 +1,549 @@
+#  Licensed under the Apache License, Version 2.0 (the "License"); you may
+#  not use this file except in compliance with the License. You may obtain
+#  a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#  WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#  License for the specific language governing permissions and limitations
+#  under the License.
+
+
+from collections import defaultdict
+import os
+import re
+import yaml
+
+
+class BlockIndex(object):
+    """Creates indices which describes the location of blocks in rst file.
+
+    These indices describe the start and end location of the strings in the rst
+    file. Different indices used to parse the file are:
+
+        AllBlocks: Contains sequential index values for all required blocks.
+        CodeBlocks: Contains index values for blocks containing code.
+        PathBlocks: Contains index values for blocks containing path.
+        DistroBlocks: Contains index values for blocks containing OS.
+
+    These indices should provide the location to extract given blocks from the
+    rst files. This class additionally provides various functionalities to
+    easily carry out different tasks like iteration and more.
+    """
+
+    def __init__(self, startIndex=tuple(), endIndex=tuple()):
+
+        self.startIndex = tuple(startIndex)
+        self.endIndex = tuple(endIndex)
+
+    def get_start_block(self, index):
+        '''Returns the value of the start index.'''
+
+        return self.startIndex[index]
+
+    def get_end_block(self, index):
+        '''Returns the value of the end index.'''
+
+        return self.endIndex[index]
+
+    def get_block(self, index):
+        '''Returns the value of the block from the start and the end index.'''
+
+        return (self.get_start_index(index), self.getEndindex(index))
+
+    def get_start_index(self, block):
+        '''Returns the index of the given block from the start index.'''
+
+        if self._block_exists(block, self.startIndex):
+            return self.startIndex.index(block)
+
+        return False
+
+    def get_end_index(self, block):
+        '''Returns the index of the given block from the end index.'''
+
+        if self._block_exists(block, self.endIndex):
+            return self.endIndex.index(block)
+
+        return False
+
+    def get_index(self, block):
+        '''Returns the index of the given block from both the indices.'''
+
+        return (self.get_start_block(block), self.get_end_index(block))
+
+    def _block_exists(self, block, index):
+        """Returns true or false if the block exists."""
+
+        return block in index
+
+    def get_startindex_generator(self):
+        """Returns a generator of startIndex."""
+
+        return self._generator(self.startIndex)
+
+    def get_endindex_generator(self):
+        """Returns a generator of endIndex."""
+
+        return self._generator(self.endIndex)
+
+    def _generator(self, index):
+        """Create a generator of the given index."""
+        for i in index:
+            yield i
+
+
+class CodeBlock(object):
+    """CodeBlock acts as a custom data-structure.
+
+    CodeBlock defines a rst block which contains a one or more lines of code or
+    configuration files. Additionally CodeBlocks also organizes metadata about
+    the rst block which could be as simple as the prompt/user or the path of
+    the configuration file.
+
+    CodeBlock at the end of the day should contain the following keys and
+    values extracted and parsed from the rst files.
+
+    commands = { <key: value_type, possible values, description>
+
+    distro: <str>, [ubuntu|rdo|obs|debian] or [all],
+                This tag specifies the distro which could be a combination of
+                different distros or all distros.
+    action: <str>, [console|config|inject],
+                This could either be a bash command, configuration or file
+                inject.
+    type: <str>, [ini|conf|apache|...],
+                Describes the content of the command. It is fetched from the
+                rst .. code-block|.. distro tag.
+    path: <str>|<os.path>,
+                If it is a config or inejct, the path of the given file. For
+                commands, the path to run the command at.
+    command: <list>, [<str>,<str>],
+                Describes the command itself. This command along with the
+                metadata provides easily BASHable datastructure.
+    output_file: <dict>, {distro: path},
+                Describes the absolute path of the given bash file where the
+                command should be written.
+    }
+
+    This class provides the datastructure along with methods to consume various
+    actions required to fill the datastructure and traverse through it.
+    """
+
+    def __init__(self):
+
+        self.command = {}
+
+    def append(self, **kwargs):
+        """Add or update values to the datastructure."""
+
+        self.command.update(kwargs)
+
+    def __dict__(self):
+
+        return self.command
+
+    def generate_code(self):
+        """Generate BASH command with it's metadata.
+
+        This method should sensibly traverse through the command dictonary and
+        generate and return the BASH code. Also return the distribution name.
+        """
+
+        command_wrapper = ''
+        bashcodelines = ''
+        bashCommands = defaultdict(list)
+        newline = '\n'
+        action = self.command['action']
+
+        path = self.command['path']
+        if path:
+            path = '{0}conf={1}{0}'.format(newline, path)
+            bashcodelines = path
+
+        if 'config' in action:
+            command_wrapper = 'iniset_sudo '
+        elif 'inject' in action:
+            command_wrapper = '{0}cat<< INJECT | sudo tee -a $conf{0}'
+            command_wrapper = command_wrapper.format(newline)
+
+        for codeline in self.command['command']:
+            bashcodelines += command_wrapper + codeline + newline
+
+        for distro in self.get_distro():
+            bashCommands[distro].append(bashcodelines)
+
+        return bashCommands
+
+    def get_distro(self):
+        """Return the distribution."""
+
+        return self.command['distro']
+
+
+class ParseBlocks(object):
+    """Convert RST block to BASH code.
+
+    Logic to convert a given RST block into BASH. This class should extract
+    given code from the RST block and consume the CodeBlocks datastructure to
+    preserve the metadata along with the code.
+
+    ParseBlocks has three logical sections:
+
+        - Metadata extraction and code type detection.
+        - Parsing Bash/Config/Inject content.
+        - Assembling all the information in CodeBlocks format.
+    """
+
+    def extract_code(self, codeBlock, cmdType, distro, path):
+        """Parse the rst block into command and extract metadata info.
+
+        This method extracts all the metadata surrounding the given line of
+        code and also detects the type of the code/config/inject before
+        invoking the respective methods.
+        """
+
+        command = CodeBlock()
+
+        # Simple helper function.
+        def getdistro(distro):
+            distro = distro.replace('.. only::', '').split('or')
+            return [d.strip() for d in distro]
+        distro = getdistro(distro) if distro else ["ubuntu", "obs", "rdo"]
+
+        if path:
+            path = path.replace('.. path', '').strip()
+
+        command.append(distro=distro, path=path)
+
+        if 'console' in cmdType:
+            action = 'console'
+            codeBlock = self._parse_code(codeBlock)
+        elif 'apache' in cmdType:
+            action = 'inject'
+            codeBlock = self._parse_inject(codeBlock)
+        elif 'ini' in cmdType or 'conf' in cmdType:
+            action = 'config'
+            codeBlock = self._parse_config(codeBlock)
+        else:
+            raise   # TODO(dbite): Raise custom exception here.
+
+        command.append(action=action, command=codeBlock)
+
+        return command
+
+    def _parse_inject(self, rstBlock):
+        """Parse inject lines.
+
+        These lines are usually configuration lines which are copy pasted or
+        appended at the end of a file. Appending newlines with EOL for better
+        visual appearance and easier BASH syntax generation.
+        """
+
+        return [rstBlock + "\nEOL\n"]
+
+    def _parse_config(self, rstBlock):
+        """Parse configuration files.
+
+        Configuration file modifications, which mostly involves setting or
+        resetting given variables and parameters. This method:
+
+            - Detects the configuration sections ``[section]``.
+            - Parses the following lines under this section iteratively.
+            - Go to step one if more lines.
+            - Generate a list of configuration lines along with it's section
+              in training-labs friendly format.
+            - Also some syntax niceness sprinkled on top.
+        """
+
+        operator = ''
+
+        # Only works for a specific sequence of configuration options.
+        parsedConfig = list()
+
+        for line in rstBlock.split('\n'):
+            line = line.strip()
+            if re.search('\[[a-zA-Z_]+\]', line):
+                operator = line[1:-1]
+            elif re.search('=', line) and not re.search('^#', line):
+                line = operator + " " + line.replace("=", " ") + "\n"
+                parsedConfig.append(line.strip())
+
+        return parsedConfig
+
+    @staticmethod
+    def _get_bash_operator(operator):
+        """Helper function to convert the operator to its equivalent syntax.
+
+        # --> root --> sudo ...
+        $ --> noroot --> ...
+        > --> mysql --> mysql_exec ...
+        """
+
+        if "#" in operator:
+            operator = "sudo "
+        elif "$" in operator:
+            operator = ""
+        elif ">" in operator:
+            operator = "mysql_exec "
+        else:
+            raise           # TODO(dbite): Create custom exceptions!
+
+        return operator
+
+    def _parse_code(self, rstBlock):
+        """Parse code lines.
+
+        Code-blocks containing bash code (console|mysql) are sent here. These
+        are bash code or mysql etc. which are to be formatted into proper bash
+        format.
+
+         - Detects type of code, replace `mysql>` with `>` if detected.
+         - Replace line continuation `asdb \` with equivalent HTML codes
+           for `\` and `\n` to properly parse multi-line commands.
+         - Iterate through all the code lines which are easily detected using
+           the operator syntax.
+         - Replace the HTML codes to it's respective ASCII/UNICODE equivalent.
+        """
+
+        parsedCmds = list()
+
+        if "mysql>" in rstBlock:
+            rstBlock = rstBlock.replace("mysql>", ">")
+
+        # Substitute HTML codes for '\' and '\n'
+        rstBlock = rstBlock.replace("\\\n", "&#10&#10&#13")
+
+        for index in re.finditer("[#\$>].*", rstBlock):
+
+            cmd = rstBlock[index.start():index.end()].replace("&#10&#10&#13",
+                                                              "\\\n")
+            operator = self._get_bash_operator(cmd[0])
+            parsedCmds.append(operator + cmd[1:].strip())
+
+        return parsedCmds
+
+
+class ExtractBlocks(object):
+    """Creates required indices form the rst code."""
+
+    def __init__(self, rstFile, bashPath):
+
+        self.rstFile = self.get_file_contents(rstFile)
+        self.blocks = None  # Lookup table.
+        self.allBlocksIterator = None
+        self.parseblocks = ParseBlocks()
+        self.bashCode = list()
+        bashFileName = os.path.basename(rstFile).replace('.rst', '.sh')
+        self.bashPath = {distro: os.path.join(path, bashFileName)
+                         for distro, path in bashPath.iteritems()}
+
+    def __del__(self):
+        """Proper handling of the file pointer."""
+
+        self.filePointer.close()
+
+    def _get_indices(self, regexStr):
+        """Helper function to return a tuple containing indices.
+
+        The indices returned contains the location of the given blocks matched
+        by the regex string. Returns the (start, end) index for the same.
+        """
+
+        searchBlocks = re.compile(regexStr, re.VERBOSE)
+        indices = [index.span()
+                   for index in searchBlocks.finditer(self.rstFile)]
+
+        return indices
+
+    def get_file_contents(self, filePath):
+        """Return the contents of the given file."""
+
+        self.filePointer = open(filePath, 'r')
+
+        return self.filePointer.read()
+
+    def get_indice_blocks(self):
+        """Should fetch regex strings from the right location."""
+
+        # Regex string for extracting particular bits from RST file.
+        # For some reason I want to keep the generic RegEX strings.
+        searchAllBlocks = '''\.\.\s     # Look for '.. '
+            (code-block::|only::|path)  # Look for required blocks
+            [a-z\s/].*
+            '''
+        searchDistroBlocksStart = '''\.\.\sonly::
+            [\sa-z].*                               # For matching all distros.
+            '''
+        searchDistroBlocksEnd = '''\.\.\sendonly\\n'''      # Match end blocks.
+
+        searchCodeBlocksStart = '''\.\.\scode-block::   # Look for code block
+            \s                                      # Include whitespace
+            (?!end)                                 # Exclude code-block:: end
+            (?:[a-z])*                              # Include everything else.
+        '''
+        searchCodeBlocksEnd = '''\.\.\send\\n'''    # Look for .. end
+        searchPath = '''\.\.\spath\s.*'''           # Look for .. path
+
+        allBlocks = BlockIndex(self._get_indices(searchAllBlocks))
+        distroBlocks = BlockIndex(self._get_indices(searchDistroBlocksStart),
+                                  self._get_indices(searchDistroBlocksEnd))
+        codeBlocks = BlockIndex(self._get_indices(searchCodeBlocksStart),
+                                self._get_indices(searchCodeBlocksEnd))
+        pathBlocks = BlockIndex(self._get_indices(searchPath))
+
+        # Point to the blocks from a dictionary to create sensible index.
+        self.blocks = {'distroBlock': distroBlocks,
+                       'codeBlock': codeBlocks,
+                       'pathBlock': pathBlocks,
+                       'allBlock': allBlocks
+                       }
+
+    def extract_codeblocks(self):
+        """Initialize the generator object and start the initial parsing."""
+
+        # Generate all blocks iterator
+        self.allBlocksIterator = \
+            self.blocks['allBlock'].get_startindex_generator()
+
+        self._extractblocks()
+
+    # Helper function for quick lookup from the blocks lookup table.
+    def _block_lookup(self, allblock):
+        """Block Lookup Helper Function.
+
+        Look for the block in blocks and return the name and index of the
+        location of the block.
+        """
+
+        for blockName in 'codeBlock', 'distroBlock', 'pathBlock':
+            blockIndex = self.blocks[blockName].get_start_index(allblock)
+            if blockIndex is not False:
+                return blockName, blockIndex
+        else:
+            # TODO(dbite): Raise custom exception.
+            raise
+
+    # Helper function for recursive-generator pattern.
+    def _extractblocks(self, distro=None, path=None, distroEnd=None):
+        """Recursive function to sequentially parse the RST file.
+
+        This method deals with traversing through the given RST file by using
+        the indices generated using regex. These indices indicate the location
+        of different chunks of blocks and also the distribution for the same.
+
+        AllBlocks provides the location of all the blocks and is used to
+        recurse and give the next block location. This block can either be
+        CodeBlock, PathBlock or DistroBlock. The lookup table provides the
+        information about which block a given index points to and fetches
+        the equivalent end index. This allows further calls to ParseBlocks
+        class to process the extracted chunk of code in the correct way.
+
+        Using recursion is more efficient as compared to iteration. It
+        simplifies the implementation logic, performance and efficiency. This
+        also allows the parsing to be accomplished with minimal variables and
+        eliminates need for keeping track, toggle flags and complicated code
+        which is hard to debug and understand.
+        """
+
+        try:
+            blockName, blockIndex = self._block_lookup(
+                self.allBlocksIterator.next())
+        except StopIteration:
+            return
+
+        block = self.blocks[blockName]
+
+        if distroEnd < block.get_start_block(blockIndex)[0]:
+            distro = None
+
+        if 'codeBlock' in blockName:
+            # Extract Code Block
+            # Use path & distro variables.
+            indexStart = block.get_start_block(blockIndex)
+            indexEnd = block.get_end_block(blockIndex)
+            codeBlock = self.rstFile[indexStart[1]:indexEnd[0]].strip()
+            cmdType = self.rstFile[indexStart[0]:indexStart[1]]
+            self.bashCode.append(
+                self.parseblocks.extract_code(codeBlock,
+                                              cmdType,
+                                              distro,
+                                              path))
+            self._extractblocks(distro=distro, distroEnd=distroEnd)
+
+        elif 'pathBlock' in blockName:
+            # Get path & recurse, the next one should be CodeBlock.
+            pathIndex = block.get_start_block(blockIndex)
+            path = self.rstFile[pathIndex[0]:pathIndex[1]]
+            self._extractblocks(distro=distro, path=path, distroEnd=distroEnd)
+
+        elif 'distroBlock' in blockName:
+            # Get distro & recurse
+            distroStart = block.get_start_block(blockIndex)
+            distro = self.rstFile[distroStart[0]:distroStart[1]]
+            distroEnd = block.get_end_block(blockIndex)[1]
+            self._extractblocks(distro=distro, distroEnd=distroEnd)
+
+        return
+
+    def get_bash_code(self):
+        """Returns bashCode which is a list containing <CodeBlock>'s."""
+
+        return self.bashCode
+
+    @staticmethod
+    def write_to_file(path, value):
+        """Static method to write given content to the file."""
+
+        with open(path, 'w') as fp:
+            fp.write(value)
+
+    def write_bash_code(self):
+        """Writes bash code to file."""
+
+        commands = defaultdict(str)
+
+        newline = "\n"
+
+        for code in self.bashCode:
+            codeLines = code.generate_code()
+            for distro, codeLine in codeLines.iteritems():
+                commands[distro] += newline.join(codeLine)
+
+        for distro, command in commands.iteritems():
+            ExtractBlocks.write_to_file(self.bashPath[distro], command)
+
+        return True
+
+
+if __name__ == '__main__':
+
+    with open("rst2bash/config/parser_config.yaml", 'r') as ymlfile:
+        cfg = yaml.load(ymlfile)
+
+    cwd = os.getcwd()
+    rst_path = os.path.join(cwd, cfg['rst_path'])
+    rst_files = cfg['rst_files']
+    bash_path = {distro: os.path.join(cwd, path)
+                 for distro, path in cfg['bash_path'].iteritems()}
+
+    for path_value in bash_path.itervalues():
+
+        if not os.path.exists(path_value):
+            os.mkdir(path_value)
+
+    for rst_file in rst_files:
+
+        rst_file_path = os.path.join(rst_path, rst_file)
+        print("Parsing: %s\n") % (rst_file)
+        code_blocks = ExtractBlocks(rst_file_path, bash_path)
+        code_blocks.get_indice_blocks()
+        try:
+            code_blocks.extract_codeblocks()
+        except Exception:
+            pass
+        bashCode = code_blocks.get_bash_code()
+        if not code_blocks.write_bash_code():
+            raise Exception("Could not write to bash")

tools/cluster

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+echo 'WIP: To be implemented.'

tools/runparser.sh

@@ -0,0 +1,14 @@
+#!/bin/bash -
+
+mkdir -p build/scripts
+git clone \
+    --depth 10 \
+    git://git.openstack.org/openstack/openstack-manuals \
+    build/openstack-manuals
+git clone \
+    git://git.openstack.org/openstack/training-labs \
+    build/training-labs
+
+python2.7 rst2bash/parser.py
+
+echo "Please find the generated BASH scripts in the build/scripts folder!".

tox.ini

@@ -1,15 +1,17 @@
 [tox]
 minversion = 2.0
-envlist = py34,py27,pypy,pep8
+envlist = py27,pep8
 skipsdist = True
 
 [testenv]
 usedevelop = True
-install_command = pip install -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt} {opts} {packages}
+install_command = pip install {opts} {packages}
 setenv =
    VIRTUAL_ENV={envdir}
    PYTHONWARNINGS=default::DeprecationWarning
-deps = -r{toxinidir}/test-requirements.txt
+deps =
+  -r{toxinidir}/requirements.txt
+  -r{toxinidir}/test-requirements.txt
 commands = python setup.py test --slowest --testr-args='{posargs}'
 
 [testenv:pep8]