Restructure the documentation according the new spec

Group the existing pages in the new top-level categories; each
category can be seen as a separate document and it has its own
index file.
The content of the pages was not changed, with the obvious
exception of the links between pages.

The autogenerated configuration has not been added yet to
configuration/; it will be fixed in a future commit.

At the same time, as suggested by the doc team, consistently
use only one separator in file names (dash, '-') instead of a mix
of dashes, dots and underscores. This may break even more links on
the Internet, but we are breaking them anyway by moving files.
Redirects can be set, but not in this commit.

Closes-Bug: #1706184
Change-Id: I5a10378d9da2603d617ad4193ea8d90e2afc5104

doc/source/admin/advanced-configuration-guide.rst

@@ -196,7 +196,7 @@ To run sahara in distributed mode pick several hosts on which
 you want to run sahara services and follow these steps:
 
  * On each host install and configure sahara using the
-   `installation guide <../installation.guide.html>`_
+   `installation guide <../install/installation-guide.html>`_
    except:
 
     * Do not run ``sahara-db-manage`` or launch sahara with ``sahara-all``
@@ -250,7 +250,7 @@ respective source trees:
 
 These options will also be present in the generated sample configuration
 file. For instructions on creating the configuration file please see the
-:doc:`configuration.guide`.
+:doc:`configuration-guide`.
 
 .. _distributed-periodic-tasks:
 
@@ -565,7 +565,7 @@ creation, the created volume will have an ``Error`` state and can not be used.
 Autoconfiguration for templates
 -------------------------------
 
-:doc:`../userdoc/configs_recommendations`
+:doc:`configs-recommendations`
 
 
 NTP service configuration

doc/source/admin/index.rst

@@ -0,0 +1,10 @@
+======================
+Operator Documentation
+======================
+
+.. toctree::
+   :maxdepth: 2
+
+   configuration-guide
+   advanced-configuration-guide
+   upgrade-guide

doc/source/admin/upgrade-guide.rst

@@ -39,7 +39,7 @@ and ``identity_uri`` parameters. These new parameters should be
 full URIs to the keystone public and admin endpoints, respectively.
 
 For more information about these configuration parameters please see
-the :doc:`configuration.guide`.
+the :doc:`../admin/configuration-guide`.
 
 Database package changes
 ++++++++++++++++++++++++
@@ -78,7 +78,7 @@ The HEAT infrastructure engine has been updated to use the same rules for
 instance user names as the direct engine. In previous releases the user
 name for instances created by sahara using HEAT was always 'ec2-user'. As
 of Juno, the user name is taken from the image registry as described in
-the :doc:`registering_image` document.
+the :doc:`../user/registering-image` document.
 
 This change breaks backward compatibility for clusters created using the HEAT
 infrastructure engine prior to the Juno release. Clusters will continue to

doc/source/configuration/index.rst

@@ -0,0 +1,8 @@
+=======================
+Configuration Reference
+=======================
+
+.. toctree::
+   :maxdepth: 1
+
+   sampleconfig

doc/source/contributor/dashboard-dev-environment-guide.rst

@@ -8,7 +8,7 @@ isolated environment and running from the command line.
 Install as a part of DevStack
 -----------------------------
 
-See the `DevStack guide <../devref/devstack.html>`_ for more information
+See the `DevStack guide <devstack.html>`_ for more information
 on installing and configuring DevStack with Sahara.
 
 Sahara UI can be installed as a DevStack plugin by adding the following line

doc/source/contributor/development-environment.rst

@@ -69,7 +69,7 @@ On openSUSE-based distributions (SLES 12, openSUSE, Factory or Tumbleweed):
 
 5. Look through the sahara.conf and modify parameter values as needed
    For details see
-   :doc:`Sahara Configuration Guide </userdoc/configuration.guide>`
+   :doc:`Sahara Configuration Guide <../admin/configuration-guide>`
 
 6. Create database schema
 
@@ -92,7 +92,7 @@ Setup local OpenStack dashboard with Sahara plugin
     :maxdepth: 1
 
 
-    ../horizon/dev.environment.guide
+    dashboard-dev-environment-guide
 
 Tips and tricks for dev environment
 -----------------------------------

doc/source/contributor/how-to-participate.rst

@@ -63,7 +63,7 @@ How to post your first patch for review
 * Apply and commit your changes
 
 * Make sure that your code passes ``PEP8`` checks and unit-tests.
-  See :doc:`development.guidelines`
+  See :doc:`development-guidelines`
 
 * Post your patch for review
 

doc/source/contributor/index.rst

@@ -0,0 +1,31 @@
+=====================
+Developer Information
+=====================
+
+Programming HowTos and Tutorials
+================================
+
+.. toctree::
+   :maxdepth: 2
+
+   development-guidelines
+   development-environment
+   devstack
+   dashboard-dev-environment-guide
+   how-to-participate
+   how-to-build-oozie
+   adding-database-migrations
+   testing
+   log-guidelines
+   apiv2
+   image-gen
+
+Other Resources
+===============
+
+.. toctree::
+   :maxdepth: 2
+
+   launchpad
+   gerrit
+   jenkins

doc/source/index.rst

@@ -10,110 +10,60 @@ Overview
 --------
 
 .. toctree::
-    :maxdepth: 1
+   :maxdepth: 2
 
-    overview
-    architecture
-    Roadmap <https://wiki.openstack.org/wiki/Sahara/Roadmap>
+   intro/index
 
 
-User guide
-----------
-
-**Installation**
-
-.. toctree::
-   :maxdepth: 1
-
-   userdoc/installation.guide
-   userdoc/configuration.guide
-   userdoc/dashboard.guide
-   userdoc/advanced.configuration.guide
-   userdoc/upgrade.guide
-   userdoc/sampleconfig
-
-**How To**
-
-.. toctree::
-   :maxdepth: 1
-
-   userdoc/overview
-   horizon/dashboard.user.guide
-   userdoc/features
-   userdoc/registering_image
-
-**Plugins**
-
-.. toctree::
-   :maxdepth: 1
-
-   userdoc/plugins
-   userdoc/vanilla_plugin
-   userdoc/ambari_plugin
-   userdoc/spark_plugin
-   userdoc/cdh_plugin
-   userdoc/mapr_plugin
-
-**Elastic Data Processing**
-
-.. toctree::
-   :maxdepth: 1
-
-   userdoc/edp
-
-**API**
+Installation
+------------
 
 .. toctree::
    :maxdepth: 2
 
-   restapi
+   install/index
 
-**Miscellaneous**
+
+Configuration
+-------------
 
 .. toctree::
-   :maxdepth: 1
+   :maxdepth: 2
 
-   userdoc/guest-requirements
-   userdoc/hadoop-swift
-   userdoc/vanilla_imagebuilder
-   userdoc/cdh_imagebuilder
+   configuration/index
 
-Developer Guide
----------------
-**Programming HowTos and Tutorials**
+
+User Guide
+----------
 
 .. toctree::
-    :maxdepth: 1
+   :maxdepth: 2
 
-    devref/development.guidelines
-    devref/development.environment
-    devref/devstack
-    horizon/dev.environment.guide
-    devref/quickstart
-    devref/how_to_participate
-    devref/how_to_build_oozie
-    devref/adding_database_migrations
-    devref/testing
-    devref/log.guidelines
-    devref/apiv2
-    devref/image-gen
+   user/index
 
-**Background Concepts for Sahara**
+
+Operator Documentation
+----------------------
 
 .. toctree::
-    :maxdepth: 1
+   :maxdepth: 2
 
-    devref/plugins
-    devref/plugin.spi
-    devref/edp.spi
-    userdoc/statuses
-    userdoc/sahara_on_ironic
+   admin/index
 
-**Other Resources**
+
+Contributor Documentation
+-------------------------
 
 .. toctree::
-   :maxdepth: 1
+   :maxdepth: 2
 
-   devref/launchpad
-   devref/gerrit
-   devref/jenkins
+   contributor/index
+
+
+Programming Reference
+---------------------
+
+.. toctree::
+   :maxdepth: 2
+
+   reference/index

doc/source/install/index.rst

@@ -0,0 +1,9 @@
+==================
+Installation Guide
+==================
+
+.. toctree::
+   :maxdepth: 2
+
+   installation-guide
+   dashboard-guide

doc/source/install/installation-guide.rst

@@ -39,7 +39,7 @@ To install with RDO
 
 3. Configure sahara as needed. The configuration file is located in
    ``/etc/sahara/sahara.conf``. For details see
-   :doc:`Sahara Configuration Guide <configuration.guide>`
+   :doc:`Sahara Configuration Guide <../admin/configuration-guide>`
 
 4. Create the database schema:
 
@@ -143,7 +143,8 @@ To install into a virtual environment
 ..
 
     Make any necessary changes to ``sahara-venv/etc/sahara.conf``.
-    For details see :doc:`Sahara Configuration Guide <configuration.guide>`
+    For details see
+    :doc:`Sahara Configuration Guide <../admin/configuration-guide>`
 
 .. _common_installation_steps:
 
@@ -209,7 +210,7 @@ installations of sahara.
 ..
 
 5. For more information on configuring sahara with the OpenStack Dashboard
-   please see :doc:`dashboard.guide`.
+   please see :doc:`dashboard-guide`.
 
 Optional installation of default templates
 ------------------------------------------
@@ -276,5 +277,6 @@ To get the list of all possible options run:
     $ sahara-venv/bin/python sahara-venv/bin/sahara-engine --help
 ..
 
-Further, consider reading :doc:`overview` for general sahara concepts and
-:doc:`plugins` for specific plugin features/requirements.
+Further, consider reading :doc:`../intro/overview` for general sahara
+concepts and :doc:`../user/plugins` for specific plugin
+features/requirements.

doc/source/intro/architecture.rst

@@ -1,7 +1,7 @@
 Architecture
 ============
 
-.. image:: images/sahara-architecture.svg
+.. image:: ../images/sahara-architecture.svg
     :width: 800 px
     :scale: 100 %
     :align: left
@@ -26,7 +26,7 @@ The Sahara architecture consists of several components:
   management solutions like Apache Ambari and Cloudera Management Console
   could be utilized for that purpose as well.
 
-* EDP - :doc:`../userdoc/edp` responsible for scheduling and managing
+* EDP - :doc:`../user/edp` responsible for scheduling and managing
   data processing jobs on clusters provisioned by sahara.
 
 * REST API - exposes sahara functionality via REST HTTP interface.

doc/source/intro/index.rst

@@ -0,0 +1,12 @@
+===============
+Sahara Overview
+===============
+
+General overview of Sahara.
+
+.. toctree::
+   :maxdepth: 2
+
+   overview
+   architecture
+   Roadmap <https://wiki.openstack.org/wiki/Sahara/Roadmap>

doc/source/intro/overview.rst

@@ -72,7 +72,7 @@ The sahara product communicates with the following OpenStack services:
 * Key manager (barbican & castellan) - persists the authentication data
   like passwords and private keys in a secure storage.
 
-.. image:: images/openstack-interop.png
+.. image:: ../images/openstack-interop.png
     :width: 800 px
     :scale: 99 %
     :align: left
@@ -176,7 +176,7 @@ This makes it possible to integrate swift with software that relies on data
 locality information to avoid network overhead.
 
 To get more information on how to enable swift support see
-:doc:`userdoc/hadoop-swift`.
+:doc:`../user/hadoop-swift`.
 
 Pluggable Deployment and Monitoring
 -----------------------------------

doc/source/reference/edp-spi.rst

@@ -4,7 +4,7 @@ Elastic Data Processing (EDP) SPI
 The EDP job engine objects provide methods for creating, monitoring, and
 terminating jobs on Sahara clusters. Provisioning plugins that support EDP
 must return an EDP job engine object from the :ref:`get_edp_engine` method
-described in :doc:`plugin.spi`.
+described in :doc:`plugin-spi`.
 
 Sahara provides subclasses of the base job engine interface that support EDP
 on clusters running Oozie, Spark, and/or Storm. These are described below.
@@ -104,7 +104,7 @@ get_possible_job_config(job_type)
 Returns hints used by the Sahara UI to prompt users for values when
 configuring and launching a job. Note that no hints are required.
 
-See :doc:`/userdoc/edp` for more information on how configuration values,
+See :doc:`../user/edp` for more information on how configuration values,
 parameters, and arguments are used by different job types.
 
 *Returns*: a dictionary of the following form, containing hints for configs,

doc/source/reference/index.rst

@@ -0,0 +1,22 @@
+=====================
+Programming Reference
+=====================
+
+Plugins and EDP
+===============
+
+.. toctree::
+   :maxdepth: 2
+
+   plugins
+   plugin-spi
+   edp-spi
+
+
+REST API
+========
+
+.. toctree::
+   :maxdepth: 2
+
+   restapi

doc/source/reference/plugin-spi.rst

@@ -112,7 +112,7 @@ get_edp_engine( cluster, job_type )
 
 Returns an EDP job engine object that supports the specified job_type on the
 given cluster, or None if there is no support. The EDP job engine object
-returned must implement the interface described in :doc:`edp.spi`.  The
+returned must implement the interface described in :doc:`edp-spi`.  The
 job_type is a String matching one of the job types listed in
 :ref:`edp_spi_job_types`.
 
@@ -166,7 +166,7 @@ get_image_arguments( self, hadoop_version ):
 
 Optional method, which gets the argument set taken by the plugin's image
 generator, or NotImplemented if the plugin does not provide image generation
-support. See :doc:`image-gen`.
+support. See :doc:`../contributor/image-gen`.
 
 *Returns*: A sequence with items of type sahara.plugins.images.ImageArgument.
 
@@ -175,7 +175,7 @@ pack_image( self, hadoop_version, remote, test_only=False, ... ):
 
 Optional method which packs an image for registration in Glance and use by
 Sahara. This method is called from the image generation CLI rather than from
-the Sahara api or engine service. See :doc:`image-gen`.
+the Sahara api or engine service. See :doc:`../contributor/image-gen`.
 
 *Returns*: None (modifies the image pointed to by the remote in-place.)
 
@@ -183,7 +183,7 @@ validate_images( self, cluster, test_only=False, image_arguments=None ):
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Validates the image to be used to create a cluster, to ensure that it meets
-the specifications of the plugin. See :doc:`image-gen`.
+the specifications of the plugin. See :doc:`../contributor/image-gen`.
 
 *Returns*: None; may raise a sahara.plugins.exceptions.ImageValidationError
 

doc/source/reference/plugins.rst

@@ -13,7 +13,7 @@ a plugin can deploy management and monitoring tools for the cluster. Sahara
 provides plugins with utility methods to work with provisioned instances.
 
 A plugin must extend the `sahara.plugins.provisioning:ProvisioningPluginBase`
-class and implement all the required methods. Read :doc:`plugin.spi` for
+class and implement all the required methods. Read :doc:`plugin-spi` for
 details.
 
 The `instance` objects provided by Sahara have a `remote` property which

doc/source/user/cdh-imagebuilder.rst

@@ -1,4 +1,4 @@
-.. _cdh_diskimage-builder-label:
+.. _cdh-diskimage-builder-label:
 
 Building Images for Cloudera Plugin
 ===================================

doc/source/user/cdh-plugin.rst

@@ -8,7 +8,7 @@ The Cloudera plugin is enabled in Sahara by default. You can manually
 modify the Sahara configuration file (default /etc/sahara/sahara.conf) to
 explicitly enable or disable it in "plugins" line.
 
-You need to build images using :doc:`cdh_imagebuilder` to produce images used
+You need to build images using :doc:`cdh-imagebuilder` to produce images used
 to provision cluster or you could download prepared images from
 http://sahara-files.mirantis.com/images/upstream/
 They already have Cloudera Express installed (version 5.0.0, 5.3.0, 5.4.0,

doc/source/user/dashboard-user-guide.rst

@@ -4,7 +4,7 @@ Sahara (Data Processing) UI User Guide
 This guide assumes that you already have the sahara service and Horizon
 dashboard up and running. Don't forget to make sure that sahara is
 registered in Keystone. If you require assistance with that, please see the
-`installation guide <../userdoc/installation.guide.html>`_.
+`installation guide <../install/installation-guide-html>`_.
 
 The sections below give a panel by panel overview of setting up clusters
 and running jobs.  For a description of using the guided cluster and job tools,
@@ -379,7 +379,7 @@ Additional Notes
    swift Data Sources. It should be noted that it is possible to configure
    sahara such that the username/password credentials are *not* required. For
    more information on that, please refer to: :doc:`Sahara Advanced
-   Configuration Guide <../userdoc/advanced.configuration.guide>`
+   Configuration Guide <../admin/advanced-configuration-guide>`
 
 Launching a cluster via the Cluster Creation Guide
 --------------------------------------------------

doc/source/user/edp.rst

@@ -27,7 +27,7 @@ Interfaces
 ----------
 
 The EDP features can be used from the sahara web UI which is described in the
-:doc:`../horizon/dashboard.user.guide`.
+:doc:`dashboard-user-guide`.
 
 The EDP features also can be used directly by a client through the
 `REST api <http://developer.openstack.org/api-ref/data-processing/>`_
@@ -63,7 +63,7 @@ stored internally.
 
 Sahara requires credentials (username and password) to access files stored in
 swift unless swift proxy users are configured as described in
-:doc:`../userdoc/advanced.configuration.guide`. The swift service must be
+:doc:`../admin/advanced-configuration-guide`. The swift service must be
 running in the same OpenStack installation referenced by sahara.
 
 To reference a binary file stored in manila, create the job binary with the
@@ -383,7 +383,7 @@ Generation of Swift Properties for Data Sources
 +++++++++++++++++++++++++++++++++++++++++++++++
 
 If swift proxy users are not configured (see
-:doc:`../userdoc/advanced.configuration.guide`) and a job is run with data
+:doc:`../admin/advanced-configuration-guide`) and a job is run with data
 source objects containing swift paths, sahara will automatically generate
 swift username and password configuration values based on the credentials
 in the data sources. If the input and output data sources are both in swift,

doc/source/user/features.rst

@@ -4,8 +4,8 @@ Features Overview
 This page highlights some of the most prominent features available in
 sahara. The guidance provided here is primarily focused on the
 runtime aspects of sahara. For discussions about configuring the sahara
-server processes please see the :doc:`configuration.guide` and
-:doc:`advanced.configuration.guide`.
+server processes please see the :doc:`../admin/configuration-guide` and
+:doc:`../admin/advanced-configuration-guide`.
 
 Anti-affinity
 -------------
@@ -92,9 +92,10 @@ documentation.
 Distributed Mode
 ----------------
 
-The :doc:`installation.guide` suggests launching sahara in distributed mode
-with ``sahara-api`` and ``sahara-engine`` processes potentially running on
-several machines simultaneously. Running in distributed mode allows sahara to
+The :doc:`../install/installation-guide` suggests launching sahara in
+distributed mode with ``sahara-api`` and ``sahara-engine`` processes
+potentially running on several machines simultaneously.
+Running in distributed mode allows sahara to
 offload intensive tasks to the engine processes while keeping the API
 process free to handle requests.
 
@@ -172,7 +173,8 @@ DNS support
 -----------
 
 Sahara can resolve hostnames of cluster instances by using DNS. For this Sahara
-uses designate. For additional details see :doc:`advanced.configuration.guide`.
+uses designate. For additional details see
+:doc:`../admin/advanced-configuration-guide`.
 
 Kerberos support
 ----------------

doc/source/user/guest-requirements.rst

@@ -35,7 +35,7 @@ To support EDP, the following components must also be installed on the guest:
 * mysql
 * hive
 
-See :doc:`vanilla_imagebuilder` for instructions on building images for this
+See :doc:`vanilla-imagebuilder` for instructions on building images for this
 plugin.
 
 Hortonworks Plugin Requirements
@@ -53,5 +53,5 @@ Cloudera Plugin Requirements
 Cloudera Plugin does not have any additional requirements, just build a CDH
 image to deploy the cluster.
 
-See :doc:`cdh_imagebuilder` for instructions on building images for this
+See :doc:`cdh-imagebuilder` for instructions on building images for this
 plugin.

doc/source/user/index.rst

@@ -0,0 +1,52 @@
+==========
+User Guide
+==========
+
+General concepts and guides
+===========================
+
+.. toctree::
+   :maxdepth: 2
+
+   overview
+   quickstart
+   dashboard-user-guide
+   features
+   registering-image
+   statuses
+   sahara-on-ironic
+
+
+Plugins
+=======
+
+.. toctree::
+   :maxdepth: 2
+
+   plugins
+   vanilla-plugin
+   ambari-plugin
+   spark-plugin
+   cdh-plugin
+   mapr-plugin
+
+
+Elastic Data Processing
+=======================
+
+.. toctree::
+   :maxdepth: 2
+
+   edp
+
+
+Guest Images
+============
+
+.. toctree::
+   :maxdepth: 2
+
+   guest-requirements
+   hadoop-swift
+   vanilla-imagebuilder
+   cdh-imagebuilder

doc/source/user/overview.rst

@@ -61,7 +61,7 @@ framework on the VMs from scratch. Some plugins might require images with
 pre-installed frameworks or Hadoop distributions.
 
 The Sahara Image Registry is a feature which helps filter out images during
-cluster creation. See :doc:`registering_image` for details on how to work
+cluster creation. See :doc:`registering-image` for details on how to work
 with Image Registry.
 
 Features

doc/source/user/plugins.rst

@@ -6,11 +6,11 @@ enables sahara to deploy a specific data processing framework (for example,
 Hadoop) or distribution, and allows configuration of topology and
 management/monitoring tools.
 
-* :doc:`vanilla_plugin` - deploys Vanilla Apache Hadoop
-* :doc:`ambari_plugin` - deploys Hortonworks Data Platform
-* :doc:`spark_plugin` - deploys Apache Spark with Cloudera HDFS
-* :doc:`mapr_plugin` - deploys MapR plugin with MapR File System
-* :doc:`cdh_plugin` - deploys Cloudera Hadoop
+* :doc:`vanilla-plugin` - deploys Vanilla Apache Hadoop
+* :doc:`ambari-plugin` - deploys Hortonworks Data Platform
+* :doc:`spark-plugin` - deploys Apache Spark with Cloudera HDFS
+* :doc:`mapr-plugin` - deploys MapR plugin with MapR File System
+* :doc:`cdh-plugin` - deploys Cloudera Hadoop
 
 Managing plugins
 ----------------

doc/source/user/quickstart.rst

@@ -5,18 +5,19 @@ Quickstart guide
 Launching a cluster via Sahara CLI commands
 ===========================================
 This guide will help you setup a vanilla Hadoop cluster using a combination
-of OpenStack command line tools and the sahara :doc:`REST API <../restapi>`.
+of OpenStack command line tools and the sahara
+:doc:`REST API <../reference/restapi>`.
 
 1. Install sahara
 -----------------
 
 * If you want to hack the code follow
-  :doc:`development.environment`.
+  :doc:`../contributor/development-environment`.
 
 OR
 
 * If you just want to install and use sahara follow
-  :doc:`../userdoc/installation.guide`.
+  :doc:`../install/installation-guide`.
 
 2. Identity service configuration
 ---------------------------------
@@ -121,7 +122,7 @@ Register the image with the username ``ubuntu``.
     a base OS wherever possible; it is better supported throughout OpenStack
     image maintenance infrastructure and its more modern filesystem is much
     more appropriate for large-scale data processing. For more please see
-    :doc:`../userdoc/vanilla_plugin`
+    :doc:`../user/vanilla-plugin`
 
 .. sourcecode:: console
 

doc/source/user/sahara-on-ironic.rst

@@ -15,7 +15,7 @@ from the bare metal performance with self-service resource provisioning.
    <https://docs.openstack.org/ironic/latest/install/index.html>`_
 
 3. Install Sahara as described in the `Sahara Installation Guide
-   <https://docs.openstack.org/sahara/latest/userdoc/installation.guide.html>`_
+   <../install/installation-guide.html>`_
 
 4. Build the Sahara image and prepare it for uploading to Glance:
 

doc/source/user/vanilla-plugin.rst

@@ -10,7 +10,7 @@ can launch Spark jobs on a Vanilla cluster.
 For cluster provisioning prepared images should be used. They already have
 Apache Hadoop 2.7.1 installed.
 
-You may build images by yourself using :doc:`vanilla_imagebuilder` or you could
+You may build images by yourself using :doc:`vanilla-imagebuilder` or you could
 download prepared images from http://sahara-files.mirantis.com/images/upstream
 
 Vanilla plugin requires an image to be tagged in Sahara Image Registry with