Add basic docs and runserver command

This commit provides some basic docs on how the
repo works. These shoudl be moved to contibutor
docs later once we have a devstack plugin and
a working install procedure. For now they
are stored in the README of discoverablity.

This change also provide a minimal settings.py
to be used in conjunction with a new
``tox -e runserver`` command. this will run
a local development instance of horizon using in memory
caching an signed cookies for session management.

in the future we may evolve this to run without
all of horizon or modify it based on how the plugin
develops once we have a working devstack jobs.

An example local.conf is also provided which deploys
nova, neutron, placement, keystone, glance an horizon.
addtionally promethus, ceilometer and sg-core are deployed
to provide metrics.

Change-Id: If9f38d88efccd29a63dd2aba4edd56f5f9fa1933

.gitignore

@@ -59,7 +59,7 @@ ChangeLog
 
 # Horizon related
 *.lock
-grian_horizon_plugin/test/.secret_key_store
+src/grian_ui/.secret_key_store
 
 # Files created by releasenotes build
 releasenotes/build

.pre-commit-config.yaml

@@ -70,8 +70,3 @@ repos:
       additional_dependencies:
         - flake8-import-order~=0.18.2
       exclude: '^(doc|releasenotes|tools)/.*$'
-
-- repo: "https://github.com/TotallyNotRobots/check-spdx-header"
-  rev: '0.1.0'
-  hooks:
-    - id: fix-spdx-header

README.rst

@@ -7,3 +7,166 @@ grian-ui - telemetry visualization dashboard
 * Release notes: https://docs.openstack.org/releasenotes/grian-ui/
 * Source: https://opendev.org/openstack/grian-ui
 * Bugs: https://bugs.launchpad.net/grian-ui
+
+
+.. TODO::
+
+   move to contributor docs.
+
+Making Changes & Contributing
+=============================
+
+This project uses `pre-commit`_, and `tox`_
+please make sure to install them before making any changes::
+
+    cd grian_ui
+    python3 -m venv .venv
+    . .venv/bin/activate
+    python3 -m pip install pre-commit tox git-review
+    pre-commit install --install-hooks
+
+It is a good practice for maintainers to update the hooks
+to the latest version every few months. this can be done with::
+
+    pre-commit autoupdate
+
+and submitting a commit.
+when updating the hooks it may be necessary to address new issues
+found. To make ci pass these issues should be fixed in the same
+commit that makes the change however you should avoid making unrelated
+fixes in the same patch.
+
+.. _pre-commit: https://pre-commit.com/
+.. _tox: https://tox.wiki
+
+This project uses tox to execute tests, build docs and automate
+other development task that are not suitable for automation via pre-commit.
+
+
+Running Tests
+-------------
+
+This repo currently has 3 type of tests but more may follow
+
+First unit tests can be run with your envs default python like this ::
+
+  ``tox -e py3``  or  ``tox -e unit``
+
+it also possible to you a non default python version by specifying a version sufix ::
+
+  tox -e py312
+
+The second  type of testing is functional testing. Where as unit
+tests should be short and test individual functions. Functional
+tests should avoid mocking code in the plugin but can use fixtures
+or other fakes/proxy to replace external systems like a fake datasoures
+that returns static data or emulates an openstack service like keystone.
+
+functional tests can be run in a similar way to unit tests ::
+
+  ``tox -e functional`` or ``tox -e functional-py312``
+
+
+The third type of testing we have today are style/lint checks.
+there are 3 ways to run these checks first with tox ::
+
+  ``tox -e pep8`` or ``tox -e lint``
+
+by convention in openstack we use a ``pep8`` tox env to execute all
+our style and linting checks not just pep8. in many other communities
+this is often called ``lint`` so we support both.
+
+internally both targets are identical and call pre-commit to run the
+style checks. So the third way to run the style checks is with
+pre-commit directly::
+
+  ``pre-commit run -a``
+
+or if you have the hooks installed they will run automatically
+when you commit a change. If you ever want to override that behavior
+you can do::
+
+  #skip all checks
+  git commit --no-verify
+
+  or
+
+  # skip a single check
+  SKIP=hacking git commit
+
+ .. NOTE::
+
+    unit, functional and style checks are enforced by our zuul
+    CI system so if they don't pass your code cant merge.
+    CI just run the tox command the same as you can do locally
+    so its encouraged to run them locally first.
+
+
+DOCS
+----
+
+docs are built via tox envs that wrap a tool called sphinx.
+At present we have two types of docs, general docs are stored in
+the doc folder and release notes that are generated by the ``reno``
+tool are stored in the releasenotes directory.
+As can be seen form this README we use a format called
+`reStructuredText`_ to write all our docs.
+
+.. _reStructuredText: https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html
+
+The main docs can be built with either ``tox -e docs`` or ``tox -e pdf-docs``
+depending on the desired format. Note that to be able to build PDF docs in
+particular you may need to install binary depencies on your system.
+These can be found in ``bindep.txt`` in the ``doc`` and ``pdf-docs`` profiles.
+
+The second type of docs we have are the release notes. New release-notes
+can be added by executing ``tox -e venv -- reno new your-note-name-here``
+and built with ``tox -e releasenotes``
+
+
+.. NOTE::
+
+   The venv example above demonstrates that many tox envs support taking
+   additional parameter that can be passed to the underlying tools. In the
+   case of the venv target it will create a virtual enve with all the
+   runtime test and docs depencies installed so tools such as sphinx,
+   reno, manage.py can all be used from that env. For the unit and
+   functional envs additional parameters can be passed to stestr to
+   contol how tests are run such as a text regex.
+
+Running Grian-UI for development
+================================
+
+Work in progress.
+
+This will evolve overtime but as of this moment its possible
+to run Grian-UI in horizon by running ``tox -e runserver``.
+This can also be done manually via ``tox -e venv -- ./manage.py runserver``
+
+For this to be useful you will need to have openstack which can
+be deployed with devstack or any other tool. A sample
+`local.conf <devstack/local.conf-all-in-one>` is provided in the
+devstack plugin.
+
+
+This local.conf expect the following file at /opt/stack/prometheus.yml ::
+
+  global:
+    scrape_interval: 10s
+  scrape_configs:
+    - job_name: "node"
+      static_configs:
+        - targets: ["localhost:3000"]
+
+.. TODO::
+   enable Grian-UI when we add any content.
+   and move the devstack exampel to the devstack plugin
+
+if you are running devstack on a remote vm you can forward the development server endpoint locally using the following command ::
+
+  ssh -N -L 8000:localhost:8000 <user@host>
+
+This will create an ssh session and forward ``localhost:8000`` on the
+remote system to localhost:8000 on your local system.
+`-N` prevents allocating a shell for the ssh session while not required
+it recommended to use a dedicated ssh command to forward ports.

devstack/local.conf-all-in-one

@@ -0,0 +1,68 @@
+[[local|localrc]]
+
+# set default password
+ADMIN_PASSWORD=password
+DATABASE_PASSWORD=${ADMIN_PASSWORD}
+RABBIT_PASSWORD=${ADMIN_PASSWORD}
+SERVICE_PASSWORD=${ADMIN_PASSWORD}
+SERVICE_TOKEN=${ADMIN_PASSWORD}
+
+HOST_IP_IFACE=enp1s0  # alternitivly set HOST_IP[v6] below
+#HOST_IP=192.168.122.51
+#HOST_IPV6=2001:db8::7
+#FLAT_INTERFACE=eth0
+
+MULTI_HOST=1
+
+CEILOMETER_ALARM_THRESHOLD="6000000000"
+CEILOMETER_BACKENDS="sg-core"
+CEILOMETER_PIPELINE_INTERVAL="15"
+
+# Enable the Ceilometer plugin
+enable_plugin ceilometer https://opendev.org/openstack/ceilometer
+enable_plugin devstack-plugin-prometheus https://opendev.org/openstack/devstack-plugin-prometheus
+enable_plugin sg-core https://github.com/openstack-k8s-operators/sg-core main
+
+
+# Enable the ceilometer api explicitly(bug:1667678)
+enable_service ceilometer-api
+enable_service prometheus
+enable_service node-exporter
+# disable cinder
+disable_service c-api
+disable_service c-bak
+disable_service c-sch
+disable_service c-vol
+# disable swift
+disable_service s-account
+disable_service s-container
+disable_service s-object
+disable_service s-proxy
+# disable etcd
+disable_service etcd
+# enable osc in server mode
+enable_service openstack-cli-server
+
+LOGFILE=$DEST/logs/stack.sh.log
+LOGDAYS=2
+
+[[post-config|$NOVA_CONF]]
+[DEFAULT]
+compute_monitors=cpu.virt_driver
+[notifications]
+# Enable both versioned and unversioned notifications. Watcher only
+# uses versioned notifications but ceilometer uses unversioned. We
+# can change this to just versioned when ceilometer handles versioned
+# notifications from nova: https://bugs.launchpad.net/ceilometer/+bug/1665449
+notification_format=both
+
+[[test-config|$TEMPEST_CONFIG]]
+[service_available]
+sg_core = True
+
+[telemetry]
+ceilometer_polling_interval = 15
+disable_ssl_certificate_validation = True
+
+[telemetry_services]
+metric_backends = prometheus

manage.py

@@ -0,0 +1,26 @@
+#!/usr/bin/env python3
+#
+# SPDX-License-Identifier: Apache-2.0
+
+"""Django's command-line utility for administrative tasks."""
+
+import os
+import sys
+
+
+def main():
+    """Run administrative tasks."""
+    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "grian_ui.dev_settings")
+    try:
+        from django.core.management import execute_from_command_line
+    except ImportError as exc:
+        raise ImportError(
+            "Couldn't import Django. Are you sure it's installed and "
+            "available on your PYTHONPATH environment variable? Did you "
+            "forget to activate a virtual environment?"
+        ) from exc
+    execute_from_command_line(sys.argv)
+
+
+if __name__ == "__main__":
+    main()

src/grian_ui/dev_settings.py

@@ -0,0 +1,107 @@
+#
+# SPDX-License-Identifier: Apache-2.0
+
+"""
+Django settings for grian_ui project.
+This is intended for running the plugin during
+development. It is not intended for production use.
+
+Generated by 'django-admin startproject' using Django 4.2.21.
+
+For more information on this file, see
+https://docs.djangoproject.com/en/4.2/topics/settings/
+
+For the full list of settings and their values, see
+https://docs.djangoproject.com/en/4.2/ref/settings/
+"""
+
+import os
+
+from pathlib import Path
+
+from horizon.utils import secret_key
+from openstack_dashboard.settings import *  # noqa
+
+
+# Build paths inside the project like this: BASE_DIR / 'subdir'.
+BASE_DIR = Path(__file__).resolve().parent.parent
+LOCAL_PATH = os.path.dirname(os.path.abspath(__file__))
+
+# Quick-start development settings - unsuitable for production
+# See https://docs.djangoproject.com/en/4.2/howto/deployment/checklist/
+
+# SECURITY WARNING: keep the secret key used in production secret!
+SECRET_KEY = secret_key.generate_or_read_from_file(
+    os.path.join(LOCAL_PATH, ".secret_key_store")
+)
+
+# SECURITY WARNING: don't run with debug turned on in production!
+DEBUG = True
+
+ALLOWED_HOSTS = []
+
+# ROOT_URLCONF = 'grian_ui.urls'
+# WSGI_APPLICATION = 'grian_ui.wsgi.application'
+CACHES = {
+    "default": {
+        "BACKEND": "django.core.cache.backends.locmem.LocMemCache",
+    }
+}
+
+
+# Database
+# https://docs.djangoproject.com/en/4.2/ref/settings/#databases
+# we will not be using a db when embeded in horizon so disable it
+# here as well.
+DATABASES = {}
+
+
+# Password validation
+# https://docs.djangoproject.com/en/4.2/ref/settings/#auth-password-validators
+
+AUTH_PASSWORD_VALIDATORS = [
+    {
+        "NAME": "django.contrib.auth.password_validation.UserAttributeSimilarityValidator",
+    },
+    {
+        "NAME": "django.contrib.auth.password_validation.MinimumLengthValidator",
+    },
+    {
+        "NAME": "django.contrib.auth.password_validation.CommonPasswordValidator",
+    },
+    {
+        "NAME": "django.contrib.auth.password_validation.NumericPasswordValidator",
+    },
+]
+
+
+# Internationalization
+# https://docs.djangoproject.com/en/4.2/topics/i18n/
+
+LANGUAGE_CODE = "en-us"
+
+TIME_ZONE = "UTC"
+
+USE_I18N = True
+
+USE_TZ = True
+
+
+# Static files (CSS, JavaScript, Images)
+# https://docs.djangoproject.com/en/4.2/howto/static-files/
+
+STATIC_URL = "static/"
+
+# Default primary key field type
+# https://docs.djangoproject.com/en/4.2/ref/settings/#default-auto-field
+
+DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"
+
+GRIAN_PLUGIN = {"datasource": os.getenv("GRIAN_DATA_SOURCE", "fake")}
+
+# COMPRESS_OFFLINE = True
+OPENSTACK_KEYSTONE_DEFAULT_ROLE = "member"
+ALLOWED_HOSTS = ["*"]
+# Send email to the console by default
+EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"
+SESSION_ENGINE = "django.contrib.sessions.backends.signed_cookies"

src/grian_ui/local/local_settings.d/_90_grian_ui.py

@@ -0,0 +1,4 @@
+#
+# SPDX-License-Identifier: Apache-2.0
+
+GRIAN_PLUGIN = {"datasource": "fake"}

src/grian_ui/urls.py

@@ -0,0 +1,26 @@
+#
+# SPDX-License-Identifier: Apache-2.0
+
+"""
+URL configuration for example_project project.
+
+The `urlpatterns` list routes URLs to views. For more information please see:
+    https://docs.djangoproject.com/en/4.2/topics/http/urls/
+Examples:
+Function views
+    1. Add an import:  from my_app import views
+    2. Add a URL to urlpatterns:  path('', views.home, name='home')
+Class-based views
+    1. Add an import:  from other_app.views import Home
+    2. Add a URL to urlpatterns:  path('', Home.as_view(), name='home')
+Including another URLconf
+    1. Import the include() function: from django.urls import include, path
+    2. Add a URL to urlpatterns:  path('blog/', include('blog.urls'))
+"""
+
+from django.contrib import admin
+from django.urls import path
+
+urlpatterns = [
+    path("admin/", admin.site.urls),
+]

src/grian_ui/wsgi.py

@@ -0,0 +1,19 @@
+#
+# SPDX-License-Identifier: Apache-2.0
+
+"""
+WSGI config for example_project project.
+
+It exposes the WSGI callable as a module-level variable named ``application``.
+
+For more information on this file, see
+https://docs.djangoproject.com/en/4.2/howto/deployment/wsgi/
+"""
+
+import os
+
+from django.core.wsgi import get_wsgi_application
+
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "grian_ui.settings")
+
+application = get_wsgi_application()

tox.ini

@@ -11,6 +11,7 @@ allowlist_externals =
   rm
   env
   make
+  ./manage.py
 install_command = python -I -m pip install -c{env:TOX_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master} {opts} {packages}
 setenv =
   VIRTUAL_ENV={envdir}
@@ -51,11 +52,12 @@ passenv =
   # there is also secret magic in subunit-trace which lets you run in a fail only
   # mode. To do this define the TRACE_FAILONLY environmental variable.
   TRACE_FAILONLY
+  DJANGO_SETTINGS_MODULE
 commands =
   stestr run {posargs}
   stestr slowest
 
-[testenv:pep8]
+[testenv:{pep8,lint}]
 description =
     Run style checks.
 deps =
@@ -115,6 +117,15 @@ commands =
   sphinx-build -W --keep-going -b latex -j auto doc/source doc/build/pdf
   make -C doc/build/pdf
 
+[testenv:runserver]
+description =
+    run horizon with grian ui enabled.
+deps =
+    {[testenv]deps}
+commands =
+    ./manage.py runserver
+
+
 [flake8]
 # We only enable the hacking (H) and local checks (N)
 select = H,N