From 15944befb6368e076c8ce175acff638ef31c0a3b Mon Sep 17 00:00:00 2001 From: Alexander Kislitsky Date: Fri, 18 Mar 2016 18:16:29 +0300 Subject: [PATCH] Docs for fuel-stats were rewritten README format changed from md to rst. Project components description added. Old README.md content moved to 'How to' recipes in the README.rst Change-Id: I3f6471f859cb4f0998fe8c1bba784f4da5368b1a Closes-Bug: #1507958 --- README.md | 134 ---------------------------------- README.rst | 211 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 211 insertions(+), 134 deletions(-) delete mode 100644 README.md create mode 100644 README.rst diff --git a/README.md b/README.md deleted file mode 100644 index 51e30f8..0000000 --- a/README.md +++ /dev/null @@ -1,134 +0,0 @@ -collector -========= - -Prototype of statistic collector - -Requirements ----------- - -System requirements: -postgresql database server of version 9.3 or greater. - -To install python requirements use command: -`pip install -r {corresponding_requirement_file}` - -Operations ----------- - -By default manage_collector.py works with prod settings. -For working with test settings use `python manage_collector.py --mode` option. - -For creating DB migration: - - python manage_collector.py --mode test db migrate -m "Migration comment" \ - -d collector/api/db/migrations/ - -Create DB user with password 'collector': - - sudo -u postgres createuser -DES collector - -or: - - sudo -u postgres psql - CREATE ROLE collector WITH NOSUPERUSER NOCREATEDB NOCREATEROLE LOGIN ENCRYPTED PASSWORD 'collector'; - -Create DB and grant privileges to it: - - sudo -u postgres psql - CREATE DATABASE collector; - GRANT ALL ON DATABASE collector TO collector; - -For apply the latest migration: - - python manage_collector.py --mode test db upgrade -d collector/api/db/migrations/ - -For revert all migrations: - - python manage_collector.py --mode test db downgrade -d collector/api/db/migrations/ - -For starting test server: - - python manage_collector.py --mode test runserver - -Example config for uWSGI is located in collector/uwsgi/collector_test.yaml - - -Local setup ----------- - -To run statistics UI locally you need to do the following: - -Install elasticsearch 1.3 - - pip install elasticsearch - -Run test_report from NodesDistribution - - prepare virtualenv: - - cd fuel-stats - virtualenv .venv - source .venv/bin/activate - pip install -r collector/test-requirements.txt - cd migration - - run tests: - - nosetests migration.test.report.test_reports:Reports.test_libvirt_type_distribution - - or - - nosetests migration.test.report.test_os_distribution:OsDistribution - - - this will create demo data from elasticsearch - -Install elasticsearch service - - you can use this helpfull gist https://gist.github.com/wingdspur/2026107 - -And don't forget to start elasticsearch service - - sudo service elasticsearch start - -Nginx installation - - sudo apt-get install nginx - - fix Nginx config: - - server { - listen 8888; // your free port - location / { - root /home/kpimenova/fuel/fuel-stats/analytics/static; // your path to fuel-stats/analytics/static - } - location ~ ^(/fuel)?(/[A-Za-z_0-9])?/(_count|_search) { - proxy_pass http://127.0.0.1:9200; - } - } - -Then restart Nginx: - - service nginx restart - - After this your local server will be available at 0.0.0.0:8888 // or any other port you've set up :) - - - Also for correct UI work you need to setup a few things - - Install nodejs packages - - cd fuel-stats/analytics/static - npm install - - Install bower packages - - cd fuel-stats/analytics/static - gulp bower - - That's all. - - You can anytime lint your code by running - - gulp lint - diff --git a/README.rst b/README.rst new file mode 100644 index 0000000..9835a03 --- /dev/null +++ b/README.rst @@ -0,0 +1,211 @@ +========== +Fuel stats +========== + +Project purpose +--------------- + +* collects stats information about OpenStack installations made by Fuel_, +* generates stat reports in the CSV format, +* provides API for fetching raw data in the JSON format, +* provides Web UI for reports generation and basic stats charts/histograms. + +Components +---------- + +Collector is the service for collecting stats. It has REST API and DB storage. +Analytics is the service for generating reports. It has REST API. +Migrator is the tool for migrating data from the DB to the Elasticsearch_. + +The collector and analytics services are started by uWSGI_. Migrator is +started by cron to migrate the fresh data into Elasticsearch_. + +Collector +--------- + +Data origin for collector is the Fuel_ master node. Stats collecting daemons +collect and send data to the collector if allowed by the cloud operator. + +Stats data is stored to the DB PostgreSQL_. + +Migrator +-------- + +Migrator periodically migrates data from the fuel-stats DB to the +Elasticsearch_ storage. This storage is used to generate basic stats charts +and histograms for the Web UI. + +Analytics +--------- + +There are two sub-components in the analytics: + +* analytics service +* Web UI + +The analytics service API provides generation of CSV reports for installation +info, plugins, and OpenStack workloads. The analytics API also provides +export of data from DB as JSON. + +The analytics Web UI provides basic summary stats charts and histograms with +the possibility of filtering data by the Fuel_ release version. Also, in the +Web UI we can generate and download stats reports on a selected time period. + +.. _howto configure dev environment: + +How to configure development environment +---------------------------------------- + +To start fuel-stats on a localhost we need to have: + +* PostgreSQL_ of version 9.3 or greater, +* Elasticsearch_ of version 1.3.4 or greater, +* Nginx_. + +Install PostgreSQL_ and development libraries: :: + + sudo apt-get install --yes postgresql postgresql-server-dev-all + +Configure Elasticsearch_ repo as described in the `elasticsearch docs`_ and +install Elasticsearch_::: + + sudo apt-get install --yes elasticsearch + +Install pip and development tools: :: + + sudo apt-get install --yes python-dev python-pip + +Install virtualenv. This step increases flexibility when dealing with +environment settings and package installation: :: + + sudo pip install virtualenv virtualenvwrapper + +You can add '. /usr/local/bin/virtualenvwrapper.sh' to .bashrc or just +execute it.:: + + . /usr/local/bin/virtualenvwrapper.sh + +Create and activate virtual env for fuel-stats: :: + + mkvirtualenv stats + workon stats + +You can use any name for the virtual env instead of 'stats'. + +Install the fuel-stats requirements: :: + + pip install -r test-requirements.txt + +Create a DB user for fuel-stats: :: + + sudo -u postgres psql + CREATE ROLE collector WITH NOSUPERUSER NOCREATEDB NOCREATEROLE LOGIN ENCRYPTED PASSWORD 'collector'; + +Create a DB and grant privileges to it: :: + + sudo -u postgres psql + CREATE DATABASE collector; + GRANT ALL ON DATABASE collector TO collector; + +Check that all tests are passed: :: + + cd fuel-stats/collector && tox + cd fuel-stats/migration && tox + cd fuel-stats/analytics && tox + +**NOTE:** The collector tests must be performed the first. + +Now you are ready to develop fuel-stats. + +How to configure Web UI +----------------------- + +We assume that you already have configured virtual env as described in +`howto configure dev environment`_. + +Install elsticsearch library and create sample data: :: + + workon stats + pip install elasticsearch + cd migration + nosetests migration.test.report.test_reports:Reports.test_libvirt_type_distribution + +Install nodejs: :: + + sudo apt-get remove nodejs nodejs-legacy npm + sudo add-apt-repository -y ppa:chris-lea/node.js + sudo apt-get update + sudo apt-get install nodejs + +Install nodejs and bower packages: :: + + cd fuel-stats/analytics/static + npm install + gulp bower + +You can anytime lint your code by running: :: + + gulp lint + +Add site configuration to Nginx_: :: + + server { + listen 8888; + location / { + root /your-path-to-fuel-stats/fuel-stats/analytics/static; + } + location ~ ^(/fuel)?(/[A-Za-z_0-9])?/(_count|_search) { + proxy_pass http://127.0.0.1:9200; + } + } + +Then restart Nginx: :: + + service nginx restart + +After this, your local server will be available at 0.0.0.0:8888 +or any other port you've set up. + +How to start local collector +---------------------------- + +You can use uWSGI_ to start the collector. Sample config can be found in +collector/uwsgi/collector_example.yaml. + +Or test web service to be used. To start the test web service, run: :: + + python collector/manage_collector.py --mode test runserver + +How to start local analytics +---------------------------- + +You can use uWSGI_ to start analytics. Sample config can be found in +analytics/uwsgi/analytics_example.yaml. + +Or test the web service to be used. To start the test web service, run: :: + + python analytics/manage_analytics.py --mode test runserver + +How to deal with DB migrations +------------------------------ + +Create new DB migration: :: + + python manage_collector.py --mode test db migrate -m "Migration comment" \ + -d collector/api/db/migrations/ + +Apply all DB migrations: :: + + python manage_collector.py --mode test db upgrade -d collector/api/db/migrations/ + +Revert all migrations: :: + + python manage_collector.py --mode test db downgrade -d collector/api/db/migrations/ + + +.. _Fuel: https://docs.mirantis.com/openstack/fuel/ +.. _Elasticsearch: https://www.elastic.co/ +.. _uWSGI: https://pypi.python.org/pypi/uWSGI/ +.. _PostgreSQL: http://www.postgresql.org/ +.. _Nginx: http://nginx.org/ +.. _elasticsearch docs: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html