Docs for fuel-stats were rewritten
README format changed from md to rst. Project components description added. Old README.md content moved to 'How to' recipes in the README.rst Change-Id: I3f6471f859cb4f0998fe8c1bba784f4da5368b1a Closes-Bug: #1507958
This commit is contained in:
parent
e9ec2111a7
commit
15944befb6
134
README.md
134
README.md
@ -1,134 +0,0 @@
|
|||||||
collector
|
|
||||||
=========
|
|
||||||
|
|
||||||
Prototype of statistic collector
|
|
||||||
|
|
||||||
Requirements
|
|
||||||
----------
|
|
||||||
|
|
||||||
System requirements:
|
|
||||||
postgresql database server of version 9.3 or greater.
|
|
||||||
|
|
||||||
To install python requirements use command:
|
|
||||||
`pip install -r {corresponding_requirement_file}`
|
|
||||||
|
|
||||||
Operations
|
|
||||||
----------
|
|
||||||
|
|
||||||
By default manage_collector.py works with prod settings.
|
|
||||||
For working with test settings use `python manage_collector.py --mode` option.
|
|
||||||
|
|
||||||
For creating DB migration:
|
|
||||||
|
|
||||||
python manage_collector.py --mode test db migrate -m "Migration comment" \
|
|
||||||
-d collector/api/db/migrations/
|
|
||||||
|
|
||||||
Create DB user with password 'collector':
|
|
||||||
|
|
||||||
sudo -u postgres createuser -DES collector
|
|
||||||
|
|
||||||
or:
|
|
||||||
|
|
||||||
sudo -u postgres psql
|
|
||||||
CREATE ROLE collector WITH NOSUPERUSER NOCREATEDB NOCREATEROLE LOGIN ENCRYPTED PASSWORD 'collector';
|
|
||||||
|
|
||||||
Create DB and grant privileges to it:
|
|
||||||
|
|
||||||
sudo -u postgres psql
|
|
||||||
CREATE DATABASE collector;
|
|
||||||
GRANT ALL ON DATABASE collector TO collector;
|
|
||||||
|
|
||||||
For apply the latest migration:
|
|
||||||
|
|
||||||
python manage_collector.py --mode test db upgrade -d collector/api/db/migrations/
|
|
||||||
|
|
||||||
For revert all migrations:
|
|
||||||
|
|
||||||
python manage_collector.py --mode test db downgrade -d collector/api/db/migrations/
|
|
||||||
|
|
||||||
For starting test server:
|
|
||||||
|
|
||||||
python manage_collector.py --mode test runserver
|
|
||||||
|
|
||||||
Example config for uWSGI is located in collector/uwsgi/collector_test.yaml
|
|
||||||
|
|
||||||
|
|
||||||
Local setup
|
|
||||||
----------
|
|
||||||
|
|
||||||
To run statistics UI locally you need to do the following:
|
|
||||||
|
|
||||||
Install elasticsearch 1.3
|
|
||||||
|
|
||||||
pip install elasticsearch
|
|
||||||
|
|
||||||
Run test_report from NodesDistribution
|
|
||||||
|
|
||||||
prepare virtualenv:
|
|
||||||
|
|
||||||
cd fuel-stats
|
|
||||||
virtualenv .venv
|
|
||||||
source .venv/bin/activate
|
|
||||||
pip install -r collector/test-requirements.txt
|
|
||||||
cd migration
|
|
||||||
|
|
||||||
run tests:
|
|
||||||
|
|
||||||
nosetests migration.test.report.test_reports:Reports.test_libvirt_type_distribution
|
|
||||||
|
|
||||||
or
|
|
||||||
|
|
||||||
nosetests migration.test.report.test_os_distribution:OsDistribution
|
|
||||||
|
|
||||||
|
|
||||||
this will create demo data from elasticsearch
|
|
||||||
|
|
||||||
Install elasticsearch service
|
|
||||||
|
|
||||||
you can use this helpfull gist https://gist.github.com/wingdspur/2026107
|
|
||||||
|
|
||||||
And don't forget to start elasticsearch service
|
|
||||||
|
|
||||||
sudo service elasticsearch start
|
|
||||||
|
|
||||||
Nginx installation
|
|
||||||
|
|
||||||
sudo apt-get install nginx
|
|
||||||
|
|
||||||
fix Nginx config:
|
|
||||||
|
|
||||||
server {
|
|
||||||
listen 8888; // your free port
|
|
||||||
location / {
|
|
||||||
root /home/kpimenova/fuel/fuel-stats/analytics/static; // your path to fuel-stats/analytics/static
|
|
||||||
}
|
|
||||||
location ~ ^(/fuel)?(/[A-Za-z_0-9])?/(_count|_search) {
|
|
||||||
proxy_pass http://127.0.0.1:9200;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
Then restart Nginx:
|
|
||||||
|
|
||||||
service nginx restart
|
|
||||||
|
|
||||||
After this your local server will be available at 0.0.0.0:8888 // or any other port you've set up :)
|
|
||||||
|
|
||||||
|
|
||||||
Also for correct UI work you need to setup a few things
|
|
||||||
|
|
||||||
Install nodejs packages
|
|
||||||
|
|
||||||
cd fuel-stats/analytics/static
|
|
||||||
npm install
|
|
||||||
|
|
||||||
Install bower packages
|
|
||||||
|
|
||||||
cd fuel-stats/analytics/static
|
|
||||||
gulp bower
|
|
||||||
|
|
||||||
That's all.
|
|
||||||
|
|
||||||
You can anytime lint your code by running
|
|
||||||
|
|
||||||
gulp lint
|
|
||||||
|
|
211
README.rst
Normal file
211
README.rst
Normal file
@ -0,0 +1,211 @@
|
|||||||
|
==========
|
||||||
|
Fuel stats
|
||||||
|
==========
|
||||||
|
|
||||||
|
Project purpose
|
||||||
|
---------------
|
||||||
|
|
||||||
|
* collects stats information about OpenStack installations made by Fuel_,
|
||||||
|
* generates stat reports in the CSV format,
|
||||||
|
* provides API for fetching raw data in the JSON format,
|
||||||
|
* provides Web UI for reports generation and basic stats charts/histograms.
|
||||||
|
|
||||||
|
Components
|
||||||
|
----------
|
||||||
|
|
||||||
|
Collector is the service for collecting stats. It has REST API and DB storage.
|
||||||
|
Analytics is the service for generating reports. It has REST API.
|
||||||
|
Migrator is the tool for migrating data from the DB to the Elasticsearch_.
|
||||||
|
|
||||||
|
The collector and analytics services are started by uWSGI_. Migrator is
|
||||||
|
started by cron to migrate the fresh data into Elasticsearch_.
|
||||||
|
|
||||||
|
Collector
|
||||||
|
---------
|
||||||
|
|
||||||
|
Data origin for collector is the Fuel_ master node. Stats collecting daemons
|
||||||
|
collect and send data to the collector if allowed by the cloud operator.
|
||||||
|
|
||||||
|
Stats data is stored to the DB PostgreSQL_.
|
||||||
|
|
||||||
|
Migrator
|
||||||
|
--------
|
||||||
|
|
||||||
|
Migrator periodically migrates data from the fuel-stats DB to the
|
||||||
|
Elasticsearch_ storage. This storage is used to generate basic stats charts
|
||||||
|
and histograms for the Web UI.
|
||||||
|
|
||||||
|
Analytics
|
||||||
|
---------
|
||||||
|
|
||||||
|
There are two sub-components in the analytics:
|
||||||
|
|
||||||
|
* analytics service
|
||||||
|
* Web UI
|
||||||
|
|
||||||
|
The analytics service API provides generation of CSV reports for installation
|
||||||
|
info, plugins, and OpenStack workloads. The analytics API also provides
|
||||||
|
export of data from DB as JSON.
|
||||||
|
|
||||||
|
The analytics Web UI provides basic summary stats charts and histograms with
|
||||||
|
the possibility of filtering data by the Fuel_ release version. Also, in the
|
||||||
|
Web UI we can generate and download stats reports on a selected time period.
|
||||||
|
|
||||||
|
.. _howto configure dev environment:
|
||||||
|
|
||||||
|
How to configure development environment
|
||||||
|
----------------------------------------
|
||||||
|
|
||||||
|
To start fuel-stats on a localhost we need to have:
|
||||||
|
|
||||||
|
* PostgreSQL_ of version 9.3 or greater,
|
||||||
|
* Elasticsearch_ of version 1.3.4 or greater,
|
||||||
|
* Nginx_.
|
||||||
|
|
||||||
|
Install PostgreSQL_ and development libraries: ::
|
||||||
|
|
||||||
|
sudo apt-get install --yes postgresql postgresql-server-dev-all
|
||||||
|
|
||||||
|
Configure Elasticsearch_ repo as described in the `elasticsearch docs`_ and
|
||||||
|
install Elasticsearch_:::
|
||||||
|
|
||||||
|
sudo apt-get install --yes elasticsearch
|
||||||
|
|
||||||
|
Install pip and development tools: ::
|
||||||
|
|
||||||
|
sudo apt-get install --yes python-dev python-pip
|
||||||
|
|
||||||
|
Install virtualenv. This step increases flexibility when dealing with
|
||||||
|
environment settings and package installation: ::
|
||||||
|
|
||||||
|
sudo pip install virtualenv virtualenvwrapper
|
||||||
|
|
||||||
|
You can add '. /usr/local/bin/virtualenvwrapper.sh' to .bashrc or just
|
||||||
|
execute it.::
|
||||||
|
|
||||||
|
. /usr/local/bin/virtualenvwrapper.sh
|
||||||
|
|
||||||
|
Create and activate virtual env for fuel-stats: ::
|
||||||
|
|
||||||
|
mkvirtualenv stats
|
||||||
|
workon stats
|
||||||
|
|
||||||
|
You can use any name for the virtual env instead of 'stats'.
|
||||||
|
|
||||||
|
Install the fuel-stats requirements: ::
|
||||||
|
|
||||||
|
pip install -r test-requirements.txt
|
||||||
|
|
||||||
|
Create a DB user for fuel-stats: ::
|
||||||
|
|
||||||
|
sudo -u postgres psql
|
||||||
|
CREATE ROLE collector WITH NOSUPERUSER NOCREATEDB NOCREATEROLE LOGIN ENCRYPTED PASSWORD 'collector';
|
||||||
|
|
||||||
|
Create a DB and grant privileges to it: ::
|
||||||
|
|
||||||
|
sudo -u postgres psql
|
||||||
|
CREATE DATABASE collector;
|
||||||
|
GRANT ALL ON DATABASE collector TO collector;
|
||||||
|
|
||||||
|
Check that all tests are passed: ::
|
||||||
|
|
||||||
|
cd fuel-stats/collector && tox
|
||||||
|
cd fuel-stats/migration && tox
|
||||||
|
cd fuel-stats/analytics && tox
|
||||||
|
|
||||||
|
**NOTE:** The collector tests must be performed the first.
|
||||||
|
|
||||||
|
Now you are ready to develop fuel-stats.
|
||||||
|
|
||||||
|
How to configure Web UI
|
||||||
|
-----------------------
|
||||||
|
|
||||||
|
We assume that you already have configured virtual env as described in
|
||||||
|
`howto configure dev environment`_.
|
||||||
|
|
||||||
|
Install elsticsearch library and create sample data: ::
|
||||||
|
|
||||||
|
workon stats
|
||||||
|
pip install elasticsearch
|
||||||
|
cd migration
|
||||||
|
nosetests migration.test.report.test_reports:Reports.test_libvirt_type_distribution
|
||||||
|
|
||||||
|
Install nodejs: ::
|
||||||
|
|
||||||
|
sudo apt-get remove nodejs nodejs-legacy npm
|
||||||
|
sudo add-apt-repository -y ppa:chris-lea/node.js
|
||||||
|
sudo apt-get update
|
||||||
|
sudo apt-get install nodejs
|
||||||
|
|
||||||
|
Install nodejs and bower packages: ::
|
||||||
|
|
||||||
|
cd fuel-stats/analytics/static
|
||||||
|
npm install
|
||||||
|
gulp bower
|
||||||
|
|
||||||
|
You can anytime lint your code by running: ::
|
||||||
|
|
||||||
|
gulp lint
|
||||||
|
|
||||||
|
Add site configuration to Nginx_: ::
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 8888;
|
||||||
|
location / {
|
||||||
|
root /your-path-to-fuel-stats/fuel-stats/analytics/static;
|
||||||
|
}
|
||||||
|
location ~ ^(/fuel)?(/[A-Za-z_0-9])?/(_count|_search) {
|
||||||
|
proxy_pass http://127.0.0.1:9200;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Then restart Nginx: ::
|
||||||
|
|
||||||
|
service nginx restart
|
||||||
|
|
||||||
|
After this, your local server will be available at 0.0.0.0:8888
|
||||||
|
or any other port you've set up.
|
||||||
|
|
||||||
|
How to start local collector
|
||||||
|
----------------------------
|
||||||
|
|
||||||
|
You can use uWSGI_ to start the collector. Sample config can be found in
|
||||||
|
collector/uwsgi/collector_example.yaml.
|
||||||
|
|
||||||
|
Or test web service to be used. To start the test web service, run: ::
|
||||||
|
|
||||||
|
python collector/manage_collector.py --mode test runserver
|
||||||
|
|
||||||
|
How to start local analytics
|
||||||
|
----------------------------
|
||||||
|
|
||||||
|
You can use uWSGI_ to start analytics. Sample config can be found in
|
||||||
|
analytics/uwsgi/analytics_example.yaml.
|
||||||
|
|
||||||
|
Or test the web service to be used. To start the test web service, run: ::
|
||||||
|
|
||||||
|
python analytics/manage_analytics.py --mode test runserver
|
||||||
|
|
||||||
|
How to deal with DB migrations
|
||||||
|
------------------------------
|
||||||
|
|
||||||
|
Create new DB migration: ::
|
||||||
|
|
||||||
|
python manage_collector.py --mode test db migrate -m "Migration comment" \
|
||||||
|
-d collector/api/db/migrations/
|
||||||
|
|
||||||
|
Apply all DB migrations: ::
|
||||||
|
|
||||||
|
python manage_collector.py --mode test db upgrade -d collector/api/db/migrations/
|
||||||
|
|
||||||
|
Revert all migrations: ::
|
||||||
|
|
||||||
|
python manage_collector.py --mode test db downgrade -d collector/api/db/migrations/
|
||||||
|
|
||||||
|
|
||||||
|
.. _Fuel: https://docs.mirantis.com/openstack/fuel/
|
||||||
|
.. _Elasticsearch: https://www.elastic.co/
|
||||||
|
.. _uWSGI: https://pypi.python.org/pypi/uWSGI/
|
||||||
|
.. _PostgreSQL: http://www.postgresql.org/
|
||||||
|
.. _Nginx: http://nginx.org/
|
||||||
|
.. _elasticsearch docs: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html
|
Loading…
Reference in New Issue
Block a user