OpenStack CI log processing tool
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
Daniel Pawlik d3506532fd Handle exception: TOO_MANY_REQUESTS 1 week ago
ansible Merge "Add performance.json file" 4 weeks ago
doc Add Opensearch configuration information; change README file; enable md 2 months ago
loggearman Change service directory permission; change python3 to python38 3 months ago
logscraper Handle exception: TOO_MANY_REQUESTS 1 week ago
opensearch-config Add Opensearch configuration information; change README file; enable md 2 months ago
.gitignore Initial project commit 6 months ago
.gitreview Added .gitreview 7 months ago
.stestr.conf Initial project commit 6 months ago
.yamllint Add job for validating logscraper and log gearman services 6 months ago
.zuul.yaml Add Logsender tool 2 months ago
Dockerfile Add Logsender tool 2 months ago
LICENSE Initial project commit 6 months ago
README.rst Add Opensearch configuration information; change README file; enable md 2 months ago
requirements.txt Add standalone get_timestamp implementation 2 months ago
setup.cfg Add Opensearch configuration information; change README file; enable md 2 months ago
setup.py Move log files to check to external config file 2 months ago
test-requirements.txt Add ansible role that is configuring logscraper 6 months ago
tox.ini Add ansible role that is configuring logscraper 6 months ago

README.rst

OpenStack CI log processing

The goal of this repository is to provide and check functionality of new log processing system base on zuul log scraper tool.

Log Scraper

The Log Scraper tool is responsible for periodical check by using Zuul CI API if there are new builds available and if there are some, it would push the informations to the log processing system.

Log Sender

The Zuul Log Sender tool is responsible for periodical check directory, if there are some files that should be send to the Elasticsearch service. NOTE: build directories that does not provide files buildinfo and inventory.yaml file are skipped.

Available workflows

The Openstack CI Log Processing project is providing two configurations for sending logs from Zuul CI to the Opensearch host.

  1. Logscraper, log gearman client, log gearman worker, logstash

With this solution, log workflow looks like:

+------------------+  1. Get last builds info  +----------------+
|                  |-------------------------> |                |
|    Logscraper    |                           |     Zuul API   |
|                  |<------------------------- |                |
+------------------+  2. Fetch data            +----------------+
         |
         |
         +-------------------+
                             |
                             |
             3. Send queue   |
             logs to gearman |
             client          |
                             v
                   +------------------+
                   |                  |
                   |  Log gearman     |
                   |     client       |
                   +------------------+
        +--------------           --------------+
        |                    |                  |
        |  4. Consume queue, |                  |
        | download log files |                  |
        |                    |                  |
        v                    v                  v
+---------------+   +----------------+  +---------------+
|  Log gearman  |   |   Log gearman  |  | Log gearman   |
|  worker       |   |   worker       |  | worker        |
+---------------+   +----------------+  +---------------+
       |                     |                  |
       |   5. Send to        |                  |
       |      Logstash       |                  |
       |                     v                  |
       |            +----------------+          |
       |            |                |          |
       +--------->  |    Logstash    |  <-------+
                    |                |
                    +----------------+
                            |
              6. Send to    |
                 Opensearch |
                            |
                   +--------v--------+
                   |                 |
                   |    Opensearch   |
                   |                 |
                   +-----------------+

On the beginning, this project was designed to use that solution, but it have a few bottlenecks: - log gearman client can use many memory, when log gearman worker is not fast, - one log gearman worker is not enough even on small infrastructure, - logstash service can fail, - logscraper is checking if log files are available, then log gearman is downloading the logs, which can make an issue on log sever, that host does not have free socket.

You can deploy your log workflow by using example Ansible playbook that you can find in ansible/playbooks/check-services.yml in this project.

  1. Logscraper, logsender

This workflow removes bottlenecks by removing: log gearman client, log gearman worker and logstash service. Logs are downloaded when available to the disk, then parsed by logsender and send directly to the Opensearch host.

With this solution, log workflow looks like:

1. Get last
+----------------+     builds info       +-----------------+
|                | --------------------> |                 |
|   Logscraper   |                       |  Zuul API       |
|                | <-------------------- |                 |
+----------------+    2. Fetch data      +-----------------+
|
3. Download logs;       |
include inventory.yaml  |
and build info          v
+----------------+
|                |
|   Logsender    |
|                |
+----------------+
|
4. Parse log files;    |
add required fields;   |
send to Opensearch     v
+-----------------+
|                 |
|   Opensearch    |
|                 |
+-----------------+

You can deploy your log workflow by using example Ansible playbook that you can find in ansible/playbooks/check-services-sender.yml in this project.

Testing

The part of OpenStack CI log processing runs a complete testing and continuous-integration environment, powered by Zuul.

Any changes to logscraper script or tests will trigger jobs to thoroughly test those changes.

To run a single test: tox -epy38 <test_name>

Benchmarking

The large Zuul CI deployments requires many CI log processing resources. In that case, we can do a benchmark with two log processing deployments. All tests will do same:

  • send 100 log builds to Elasticsearch that is running on same host
  • logscraper will be using 4 workers
  • VM will have 8 vcpus, 16 GB of RAM

Testing workflows:

  • loggearman and logstash

This workflow will spawn 3 additional loggearman workers because it this service is a bottleneck in that log ci workflow.

You can do it with command:

for i in {1..3}; do \
  podman run --network host -d --name loggearman-worker-$i \
   --volume /etc/loggearman:/etc/loggearman:z \
   --volume /var/log/loggearman:/var/log/loggearman:z \
   quay.rdoproject.org/software-factory/loggearman:latest \
   log-gearman-worker -c /etc/loggearman/worker.yml --foreground  -d /var/log/loggearman/worker.log

To remove:

for i in {1..3}; do \
  podman stop loggearman-worker-$i ; podman rm loggearman-worker-$i

On the end, basic calucations:

import datetime
start = datetime.datetime.fromisoformat("2022-02-28 16:44:59")
stop = datetime.datetime.fromisoformat("2022-02-28 16:46:01")
print((stop-start).total_seconds())

Time spend to run logscraper and wait for finish all loggearman workers took: 62 seconds and it takes 680MB of RAM.

  • logsender workflow

This workflow will only use logsender tool and it will push the logs directly to the Elasticsearch service. Same as in previous test, it will be executed on 4 processes.

To download logs:

logscraper \
 --zuul-api-url https://zuul.opendev.org/api/tenant/openstack \
 --checkpoint-file /tmp/results-checkpoint \
 --worker 8 \
 --max-skipped 100 \
 --download \
 --directory /tmp/logscraper

This operation took: 30 seconds and it uses 130 MB of RAM.

logsender --username admin --password mypassword --host localhost --port 9200 --insecure --workers 4

Time spend to run logscraper and wait for finish all loggearman workers took: 35 second and it takes 520 MB of RAM.

Conclusion:

The logsender way seems to use less memory (on Opendev deployment, logstash service is on different host, but 4096 MB of RAM is not enough) and it is faster, but the logscraper and logsender process was executed one by one - on the beginning logscraper download logs, then logsender send them to Elasticsearch.

Continuous Deployment

Once changes are reviewed and committed, they will be applied automatically to the production hosts.

Contributing

Contributions are welcome!

Currently only unit tests are available. In the future, functional tests would be added.

Documentation

The latest documentation is available at http://docs.openstack.org/infra/ci-log-processing

That documentation is generated from this repository. You can generate it yourself with tox -e docs.