openstack/ci-log-processing
Code Issues Proposed changes
Files
a6bdcc96b46bf88d48875cee7c76d187e257e82d
ci-log-processing/doc/source/logscraper.rst
Daniel Pawlik a6bdcc96b4 Initial project commit 
This commit provides Zuul logscraper tool responsible for
getting newest logs by using Zuul API and pushing them to
log processing services.

Change-Id: Iff45fbcb6d4813cc60dc705c60e95f3045984b5c
2021-11-16 11:50:55 +01:00

1.7 KiB
Raw Blame History

Logscraper

The logscraper tool can be running as a one-shot log scrape or as periodical check, if some new log jobs are available.

The tool have help function, that is showing available options for it. It is available by typing:

logscraper --help

Basic usage

Base on the use case, we can run logscraper.

Example:

  • periodical check if there are some new logs for openstack tenant:
logscraper --gearman-server somehost --zuul-api-url https://zuul.opendev.org/api/tenant/openstack --checkpoint-file /tmp/results-checkpoint.txt --follow
  • one shot on getting logs from zuul tenant:
logscraper --gearman-server localhost --zuul-api-url https://zuul.opendev.org/api/tenant/zuul --checkpoint-file /tmp/zuul-result-timestamp.txt
  • periodically scrape logs from tenants: openstack, zuul and local
logscraper --gearman-server localhost --zuul-api-url https://zuul.opendev.org/api/tenant/openstack --zuul-api-url https://zuul.opendev.org/api/tenant/zuul --zuul-api-url https://zuul.opendev.org/api/tenant/local --checkpoint-file /tmp/someresults.txt --follow

Containerize tool

Instead of using pip tool, you can build your own container image that contains logscraper tool, for example:

docker build -t logscraper -f Dockerfile

Then you can execute commands that are described above.

NOTE: if you want to use parameter --checkpoint-file, you need to mount a volume to the container, for example:

docker run -v $(pwd):/checkpoint-dir:z -d logscraper logscraper --gearman-server somehost --zuul-api-url https://zuul.opendev.org/api/tenant/openstack --checkpoint-file /checkpoint-dir/checkpoint.txt --follow