* modules/openstack_project/files/log_archive_maintenance.sh: The
refactoring in I5ff6defb4d403f3017ab098c2ac1df3777662933 orphaned a
stray -not in the directory removal expression, inverting the
meaning of -empty as a result. Luckily non-empty directories can't
be removed without additional effort, so no data was lost (just some
very large cronspam sent instead). This removes the remaining cruft.
Change-Id: Ie6ce39ed46eedfaaf0771ab1a91f5f50c90c1b62
* modules/openstack_project/files/log_archive_maintenance.sh: Start
matching for removal on any files in the logs tree besides those we
explicitly don't compress. Also do it via a refactor to allow for
improved optimization and avoid unnecessary redundancy.
Change-Id: I5ff6defb4d403f3017ab098c2ac1df3777662933
Logs are now 4 months instead of 6, docs-draft 3 weeks instead of 4.
Both have unsustainable upward trajectories.
Change-Id: If8ae334c9b48bdacc79b71015fbf386c0f3613f6
The I/O bandwidth starvation inflicted on static.openstack.org
during log compression and deletion runs is overwhelming the server
and preventing other activities from completing in a timely fashion.
Use optimization level 3 to hopefully improve performance of each
run, and schedule them to only run weekly around or shortly after
midnight Saturday mornings American time instead of every six hours.
Filesystems on static.openstack.org will also be increased in size
accordingly to provide more cushion for data to build up between
runs.
Change-Id: I0d9f0c4c1a254bbba2d1cbfca4603ceb34274f9d
* modules/openstack_project/files/log_archive_maintenance.sh: On
occasion the compressor will be engaged on a file which is still
being written to the log archive, resulting in a compressed but
truncated copy along with the original it lacked the write lock to
unlink. Then on a subsequent pass, this file will be skipped because
the destination filename already exists. Instead, don't compress a
file unless its last modified time is more than 10 minutes in the
past.
Change-Id: Ife874784babd41aa45b4183f4f0a5a5730cf5a23
* modules/openstack_project/files/log_archive_maintenance.sh:
According to the gzip manpage, there are a variety of filename
endings it will consider to indicate a file is already compressed
and will then refuse to compress. Some tempfiles get
randomly-generated names which match these endings, causing the
compression cron job to emit errors. Instead we make the script skip
over them.
Change-Id: Ib78a19362eaa54c43f44a9e6bd0c265dddee92be
Reviewed-on: https://review.openstack.org/34419
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
* modules/openstack_project/files/log_archive_maintenance.sh: The
log compression script no longer attempts to compress files stored
in the help subdirectory of the site document root.
Change-Id: I1db1dead0608c10b71dd3d91c1fedc2d7cc06926
Reviewed-on: https://review.openstack.org/34263
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
They can be recreated if needed so let's just keep them around long
enough to be useful in code reviews.
Change-Id: I2cac9a75190b1db23df67568b1973b25c4179590
Reviewed-on: https://review.openstack.org/32039
Reviewed-by: Anne Gentle <anne@openstack.org>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Add script and cron job to consolidate find through
/srv/static/logs/ to gzip files and to search for .gz files
over 183 days old and empty directories over a day old and
remove them.
Change-Id: I78f08852c8ee45fe8234d6b8ea0e25e199216cfd
Reviewed-on: https://review.openstack.org/26049
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins