10 Commits

Author SHA1 Message Date
Jeremy Stanley
714fa40191 Fix empty directory removal for log expiration
* modules/openstack_project/files/log_archive_maintenance.sh: The
refactoring in I5ff6defb4d403f3017ab098c2ac1df3777662933 orphaned a
stray -not in the directory removal expression, inverting the
meaning of -empty as a result. Luckily non-empty directories can't
be removed without additional effort, so no data was lost (just some
very large cronspam sent instead). This removes the remaining cruft.

Change-Id: Ie6ce39ed46eedfaaf0771ab1a91f5f50c90c1b62
2014-06-02 18:15:37 +00:00
Jeremy Stanley
7e480a1ed1 Remove old non-gz files in log archive maintenance
* modules/openstack_project/files/log_archive_maintenance.sh: Start
matching for removal on any files in the logs tree besides those we
explicitly don't compress. Also do it via a refactor to allow for
improved optimization and avoid unnecessary redundancy.

Change-Id: I5ff6defb4d403f3017ab098c2ac1df3777662933
2014-05-20 20:56:32 +00:00
James E. Blair
86830dc5ad Reduce logs and docs-draft retention
Logs are now 4 months instead of 6, docs-draft 3 weeks instead of 4.

Both have unsustainable upward trajectories.

Change-Id: If8ae334c9b48bdacc79b71015fbf386c0f3613f6
2014-04-16 07:10:18 -07:00
Jeremy Stanley
7eb7e1157f Reduce log compression frequency and optimize
The I/O bandwidth starvation inflicted on static.openstack.org
during log compression and deletion runs is overwhelming the server
and preventing other activities from completing in a timely fashion.
Use optimization level 3 to hopefully improve performance of each
run, and schedule them to only run weekly around or shortly after
midnight Saturday mornings American time instead of every six hours.
Filesystems on static.openstack.org will also be increased in size
accordingly to provide more cushion for data to build up between
runs.

Change-Id: I0d9f0c4c1a254bbba2d1cbfca4603ceb34274f9d
2013-08-23 16:09:33 +00:00
Jeremy Stanley
968e77129b Don't compress very, very recent log files
* modules/openstack_project/files/log_archive_maintenance.sh: On
occasion the compressor will be engaged on a file which is still
being written to the log archive, resulting in a compressed but
truncated copy along with the original it lacked the write lock to
unlink. Then on a subsequent pass, this file will be skipped because
the destination filename already exists. Instead, don't compress a
file unless its last modified time is more than 10 minutes in the
past.

Change-Id: Ife874784babd41aa45b4183f4f0a5a5730cf5a23
2013-08-12 21:23:40 +00:00
Jeremy Stanley
b1b421fa9d Don't remove lost+found from static.o.o.
Change-Id: I566a38da017447d13014739ba37ba42bca21d09c
2013-07-23 16:02:47 +00:00
Jeremy Stanley
d241c0575c Don't try to compress patterns ignored by gzip.
* modules/openstack_project/files/log_archive_maintenance.sh:
According to the gzip manpage, there are a variety of filename
endings it will consider to indicate a file is already compressed
and will then refuse to compress. Some tempfiles get
randomly-generated names which match these endings, causing the
compression cron job to emit errors. Instead we make the script skip
over them.

Change-Id: Ib78a19362eaa54c43f44a9e6bd0c265dddee92be
Reviewed-on: https://review.openstack.org/34419
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-07-02 16:52:15 +00:00
Jeremy Stanley
fa1930aaf9 Don't compress log help files.
* modules/openstack_project/files/log_archive_maintenance.sh: The
log compression script no longer attempts to compress files stored
in the help subdirectory of the site document root.

Change-Id: I1db1dead0608c10b71dd3d91c1fedc2d7cc06926
Reviewed-on: https://review.openstack.org/34263
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
2013-06-24 22:31:12 +00:00
James E. Blair
64ca895bd0 Delete docs-draft files after 30 days.
They can be recreated if needed so let's just keep them around long
enough to be useful in code reviews.

Change-Id: I2cac9a75190b1db23df67568b1973b25c4179590
Reviewed-on: https://review.openstack.org/32039
Reviewed-by: Anne Gentle <anne@openstack.org>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-06-07 16:39:42 +00:00
Elizabeth Krumbach
5128ee3457 Add script to static to gzip logs & rm >183d old
Add script and cron job to consolidate find through
/srv/static/logs/ to gzip files and to search for .gz files
over 183 days old and empty directories over a day old and
remove them.

Change-Id: I78f08852c8ee45fe8234d6b8ea0e25e199216cfd
Reviewed-on: https://review.openstack.org/26049
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-04-08 17:03:58 +00:00