3 Commits

Author SHA1 Message Date
Clark Boylan
a6f2ca827e Stop compressing files during intermediate steps part 2
Stop compressing files that would otherwise be handled uncompressed in
intermediate log handling. The reason for this is final log/artifact
upload should decide if a file need to be compressed or not as some file
hosting services (like swift) may not return the expected data if we get
this wrong.

This was noticed when we used swift to store intermediate tar.gz
tarballs before final upload to permanent storage. When retrieving the
tar.gz files from swift we got them back as uncompressed .tar files
(because swift was being helpful) and then later when we try to
uncompress these files we break.

Instead we'll let the upload to swift (or other storage) decide if files
should be compressed further than their existing state.

This is the second change in the stack and update the stage-output role.

Change-Id: I2fa6f123d23568ba1537a7c26d2672f04110c35b
2020-01-08 09:24:45 -08:00
Andrea Frittoli (andreaf)
2ea8661801 Add compress capabilities to stage artifacts
Compress files and files in folders. Use gz by default.

Change-Id: I72796e64a65fe88d85168734ac881ee3ec2ce100
2017-11-16 12:02:00 +00:00
Andrea Frittoli (andreaf)
786c48f051 Add a generic stage-output role
This role does roughly what save_dir and save_file do in d-g
functions, only it can be used for a list of files and/or
folders. Jobs can use this role to rename artifacts and place
them in a staging folder which is then used for mass sync to
the executor.

This is typically used in post - the zuul user won't necessarily
have write access to the folders with the artifacts - so it's
easier to stage everything to a folder owned by the zuul user.

Change-Id: Id45a50821539985710da7172a9185fd91e803603
2017-11-02 07:22:26 +00:00