e3721d1ab7
Buck version was upgraded in Ib3f22e70b7. This version contains
behavioral change for genrule()'s rules. Since: [1] the caching
behavior was switched to input-based keys. So that the command in
genrule() is executed only when one of the input keys has changed.
This massive change in behavior broke Gerrit build toolchain in
very subtle way: when gerrit.war was built once, it wasn't rebuilt
any more unless caches were invalidated. To understand why this
is the case, some background understanding needed how gerrit.war
is packaged in our build toolchain. Gathering of gerrit.war is done
by genrule() that executes python script that retrieves the needed
libraries by issuing buck audit classpath. However, since: [1] this
python script is not invoked any more, when some libraries in
transitive dependency chain have changed, because only very few
libraries specified as the input to this python rule and the rest of
the dependencies is retrieved in dynamic way in the script itself.
So when some source code was changed, and corresponding library, say
gerrit-server was changed as well, the first order dependency to the
python script, like gerrit-pgm wasn't changed, and thus its rule key
remained stable, so that pack_war.py wasn't invoked any more, leaving
gerrit.war stale.
The reproducer for this problem can be found here: [2]. The upstream
issue is here: [3].
The fix is straight forward: instead of passing only the first order
dependencies to the war packager script (that cannot reliably be done
after changed caching behavior of genrule() in: [1]) and issue buck
audit classpath in python script, pass the whole classpath from Buck
build file to war packager. For one this will fix the caching issue,
as the rule key now reflects the whole classpath and thus would change
if some deps in transitive dependency chain would change, for another
we wouldn't have to retrieve the classpath twice: in Buck build itself
and in war packager script.
Extend the test plan documentation for Buck upgardes and include
re-creation of gerrit.war and test it for not being stale.
[1] 143646f4cc
[2] https://github.com/davido/buck_genrule_changed_caching_behaviour_143646f
[3] https://github.com/facebook/buck/issues/470
Test plan:
* buck build gerrit
* change some sources in gerrit-server project
* buck build gerrit
* verify that gerrit.war was rebuilt in previous step and reflect the
changes made in gerrit-server project
Reported-By: Doug Kelly <dougk.ff7@gmail.com>
Change-Id: I09d95176db62ea5846a4791b9a21b1c3a577f548
59 lines
1.9 KiB
Python
Executable File
59 lines
1.9 KiB
Python
Executable File
#!/usr/bin/env python
|
|
# Copyright (C) 2013 The Android Open Source Project
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
from __future__ import print_function
|
|
from optparse import OptionParser
|
|
from os import chdir, makedirs, path, symlink
|
|
from subprocess import check_call
|
|
import sys
|
|
|
|
opts = OptionParser()
|
|
opts.add_option('-o', help='path to write WAR to')
|
|
opts.add_option('--lib', action='append', help='target for WEB-INF/lib')
|
|
opts.add_option('--pgmlib', action='append', help='target for WEB-INF/pgm-lib')
|
|
opts.add_option('--tmp', help='temporary directory')
|
|
args, ctx = opts.parse_args()
|
|
|
|
war = args.tmp
|
|
root = war[:war.index('buck-out')]
|
|
jars = set()
|
|
|
|
def prune(l):
|
|
return [j[j.find('buck-out'):] for e in l for j in e.split(':')]
|
|
|
|
def link_jars(libs, directory):
|
|
makedirs(directory)
|
|
while not path.isfile('.buckconfig'):
|
|
chdir('..')
|
|
for j in libs:
|
|
if j not in jars:
|
|
jars.add(j)
|
|
n = path.basename(j)
|
|
if j.startswith('buck-out/gen/gerrit-'):
|
|
n = j.split('/')[2] + '-' + n
|
|
symlink(path.join(root, j), path.join(directory, n))
|
|
|
|
if args.lib:
|
|
link_jars(prune(args.lib), path.join(war, 'WEB-INF', 'lib'))
|
|
if args.pgmlib:
|
|
link_jars(prune(args.pgmlib), path.join(war, 'WEB-INF', 'pgm-lib'))
|
|
try:
|
|
for s in ctx:
|
|
check_call(['unzip', '-q', '-d', war, s])
|
|
check_call(['zip', '-9qr', args.o, '.'], cwd=war)
|
|
except KeyboardInterrupt:
|
|
print('Interrupted by user', file=sys.stderr)
|
|
exit(1)
|