gerrit/Documentation/gen_licenses.py
Dave Borowitz ffe301570c Build bower_components with buck
Add support for downloading npm binaries including dependencies and
running them in buck genrules. In npm land, transitive dependencies
are generally included in the package distribution, and there are a
*lot* of them. Since we aren't redistributing these binaries and
they're only part of the build process, we don't have to worry too
much about licensing, only that they don't have anything totally
crazy.

We assume packages have a certain format and we can detect the binary
to run from the genrule output filename. Actually running the binary
is tricky as well, since we have to extract it first. But it might be
large, so we don't want to extract it on every invocation; and naive
extraction to a common location (in buck-out) is racy. So we need a
custom extractor scheme using atomic rename to make this work.

Download bower as an npm package and use it to download bower
packages. Bower packages can come from a variety of sources, usually
git repositories, so we can't simply use download_file. There is
additional logic in bower to read bower.json and strip out unneeded
files, so I didn't want to get into reimplementing that. The tricky
thing about bower is convincing it to avoid transitive dependencies so
we can let Buck handle parallelism and caching. To do this, we need to
read the package information from the upstream bower repository, and
explicitly ignore all listed dependencies when downloading.

We combine the flattened list of bower packages in a single
bower_components rule. It would be nice to have deps of each
bower_component so we didn't need to flatten these, but Buck genrules
don't have deps so this is a nonstarter. Considering we only expect to
have a single bower_components for the whole project, hopefully this
is not too onerous.

This change just gets us the bower_components directory. We still have
some work to do to use this from Gerrit. Plus even more work to
replace the gulpfile and actually package this stuff together into a
compiled JS app for the war distribution.

Change-Id: Id277d2d812ffcc3bce87ff00b5894bacdffc038e
2015-11-12 19:01:53 -05:00

191 lines
5.7 KiB
Python
Executable File

#!/usr/bin/env python
# Copyright (C) 2013 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# TODO(sop): Be more detailed: version, link to Maven Central
from __future__ import print_function
import argparse
from collections import defaultdict, deque
import json
from os import chdir, path
from shutil import copyfileobj
from subprocess import Popen, PIPE
from sys import stdout, stderr
parser = argparse.ArgumentParser()
parser.add_argument('--asciidoc', action='store_true')
parser.add_argument('--partial', action='store_true')
parser.add_argument('targets', nargs='+')
args = parser.parse_args()
KNOWN_PROVIDED_DEPS = [
'//lib/bouncycastle:bcpg',
'//lib/bouncycastle:bcpkix',
'//lib/bouncycastle:bcprov',
]
for target in args.targets:
if not target.startswith('//'):
print('Target must be absolute: %s' % target, file=stderr)
def parse_graph():
graph = defaultdict(list)
while not path.isfile('.buckconfig'):
chdir('..')
query = ' + '.join('deps(%s)' % t for t in args.targets)
p = Popen([
'buck', 'query', query,
'--output-attributes=buck.direct_dependencies'], stdout=PIPE)
obj = json.load(p.stdout)
for target, attrs in obj.iteritems():
for dep in attrs['buck.direct_dependencies']:
# bower is only used by the build process; skip it in those cases, even if
# --partial is not passed.
if ((target == '//tools/js:download_bower'
or target.startswith('//lib/js:')
and dep == '//lib/js:bower')):
continue
if target in KNOWN_PROVIDED_DEPS:
continue
if args.partial:
if dep == '//lib/codemirror:js_minifier':
if target == '//lib/codemirror:js':
continue
if target.startswith('//lib/codemirror:mode_'):
continue
if (target == '//gerrit-gwtui:ui_module'
and dep == '//gerrit-gwtexpui:CSS'):
continue
graph[target].append(dep)
r = p.wait()
if r != 0:
exit(r)
return graph
graph = parse_graph()
licenses = defaultdict(set)
do_not_distribute = False
queue = deque(args.targets)
while queue:
target = queue.popleft()
for dep in graph[target]:
if not dep.startswith('//lib:LICENSE-'):
continue
if 'DO_NOT_DISTRIBUTE' in dep:
do_not_distribute = True
licenses[dep].add(target)
queue.extend(graph[target])
if do_not_distribute:
print('DO_NOT_DISTRIBUTE license found', file=stderr)
for target in args.targets:
print('...via %s:' % target)
Popen(['buck', 'query',
'allpaths(%s, //lib:LICENSE-DO_NOT_DISTRIBUTE)' % target],
stdout=stderr).communicate()
exit(1)
used = sorted(licenses.keys())
if args.asciidoc:
print("""\
Gerrit Code Review - Licenses
=============================
Gerrit open source software is licensed under the <<Apache2_0,Apache
License 2.0>>. Executable distributions also include other software
components that are provided under additional licenses.
[[cryptography]]
Cryptography Notice
-------------------
This distribution includes cryptographic software. The country
in which you currently reside may have restrictions on the import,
possession, use, and/or re-export to another country, of encryption
software. BEFORE using any encryption software, please check
your country's laws, regulations and policies concerning the
import, possession, or use, and re-export of encryption software,
to see if this is permitted. See the
link:http://www.wassenaar.org/[Wassenaar Arrangement]
for more information.
The U.S. Government Department of Commerce, Bureau of Industry
and Security (BIS), has classified this software as Export
Commodity Control Number (ECCN) 5D002.C.1, which includes
information security software using or performing cryptographic
functions with asymmetric algorithms. The form and manner of
this distribution makes it eligible for export under the License
Exception ENC Technology Software Unrestricted (TSU) exception
(see the BIS Export Administration Regulations, Section 740.13)
for both object code and source code.
Gerrit includes an SSH daemon (Apache SSHD), to support authenticated
uploads of changes directly from `git push` command line clients.
Gerrit includes an SSH client (JSch), to support authenticated
replication of changes to remote systems, such as for automatic
updates of mirror servers, or realtime backups.
For either feature to function, Gerrit requires the
link:http://java.sun.com/javase/technologies/security/[Java Cryptography extensions]
and/or the
link:http://www.bouncycastle.org/java.html[Bouncy Castle Crypto API]
to be installed by the end-user.
Licenses
--------
""")
for n in used:
libs = sorted(licenses[n])
name = n[len('//lib:LICENSE-'):]
if args.asciidoc:
print()
print('[[%s]]' % name.replace('.', '_'))
print(name)
print('~' * len(name))
print()
else:
print()
print(name)
print('--')
for d in libs:
if d.startswith('//lib:') or d.startswith('//lib/'):
p = d[len('//lib:'):]
else:
p = d[d.index(':')+1:].lower()
print('* ' + p)
if args.asciidoc:
print()
print('[[license]]')
print('[verse]')
print('--')
with open(n[2:].replace(':', '/')) as fd:
copyfileobj(fd, stdout)
print('--')
if args.asciidoc:
print("""
GERRIT
------
Part of link:index.html[Gerrit Code Review]
""")