Allow superprojects to subscribe to submodules updates
Before discussing about the feature introduced in this commit, let's discuss a little about the Submodule feature. It is a git feature that allows an external repository to be attached inside one repository at a specific path. Imagine a repository called 'super' and another one called 'a'. Also consider 'a' available in a running gerrit instance on "server". With this feature, one could attach 'a' inside of 'super' repository in path 'a' by executing following command when inside 'super': "git submodule add ssh://server/a a". Still considering above example, after its execution notice inside 'super' local repository the 'a' folder is considered a gitlink to external repository 'a'. Also notice a file called .gitmodules is created (it is a config file containing the subscription of 'a'). The "git submodule status" command provides the sha-1 each gitlink points to in the external repository. In the example provided, if 'a' is updated and 'super' is supposed to see latest sha-1 (considering here 'a' has only the master branch), one should then commit 'a' modified gitlink in 'super' project. Actually it would not need to be even an external update, one could move to 'a' folder (insider 'super'), modify its content, commit, then move back to 'super' and commit 'a' gitlink. The feature introduced in this commit allows superprojects to subscribe to submodules updates. When a commit is merged to a project, the commit content is scanned to identify if it registers submodules (if the commit contains new gitlinks and .gitmodules file with required info) and if so, a new submodule subscription is registered. When a new commit of a registered submodule is merged, gerrit automatically updates the subscribers to the submodule with new commit having the updated gitlinks. The most notable benefit of the feature is to not require to push/merge commits of super projects (subscribers) with gitlinks whenever a project being a submodule is updated. It is only required to push commits with gitlinks when they are created (and in this case it is also required to push .gitmodules file). Submodule subscription is actually the subscription of a submodule project and one of its branches for a branch of a super project. Whenever a submodule subscription is created, the user should certify that the branch field is filled in the .gitmodules configuration file, otherwise the subscriptions are not registered in gerrit. It is not automatically filled by git submodule command. Its value should indicate the branch of submodule project that when updated will trigger automatic update of its registered gitlink. The branch value could be '.' if the submodule project branch has the same name that the destination branch of commit having gitlinks/.gitmodules file. Since it manages subscriptions in the branch scope, we could have a scenario having a project called 'super' having a branch 'integration' subscribed to a project called 'a' in branch 'integration', and also having same 'super' project but in branch 'dev' subscribed to the 'a' project in a branch called 'local-dev'. If one does not want to have use of this feature, (s)he should not add the branch field in an added submodule section of .gitmodules file. If one have added a submodule subscription and want to drop it, it is required to merge a commit updating subscribed super project/branch removing the gitlink and the submodule section of .gitmodules file. The branch field of a submodule section is a custom git submodule feature introduced by this one and for gerrit use. One should always certify to fill it by editing .gitmodules file after adding submodules to a super project, if it is the intention to make use of the gerrit feature introduced here. The feature also requires the canonical web url provided in the gerrit configuration file. It will automatically updates only the subscribers of project urls of the running server (the one described in canonical web url value). Considering the functionalities introduced in this feature, we have actually an option to the current way Android platform code is handled (using manifest.xml files). And it offers the benefit of not requiring users to introduce commits in the super project (platform repository considering Android source code example) and not using the repo tool. Also notice that the feature makes possible to have recursive update: a 'super' project could be the subscriber of an 'a' project, and the 'a' project could be the subscriber of a 'a-subproject' project. Change-Id: Id067e85d914e13324f1f28418233cc96a60e3ab2
This commit is contained in:
Goran Lungberg
committed by
Gustaf Lundh
Gustaf Lundh
parent
a3f73aab40
commit
d15704079c
@@ -0,0 +1,379 @@
|
||||
// Copyright (C) 2011 The Android Open Source Project
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package com.google.gerrit.server.git;
|
||||
|
||||
import com.google.gerrit.reviewdb.Branch;
|
||||
import com.google.gerrit.reviewdb.Change;
|
||||
import com.google.gerrit.reviewdb.Project;
|
||||
import com.google.gerrit.reviewdb.ReviewDb;
|
||||
import com.google.gerrit.reviewdb.SubmoduleSubscription;
|
||||
import com.google.gerrit.server.GerritPersonIdent;
|
||||
import com.google.gerrit.server.config.CanonicalWebUrl;
|
||||
import com.google.gerrit.server.util.SubmoduleSectionParser;
|
||||
import com.google.gwtorm.client.OrmException;
|
||||
import com.google.gwtorm.client.SchemaFactory;
|
||||
import com.google.inject.Inject;
|
||||
import com.google.inject.Provider;
|
||||
import com.google.inject.assistedinject.Assisted;
|
||||
|
||||
import org.eclipse.jgit.dircache.DirCache;
|
||||
import org.eclipse.jgit.dircache.DirCacheBuilder;
|
||||
import org.eclipse.jgit.dircache.DirCacheEditor;
|
||||
import org.eclipse.jgit.dircache.DirCacheEntry;
|
||||
import org.eclipse.jgit.dircache.DirCacheEditor.PathEdit;
|
||||
import org.eclipse.jgit.errors.ConfigInvalidException;
|
||||
import org.eclipse.jgit.errors.IncorrectObjectTypeException;
|
||||
import org.eclipse.jgit.errors.MissingObjectException;
|
||||
import org.eclipse.jgit.lib.BlobBasedConfig;
|
||||
import org.eclipse.jgit.lib.CommitBuilder;
|
||||
import org.eclipse.jgit.lib.Constants;
|
||||
import org.eclipse.jgit.lib.FileMode;
|
||||
import org.eclipse.jgit.lib.ObjectId;
|
||||
import org.eclipse.jgit.lib.ObjectInserter;
|
||||
import org.eclipse.jgit.lib.PersonIdent;
|
||||
import org.eclipse.jgit.lib.Ref;
|
||||
import org.eclipse.jgit.lib.RefUpdate;
|
||||
import org.eclipse.jgit.lib.Repository;
|
||||
import org.eclipse.jgit.revwalk.RevCommit;
|
||||
import org.eclipse.jgit.revwalk.RevWalk;
|
||||
import org.eclipse.jgit.treewalk.TreeWalk;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.net.URI;
|
||||
import java.net.URISyntaxException;
|
||||
import java.util.HashMap;
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
|
||||
import javax.annotation.Nullable;
|
||||
|
||||
public class SubmoduleOp {
|
||||
public interface Factory {
|
||||
SubmoduleOp create(Branch.NameKey destBranch, RevCommit mergeTip,
|
||||
RevWalk rw, Repository db, Project destProject, List<Change> submitted,
|
||||
Map<Change.Id, CodeReviewCommit> commits);
|
||||
}
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(SubmoduleOp.class);
|
||||
private static final String GIT_MODULES = ".gitmodules";
|
||||
|
||||
private final Branch.NameKey destBranch;
|
||||
private RevCommit mergeTip;
|
||||
private RevWalk rw;
|
||||
private final Provider<String> urlProvider;
|
||||
private ReviewDb schema;
|
||||
private Repository db;
|
||||
private Project destProject;
|
||||
private List<Change> submitted;
|
||||
private final Map<Change.Id, CodeReviewCommit> commits;
|
||||
private final PersonIdent myIdent;
|
||||
private final GitRepositoryManager repoManager;
|
||||
private final ReplicationQueue replication;
|
||||
private final SchemaFactory<ReviewDb> schemaFactory;
|
||||
private final Set<Branch.NameKey> updatedSubscribers;
|
||||
|
||||
@Inject
|
||||
public SubmoduleOp(@Assisted final Branch.NameKey destBranch,
|
||||
@Assisted RevCommit mergeTip, @Assisted RevWalk rw,
|
||||
@CanonicalWebUrl @Nullable final Provider<String> urlProvider,
|
||||
final SchemaFactory<ReviewDb> sf, @Assisted Repository db,
|
||||
@Assisted Project destProject, @Assisted List<Change> submitted,
|
||||
@Assisted final Map<Change.Id, CodeReviewCommit> commits,
|
||||
@GerritPersonIdent final PersonIdent myIdent,
|
||||
GitRepositoryManager repoManager, ReplicationQueue replication) {
|
||||
this.destBranch = destBranch;
|
||||
this.mergeTip = mergeTip;
|
||||
this.rw = rw;
|
||||
this.urlProvider = urlProvider;
|
||||
this.schemaFactory = sf;
|
||||
this.db = db;
|
||||
this.destProject = destProject;
|
||||
this.submitted = submitted;
|
||||
this.commits = commits;
|
||||
this.myIdent = myIdent;
|
||||
this.repoManager = repoManager;
|
||||
this.replication = replication;
|
||||
|
||||
updatedSubscribers = new HashSet<Branch.NameKey>();
|
||||
}
|
||||
|
||||
public void update() throws SubmoduleException {
|
||||
try {
|
||||
schema = schemaFactory.open();
|
||||
|
||||
updateSubmoduleSubscriptions();
|
||||
updateSuperProjects(destBranch, mergeTip.getId().toObjectId(), null);
|
||||
} catch (OrmException e) {
|
||||
throw new SubmoduleException("Cannot open database", e);
|
||||
} finally {
|
||||
if (schema != null) {
|
||||
schema.close();
|
||||
schema = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private void updateSubmoduleSubscriptions() throws SubmoduleException {
|
||||
if (urlProvider.get() == null) {
|
||||
logAndThrowSubmoduleException("Cannot establish canonical web url used to access gerrit."
|
||||
+ " It should be provided in gerrit.config file.");
|
||||
}
|
||||
|
||||
try {
|
||||
final TreeWalk tw = TreeWalk.forPath(db, GIT_MODULES, mergeTip.getTree());
|
||||
if (tw != null
|
||||
&& (FileMode.REGULAR_FILE.equals(tw.getRawMode(0)) || FileMode.EXECUTABLE_FILE
|
||||
.equals(tw.getRawMode(0)))) {
|
||||
|
||||
BlobBasedConfig bbc =
|
||||
new BlobBasedConfig(null, db, mergeTip, GIT_MODULES);
|
||||
|
||||
final String thisServer = new URI(urlProvider.get()).getHost();
|
||||
|
||||
final Branch.NameKey target =
|
||||
new Branch.NameKey(new Project.NameKey(destProject.getName()),
|
||||
destBranch.get());
|
||||
|
||||
final Set<SubmoduleSubscription> oldSubscriptions =
|
||||
new HashSet<SubmoduleSubscription>(schema.submoduleSubscriptions()
|
||||
.bySuperProject(destBranch).toList());
|
||||
final List<SubmoduleSubscription> newSubscriptions =
|
||||
new SubmoduleSectionParser(bbc, thisServer, target, repoManager)
|
||||
.parseAllSections();
|
||||
|
||||
final Set<SubmoduleSubscription> alreadySubscribeds =
|
||||
new HashSet<SubmoduleSubscription>();
|
||||
for (SubmoduleSubscription s : newSubscriptions) {
|
||||
if (oldSubscriptions.contains(s)) {
|
||||
alreadySubscribeds.add(s);
|
||||
}
|
||||
}
|
||||
|
||||
oldSubscriptions.removeAll(newSubscriptions);
|
||||
newSubscriptions.removeAll(alreadySubscribeds);
|
||||
|
||||
if (!oldSubscriptions.isEmpty()) {
|
||||
schema.submoduleSubscriptions().delete(oldSubscriptions);
|
||||
}
|
||||
schema.submoduleSubscriptions().insert(newSubscriptions);
|
||||
}
|
||||
} catch (OrmException e) {
|
||||
logAndThrowSubmoduleException(
|
||||
"Database problem at update of subscriptions table from "
|
||||
+ GIT_MODULES + " file.", e);
|
||||
} catch (ConfigInvalidException e) {
|
||||
logAndThrowSubmoduleException(
|
||||
"Problem at update of subscriptions table: " + GIT_MODULES
|
||||
+ " config file is invalid.", e);
|
||||
} catch (IOException e) {
|
||||
logAndThrowSubmoduleException(
|
||||
"Problem at update of subscriptions table from " + GIT_MODULES + ".",
|
||||
e);
|
||||
} catch (URISyntaxException e) {
|
||||
logAndThrowSubmoduleException(
|
||||
"Incorrect gerrit canonical web url provided in gerrit.config file.",
|
||||
e);
|
||||
}
|
||||
}
|
||||
|
||||
private void updateSuperProjects(final Branch.NameKey updatedBranch,
|
||||
final ObjectId mergedCommit, final String msg) throws SubmoduleException {
|
||||
try {
|
||||
final List<SubmoduleSubscription> subscribers =
|
||||
schema.submoduleSubscriptions().bySubmodule(updatedBranch).toList();
|
||||
|
||||
if (!subscribers.isEmpty()) {
|
||||
String msgbuf = msg;
|
||||
if (msgbuf == null) {
|
||||
// The first updatedBranch on a cascade event of automatic
|
||||
// updates of repos is added to updatedSubscribers set so
|
||||
// if we face a situation having
|
||||
// submodule-a(master)-->super(master)-->submodule-a(master),
|
||||
// it will be detected we have a circular subscription
|
||||
// when updateSuperProjects is called having as updatedBranch
|
||||
// the super(master) value.
|
||||
updatedSubscribers.add(updatedBranch);
|
||||
|
||||
for (final Change chg : submitted) {
|
||||
final CodeReviewCommit c = commits.get(chg.getId());
|
||||
if (c != null
|
||||
&& (c.statusCode == CommitMergeStatus.CLEAN_MERGE || c.statusCode == CommitMergeStatus.CLEAN_PICK)) {
|
||||
msgbuf += "\n";
|
||||
msgbuf += c.getFullMessage();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// update subscribers of this module
|
||||
for (final SubmoduleSubscription s : subscribers) {
|
||||
if (!updatedSubscribers.add(s.getSuperProject())) {
|
||||
log.error("Possible circular subscription involving "
|
||||
+ s.toString());
|
||||
} else {
|
||||
|
||||
Map<Branch.NameKey, ObjectId> modules =
|
||||
new HashMap<Branch.NameKey, ObjectId>(1);
|
||||
modules.put(updatedBranch, mergedCommit);
|
||||
|
||||
Map<Branch.NameKey, String> paths =
|
||||
new HashMap<Branch.NameKey, String>(1);
|
||||
paths.put(updatedBranch, s.getPath());
|
||||
|
||||
try {
|
||||
updateGitlinks(s.getSuperProject(), modules, paths, msgbuf);
|
||||
} catch (SubmoduleException e) {
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (OrmException e) {
|
||||
logAndThrowSubmoduleException("Cannot read subscription records", e);
|
||||
}
|
||||
}
|
||||
|
||||
private void updateGitlinks(final Branch.NameKey subscriber,
|
||||
final Map<Branch.NameKey, ObjectId> modules,
|
||||
final Map<Branch.NameKey, String> paths, final String msg)
|
||||
throws SubmoduleException {
|
||||
PersonIdent author = null;
|
||||
|
||||
final StringBuilder msgbuf = new StringBuilder();
|
||||
msgbuf.append("Updated " + subscriber.getParentKey().get());
|
||||
Repository pdb = null;
|
||||
|
||||
try {
|
||||
boolean sameAuthorForAll = true;
|
||||
|
||||
for (final Map.Entry<Branch.NameKey, ObjectId> me : modules.entrySet()) {
|
||||
RevCommit c = rw.parseCommit(me.getValue());
|
||||
|
||||
msgbuf.append("\nProject: ");
|
||||
msgbuf.append(me.getKey().getParentKey().get());
|
||||
msgbuf.append(" " + me.getValue().getName());
|
||||
msgbuf.append("\n");
|
||||
if (modules.size() == 1 && msg != null) {
|
||||
msgbuf.append(msg);
|
||||
} else {
|
||||
msgbuf.append(c.getShortMessage());
|
||||
}
|
||||
msgbuf.append("\n");
|
||||
|
||||
if (author == null) {
|
||||
author = c.getAuthorIdent();
|
||||
} else if (!author.equals(c.getAuthorIdent())) {
|
||||
sameAuthorForAll = false;
|
||||
}
|
||||
}
|
||||
|
||||
if (!sameAuthorForAll || author == null) {
|
||||
author = myIdent;
|
||||
}
|
||||
|
||||
pdb = repoManager.openRepository(subscriber.getParentKey());
|
||||
if (pdb.getRef(subscriber.get()) == null) {
|
||||
throw new SubmoduleException(
|
||||
"The branch was probably deleted from the subscriber repository");
|
||||
}
|
||||
|
||||
final ObjectId currentCommitId =
|
||||
pdb.getRef(subscriber.get()).getObjectId();
|
||||
|
||||
DirCache dc = readTree(pdb, pdb.getRef(subscriber.get()));
|
||||
DirCacheEditor ed = dc.editor();
|
||||
for (final Map.Entry<Branch.NameKey, ObjectId> me : modules.entrySet()) {
|
||||
ed.add(new PathEdit(paths.get(me.getKey())) {
|
||||
public void apply(DirCacheEntry ent) {
|
||||
ent.setFileMode(FileMode.GITLINK);
|
||||
ent.setObjectId(me.getValue().copy());
|
||||
}
|
||||
});
|
||||
}
|
||||
ed.finish();
|
||||
|
||||
ObjectInserter oi = pdb.newObjectInserter();
|
||||
ObjectId tree = dc.writeTree(oi);
|
||||
|
||||
final CommitBuilder commit = new CommitBuilder();
|
||||
commit.setTreeId(tree);
|
||||
commit.setParentIds(new ObjectId[] {currentCommitId});
|
||||
commit.setAuthor(author);
|
||||
commit.setCommitter(myIdent);
|
||||
commit.setMessage(msgbuf.toString());
|
||||
oi.insert(commit);
|
||||
|
||||
ObjectId commitId = oi.idFor(Constants.OBJ_COMMIT, commit.build());
|
||||
|
||||
final RefUpdate rfu = pdb.updateRef(subscriber.get());
|
||||
rfu.setForceUpdate(false);
|
||||
rfu.setNewObjectId(commitId);
|
||||
rfu.setExpectedOldObjectId(currentCommitId);
|
||||
rfu
|
||||
.setRefLogMessage("Submit to " + subscriber.getParentKey().get(),
|
||||
true);
|
||||
|
||||
switch (rfu.update()) {
|
||||
case NEW:
|
||||
case FAST_FORWARD:
|
||||
replication.scheduleUpdate(subscriber.getParentKey(), rfu.getName());
|
||||
// TODO since this is performed "in the background" no mail will be
|
||||
// sent to inform users about the updated branch
|
||||
break;
|
||||
|
||||
default:
|
||||
throw new IOException(rfu.getResult().name());
|
||||
}
|
||||
|
||||
// Recursive call: update subscribers of the subscriber
|
||||
updateSuperProjects(subscriber, commitId, msgbuf.toString());
|
||||
} catch (IOException e) {
|
||||
logAndThrowSubmoduleException("Cannot update gitlinks for "
|
||||
+ subscriber.get(), e);
|
||||
} finally {
|
||||
if (pdb != null) {
|
||||
pdb.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static DirCache readTree(final Repository pdb, final Ref branch)
|
||||
throws MissingObjectException, IncorrectObjectTypeException, IOException {
|
||||
final RevWalk rw = new RevWalk(pdb);
|
||||
|
||||
final DirCache dc = DirCache.newInCore();
|
||||
final DirCacheBuilder b = dc.builder();
|
||||
b.addTree(new byte[0], // no prefix path
|
||||
DirCacheEntry.STAGE_0, // standard stage
|
||||
pdb.newObjectReader(), rw.parseTree(branch.getObjectId()));
|
||||
b.finish();
|
||||
return dc;
|
||||
}
|
||||
|
||||
private static void logAndThrowSubmoduleException(final String errorMsg,
|
||||
final Exception e) throws SubmoduleException {
|
||||
log.error(errorMsg, e);
|
||||
throw new SubmoduleException(errorMsg, e);
|
||||
}
|
||||
|
||||
private static void logAndThrowSubmoduleException(final String errorMsg)
|
||||
throws SubmoduleException {
|
||||
log.error(errorMsg);
|
||||
throw new SubmoduleException(errorMsg);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user