6

I've found several simple examples using both filter-branch and subtree, but they always are just moving 1 directory around. I'd like to take the following repo:

/ Project1.sln Project2.sln Source/ CommonLib.Data/ CommonLib.Web/ Project1.Data/ Project1.Web/ Project1.Other/ Project2.Data/ Project2.Web/

And move things out to their own repos, with the following structure:

# CommonRepo / CommonLib.Data/ CommonLib.Web/ # Project1Repo / Project1.sln Project1.Data/ Project1.Web/ Project1.Other/ # Project2Repo / Project2.sln Project2.Data/ Project2.Web/

While maintaining the entire history. To complicate things, there are 1 or more branches of the original repo that correspond to each project, and thus the version of CommonLib the other projects referred to may be slightly different.

I'd like to use git subtree add to add a reference back to the CommonLib in each of the new repos at the correct tag/revision, but first I need a way to split several directories at once off into their own location.

git subtree split -P seems to only want 1 directory, and I haven't been able to get filter-branch to grab the multiples, either. I'm on a windows box so don't have all the scripting niceties set up to make this easier.

Any advice?

Improve this question

3 Answers 3

Reset to default
5

In the end, I recommend you keep the common lib included in your projects especially due to the divergence you spoke about, so your ideal structure should be:

# CommonRepo / CommonLib.Data/ CommonLib.Web/ # Project1Repo / Project1.sln Project1.Data/ Project1.Web/ Project1.Other/ CommonLib/ # I recommend that you do whatever restructuring needed to support this in a sub-directory CommonLib.Data/ CommonLib.Web/ # Project2Repo / Project2.sln Project2.Data/ Project2.Web/ CommonLib/ # I recommend that you do whatever restructuring needed to support this in a sub-directory CommonLib.Data/ CommonLib.Web/

Now to handle the splitting:

when you split, as long as you don't use different annotations or something the commit ids will be compatible and should play nicely with merge. So you can start by extracting the CommonLib by itself.

  1. I recommend you clone your whole depo before starting just to be sure you don't lose anything.

    git clone <big-repo> <big-repo-clone>

  2. Prepare the old repo

    pushd <big-repo-clone> # split for the common lib git checkout master # assuming you want your common lib at master git subtree split --prefix=Source --branch=temp-commonLib # split the projects from their respective branches git checkout <branch-for-project1> git subtree split --prefix=Source --branch=temp-project1 # split the projects from their respective branches git checkout <branch-for-project2> git subtree split --prefix=Source --branch=temp-project2

  3. Now we need to clean out the parts of those projects that we don't want there. Since they're mixed in you can't really use sub-tree but you can filter-branch to rewrite the history without the other parts.

    # strip unrelated parts from the CommonLib git checkout temp-commonLib git filter-branch --tag-name-filter cat --prune-empty --index-filter 'git rm -rf --cached --ignore-unmatch Project1* Project2*' HEAD # strip unrelated parts from the Project1 git checkout temp-project1 git filter-branch --tag-name-filter cat --prune-empty --index-filter 'git rm -rf --cached --ignore-unmatch CommonLib* Project2*' HEAD # strip unrelated parts from the Project2 git checkout temp-project2 git filter-branch --tag-name-filter cat --prune-empty --index-filter 'git rm -rf --cached --ignore-unmatch CommonLib* Project1*' HEAD

    The prune empty will strip the commits that become empty because they only contained changes that were in the folders you removed.

    Note: All of these changes are at the /source level so that it can be the new root for each project. You can later add your solution back in. Or you can use this prune technique with clones instead of subtrees, and when you're all done you can just move all the contents from '/Source' to '/'

    Now your is going to have extra branches and backups in refs/original/refs/heads/<branch-name>. If during the process you get a fatal error with filter-branch, you can re-create the branch and start again, or if you're confident it didn't do anything yet you can delete this backup with: git update-ref -d refs/original/refs/heads/<branch-name>.

  4. Now just create new repos to store the projects created from those branches

    popd # to get out of <big-repo-clone> mkdir <new-repo> pushd <new-repo> git init git pull <big-repo-clone> <name-of-branch> # like temp-project1 popd # to get out of the <new-repo>

  5. One last thing, lets pull the CommonRepo into the projects.

    pushd <new-project-repo> git subtree add --prefix=CommonLib <new-commonlib-repo>

You then just need to bring in the .sln files (I'll leave this last step up to you).

Improve this answer
Sign up to request clarification or add additional context in comments.

9 Comments

I don't think you need to specify multiple directories at a time to accomplish your goal
In your example and edit of my post I think you've reorganized my hierarchy. The web and non-web portions of each project are peers, the web directory does not exist under the other directory that needs to move.
oh apologies. I couldn't really understand your directory structure as it was written. Is this what you mean (see edit in my post)
No, there is no common path for the two directories associated with a given project. I'll edit yours to reflect it more accurately, and show the additional complexity as I've simplified things.
I've updated my answer based on the updated initial structure, and I tested it.
|
1

My program git_filter can do the split for you. I wrote it because all the other solutions were very slow on our large repository. It is here:

https://github.com/slobobaby/git_filter

It creates multiple branches for each extract of the original repository. At the moment I have a test branch here:

https://github.com/slobobaby/git_filter/tree/subdir

which will create a new branch containing a subdirectory of the original repository renamed to the root of the new.

It takes a few minutes to run compared to hours or days for the git-core based solutions.

There is a script included which then pushes these new branches to new clean repositories.

Improve this answer

6 Comments

Oops I didn't see you were on windows. You could always start a virtual machine with linux in it... or port my code to windows.
I found your project while searching yesterday, but I wasn't sure how to configure it to do what I want. If there are multiple branches on the original repo, will those branches be carried over into the new repos by the tool?
No. Each branch is pushed as a new separate repository as a second stage of the process. It really helps the process speed to reuse the original objects instead of creating new ones. The push_clean_repos script creates the clean repositories.
If I had read my own code I could have seen the subdir branh does the opposite of what you want and moves the root directory down to a subdirectory. This new branch at github.com/slobobaby/git_filter/tree/flatten does what you want. I even added the configuration which rewrites the example repository you describe above. Just checkout and make and run ./git_filter test.cfg with your repo in test under the current directory :)
In answer to the question about multiple branches the answer is per default it only takes one branch and filters it. You need to run it several times to filter several branches. I could probably change it to execute on all commits instead of just a certain branch's.
|
1

For me, instead of trying to figure out how to split out many parts of a repo, I prefer to clone the original, then pare it down by slicing away what I don't want. Then in the original (or another clone) I pare away the parts I've split out. I've found it to be a much easier, iterative approach, with much greater visibility over what's happening.

See my recent answer on this topic here: https://stackoverflow.com/a/22210682/955926

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.