How do I "git clone" a repo, including its submodules?

Question

How do I clone a git repository so that it also clones its submodules?

Running git clone $REPO_URL merely creates empty submodule directories.

@LiamCrowley , the parent (hosting, containing) repo might depend on a particular version of the submodule for a variety of reasons. For example, the maintainers of the host repo might not be ready to deal with updates just yet. — arkadianriver
– arkadianriver, Commented May 21, 2021 at 12:13
if your repo already exists then you do git submodule init then git submodule update --init potentially with the --remote flag too e.g. stackoverflow.com/a/74999430/1601580 — Charlie Parker
– Charlie Parker, Commented Jan 3, 2023 at 23:01
for ref, do git clone --recurse-submodules --remote-submodules <repo-URL> — Charlie Parker
– Charlie Parker, Commented Jan 3, 2023 at 23:19

Mathias Bynens · Accepted Answer · 2018-10-19 06:46:07Z

With version 2.13 of Git and later, --recurse-submodules can be used instead of --recursive:

git clone --recurse-submodules -j8 git://github.com/foo/bar.git cd bar

^{Editor’s note: -j8 is an optional performance optimization that became available in version 2.8, and fetches up to 8 submodules at a time in parallel — see man git-clone.}

With version 1.9 of Git up until version 2.12 (-j flag only available in version 2.8+):

git clone --recursive -j8 git://github.com/foo/bar.git cd bar

With version 1.6.5 of Git and later, you can use:

git clone --recursive git://github.com/foo/bar.git cd bar

For already cloned repos, or older Git versions, use:

git clone git://github.com/foo/bar.git cd bar git submodule update --init --recursive

Is there any way to specify this behavior as default in your git repository, so that less-informed cloners will automatically get an initialized submodule?
And logically thinking git clone --recursive will also populate any submodules of a submodule, right?
I think I come back to this answer once a month... Why don't git just ask upon cloning if it should also download the submodules?
Also to make clear (since I wanted to know and couldn't find an answer except by looking at the source), the git clone --recursive and --recurse-submodules options behave identically. They result in a call to the same function.

LiraNuna · Accepted Answer · 2010-09-26 07:19:04Z

682

You have to do two things before a submodule will be filled:

git submodule init git submodule update

answered Sep 26, 2010 at 7:19

LiraNuna

67.7k15 gold badges121 silver badges141 bronze badges

10 Comments

user578895 Over a year ago

I was afraid of that... it doesn't make any sense since you're checking out a partial project in that case. I understand that the submodule updates aren't automatic, but why isn't the bound version automatically checked out?? Is there any way to force it? I have a project with 3-levels of submodules and it seems absurd to have to traverse that far just to do a checkout.

joschi Over a year ago

Please read the git-submodule(1) man page (kernel.org/pub/software/scm/git/docs/git-submodule.html). You'll find out that git submodule update supports a nice parameter called --recursive.

Mathias Bynens Over a year ago

Why not just do both of them in one command? git submodule update --init (Also see my answer).

schmijos Over a year ago

I think its better to answer the question with these two commands. Its explains better how to accomplish the task.

Jack Poulson Over a year ago

@MathiasBynens A machine that I just logged into only has git 1.5.5.6, which apparently does not support the shortened instruction, but does support it as two commands.

|

VonC · Accepted Answer · 2023-08-20 12:19:53Z

Git 2.23 (Q3 2019): if you want to clone and update the submodules to their latest revision:

git clone --recurse-submodules --remote-submodules <repo-URL>

If you just want to clone them at their recorded SHA1:

git clone --recurse-submodules <repo-URL>

See below.

Note that Git 2.29 (Q4 2020) brings a significant optimization around submodule handling.

See commit a462bee (06 Sep 2020) by Orgad Shaneh (orgads).
^{(Merged by Junio C Hamano -- gitster -- in commit 2ce9d4e, 18 Sep 2020)}

submodule: suppress checking for file name and ref ambiguity for object ids

^{Signed-off-by: Orgad Shaneh}

The argv argument of collect_changed_submodules() contains only object ids (the objects references of all the refs).

Notify setup_revisions() that the input is not filenames by passing assume_dashdash, so it can avoid redundant stat for each ref.

Also suppress refname_ambiguity flag to avoid filesystem lookups for each object. Similar logic can be found in cat-file, pack-objects and more.

This change reduces the time for git fetch^(man) in my repo from 25s to 6s.

Original answer 2010

As joschi mentions in the comments, git submodule now supports the --recursive option (Git1.6.5 and more).

If --recursive is specified, this command will recurse into the registered submodules, and update any nested submodules within.

See Working with git submodules recursively for the init part.
See git submodule explained for more.

With version 1.6.5 of git and later, you can do this automatically by cloning the super-project with the –-recursive option:
git clone --recursive git://github.com/mysociety/whatdotheyknow.git 

Update 2016, with git 2.8: see "How to speed up / parallelize downloads of git submodules using git clone --recursive?"

You can initiate fetching the submodule using multiple threads, in parallel.
For instances:

git fetch --recurse-submodules -j2

Even better, with Git 2.23 (Q3 2019), you can clone and checkout the submodule to their tracking branch in one command!

See commit 4c69101 (19 May 2019) by Ben Avison (bavison).
^{(Merged by Junio C Hamano -- gitster -- in commit 9476094, 17 Jun 2019)}

clone: add --remote-submodules flag

When using git clone --recurse-submodules there was previously no way to pass a --remote switch to the implicit git submodule update command for any use case where you want the submodules to be checked out on their remote-tracking branch rather than with the SHA-1 recorded in the superproject.

This patch rectifies this situation.
It actually passes --no-fetch to git submodule update as well on the grounds the submodule has only just been cloned, so fetching from the remote again only serves to slow things down.

That means:

--[no-]remote-submodules: 
All submodules which are cloned will use the status of the submodule’s remote-tracking branch to update the submodule, rather than the superproject’s recorded SHA-1. Equivalent to passing --remote to git submodule update.

jhu notes in the comments:

If you want to use git clone --recurse-submodules --remote-submodules <repo-URL> to clone and update to the latest version, your submodules must either:

have a branch master, assumed by git when running the above, or

record a valid branch name in the .gitmodules of the cloned repo. > Otherwise you will get an error about a missing head, and cloning will fail.
So if you have a submodule without branch master, say submodule sub with branch main, run in the root dir of the cloned repo git config -f .gitmodules submodule.sub.branch main and push your changes to the remote.

So it took Git 14 years to start adding proper support for submodules, huh. Thanks for the update! What if I already have a clone of the main repo without submodules and without a recorded SHA1, and I want to pull in the latest version of each submodule. Is it doable?
@VioletGiraffe If that cloned repository has submodules, it has "recorded SHA1". And a git submodule update --init --recursive --remote should update them to the latest commit of their respective branch. (ex: stackoverflow.com/a/56981834/6309)
Let me clarify with an example: I have a template project on Github that uses submodules, and I even commited specific revisions of the submodules into this template repo. But when I create a new project out of this repo, none of the commands you listed (neither clone --recurse-submodules --remote-submodules nor submodule update --init --recursive --remote) let me actually fetch the subrepos. All I get is a .gitmodules file, and I couldn't find any way to init the subrepos other than manually cloning them one by one. I'd like to at least have a script to do it with submodule foreach...
@VioletGiraffe That is because you have added and committed the .gitmodules but not the gitlink (stackoverflow.com/a/16581096/6309, special entries in the index: stackoverflow.com/a/19354410/6309) Here is a repository which does have the proper gitlink registered: github.com/tiagomazzutti/antlr4dart
If you want to use git clone --recurse-submodules --remote-submodules <repo-URL> to clone and update to the latest version, your submodules must either 1. have a branch master, assumed by git when running the above, or 2. record a valid branch name in the .gitmodules of the cloned repo. Otherwise you will get an error about a missing head, and cloning will fail. So if you have a submodule without branch master, say submodule sub with branch main, run in the root dir of the cloned repo git config -f .gitmodules submodule.sub.branch main and push your changes to the remote.

wovano · Accepted Answer · 2023-07-12 14:18:37Z

175

You can use this command to clone your repo with all the submodules:

git clone --recursive YOUR-GIT-REPO-URL

Or if you have already cloned the project, you can use:

git submodule init git submodule update

edited Jul 12, 2023 at 14:18

wovano

5,1915 gold badges33 silver badges58 bronze badges

answered Oct 16, 2017 at 15:47

Javier C.

8,3155 gold badges47 silver badges57 bronze badges

3 Comments

BertC Over a year ago

On git version 2.24.3 the above command gives me the error: error: Server does not allow request for unadvertised object e635630d55682951eb2da35630d5da15b6cc Fetched in submodule path 'ui-library', but it did not contain e635630d55682951eb2da35630d5da15b6cc. Direct fetching of that commit failed.

Pedro77 Over a year ago

ran the second option, nothing happens

Charlie Parker Over a year ago

what happens if you do them in reverse?

Benyamin Jafari · Accepted Answer · 2023-07-13 20:14:58Z

132

[Quick Answer]

After cloning the parent repo (including some submodule repos), do the following:

git submodule update --init --recursive

edited Jul 13, 2023 at 20:14

answered Jul 1, 2019 at 9:39

Benyamin Jafari

35.8k35 gold badges161 silver badges184 bronze badges

1 Comment

Charlie Parker Over a year ago

or --remote if you want the right branch

Antier Solutions · Accepted Answer · 2021-03-18 12:44:09Z

49

Use this command to clone repo with all submodules

git clone --recurse-submodules [email protected]:yourproject

To update code for all submodules

git submodule update --recursive --remote

answered Mar 18, 2021 at 12:44

Antier Solutions

1,4229 silver badges10 bronze badges

2 Comments

Charlie Parker Over a year ago

--remote also makes sure the right branch from remote is used. Right?

Gangula Over a year ago

from the documentation: If you run git submodule update --remote, Git will go into your submodules and fetch and update for you.

Mars Redwyne · Accepted Answer · 2012-03-07 22:28:58Z

40

If your submodule was added in a branch be sure to include it in your clone command...

git clone -b <branch_name> --recursive <remote> <directory>

answered Mar 7, 2012 at 22:28

Mars Redwyne

1,28713 silver badges8 bronze badges

1 Comment

AceFunk Over a year ago

This was more like what I was looking for... but the submodules list their branch as 'detached'. :(

nweiler · Accepted Answer · 2013-01-16 18:39:31Z

35

Try this:

git clone --recurse-submodules

It automatically pulls in the submodule data assuming you have already added the submodules to the parent project.

answered Jan 16, 2013 at 18:39

nweiler

1,1901 gold badge13 silver badges23 bronze badges

2 Comments

Joel Purra Over a year ago

Note that --recurse-submodules and --recursive are equivalent aliases.

Enrico Over a year ago

@SuperUberDuper in that case you can do git submodule update --init --recursive as explained in this answer

Benjamin Loison · Accepted Answer · 2023-12-23 21:28:50Z

32

You can use the --recursive flag when cloning a repository. This parameter forces git to clone all defined submodules in the repository.

git clone --recursive [email protected]:your_repo.git

After cloning, sometimes submodules branches may be changed, so run this command after it:

git submodule foreach "git checkout master"

edited Dec 23, 2023 at 21:28

Benjamin Loison

5,7514 gold badges20 silver badges37 bronze badges

answered Jan 17, 2019 at 4:44

Ahmad Azimi

7231 gold badge8 silver badges12 bronze badges

2 Comments

Charlie Parker Over a year ago

+ 1 foreach but not enough details. see this for a more detailed answer: stackoverflow.com/questions/74988223/…

jhu Over a year ago

The checkout done here switches from a snapshot (detached head) to a working branch. And, say you have some submodules where the branch you want to work with is master, and some submodules where the branch you want to work with is main. (This is quite common now, since master used to be the default, but now it is typically main.) Then you can check out a working branch of all submodules by running git submodule foreach --recursive "git checkout master || git checkout main"

NG_ · Accepted Answer · 2019-05-31 09:29:13Z

29

I think you can go with 3 steps:

git clone git submodule init git submodule update

edited May 31, 2019 at 9:29

NG_

7,2518 gold badges50 silver badges70 bronze badges

answered Jan 16, 2019 at 14:11

muhammad ali e

6876 silver badges9 bronze badges

Comments

kaiser · Accepted Answer · 2013-02-16 16:10:25Z

21

late answer

// git CLONE INCLUDE-SUBMODULES ADDRESS DESTINATION-DIRECTORY git clone --recursive https://[email protected]/USERNAME/REPO.git DESTINATION_DIR

As I just spent a whole hour fiddling around with a friend: Even if you have Admin rights on BitBucket, always clone the ORIGINAL repository and use the password of the one who owns the repo. Annoying to find out that you ran into this minetrap :P

answered Feb 16, 2013 at 16:10

kaiser

22.4k18 gold badges92 silver badges112 bronze badges

9 Comments

jsleuth Over a year ago

That's exactly what I'm dealing with. So, are you saying that anyone who needs to develop on a bitbucket repository that has submodules must use the repository creator's credentials? Blech.

kaiser Over a year ago

@jsleuth Seems so - it sucks BIG TIME... and I know it.

Treffynnon Over a year ago

@kaiser This answer should really be deleted

Treffynnon Over a year ago

It doesn't descriptively answer the OPs question, but details an unrelated bug in Bitbucket; which, incidentally, could just be shortened to "use SSH key authentication".

niziak Over a year ago

Still not resolved on Bitbucked. I'm using relative paths to submodules and as a trick I have to perform following steps: - git remote set-url origin [email protected]:namespace/main-repo.git - git submodule update --init --recursive

|

4b0 · Accepted Answer · 2018-06-18 08:31:21Z

Try this for including submodules in git repository.

git clone -b <branch_name> --recursive <remote> <directory>

or

git clone --recurse-submodules

Rajitha Udayanga · Accepted Answer · 2021-03-08 04:54:29Z

19

Just do these in your project directory.

$ git submodule init $ git submodule update

answered Mar 8, 2021 at 4:54

Rajitha Udayanga

1,7622 gold badges15 silver badges24 bronze badges

1 Comment

Fred Over a year ago

This is the right answer if you had already cloned your repo before.

Mafii · Accepted Answer · 2016-10-07 19:53:14Z

Submodules parallel fetch aims at reducing the time required to fetch a repositories and all of its related submodules by enabling the fetching of multiple repositories at once. This can be accomplished by using the new --jobs option, e.g.:

git fetch --recurse-submodules --jobs=4

According to Git team, this can substantially speed up updating repositories that contain many submodules. When using --recurse-submodules without the new --jobs option, Git will fetch submodules one by one.

Source: http://www.infoq.com/news/2016/03/git28-released

nzrytmn · Accepted Answer · 2019-08-26 09:19:26Z

If it is a new project simply you can do like this :

$ git clone --recurse-submodules https://github.com/chaconinc/YourProjectName

If it is already installed than :

$ cd YourProjectName (for the cases you are not at right directory) $ git submodule init $ git submodule update

FatalError · Accepted Answer · 2019-08-26 20:51:34Z

I had the same problem for a GitHub repository. My account was missing SSH Key. The process is

Then, you can clone the repository with submodules (git clone --recursive YOUR-GIT-REPO-URL)

or

Run git submodule init and git submodule update to fetch submodules in already cloned repository.

Yes, that is Permission denied (publickey). fatal: Could not read from remote repository. error

4b0 · Accepted Answer · 2018-06-18 08:30:38Z

Try this.

git clone -b <branch_name> --recursive <remote> <directory>

If you have added the submodule in a branch make sure that you add it to the clone command.

2 revs, 2 users 98% · Accepted Answer · 2023-12-23 21:30:11Z

I recommend:

# - git submodule init initializes your local configuration file to track the submodules your repository uses, it just sets up the configuration so that you can use the git submodule update command to clone and update the submodules. git submodule init # - git submodule update --init initializes your local configuration file and clones the submodules for you, using the commit specified in the main repository. # note, command bellow will not pull the right branch -- even if it's in your .gitmodules file, for that you need remote. Likely because it looks at the origin (pointer to remote) in github for the available branches. # note, bellow pulls the submodules if you didn't specify them when cloning parent project, ref: https://youtu.be/wTGIDDg0tK8?t=119 git submodule update --init

if you have a specific branch for your submodules then change it to:

# - git submodule init initializes your local configuration file to track the submodules your repository uses, it just sets up the configuration so that you can use the git submodule update command to clone and update the submodules. git submodule init # - The --remote option tells Git to update the submodule to the commit specified in the upstream repository, rather than the commit specified in the main repository. #git submodule update --init --remote git submodule update --init --recursive --remote meta-dataset

For a full example that was testing:

# decided against this because it seems complicated # - note to clone uutils with its submodule do (cmd not tested): cd $HOME git clone --recurse-submodules [email protected]:brando90/ultimate-utils.git # - git submodules cd $HOME/diversity-for-predictive-success-of-meta-learning # - in case it's needed if the submodules bellow have branches your local project doesn't know about from the submodules upstream git fetch # -- first repo # - adds the repo to the .gitmodule & clones the repo git submodule add -f -b hdb --name meta-dataset [email protected]:brando90/meta-dataset.git meta-dataset/ # - ref for init then update: https://stackoverflow.com/questions/3796927/how-do-i-git-clone-a-repo-including-its-submodules/3796947#3796947 #git submodule init #git submodule update # - git submodule init initializes your local configuration file to track the submodules your repository uses, it just sets up the configuration so that you can use the git submodule update command to clone and update the submodules. git submodule init # - git submodule update --init initializes your local configuration file and clones the submodules for you, using the commit specified in the main repository. # note, command bellow will not pull the right branch -- even if it's in your .gitmodules file, for that you need remote. Likely because it looks at the origin (pointer to remote) in github for the available branches. # note, bellow pulls the submodules if you didn't specify them when cloning parent project, ref: https://youtu.be/wTGIDDg0tK8?t=119 git submodule update --init # - The --remote option tells Git to update the submodule to the commit specified in the upstream repository, rather than the commit specified in the main repository. #git submodule update --init --remote git submodule update --init --recursive --remote meta-dataset # - check we are using the right branch https://stackoverflow.com/questions/74998463/why-does-git-submodule-status-not-match-the-output-of-git-branch-of-my-submodule git submodule status cd meta-dataset git branch # should show hdb cd .. # pip install -r $HOME/meta-dataset/requirements.txt # -- 2nd repo, simplified commands from above git submodule add -f -b hdb --name pytorch-meta-dataset [email protected]:brando90/pytorch-meta-dataset.git pytorch-meta-dataset/ git submodule init git submodule update --init --recursive --remote meta-dataset # - check it's in specified branch git submodule status cd pytorch-meta-dataset git branch # should show hdb cd .. # pip install -r $HOME/pytorch-meta-dataset/requirements.txt

wovano · Accepted Answer · 2023-07-12 14:26:50Z

3

If you want to tell your git client to do all actions with --recurse-submodules then you can set this to your git config:

git config submodule.recurse true

See: https://www.git-scm.com/docs/git-config#Documentation/git-config.txt-submodulerecurse

edited Jul 12, 2023 at 14:26

wovano

5,1915 gold badges33 silver badges58 bronze badges

answered Mar 14, 2023 at 8:42

Nico Hollmann

635 bronze badges

3 Comments

Compholio Over a year ago

Unless this is for a newer git than I have, this does not appear to work (even with --global).

Guildenstern Over a year ago

@Compholio submodule.recurse is documented in man git config in my git version 2.39.2

Compholio Over a year ago

Yeah, the documentation that you have linked to says "clone and ls-files are not supported." So, I do not believe that config option works to do this (it works for other things, just not clone).

wovano · Accepted Answer · 2023-07-12 14:31:01Z

1

git submodule init git submodule update

or maybe:

git stash -u git pull origin master git stash p

edited Jul 12, 2023 at 14:31

wovano

5,1915 gold badges33 silver badges58 bronze badges

answered Apr 13, 2022 at 15:51

Gangbin chen

212 bronze badges

1 Comment

Community Over a year ago

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

eQ19 · Accepted Answer · 2022-11-06 19:17:06Z

If the submodules are private submodules you can use credential store so that it also clones its private submodules recursively.

USER=${GITHUB_ACTOR} TOKEN=${{ secrets.JEKYLL_GITHUB_TOKEN }} git config --global credential.helper store echo "https://${USER}:${TOKEN}@github.com" > ~/.git-credentials git clone --recurse-submodules -j8 git://github.com/foo/bar.git cd bar

I use it to clone my submodules where the private one is in 5th level deep. Please allow me to show you how it goes in action:

wovano · Accepted Answer · 2023-07-12 14:30:04Z

0

You can copy clone url from github.

Then use:

git clone --recursive <url of clone copied above>

edited Jul 12, 2023 at 14:30

wovano

5,1915 gold badges33 silver badges58 bronze badges

answered Jan 16, 2023 at 13:59

Poonam Soni

11

1 Comment

Community Over a year ago

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

Mahalakshmi B S · Accepted Answer · 2024-08-13 10:28:50Z

we can do by executing script file which will generate submodule code at a once If we don't want to do any post cloning process and git clone submodule--recursive

1.create script file 2.set environment variable in your local system #!/bin/bash # Define the repository URL and branch REPO_URL="<Repo_URL>" BRANCH_NAME="branch_name" # Clone the repository with the specified branch and initialize submodules in one go git clone --branch "$BRANCH_NAME" "$REPO_URL" && \ cd "$(basename "$REPO_URL" .git)" && \ git submodule update --init --recursive

Collectives™ on Stack Overflow

How do I "git clone" a repo, including its submodules?

23 Answers 23

27 Comments

10 Comments

`submodule`: suppress checking for file name and ref ambiguity for object ids

`clone`: add `--remote-submodules` flag

14 Comments

3 Comments

[Quick Answer]

1 Comment

2 Comments

1 Comment

2 Comments

2 Comments

Comments

9 Comments

Comments

1 Comment

Comments

Comments

1 Comment

Comments

Comments

3 Comments

1 Comment

Comments

1 Comment

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

23 Answers 23

27 Comments

10 Comments

submodule: suppress checking for file name and ref ambiguity for object ids

clone: add --remote-submodules flag

14 Comments

3 Comments

[Quick Answer]

1 Comment

2 Comments

1 Comment

2 Comments

2 Comments

Comments

9 Comments

Comments

1 Comment

Comments

Comments

1 Comment

Comments

Comments

3 Comments

1 Comment

Comments

1 Comment

Comments

Linked

Related

`submodule`: suppress checking for file name and ref ambiguity for object ids

`clone`: add `--remote-submodules` flag