GitLab Clone Group

This tool allows you to download all repos in a GitLab group. It uses the git clone and git fetch command to download the repo.

Example usage:

cargo run -- 9655804 --download-folder ~/Projects/GitLab
# or
gitlab-clone-group 30 --gitlab-url gitlab.example.com --download-folder ~/Projects/GitLab --exclude-group-names "old-group"

With the --gitlab-url this tool can also be used in corporate environments. If the GitLab API is not available from you system, you can download it by any other means and use the --project-list-from-file option.

If the --project-list-from-file is not given it requires a GitLab API key.

Cloning repos

This tool will download the repos using the systems git clone command and uses the ssh. For example:

git clone git@gitlab.com:df-modding-tools/df-raw-language-server.git ~/Projects/GitLab/df-modding-tools__df-raw-language-server`

By default all / in the subgroups will be replaces with __ (2 underscores). This is done so all repos are just nested in 1 folder.

After each clone the application sleeps for a little bit in order to not overrun/DoS the GitLab server.

Update repos

If a repo was already present it will run git fetch in order to download all new changes. This will not overwrite existing files or projects that have uncommitted changes.

You can use the --checkout-default-branch in order to do a git checkout on each repo. It will checkout the default branch as set in GitLab (usually master or main).

Command options

> gitlab-clone-group --help
Usage: gitlab-clone-group [OPTIONS] <GITLAB_GROUP_ID>

Arguments:
  <GITLAB_GROUP_ID>  The ID of the GitLab group. This is usually a number

Options:
      --gitlab-url <GITLAB_URL>
          The main URL where the gitlab instance can be found. Default: `gitlab.com`
      --download-folder <FOLDER>
          The folder the GitLab instance will be cloned to. Default is the current working directory
      --checkout-default-branch
          If set to `true` it will try to check out the default branch again. This will not succeed when local changes are made that are not committed
      --exclude-group-names <EXCLUDE_GROUP_NAMES>
          List of group names to not download. Group name must be an exact match
      --project-list-from-file <PROJECT_LIST_FROM_FILE>
          JSON File where the list of projects will be found. If this option is present it will not use the GitLab API to download the list of projects. This expected format is the same json format as it would have received from the GitLab API
  -h, --help
          Print help
  -V, --version
          Print version

Example command usage:

gitlab-clone-group 9655804 --download-folder ./test

Output (terminal output has colors):

To download the list of projects we need a GitLab API Key.
You can create a personal access token here:
https://gitlab.com/-/profile/personal_access_tokens?name=Download+GitLab+Projects&scopes=read_api
Please enter your API Key:
INFO  [gitlab_clone_group] Downloading page 1 from GitLab API.
INFO  [gitlab_clone_group] Downloading page 2 from GitLab API.
INFO  [gitlab_clone_group] Found in GitLab: 5
INFO  [gitlab_clone_group] After filtering: 5
Projects: [
    "df-modding-tools/df-test-mod",
    "df-modding-tools/df-raw-language-server",
    "df-modding-tools/df-raw-collection",
    "df-modding-tools/df-raw-syntax",
    "df-modding-tools/df-syntax-highlighting",
]
Going to downloading the 5 repo's listed above. Press any key to continue. Press `ctrl+c` (`^c`) to quit.
INFO  [gitlab_clone_group] Project 0/5 DF Test Mod
INFO  [gitlab_clone_group] Cloning repo
Cloning into 'df-modding-tools__df-test-mod'...
remote: Enumerating objects: 15, done.
remote: Counting objects: 100% (15/15), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 15 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (15/15), 8.16 KiB | 8.16 MiB/s, done.
INFO  [gitlab_clone_group] Waiting
INFO  [gitlab_clone_group] Project 1/5 DF Raw Language Server
INFO  [gitlab_clone_group] Cloning repo
Cloning into 'df-modding-tools__df-raw-language-server'...
remote: Enumerating objects: 6649, done.
remote: Counting objects: 100% (474/474), done.
remote: Compressing objects: 100% (456/456), done.
remote: Total 6649 (delta 218), reused 54 (delta 16), pack-reused 6175
Receiving objects: 100% (6649/6649), 16.64 MiB | 6.70 MiB/s, done.
Resolving deltas: 100% (4286/4286), done.
INFO  [gitlab_clone_group] Waiting
[...]

License

The code in this project is licensed under the MIT or Apache 2.0 license. All contributions, code and documentation, to this project will be similarly licensed.