[Feature request / searching for solution] Import recursively

[rsarrazin2]

Also using vcs for the same reasons mentioned in #221 (git submodules being too error-prone, especially for non-git-affine developers), I still see a downside to vcs, which is the (apparent) inability to import / pull recursively into the pointed-to repositories.

The --recursive is obviously not an option since it involves git submodules - which we want to avoid at all costs. Since there can be multiple .repos files in the imported repos - and we may not want to recursively import all of them - the manifest file could contain an optional dependencies (or any other good name) list field that points to the .repos files inside the repo that vcs then shall use when importing recursively.

For the time being, and since vcs doesn't allow for custom fields, I plan to write a wrapper around vcs that imports all new dependencies.repos files found after an import. Here some pseudo-code:

vcs import < $initial_repos_file

repos_files = find_new_repos_files("dependencies.repos")

while(repos_files.not_empty()):
  for(file in repos_files):
    vcs import < $file

  repos_files = find_new_repos_files("dependencies.repos")

My question here: Is there a more straightforward way to import recursively with the current vcs version? I may have missed some very simple way.

Thanks in advance for the answers.

[rsarrazin2]

Quick addition: It looks like vcs does accept (and obviously ignore) additional fields in the YAML .repos file so that I could store the "subdependencies" in the .repos file. Be like:

repositories:
  other/repo2:
    type: git
    url: /home/rolands/workspace/_test/test_vcs/repo2/
    version: 2ea2ca5138d042eca88fdbd119bd2661f8de5213
    subdependencies: [test.repos, other_dependency.repos]

The wrapper is then a lot easier as I just need to take the location (other/repo2), join with the subdependency (test.repos) and import from there by calling the wrapper recursively. The corner case of possible infinite recursion can be mitigated by having a configurable maximum import depth to prevent it.

Still, if there's an easier solution working with the current vcs version, my question holds.

[andry81]

repositories:
other/repo2:
type: git
url: /home/rolands/workspace/_test/test_vcs/repo2/
version: 2ea2ca5138d042eca88fdbd119bd2661f8de5213
subdependencies: [test.repos, other_dependency.repos]

Why not to make a conditional import?

repositories:
  other/repo2:
    type: git
    url: /home/rolands/workspace/_test/test_vcs/repo2/
    version: 2ea2ca5138d042eca88fdbd119bd2661f8de5213
    if:
      expr: (( TESTS )) # a bash shell condition
      then:
        - import: test.repos # relative to this file
        - import:
          - other/test.repos # relative to this file
          - ./_externals/myproj1/.repos # conditional import from an external repository of the `other/repo2` repository
          - $PROJECT_ROOT/_externals/myproj2/.repos # conditional import from an external repository of an arbitrary root repository
          - $UNEXISTED/_externals/myproj3/.repos # invalid import, `$UNEXISTED` is empty
    if:
      expr: (( OTHER_DEPS ))
      then:
        - import: other/other_dependency.repos

PROJECT_ROOT=~/myreporoot
TESTS=1 OTHER_DEPS=1 vcs import < .repos

[jfrascon]

The corner case of possible infinite recursion can be mitigated by having a configurable maximum import depth to prevent it.

Or, before downloading the remote repository, you could check whether the local path (e.g., <your_workspace>/src/other/ros2) already exists in your filesystem. If the folder exists, you could throw an error or warning indicating that the repository has already been downloaded and take appropriate action accordingly.

[Ryanf55]

We use yq in a script to handle merging the repos files manually, but it suffers from issues where we want a colcon workspace with only one repository clone per workspace, and inability to deal well with conflicting version specifiers for repos.

yq eval-all '. as $item ireduce ({}; . * $item )' "${REPOS_FILES_REVERSED[@]}" | grep "" 1>&2 > "$(pwd)/merged.repos"

https://mikefarah.gitbook.io/yq/operators/multiply-merge

mikefarah/yq#1943

dependency_a.repos on main

repositories:
  Common/dependency_p:
    type: git
    url: [email protected]:Common/dependency_p.git
    version: foo

dependency_b.repos on main

repositories:
  Common/dependency_p:
    type: git
    url: [email protected]:Common/dependency_p.git
    version: bar

top.repos

repositories:
  Common/dependency_a:
    type: git
    url: [email protected]:Common/dependency_a.git
    version: main
  Common/dependency_b:
    type: git
    url: [email protected]:Common/dependency_b.git
    version: main

When you merge these at the top level project, somehow you need to make a decision on whether to import dependency_p on branch foo or bar. There isn't a great way to decide, other than the order we manually put it in yq.

Something built into vcs that can error out on conflicts like this would be helpful, as we currenty just wait till CI breaks.