support v1 recipe format via rattler-build

[beckermr]

This issue is to track progress on supporting the v1 recipe format via rattler build.

Minimal Work Items for Uploads:

• ensure conda-forge-feedstock-check-solvable works for the new format (feat: solvability implementation for rattler-build regro/conda-forge-feedstock-check-solvable#58)
• ensure the other migrators in the bot don't choke (feat: skip all migrators on schema v1 for now regro/cf-scripts#3006 misc: disable recipe.yaml for migrators regro/cf-scripts#2918)
• ensure ABI migrations in the bot can go through
  • enable build number bumps (feat: add build number editing for v1 recipes regro/cf-scripts#3009)
  • add some tests in the bot for ABI+v1 recipe migrations (replicate the tests of yaml-based migrations in this file: https://github.com/regro/cf-scripts/blob/main/tests/test_migrators.py, feat: test v1 recipe.yaml migrations regro/cf-scripts#3011)
• fix v1 recipe linting (v1 recipe linting broken conda-smithy#2067)
• fully turn on the linter (it still hints) (feat: enable v1 lint for PRs conda-forge-webservices#685, feat: enable v1 linting conda-forge-feedstock-ops#14)
• turn on uploads (last one! feat: allow uploading on main channel for v1 recipes conda-forge-ci-setup-feedstock#337)

Next Steps + Bug Reports

• rerendering readme fails for v1 recipes and c stdlib conda-smithy#2085
• move variants.yaml to a backup file when rendering conda-smithy#2082
• Support variants.yml staged-recipes#27750
• examples break build-locally.py staged-recipes#27695
• fix bug with test: None regro/cf-scripts#3018
• Support conda_build_config_vars in rattler-build conda-forge-ci-setup-feedstock#359
• feat: add version bumping for v1 recipes regro/cf-scripts#3525
• enable selected minimigrators as they are needed (Migrators for v1 recipes regro/cf-scripts#3642)
• v1 recipe format solvable tests are failing regro/conda-forge-feedstock-check-solvable#61
• add linter lint+hint for recipe being able to be parsed so we can run automatic conversion (feat: add lint+hint for recipe parsing conda-smithy#2141)

Running List of Blockers for Automatic Migration PRs from the Bot

• rattler-build produces bad package version if 0 involved prefix-dev/rattler-build#1455
• Values of context variables can have different types depending on the underlying shell prefix-dev/rattler-build#1451
• ignore_run_exports doesn't work with ${{ compiler('cxx') }} prefix-dev/rattler-build#1445
• rattler-build doesn't include licenses if it's a directory prefix-dev/rattler-build#1443
• test environment not activated when running the python test type prefix-dev/rattler-build#1437
• test environment not recognized by conda as an environment prefix-dev/rattler-build#1439
• test env not properly activated in another case prefix-dev/rattler-build#1440
• cannot properly capture stderr and stdout in commands run in subshells via backticks for tests prefix-dev/rattler-build#1438
• variant env vars not set for test scripts prefix-dev/rattler-build#1317
• enable more flexible patching like conda-build prefix-dev/rattler-build#1474
• BUG: mac relinking doesn't rewrite default install_name in dependencies like conda-build prefix-dev/rattler-build#1476
• test.sources requires trailing slash for directories prefix-dev/rattler-build#1480
• Check applying patches with tabulations prefix-dev/rattler-build#1491
• rattler-build build panics when test.python.python_version match spec is invalid prefix-dev/rattler-build#1493
• [bug] rpath subdirectories on linux prefix-dev/rattler-build#1498
• Missing env vars in build script prefix-dev/rattler-build#889
• How to reference relative to $PREFIX in license_file? prefix-dev/rattler-build#1521
• bug: too many open files again and RATTLER_IO_CONCURRENCY_LIMIT env var not helping prefix-dev/rattler-build#1638
• Credential leaks in about.json when sending HTTP credentials via CLI prefix-dev/rattler-build#1636
• test env is in tempdir and so cannot debug by hand prefix-dev/rattler-build#1436
• turn off set -x for test scripts prefix-dev/rattler-build#1639
• test environment not recognized by conda prefix-dev/rattler-build#1643

supersedes #1612

[wolfv]

I think for the conda-forge-feedstock-check solvable I will have a good grip on what needs to be done.

For the second point, ensure that ABI migrations in the bot can go through, I actually don't really know where to start. Do you have additional pointers?

[beckermr]

Right so for that, once the solver checks work, we should add a test to the bot repo that does an abi migration and make sure it all runs.

[wolfv]

regro/conda-forge-feedstock-check-solvable#58 this is the PR for adding solvability checks with rattler-build.

[wolfv]

conda-forge/conda-forge-webservices#685 opened to enable lints on PRs etc.

[wolfv]

@beckermr - I think I need some help / guidance in order to understand what's needed for the cf-graph items. How can I ensure that?

[wolfv]

Thanks @beckermr for updating the todo items. I can make the build number PR right away (it's a subset of the version bump PR) so we can get that one in straight away! If you let me know how / where to add tests for migrations I can do it tomorrow, too :)

[beckermr]

I updated the issue list above with a pointer to the correct test file. We should replicate all of the tests there for yaml migrations to ensure they work on v1 recipes.

[wolfv]

Added PR for build number bumps: regro/cf-scripts#3009

[wolfv]

Only the tests to go! :)

[wolfv]

Ticked off the last one (uploading packages) :)

There are two more on staged-recipes:

• Support variants.yml staged-recipes#27750
• examples break build-locally.py staged-recipes#27695

[isuruf]

Opened

• Support conda_build_config_vars in rattler-build conda-forge-ci-setup-feedstock#359

[wolfv]

@beckermr are you OK if I edit your initial comment to include the additional TODO items?

[beckermr]

@wolfv there are some seeming important issues here (e.g., not including licenses in some cases).

I suspect it may be prudent to hold off on more conversions to v1 until more of them are fixed.

[wolfv]

@beckermr that particular issue (folders not being globbed if there is no / at the end) has been fixed since ~2-3 releases.

What are the big issues you are seeing?

[beckermr]

Oh that's great! Can we go through the rattler-build issue tracker and list of issues above and mark any that are fixed as closed?

I also think we still have open issues on recipes with merging build host and test env activation.

[beckermr]

Wow that's a lot of work on the issue tracker! Thanks for the big push on this!

[beckermr]

cc @wolfv