Support for creating environment files from conda lock files

[tylerpotts]

A very frequent pattern for the data scientists I manage is using jupyter notebooks for development and the productionalizing with Metaflow.

Right now the Jupyter Notebook -> Metaflow python conda environment transition is a friction point.

Ideally we'd like to solve the environment once, lock it with a conda lock file, and then use that lock file throughout the life of that version of the code. We also cache our python packages with an artifactory so this adds a fairly nice confidence of reproducibility.

@romain-intel I put some work in trying to figure out how to get the conda lock file input working a while ago but got pulled into other projects. I seem to remember you saying it wouldn't be a big lift. Any chance you could consider implementing this feature?

In the short term I don't anticipate having the bandwidth to get the necessary context in this project to add the functionality myself.

[romain-intel]

I think it would be easily possible to do in some cases. If you always go through conda to create your environment, that would work well (ie: conda would know about everything). If, however, you used conda and then pip, it may be harder to determine. It actually came up a few days ago internally as well. It sounds like your use case would fall more in the first category (ie: conda knows everything). I can't promise any timeline but let me see when I can get this in. You are right that it would be a very cool feature.

[npow]

Status check (metaflow-netflixext 1.3.8): still not supported. The -f/--file option's help text says "(conda-lock format)" but parse_yml_value only parses standard environment.yml keys (channels, dependencies, pip, sys). There's no handling of an actual *.conda-lock.yml lockfile (metadata header, per-platform package URLs/hashes). Issue remains valid.

[npow]

Scope of the gap and path forward

To make this actionable, here's what's still missing and what an implementation needs to touch. Posting in case anyone wants to pick it up.

What's already there

• *.lock-style state for the Python side is supported:
  • uv.lock via the uv flow mutator (see conda_flow_mutator.py)
  • pylock.toml (PEP 751) as an internal decorator + CLI input
• environment.yml and requirements.txt are accepted as inputs to resolution.

What's still missing

Actual *.conda-lock.yml files (the output of conda-lock lock) are not consumed. The existing -f option's help text says "(conda-lock format)" but parse_yml_value only handles the standard environment.yml keys (channels, dependencies, pip, sys). It does not handle the conda-lock schema (metadata.{content_hash,channels,platforms,sources} + a flat package: list with per-package manager, platform, url, hash, dependencies).

Implementation sketch for whoever picks this up

1. New parser in parsers.py: parse_conda_lock_yml(content) that returns a per-platform ResolvedEnvironment-shaped dict. Use the existing yaml deps + metadata.platforms to produce one entry per platform.
2. New CLI flag in cmd/environment/environment_cmd.py: --lockfile <path> (distinct from -f, which today expects a non-locked yml).
3. New code path: lockfiles are already resolved, so instead of going through EnvsResolver.add_environment -> resolve_environments, construct ResolvedEnvironment and PackageSpecification instances directly from the lockfile rows (name, version, url, hash, manager). Register with the conda cache via the same caching codepath that _resolve_environments already uses for re-uploads.
4. Map manager: manager: conda -> conda PackageSpecification; manager: pip -> pypi PackageSpecification. Skip category: dev unless --include-dev.
5. Multi-platform: a single conda-lock.yml is multi-platform; iterate metadata.platforms and register one ResolvedEnvironment per arch under the same req_id (so co-resolution semantics hold).
6. Update help text on the existing -f option to drop the misleading "(conda-lock format)" mention.

Why this matters

Without this, teams using conda-lock lock as their reproducibility tool (the Jupyter -> Metaflow handoff the original report describes) have to re-resolve from the source environment.yml every time, which can produce different package sets than the lockfile they validated against. The uv.lock / pylock.toml paths only cover the Python-only case, not the Conda + Python case conda-lock is designed for.

Leaving open. PRs welcome.