Support Bzlmod and add rules_scala to bazel-central-registry

[sfc-gh-pbennes]

Greetings!

It looks like rules_scala isn't part of the Bzlmod effort or added to bazel-central-registry yet. I've opened an issue both here and in https://github.com/bazelbuild/bazel-central-registry to request it: bazelbuild/bazel-central-registry#522

[chrislovecnm]

Just a note to anyone that starts the work native.register_toolchains is not supported when using bzlmod.

[sluongng]

@chrislovecnm it's still available here no? https://bazel.build/rules/lib/globals/module#register_toolchains

[chrislovecnm]

@sluongng When I last tried it, it did not work when using it inside an extension. In rules_python we are skipping the native call:

https://github.com/bazelbuild/rules_python/blob/36082079b514529d4009a2b104475fde1cdd5b30/python/repositories.bzl#L569

And build the toolchains here:

https://github.com/bazelbuild/rules_python/blob/36082079b514529d4009a2b104475fde1cdd5b30/python/extensions/private/pythons_hub.bzl#L56

We build the toolchains and use the following:

https://github.com/bazelbuild/rules_python/blob/36082079b514529d4009a2b104475fde1cdd5b30/MODULE.bazel#L49

[sluongng]

Yeah you would only want to prepare the toolchain (as a new repository) inside the extension I think. Registering it must be called from the MODULE.bazel file.

So native.register_toolchains call from the extension's starlark does not work, but having the extension create @pythons_hub//:all and then MODULE.bazel calling register_toolchains("@pythons_hub//:all") worked. 👍

[mateuszkuta256]

I prepared a minimal bzlmod support here and gonna split it into a few PRs
TODOs after merging it:

• write documentation on how to use rules_scala with bzlmod (some properties like SCALA_VERSION will probably be configurable via repo-env)
• prepare release notes
• contribute repo to the Bazel Central Registry
• add tests that the project builds using both WORKSPACE and MODULE.bazel
• migrate test/external workspaces to blzmod too

[pcj]

@mateuszkuta256 thanks for getting this started! Any progress updates or blockers?

[mateuszkuta256]

@mateuszkuta256 thanks for getting this started! Any progress updates or blockers?

hi, unfortunately I'm working on other things now and won't continue with this PR in the foreseeable future
I remember this migration was on hold because interfered with 'cross compilation support'
It looks like great progress was done it this area, so maybe someone can already take over the PR

[mbland]

I'd like to take on the task of Bzlmodifying this repo through a series of pull requests.

I've already created a (mostly) working branch in my own fork. Though I saw a couple draft pull requests here, I ended up taking a different approach and got it mostly working. In fact, I did exactly what @sluongng suggested in #1482 (comment). (I didn't notice this comment before just now—I might've read it, but not understood it at the time—but I did study rules_python and rules_go, and ended up doing exactly that.)

There are still issues I need help to address (recorded in some of the commit messages), I didn't strictly maintain WORKSPACE compatibility (fixable), and I only tested against Scala 2.13. But I can start teasing chunks out of it in a methodical fashion, to ensure that I maintain WORKSPACE and cross-version compatibility.

For an example of what it looks like from a client perspective, here's what the MODULE.bazel stanza from my local EngFlow/example repo looks like (as opposed to the current stanza, explanatory comment notwithstanding):

bazel_dep(name = "rules_scala", repo_name = "io_bazel_rules_scala")
local_path_override(
    module_name = "rules_scala",
    path = "../../bazelbuild/rules_scala"
)   

scala_dev_deps = use_extension(
    "@io_bazel_rules_scala//scala/extensions:deps.bzl",
    "scala_deps",
    dev_dependency = True,
)   
scala_dev_deps.toolchains(
    scalatest = True,
)

So if folks are game for me to do this, I'll start carving off pieces as separate pull requests, and we can resolve any outstanding problems in the process.

cc: @BillyAutrey @jayconrod @benjaminp @TheGrizzlyDev

[mbland]

A quick update for visibility: I'm very close to having the Bzlmodified rules_scala passing 100% of the tests, down to the last couple of failures:

• I need to replace the bind() calls from twitter_scrooge with alias() or some such to get test_version.sh to pass.
• I need to fix the Scala 3.1.2 and Scala 3.3.3 failures for the Scalafmt targets from test/shell/test_cross_build.sh (corresponding to the test3 and library3 test cases in test_scalafmt()).

I also need to get test_lint.sh working again, and update the dt_patches repos for Bzlmod. But the following all pass:

• test_rules_scala.sh
• test_reproducibility.sh
• test_examples.sh
• test_coverage.sh

Hopefully I can get these fixed up today, and I'll start peeling off the next pull request or two. And thanks to @simuons and @liucijus for reviewing and merging #1619 and #1620 already.

[mbland]

Weekly update!

Overview

#1722 is in, and MODULE.bazel exists on master. Praise Kier!

No, actually: Praise @simuons and @liucijus! 🙂 Not just for landing #1722, but for following up so quickly on all the refinements I've sent after that.

Now we're down to just three pull requests, all orthogonal to one another, and no pending branches! As noted below:

• Add toolchain options API to WORKSPACE and Bzlmod #1730 contains a minor API change affecting both WORKSPACE and Bzlmod.
• Use the Publish to BCR reusable GitHub workflow #1731 adds the Publish to BCR workflow, now that it's ready for prime time.

Once those two land (plus maybe #1729), we're ready to release and publish this thing and close out here.

Merged

• Enable Bzlmod #1722
• Add src/... to test_rules_scala, fix Windows break #1724
• Replace @scalafmt_default with toolchains target #1725
• Establish minimum compatible Bazel, dep versions #1726
• Bump Bazel, rules_*, Guava, Go, golang tools, etc. #1727
• Remove Label wrappers for target string literals #1728

Open

• Add test_dependency_versions, update test_runner #1729
• Add toolchain options API to WORKSPACE and Bzlmod #1730
• Use the Publish to BCR reusable GitHub workflow #1731

Pending branches

None! That's it! No more!

Preview branches

As always, if you want to try everything out before the next release, please only use either my bzlmod or bzlmod-bazel-8 branches. Both contain all the changes from the open pull requests.

Updated scala_toolchains() and scala_deps module extension APIs

I've added what I call "toolchains options support" in #1730. This means that scala_toolchains() accepts either boolean or dict arguments for specific toolchains, and the scala_deps module extension generates them from tag classes. This yields similar changes in both the WORKSPACE and Bzlmod APIs, specifically:

• Add toolchain options API to WORKSPACE and Bzlmod #1730 adds toolchain options support to the scala_toolchains() parameters scalafmt, scala_proto, and twitter_scrooge, and to the scalafmt tag class.

• Add toolchain options API to WORKSPACE and Bzlmod #1730 eliminates the scalafmt_default_config, scala_proto_options, and twitter_scrooge_deps option parameters from scala_toolchains().

• The scala_proto and twitter_scrooge tag class APIs haven't changed, but now they and the scalafmt tag class share a common implementation.

• All these toolchains options dicts are aligned with the underlying toolchain macro arguments, making the relationship between them clear.

• Thanks to the shared implementation, it will be easy to add options support to additional toolchains over time. These updates will require only a minor release, not a major one.

For example, under WORKSPACE, enabling scalafmt and setting a custom config changed like this:

# Before
scala_toolchains(
    # Other toolchains settings...
    scalafmt = True,
    scalafmt_default_config = "//path/to/my/custom:scalafmt.conf",
)

# After
scala_toolchains(
    # Other toolchains settings...
    scalafmt = {"default_config": "//path/to/my/custom:scalafmt.conf"},
)

scalafmt = True will still work if you want to use the default options, but it's implied if you supply your own options object.

Under Bzlmod, the scala_proto tag class changed thus to align the interface with setup_scala_proto_toolchains from //scala_proto:toolchains.bzl:

# Before
scala_deps.scala_proto(
    options = ["grpc"],
)

# After
scala_deps.scala_proto(
    default_gen_opts = ["grpc"],
)

scala_deps.scala_proto() will work in this case, since under the hood, {"default_gen_opts": ["grpc"]} is the default.

Please see the description of #1730 for further details on all the good things.

This came about after @simuons asked in #1722 (comment) to replace the boolean flags in scala_deps.toolchains() with individual tag classes. The thought being, each toolchain can potentially define its own set of attributes via its own tag class. (I mentioned all this in last week's update.)

In the same comment, he also asked whether we could replace toolchain macros (invoked by users in BUILD files) entirely with a module extension. Though I noted this won't be trivial to do, and shouldn't hold up the release, I later experimented with the toolchain macro replacement idea. I'm not yet near achieving that goal, but my experimentation yielded the immediate improvements described above and in #1730.

[mbland]

Weekly update!

Overview

We're down to one last pull request (for real, this time!): #1730.

With this last API change (described in last week's update), pushing the v7.0.0 tag should publish the rules_scala module to the Bazel Central Registry. This is because #1731 put the necessary Publish to BCR configuration into place. Future dependency version bumps and backward compatible API changes can happen in minor version updates easily after that.

Ideally, #1729 would make it in before publishing v7.0.0 as well, but it's not strictly required. It adds tests to enforce the minimum dependency version guarantees from #1726, but it's not technically blocking v7.0.0. (Same with #1732, a routine dependency version bump.)

Merged

• Use the Publish to BCR reusable GitHub workflow #1731

Open

• Add test_dependency_versions, update test_runner #1729
• Add toolchain options API to WORKSPACE and Bzlmod #1730
• Bump dependency versions #1732

Preview branches

As always, if you want to try everything out before the next release, please only use either my bzlmod or bzlmod-bazel-8 branches. Both contain all the changes from the open pull requests.

Move to the bazel-contrib GitHub org

Thanks to @meteorcloudy, the repo has moved from bazelbuild/rules_scala to bazel-contrib/rules_scala per #1616. The lack of Bzlmod support was the last blocker. Now that #1722 is in, and publishing the first rules_scala module is imminent, moving to bazel-contrib now streamlines the publishing process.

Toolchains blog post

I'm now working in earnest on the next installment in my Bzlmod blog post series, based on this rules_scala experience. I don't know for sure yet when it'll be ready, but I'm having fun retracing all the steps. If I can manage to explain them to myself, hopefully the result will be useful to others, too.

Module extension toolchains experiment

To largely repeat #1730 (comment): I've continued to experiment with generating toolchains from the scala_deps module extension in the bzlmod-toolchains-experiment branch of mbland/rules_scala. I've managed to generate a separate toolchain in @rules_scala_toolchains//scala after updating the scala_deps.scala() tag class.

This confirms that, after #1730 lands, and after the new release, we can expand the tag classes to pass through more parameters. It also confirms that we can generate new toolchains in @rules_scala_toolchains using a module extension.

Such updates can remain backward compatible, requiring only minor version releases. It may even be possible to support an arbitrary number of toolchains and/or separate toolchain repos in a minor version update. I've yet to get around to experimenting with that yet, however.

The goal is to follow through on @simuons's idea of configuring all toolchains via module extensions from #1722 (comment). It's not worth blocking v7.0.0, as existing setup_scala_toolchain macro calls from BUILD files will continue to work. But it might be more idiomatic, following toolchain configuration patterns like those in rules_go and rules_python. (I studied those two rule sets to develop scala_toolchains() and the scala_deps extension in the first place.)

If/when I reach a point where replacing BUILD macros with module extensions is a possibility, I don't think it'll require a major version release. We can publish the new mechanism in a minor release and see how well it works in practice. Then, if we want to make the toolchain setup macros private, that can happen in v8.0.0 or some other future major version release.

[mbland]

Daily update! 😛

@simuons merged #1729, #1730, and #1732 today, as well as @WojciechMazur's #1733.

So @simuons and @liucijus, I suppose it's up to you to publish v7.0.0 when you like. The only thing is you may consider creating your own Personal Access Token to publish to the BCR, per the dicsussion in #1731.

Then once https://registry.bazel.build/modules/rules_scala goes from HTTP 404 to HTTP 200, we're done here! 😄

---

In other news, I did just open #1734 with a very small documentation update (on a tip from @gergelyfabian); nothing that should block the release.

I'll continue to experiment with my bzlmod-toolchains-experiment branch for post-v7.0.0 releases.