Skip to content

Commit 116dd19

Browse files
committed
Only support protoc toolchainization for >= v29.0
Updates `README.md` to explain why this requirement exists, and why Bazel 6.5.0 is capped at `protobuf` v29 in general, even without protocol compiler toolchainization. Removes the `v28.2` and `v28.3` entries from `scripts/update_protoc_integrity.py` and `protoc/private/protoc_integrity.bzl`. I tried several things to get it to work with `protobuf` v28.2, as described below. However, each path only led to the same suffering described in the new "Why this requires `protobuf` v29 or later" section of the README. Without `--incompatible_enable_proto_toolchain_resolution`, everything still worked (i.e., `bazel build` compiled `protoc` and finished successfully). But there's no point in keeping any of the new patches or flags if `--incompatible_enable_proto_toolchain_resolution` still breaks. I discovered along the way that `protobuf` v30 isn't compatible with Bazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0 compatibility" section of `README.md`. --- I experimented with using `protobuf` v28.2, `rules_proto` 6.0.2, and `rules_java` 7.12.4 and 8.10.0. I updated the `protobuf` patch for v28.2 with the following statements: ```py load("//bazel/common:proto_common.bzl", "proto_common") load("@rules_proto//proto:proto_common.bzl", "toolchains") _PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type" _PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION" _PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN) def _protoc_files_to_run(ctx): if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False): ``` I updated `protoc/private/protoc_toolchain.bzl` to use `proto_common` from `rules_proto`. I also created a `rules_proto` 6.0.2 patch for `proto_toolchain()` to fix a "no such package: //proto" breakage: ```diff 6.0.2 patch for `proto_toolchain()`: ```diff diff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzl index a091b80..def2699 100644 --- i/proto/private/rules/proto_toolchain.bzl +++ w/proto/private/rules/proto_toolchain.bzl @@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []): native.toolchain( name = name + "_toolchain", - toolchain_type = "//proto:toolchain_type", + toolchain_type = Label("//proto:toolchain_type"), exec_compatible_with = exec_compatible_with, target_compatible_with = [], toolchain = name, ``` I tried adding combinations of the following `--incompatible_autoload_externally` flag values to .bazelrc`: ```txt common --incompatible_autoload_externally=+@protobuf,+@rules_java ``` Nothing worked.
1 parent d7352d6 commit 116dd19

File tree

3 files changed

+91
-39
lines changed

3 files changed

+91
-39
lines changed

README.md

+91-17
Original file line numberDiff line numberDiff line change
@@ -196,16 +196,17 @@ load(
196196

197197
### <a id="protoc"></a>Using a precompiled protocol compiler
198198

199-
`rules_scala` now supports
200-
[`--incompatible_enable_proto_toolchain_resolution`][]. When using this flag
201-
with the `MODULE.bazel` or `WORKSPACE` configurations below, `rules_scala` will
202-
use a precompiled protocol compiler binary by default.
199+
`rules_scala` now supports the
200+
[`--incompatible_enable_proto_toolchain_resolution`][] flag when using
201+
[`protobuf` v29 or later](#why-proto-v29). When using this flag with the
202+
`MODULE.bazel` or `WORKSPACE` configurations below, `rules_scala` will use a
203+
precompiled protocol compiler binary by default.
203204

204205
[`--incompatible_enable_proto_toolchain_resolution`]: https://bazel.build/reference/command-line-reference#flag--incompatible_enable_proto_toolchain_resolution
205206

206-
__Windows builds now require the precompiled protocol compiler toolchain.__ See
207-
the [Windows MSVC builds of protobuf broken by default](#protoc-msvc) section
208-
below for details.
207+
__Windows builds now require using `protobuf` v29 or later with the precompiled
208+
protocol compiler toolchain.__ See the [Windows MSVC builds of protobuf broken
209+
by default](#protoc-msvc) section below for details.
209210

210211
#### Common setup
211212

@@ -226,12 +227,11 @@ register_toolchains("@rules_scala//protoc:all")
226227

227228
#### Temporary required `protobuf` patch
228229

229-
As of `protobuf` v29.3, enabling protocol compiler toolchainization requires
230-
applying [protoc/0001-protobuf-19679-rm-protoc-dep.patch][]. It is the `git
231-
diff` output from the branch used to create protocolbuffers/protobuf#19679.
232-
Without it, there remains a transitive dependency on
233-
`@com_google_protobuf//:protoc`, causing it to recompile even with the
234-
precompiled toolchain registered first.
230+
At the moment, enabling protocol compiler toolchainization requires applying
231+
[protoc/0001-protobuf-19679-rm-protoc-dep.patch][]. It is the `git diff` output
232+
from the branch used to create protocolbuffers/protobuf#19679. Without it, a
233+
transitive dependency on `@com_google_protobuf//:protoc` remains, causing
234+
`protoc` to recompile even with the precompiled toolchain registered first.
235235

236236
[protoc/0001-protobuf-19679-rm-protoc-dep.patch]: ./protoc/0001-protobuf-19679-rm-protoc-dep.patch
237237

@@ -289,6 +289,55 @@ load("@platforms//host:extension.bzl", "host_platform_repo")
289289
host_platform_repo(name = "host_platform")
290290
```
291291

292+
#### <a id="why-proto-v29"></a>Why this requires `protobuf` v29 or later
293+
294+
Using `--incompatible_enable_proto_toolchain_resolution` with versions of
295+
`protobuf` before v29 causes build failures due to a missing internal Bazel
296+
dependency.
297+
298+
Bazel's builtin `bazel_java_proto_aspect` transitively depends on a toolchain
299+
with a [`toolchain_type`][] of `@rules_java//java/proto:toolchain_type`.
300+
Experimentation with `protobuf` v28.2 using both Bazel 6.5.0 and 7.5.0 led to
301+
the following error:
302+
303+
```txt
304+
ERROR: .../external/bazel_tools/src/main/protobuf/BUILD:28:15:
305+
in @@_builtins//:common/java/proto/java_proto_library.bzl%bazel_java_proto_aspect
306+
aspect on proto_library rule
307+
@@bazel_tools//src/main/protobuf:worker_protocol_proto:
308+
309+
Traceback (most recent call last):
310+
File "/virtual_builtins_bzl/common/java/proto/java_proto_library.bzl",
311+
line 53, column 53, in _bazel_java_proto_aspect_impl
312+
File "/virtual_builtins_bzl/common/proto/proto_common.bzl",
313+
line 364, column 17, in _find_toolchain
314+
Error in fail: No toolchains registered for
315+
'@rules_java//java/proto:toolchain_type'.
316+
317+
ERROR: Analysis of target
318+
'@@bazel_tools//src/main/protobuf:worker_protocol_proto' failed
319+
```
320+
321+
See the commit "Only support protoc toolchainization for >= v29.0" from
322+
bazelbuild/rules_scala#1710 for details of the experiment.
323+
324+
For `protobuf` v29.0, protocolbuffers/protobuf#18308 added the
325+
[`@protobuf//bazel/private/toolchains`][proto-private-tc] package and updated
326+
`protobuf_deps()` from `@protobuf//:protobuf_deps.bzl` to register it:
327+
328+
```py
329+
native.register_toolchains("//bazel/private/toolchains:all")
330+
```
331+
332+
[`toolchain_type`]: https://bazel.build/extending/toolchains#writing-rules-toolchains
333+
[proto-private-tc]: https://github.com/protocolbuffers/protobuf/blob/v29.0/bazel/private/toolchains/BUILD.bazel
334+
335+
protocolbuffers/protobuf#18435 then introduced
336+
[`java_source_toolchain_bazel7`][java-proto-tc] with the required
337+
`toolchain_type`.
338+
339+
[java-proto-tc]: https://github.com/protocolbuffers/protobuf/blob/v29.0/bazel/private/toolchains/BUILD.bazel#L50-L74
340+
292341
#### More background on protocol compiler toolchainization
293342

294343
- [Proto Toolchainisation Design Doc](
@@ -905,11 +954,36 @@ At the moment, `WORKSPACE` builds mostly continue to work with Bazel 6.5.0, but
905954
not out of the box, and may break at any time. You will have to choose one of
906955
the following approaches to resolve `protobuf` compatibility issues.
907956

908-
First, you may choose to use protocol compiler toolchainization. See the [Using
909-
a precompiled protocol compiler](#protoc) section for details.
957+
First of all, you _must_ use `protobuf` v29 or earlier. `rules_scala` now uses
958+
v30 by default, which removes `py_proto_library` and other symbols that Bazel
959+
6.5.0 requires:
960+
961+
```txt
962+
ERROR: Traceback (most recent call last):
963+
File ".../external/bazel_tools/src/main/protobuf/BUILD",
964+
line 1, column 46, in <toplevel>
965+
load("@com_google_protobuf//:protobuf.bzl", "py_proto_library")
966+
967+
Error: file '@com_google_protobuf//:protobuf.bzl'
968+
does not contain symbol 'py_proto_library'
969+
970+
ERROR: .../src/java/io/bazel/rulesscala/worker/BUILD:3:13:
971+
no such target '@bazel_tools//src/main/protobuf:worker_protocol_java_proto':
972+
target 'worker_protocol_java_proto'
973+
not declared in package 'src/main/protobuf'
974+
defined by .../external/bazel_tools/src/main/protobuf/BUILD
975+
(Tip: use `query "@bazel_tools//src/main/protobuf:*"`
976+
to see all the targets in that package)
977+
and referenced by '//src/java/io/bazel/rulesscala/worker:worker'
978+
```
979+
980+
You may use protocol compiler toolchainization with `protobuf` v29 to avoid
981+
recompiling `protoc`. See the [Using a precompiled protocol compiler](#protoc)
982+
section for details.
910983

911-
Otherwise, per bazelbuild/rules_scala#1647, you must add the following flags to
912-
`.bazelrc`, required by the newer `abseil-cpp` version used by `protobuf`:
984+
Otherwise, per bazelbuild/rules_scala#1647, add the following flags to
985+
`.bazelrc` to compile the newer `abseil-cpp` versions used by newer `protobuf`
986+
versions:
913987

914988
```txt
915989
common --enable_platform_specific_config

protoc/private/protoc_integrity.bzl

-20
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,6 @@ PROTOC_VERSIONS = [
1515
"29.2",
1616
"29.1",
1717
"29.0",
18-
"28.3",
19-
"28.2",
2018
]
2119

2220
PROTOC_BUILDS = {
@@ -31,8 +29,6 @@ PROTOC_BUILDS = {
3129
"29.2": "sha256-Kc9IPi+yGCfl+sSWTjXq5HKiOOKMdi8C+xfc2T/4uJ8=",
3230
"29.1": "sha256-H3Sj8zVd58Bma8ElYRwTUywlmPhTUh0NPmIaWwnyR5k=",
3331
"29.0": "sha256-MF8b5a57LzlFGHCzErRcHguiaZAcg7oW2F+fnRRBs0g=",
34-
"28.3": "sha256-HeUiAyqLGUAC/jXKuG10eEgji15N5PmWSDcgefW0b5o=",
35-
"28.2": "sha256-kdglPNwPDw/FHCtpyAZ3mWYy9SWthFBL+ltO44rT5Jw=",
3632
},
3733
},
3834
"linux-ppcle_64": {
@@ -46,8 +42,6 @@ PROTOC_BUILDS = {
4642
"29.2": "sha256-uiCJWht/NKb/ql5Rw0ExbErrxMFEN3hjQITpn262f/k=",
4743
"29.1": "sha256-B1vWZq1B60BKkjv6+pCkr6IHSiyaqLnHfERF5hbo75s=",
4844
"29.0": "sha256-EJAnjNB1e3AsNrA+6t9KvTYCAnm7B7DfX4C9iW8+IDM=",
49-
"28.3": "sha256-dSKdPN5z5wYZcXgU9R+m9K16NiwzUe5ZGXuxV8sAgsY=",
50-
"28.2": "sha256-xcFrR2f/iGYJDuEOhwkDABBSAbXbHs7zgO3XPCM03Ng=",
5145
},
5246
},
5347
"linux-s390_64": {
@@ -61,8 +55,6 @@ PROTOC_BUILDS = {
6155
"29.2": "sha256-LwpVmdprgpMqCNQz+nkTux1oWIl8zTxrwyhvAYt2xOA=",
6256
"29.1": "sha256-J5fNVlyn/7/ALaVsygE7/0+X5oMkizuZ3PfUuxf27GE=",
6357
"29.0": "sha256-LhXZqwaFbCXKbeYi4RR4XZGHHb25HCwQPrYouH8m3Ew=",
64-
"28.3": "sha256-jhtvqCX7CVlqiS5d6bgRLXoJtL3ftxzLv1+GgKT1aKc=",
65-
"28.2": "sha256-ESIsQ4+G6Hsv2vaKKmzB2ytiKBP9btiRUJ4vOw4F7hs=",
6658
},
6759
},
6860
"linux-x86_32": {
@@ -76,8 +68,6 @@ PROTOC_BUILDS = {
7668
"29.2": "sha256-FU+NR+6YO8bxoa4tsUl+53E8Qt7vY1EXDxhmG/vNouw=",
7769
"29.1": "sha256-nd/EAbEqC4dHHg4POg0MKpByRj8EFDUUnpSfiiCCu+s=",
7870
"29.0": "sha256-tKyBCfKrSGLV5WuH4/cVMPug46YeyLnZQOKUdskGAQE=",
79-
"28.3": "sha256-DJ6zLLnl06rHLGfPc38In+Mr8kVssBc+EpWQfNGIXhI=",
80-
"28.2": "sha256-ucjToo5Lq5WcwQ5smjjxfFlGc3Npv+AT9RqG19Ns//E=",
8171
},
8272
},
8373
"linux-x86_64": {
@@ -91,8 +81,6 @@ PROTOC_BUILDS = {
9181
"29.2": "sha256-Uunn7OVcfjDn6LvSVLSyG0CKUwm8qCZ2PHEktpahMuk=",
9282
"29.1": "sha256-AMg/6XIthelsgblBsp8Xp0SzO0zmbg8YAJ/Yk33iLGA=",
9383
"29.0": "sha256-PFEGWvO5pgbZ4Yob9igUNzT/S55pcl1kWYV0MLp6eN8=",
94-
"28.3": "sha256-CtlJ8EpqF02oPNy9s23uCkklJypbbYP3mmv5hSB21T8=",
95-
"28.2": "sha256-L+v9QrWc6Too63iQGaRwo90ESWGbwE+E2tEzPaJh3sE=",
9684
},
9785
},
9886
"osx-aarch_64": {
@@ -106,8 +94,6 @@ PROTOC_BUILDS = {
10694
"29.2": "sha256-DhU6ONbaGVlMmA5/fNPqDd1SydoQaMA8DYUzNp+/6yA=",
10795
"29.1": "sha256-uP1ZdpJhmKfE6lxutL94lZ1frtJ7/GGCVMqhBD93BEU=",
10896
"29.0": "sha256-srWfA7AwyKdIYj1oKotbycwJnkvP0GuJZM6J7AZbMQM=",
109-
"28.3": "sha256-ks7v2mpyk+wBTm7KyC1kcZNXFFy2/ChlutreteYsBDE=",
110-
"28.2": "sha256-e7BI9ShBeJ2exhmDvgzkyeT7O9mhQ0YoILqaO+CgN5c=",
11197
},
11298
},
11399
"osx-x86_64": {
@@ -121,8 +107,6 @@ PROTOC_BUILDS = {
121107
"29.2": "sha256-uivZg7XwbsONZjtgKISll96jmQpDgD1+FT7Y98VCaeE=",
122108
"29.1": "sha256-2wK0uG3k1MztPqmTQ0faKNyV5/OIY//EzjzCYoMCjaY=",
123109
"29.0": "sha256-56HP/ILiHapngzARRJxw3f8eujsRWTQ4fm6BQe+rCS8=",
124-
"28.3": "sha256-l/5dRCCQtNvCPNE4T7m0RPodxuZ9FbteH+TeDadjiyA=",
125-
"28.2": "sha256-Iy8H0Sv0gGIHp57CxzeDAcUuby9+/dIcDdQW8L2hA+w=",
126110
},
127111
},
128112
"win32": {
@@ -136,8 +120,6 @@ PROTOC_BUILDS = {
136120
"29.2": "sha256-73CfcaUbOompsm3meCBc7zyV4h0IGLFn+/WwOackr9E=",
137121
"29.1": "sha256-EQXg+mRFnwsa9e5NWHfauGTy4Q2K6wRhjytpxsOm7QM=",
138122
"29.0": "sha256-154nzOTEAXRUERc8XraBFsPiBACxzEwt0stHeneUGP4=",
139-
"28.3": "sha256-sI/m/M9DE+LMxv1ybchV7okFM+JH9MVDyYf/2YDP1bI=",
140-
"28.2": "sha256-V6hpbqvtUgl19PpGkUGwuC+iEIYj0YtyOqi+yae0u1g=",
141123
},
142124
},
143125
"win64": {
@@ -151,8 +133,6 @@ PROTOC_BUILDS = {
151133
"29.2": "sha256-Weph77JLnYohQXHiyj/sVcPxUX7/BnZWyHXYoc0Gzk8=",
152134
"29.1": "sha256-fqSCJYV//BIkWIwzXCsa+deKGK+dV8BSjMoxk+M26c4=",
153135
"29.0": "sha256-0DuSGYWLikyogGO3i/Clzec7UYCLkwxLZvBuhILDq+Y=",
154-
"28.3": "sha256-zmT0m97d70nOS9MTqPWbz5L89ntYMe+/ZhcDhtLmaUg=",
155-
"28.2": "sha256-S94ZJx7XyrkANXDyjG5MTXGWPq8SEahr87sl2biVF3o=",
156136
},
157137
},
158138
}

scripts/update_protoc_integrity.py

-2
Original file line numberDiff line numberDiff line change
@@ -32,8 +32,6 @@
3232
"29.2",
3333
"29.1",
3434
"29.0",
35-
"28.3",
36-
"28.2",
3735
]
3836

3937
PROTOC_RELEASES_URL = "https://github.com/protocolbuffers/protobuf/releases"

0 commit comments

Comments
 (0)