Release 2.12.15

[SethTisue]

SCALA_VER_BASE="2.12.15"
SCALA_VER_SUFFIX=""
SCALA_SHA=96a666989b6bee067b1029553c6684ef1cb6f6b1
DIST_SHA=72e412c54705d343a892282a0f80dc0eb69767a5
SCALA_VER="$SCALA_VER_BASE$SCALA_VER_SUFFIX"

Key links:

• scala/scala milestone: https://github.com/scala/scala/milestones/2.12.15
• scala/bug milestone: https://github.com/scala/bug/milestones/2.12.15
• scala/scala-dev milestone: https://github.com/scala/scala-dev/milestones/2.12.15
• Discourse topic: https://contributors.scala-lang.org/t/scala-2-12-15-planning/5152/5
• release notes draft: draft 2.12.15 release notes #784

N weeks before the release

• Wind down PR queue. There has to be enough time after the last (non-trivial) PR is merged and the next phase. The core of the eco-system needs time to prepare for the final!
• Triage scala/bug and scala/scala-dev tickets
• Create next scala/scala milestone, move the magical "Merge to 2.13.x" description to it (so Scabot uses it as default for new PRs), move pending PRs
• Create next scala/bug milestone, move pending issues
• Create next scala/scala-dev milestone, move pending issues
• Check PRs assigned to the milestone, also check WIP
• Announce expected release date and current nightly "release candidate" (nightly sha-mangled version) at https://scala-ci.typesafe.com/artifactory/scala-integration/ on https://contributors.scala-lang.org/c/announcements
• Also notify Scala Center advisory board members of the upcoming release, so they can help test if they want (Seth can handle this, if asked)

Release announcement / notes

• Notify community on https://contributors.scala-lang.org/c/announcements
• Review merged PRs, make sure release-notes label is applied appropriately
• PRs with release-notes label must have excellent title & description (title will be pasted literally in release note bullet list)
• Draft release notes (PR and self-merge, so others can comment there rather than on the commits)
  • Starting point: gh api --paginate -X GET search/issues -f q='repo:scala/scala is:pull-request is:merged milestone:2.12.14 label:release-notes' -q '.items[] | " * \(.title) ([#\(.number)](\(.html_url)) by [@\(.user.login)](\(.user.html_url)))"'
• On contributors thread, link to release note file and request feedback

N days before release

• Announce no more PRs will be merged unless last-minute regressions are found. Re-iterate current nightly sha version for testing.
• Community build
  • JDK 8: https://scala-ci.typesafe.com/job/scala-2.12.x-jdk8-integrate-community-build/6928/
  • JDK 11: https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk11-integrate-community-build/2100/
  • JDK 17: https://scala-ci.typesafe.com/job/scala-2.12.x-jdk17-integrate-community-build/120/
• Windows build on GitHub Actions: https://github.com/scala/scala/actions/runs/1174644991
• Check any merged PRs accidentally assigned to the next milestone in this branch, and re-assign them to this milestone
• Merge in any older release branch
• Check module versioning (is everything in versions.properties up to date?)
  • including make sure the version of scala-asm we're using is using latest ASM
• On major release, bump PickleFormat version
• Close the scala/scala and scala/bug milestones

[SethTisue]

Point of no return

Once sufficient time has passed since last merged PR, it's time to cut the release!

How long we wait depends on what kind of release it is. For a major release, it might be 1-2 weeks, to give core projects time to try out the preceding release candidate and/or the current candidate nightly. For point releases, assuming we've given the community ahead-of-time warning and kept them appraised of progress on the Discourse thread, and assuming the last changes merged seem sufficiently safe, we might build the next day.

Be mindful of others' schedules; even minor releases make work downstream (for Scala.js and Scala Native, for the Scala 3 team, for compiler plugin authors, and so on). It's better not to release on Friday or before a holiday.

• Make sure there are no stray staging repos on sonatype
• Trigger a custom build on travis
  • Select the correct branch
  • Custom config: before_script: export SCALA_VER_BASE=2.12.15 SCALA_VER_SUFFIX=
  • https://app.travis-ci.com/github/scala/scala/jobs/537392523
  • Check the build status on https://github.com/scala/scala/commits/2.12.x
  • Check that the scala/scala job also triggered a following scala/scala-dist job: https://app.travis-ci.com/github/scala/scala-dist/builds/?
  • Create the scala/scala tag locally: git tag -s -m "Scala $SCALA_VER" v$SCALA_VER $SCALA_SHA
  • Create scala-dist tag locally: git tag -s -m "Scala $SCALA_VER" v$SCALA_VER $DIST_SHA
  • Note the repos to be promoted after tag is cut (see travis log)
    • https://oss.sonatype.org/content/repositories/orgscala-lang-????
    • https://oss.sonatype.org/content/repositories/orgscala-lang-????
• Sanity check jar/pom
  • https://oss.sonatype.org/content/repositories/staging/org/scala-lang/scala-compiler/$SCALA_VER/
  • in particular, if the release was staged multiple times, double check that https://oss.sonatype.org/content/repositories/staging/ has the files from the most recent build
• Check that JARs haven't mysteriously bloated — compare sizes to previous release. We have no other backstop for this.
• Remember, tags are forever, so are maven artifacts (even staged ones, as they could end up in local caches) and S3 uploads (S3 buckets can be changed, but it can takes days to become consistent)
• Push scala/scala tag: git push https://github.com/scala/scala.git v$SCALA_VER
• Add release notes to https://github.com/scala/scala/releases/tag/v2.12.15
• Push scala/scala-dist tag: git push https://github.com/scala/scala-dist.git v$SCALA_VER
• Trigger two scala-dist jobs on travis (https://travis-ci.com/scala/scala-dist) with custom config. must use full-length SHAs!
  • before_script: export version=2.12.15 scala_sha=96a666989b6bee067b1029553c6684ef1cb6f6b1 mode=archives: https://app.travis-ci.com/github/scala/scala-dist/builds/237658942
  • before_script: export version=2.12.15 scala_sha=96a666989b6bee067b1029553c6684ef1cb6f6b1 mode=update-api: https://app.travis-ci.com/github/scala/scala-dist/builds/237658990
• Promote staging repos: st_stagingRepoPromote [scala-repo], st_stagingRepoPromote [modules-repo] (or use oss.sonatype.org web UI)

Check availability

• Check release on sonatype: https://oss.sonatype.org/content/repositories/releases/org/scala-lang/scala-compiler/2.12.15/
• Check the release on maven central: https://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.12.15/

When everything is on maven central

• Prepare PR to https://github.com/scala/scala-lang/ (using scala/make-release-notes which requires a staged release and a pushed tag)
  • download/index.md
  • _config.yml (update devscalaversion or scalaversion)
  • index.md (update currentScalaVersion)
• Prepare PR to https://github.com/scala/docs.scala-lang/
  • api/all.md, _config.yml, _overviews/jdk-compatibility/overview.md
• make sure the draft release notes are on GitHub tag: https://github.com/scala/scala/releases/tag/v2.12.15
• Pre-announce the release on https://contributors.scala-lang.org/c/announcements
• On 2.13.0 only: (manually) update the current symlink for the API docs
  • https://github.com/scala/scala-dist/blob/2.13.x/scripts/jobs/release/website/update-api#L15
• Check that the API docs are published
  • http://www.scala-lang.org/api/2.12.15/
  • https://docs.scala-lang.org/api/all.html ?
  • if they don't show up, possible troubleshooting steps include:
    • review the two scala-dist job logs to make sure that
      • the first one appears to have succeeded putting files in /home/linuxsoft/archives/scala/api on chara.epfl.ch
      • the second one appears to have succeeded in updating the symlink (from 2.1x.y to 2.12.15)
    • ssh to chara.epfl.ch and poke around to see if things are where they should be
      • if you don't have the credential for this locally but you are able to bring jenkins-worker-publish up at ssh jenkins-worker-publish, then from there you can ssh -i ~/.ssh/jenkins_lightbend_chara [email protected]
    • see if https://scala-webapps.epfl.ch/jenkins/view/All/job/production_scala-lang.org-scala-dist-archive-sync/ has run a job yet to sync the changes into production
      • if not, you can manually trigger a job. Seth has access to do that, probably others on the team do too. if we get stuck, Fabien can help
• Merge the scala-lang PR and the docs.scala-lang.org PR
  • wait for them to arrive on the websites and make sure they look okay
    • if the scala-lang changes don't show up, possible troubleshooting steps include:
      • see if https://scala-webapps.epfl.ch/jenkins/view/All/job/production_scala-lang.org-builder/ has run a job yet to actually publish the changes
      • see note above about permissions to trigger a job

Modules

• build and release scala-collection-compat and other modules (or open tickets asking that the maintainers do so)
  • this work has moved to https://github.com/scala/make-release-notes/blob/2.13.x/projects-2.13.md
• if it's a 2.12.x release, publish macro paradise for the new version

Announcements

• Scala Users discourse https://users.scala-lang.org
• add a reply on the same https://contributors.scala-lang.org thread where you announced that the release process was starting
• Tweet from @scala_lang
• https://gitter.im/scala/scala
  • if you feel like it, say something in https://gitter.im/scala/contributors too
• ask Seth to announce on #scala IRC

Afterwards

• Check the release on maven search (may take a few hours):
  • https://search.maven.org/#search%7Cga%7C1%7Cg%3Aorg.scala-lang%20v%3A2.12.15
• Create PR to add/update spec links on scala-lang.org (example: update spec links and advisements scala-lang#1050)
• Create a scala/scala PR to:
  • update starr.version in /versions.properties
  • update Global / baseVersion in /build.sbt
  • update mimaReferenceVersion in /project/MimaFilters.scala
  • clear out mimaFilters in /project/MimaFilters.scala, except the one(s) labeled "KEEP"
  • spec/_config.yml, if it's a major bump
• If it's a major bump:
  • Update latestSpecVersion in spec/_config.yml on the old branch, so that spec is marked as no longer current
  • Ditto for the nightly build and spec links in _data/footer.yml and _data/doc-nav-header.yml on docs.scala-lang.org
• Create PR to update https://github.com/lightbend/lightbend-technology-intro-doc/blob/master/docs/modules/getting-help/pages/build-dependencies.adoc
• Consider updating https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html
• (Lightbend) Publish scala-fortify-plugin
• (Lightbend) Update WhiteSource
• (Lightbend) Notify eng-updates
• Close this ticket and close the scala-dev milestone

[SethTisue]

publishing failed, some Sonatype problem that isn't obvious to me. I'll investigate, but it's no longer feasible to publish this week. hopefully next week

[SethTisue]

the failed build on Sep 1: https://app.travis-ci.com/github/scala/scala/jobs/535138405

the error:

[error] java.net.ProtocolException: Server redirected too many  times (20)
[error] 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[error] 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
...
[error] 	at sbt.internal.librarymanagement.IvyActions$.publish(IvyActions.scala:501)

successful 2.12.14 publishing run, for comparison: https://app.travis-ci.com/github/scala/scala/jobs/508816266

for diffing purposes, it helps to pipe the raw logs through gsed 's/\x1b\[[0-9;]*m//g' | grep -v ^travis_time:

but actually the diffs don't give any clues. maybe I need to run with sbt -debug

this is pretty rough to troubleshoot since the build takes so long. it's tempting to try to publish from my local machine instead (without having to rebuild everything every time) and hope to learn something from that -- though that won't help if the problem has to do with the encrypted secrets on Travis-CI

[SethTisue]

note that I published scala-async from Travis-CI last week and that went fine, so there isn't an org-wide issue

note that since 2.12.14, we upgraded from sbt 1.3.13 to 1.5.5 as part of an effort to backport build changes from 2.13 to 2.12

perhaps I should do a trial run of the 2.13.7 release (and drop the staging repos, of course!) to see if the same problem occurs there

another possible source of clues is to re-run the 2.12.14 and/or 2.13.6 releases and see if they succeed (and then drop the staging repos)

[SethTisue]

another end to approach this from is "what could cause this error?"

at xerial/sbt-sonatype#214 someone got the same error because of oss.sonatype.org vs s01.oss.sonatype.org, but we're publishing from a longstanding Sonatype account and not a recently created one, so oss.sonatype.org should still be right. and anyway the 2.12.14 release was in May 2021 which is after Sonatype started expecting new accounts to use the new host (in February 2021)

at sbt/sbt-pgp#182 someone got the same error simply because they were using wrong credentials. if we were setting up publishing in the scala/scala repo for the first time, wrong credentials is the first explanation I'd consider, but in this case, we know that everything was set up right before, so how could it be wrong now? regardless, maybe I need to accept that it's somehow a wrong-credentials problem and set the credentials up again

I think the next thing I'll try is a dry run of 2.13.7 publishing, since that will tell us if the problem is repo-wide or is somehow specific to the 2.12.x branch

[SethTisue]

2.13.x build triggered with before_script: export SCALA_VER_BASE=2.13.7 SCALA_VER_SUFFIX=-M1 at https://app.travis-ci.com/github/scala/scala/builds/237558813

(but it might take a while because we're now down to only one concurrent Travis-CI job in scala/* and a PR validation job is running)

[SethTisue]

OMG might this just be #764 ...? I HOPE SO

it seems like a distant hope since the symptoms here aren't the same as those at #762 , but regardless: scala/scala#9758

UPDATE: never mind

[SethTisue]

2.13.x build triggered [...]

It failed with the same error as we're getting on 2.12.x. That narrows down the possible causes — we no longer need to consider 2.12-specific causes. I'm not sure whether that's good news or bad news :-), but I guess any narrowing is good 🤷

[SethTisue]

Now that we know both 2.12 and 2.13 are broken, I'm contemplating just replacing the publishing secrets and hope that helps. (Before, I was afraid to do that for fear of breaking 2.13 too.)

I don't know enough about PGP stuff to know whether it's plausible that we're using an expired key or something like that.

[SethTisue]

I reviewed the diffs on 2.13.x since v2.13.6, and scala/scala#9658 stood out as a plausible culprit. (Jason also suggested at team meeting today that this PR might be relevant.)

9658 was backported to 2.12.x (scala/scala#9659), so that's consistent with both branches being broken.

I think I'll try simply reverting the 2.12.x backport and hope that allows us to get 2.12.15 out the door. Then with that done, we can slow down and try and understand what went wrong, in time for 2.13.7.

[SethTisue]

pushed the reversion to a test-revert-9659 branch and kicked off https://app.travis-ci.com/github/scala/scala/builds/237568849 with before_script: export SCALA_VER_BASE=2.12.15 SCALA_VER_SUFFIX=-M1

[SethTisue]

gah that didn't help 🙀

maybe the credentials really are expired/bad somehow

[SethTisue]

this seems suspicious:

$ (cd admin && ./init.sh)
/home/travis/.gnupg/pubring.gpg
-------------------------------
pub   4096R/6D92E560 2018-02-13 [expired: 2020-02-13]
uid                  Scala Project <[email protected]>

/home/travis/.gnupg/secring.gpg
-------------------------------
sec#  4096R/6D92E560 2018-02-13 [expires: 2020-02-13]
uid                  Scala Project <[email protected]>
ssb   4096R/D692BCFF 2018-02-14

but if expires: 2020-02-13 were a problem, surely that would have caused release failures last year, not this year? the same thing appears in the successful 2.12.14 log

and anyway if it was a GPG key problem we'd get some other kind of error, I think

[SethTisue]

https://app.travis-ci.com/github/scala/scala/builds/237577842 is a re-run of the 2.13.6 publishing and fails the same way

plus I had already done a similar 2.12.x experiment at https://app.travis-ci.com/github/scala/scala/jobs/535148973 (I don't recall how I picked the commit I used, but it was around 2.12.14 release time) with the same result

so it looks like I can stop groveling around trying to identify what commit or pull request is the culprit and just redo the secrets... though I still wish I had a theory about what changed to make this stop working, since if I redo the secrets and it still doesn't work we'll still be right on square one

[SethTisue]

We have secrets in two places: in the Travis-CI web UI, and in .travis.yml itself

The ones in the Travis-CI web UI are encrypted_40abf89dbafd_iv and encrypted_40abf89dbafd_iv and I think they have to do with publishing the spec? Not sure. (But I really doubt these are in play here.)

In .travis.yml we have:

• PRIV_KEY_SECRET: for spec publishing
• TRAVIS_TOKEN: for triggering scala-dist via the Travis-CI API
• GPG_SUBKEY_SECRET: for signing JARs
• PRIVATE_REPO_PASS: for publishing to scala-ci Artifactory (as opposed to Sonatype)
• SONA_USER, SONA_PASS: Sonatype

afaics only SONA_* are relevant here.

I don't know whether they were set up to use the lamp username or password, or that account's API token (the "username" and "password" of the token are accessible, if you know the account password, via the Sonatype web UI). I always use the token when setting up module publishing (as per advice in sbt-ci-release readme), so I guess I'll use that here as well.

[SethTisue]

redid the secrets (following https://docs.travis-ci.com/user/encryption-keys/) as follows:

• travis encrypt --add -r scala/scala and then entered SONA_USER=.... at keyboard
• ditto for SONA_PASS=...

pushed to redo-secrets branch and triggered https://app.travis-ci.com/github/scala/scala/jobs/537206661

at the top of the log I see this, as expected:

$ export SONA_USER=[secure]
$ export SONA_PASS=[secure]

20 minutes later:

Closed sonatype staging repos:
https://oss.sonatype.org/content/repositories/orgscala-lang-2824
https://oss.sonatype.org/content/repositories/orgscala-lang-2825
https://oss.sonatype.org/content/repositories/orgscala-lang-2826
https://oss.sonatype.org/content/repositories/orgscala-lang-2827.

🎉

and I shall not throw my laptop into the sea after all.... not today, anyway

[SethTisue]

I dropped the staging repos. I'll re-run this after merging scala/scala#9759 . (Maybe it's paranoia talking, but I feel more comfortable having the real release come from an actual merge commit on the 2.12.x branch.)