Skip to content

Commit 6ec2f22

Browse files
sgilmore10kou
sgilmore10
and
kou
authored
apacheGH-41102: [Packaging][Release] Create unique git tags for release candidates (e.g. apache-arrow-{MAJOR}.{MINOR}.{PATCH}-rc{RC_NUM}) (apache#41131)
### Rationale for this change As per @ kou's [suggestion](apache#40956 (comment)) in apache#40956, we should create unique git tags (e.g. `apache-arrow-{MAJOR}.{MINOR}.{VERSION}-rc{RC_NUM}`) instead re-using the same git tag (`apache-arrow-{MAJOR}.{MINOR}.{VERSION}`) for each release candidate. The official release candidate tag (`apache-arrow-{MAJOR}.{MINOR}.{VERSION}`) should be created **only** after a release candidate is voted on and accepted. This "official" release tag should point to the same object in the git database as the accepted release candidate tag. The new release workflow could look like the following: > 1. Create a apache-arrow-X.Y.Z-rc0 tag for X.Y.Z RC0 > 2. (Found a problem for X.Y.Z RC0) > 3. Create a apache-arrow-X.Y.Z-rc1 tag for X.Y.Z RC1 > 4. Vote > 5. Passed > 6. Create a apache-arrow-X.Y.Z tag from apache-arrow-X.Y.Z-rc1 ike apache/arrow-adbc and apache/arrow-flight-sql-postgresql do See @ kou's [comment](apache#40956 (comment)) for more details. ### What changes are included in this PR? 1. Updated `dev/release/01-prepare.sh` to create release-candidate-specific git tags (e.g. `apache-arrow-{MAJOR}.{MINOR}.{PATCH}-rc{RC_NUM}`). 2. Updated scripts in `dev/release` to use the new git tag name. 3. Added GitHub Workflow file `publish_release_candidate.yml`. This workflow is triggered when a release candidate git tag is pushed and creates a Prerelease GitHub Release. 4. Added logic to `dev/release/02-post-binary.sh` to create and push the release git tag (i.e. `apache-arrow-{MAJOR}.{MINOR}.{PATCH}`). 5. Added GitHub Workflow `publish_release.yml`. This workflow is triggered when the release tag is pushed and creates a GitHub Release for the approved release (i.e. the voted upon release). 6. Added `dev/release/post-16-delete-release-candidates.sh` to delete the release candidate git tags and their associated GitHub Releases. 7. Updated `docs/developers/release.rst` with the new steps. ### Are these changes tested? 1. We were not able to verify the changes made to the scripts in `dev/release`. Any suggestions on how we can verify these scripts would be much appreciated :) 2. We did test the new GitHub Workflows (`publish_release_candidate.yml` and `publish_release.yml`) work as intended by pushing git tags to [`mathworks/arrow`](https://github.com/mathworks/arrow). ### Are there any user-facing changes? No. ### Open Questions 1. We noticed that [apache/arrow-flight-sql-postgresql](https://github.com/apache/arrow-flight-sql-postgresql/releases) does **not** delete the release candidate Prereleases from their GitHub Releases area. Should we be doing the same? Or would it be preferable to just delete the the release candidates **without** deleting the release candidate tags. 2. We're not that familiar with ruby, so we're not sure if the changes we made to `dev/release/02-source-test.rb` make sense. ### Future Directions 1. Continue working on apache#40956 2. Add logic to auto-sign release artifacts in GitHub Actions Workflows. * GitHub Issue: apache#41102 Lead-authored-by: Sarah Gilmore <[email protected]> Co-authored-by: Sarah Gilmore <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sarah Gilmore <[email protected]>
1 parent 680980e commit 6ec2f22

30 files changed

+523
-90
lines changed

‎.github/workflows/release.yml

+74
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: Release
19+
20+
on:
21+
push:
22+
tags:
23+
# Trigger workflow when a tag whose name matches the pattern
24+
# pattern "apache-arrow-{MAJOR}.{MINOR}.{PATCH}" is pushed.
25+
- "apache-arrow-[0-9]+.[0-9]+.[0-9]+"
26+
27+
permissions:
28+
contents: write
29+
30+
env:
31+
GH_TOKEN: ${{ github.token }}
32+
33+
jobs:
34+
publish:
35+
name: Publish
36+
runs-on: ubuntu-latest
37+
timeout-minutes: 5
38+
steps:
39+
- name: Get Tag Name of Latest Release Candidate
40+
run: |
41+
rc_tag=$(gh release list --repo apache/arrow | \
42+
cut -f3 | \
43+
grep -F "${GITHUB_REF_NAME}-rc" | \
44+
head -n1)
45+
echo "Latest Release Candidate Tag: ${rc_tag}"
46+
echo "RELEASE_CANDIDATE_TAG_NAME=${rc_tag}" >> ${GITHUB_ENV}
47+
- name: Store Version and Release Candidate Number
48+
run: |
49+
version_with_rc=${RELEASE_CANDIDATE_TAG_NAME#apache-arrow-}
50+
version=${version_with_rc%-rc*}
51+
rc_num=${version_with_rc#${version}-rc}
52+
echo "VERSION_WITH_RC=${version_with_rc}" >> ${GITHUB_ENV}
53+
echo "VERSION=${version}" >> ${GITHUB_ENV}
54+
echo "RC_NUM=${rc_num}" >> ${GITHUB_ENV}
55+
- name: Download Release Candidate Artifacts
56+
run: |
57+
mkdir release_candidate_artifacts
58+
gh release download ${RELEASE_CANDIDATE_TAG_NAME} --repo apache/arrow --dir release_candidate_artifacts
59+
- name: Create Release Title
60+
run: |
61+
title="Apache Arrow ${VERSION}"
62+
echo "RELEASE_TITLE=${title}" >> ${GITHUB_ENV}
63+
# Set the release notes to "TODO" temporarily. After the release notes page
64+
# (https://arrow.apache.org/release/{VERSION}.html) is published, use
65+
# gh release edit to update the release notes to refer to the newly
66+
# pushed web page. See dev/post/post-05-update-gh-release-notes.sh
67+
- name: Create GitHub Release
68+
run: |
69+
gh release create ${GITHUB_REF_NAME} \
70+
--repo apache/arrow \
71+
--verify-tag \
72+
--title "${RELEASE_TITLE}" \
73+
--notes "TODO" \
74+
release_candidate_artifacts/*

‎.github/workflows/release_candidate.yml

+70
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: Release
19+
20+
on:
21+
push:
22+
tags:
23+
# Trigger workflow when a tag whose name matches the pattern
24+
# "apache-arrow-{MAJOR}.{MINOR}.{PATCH}-rc{RC_NUM}" is pushed.
25+
- "apache-arrow-[0-9]+.[0-9]+.[0-9]+-rc[0-9]+"
26+
27+
permissions:
28+
contents: write
29+
30+
env:
31+
GH_TOKEN: ${{ github.token }}
32+
33+
jobs:
34+
publish:
35+
name: Publish
36+
runs-on: ubuntu-latest
37+
timeout-minutes: 5
38+
steps:
39+
- name: Checkout Arrow
40+
uses: actions/checkout@v4
41+
with:
42+
fetch-depth: 0
43+
- name: Store Version and Release Candidate Number
44+
run: |
45+
version_with_rc=${GITHUB_REF_NAME#apache-arrow-}
46+
version=${version_with_rc%-rc*}
47+
rc_num=${version_with_rc#${version}-rc}
48+
echo "VERSION_WITH_RC=${version_with_rc}" >> ${GITHUB_ENV}
49+
echo "VERSION=${version}" >> ${GITHUB_ENV}
50+
echo "RC_NUM=${rc_num}" >> ${GITHUB_ENV}
51+
- name: Create Release Candidate Title
52+
run: |
53+
title="Apache Arrow ${VERSION} RC${RC_NUM}"
54+
echo "RELEASE_CANDIDATE_TITLE=${title}" >> ${GITHUB_ENV}
55+
- name: Create Release Candidate Notes
56+
run: |
57+
release_notes="Release Candidate: ${VERSION} RC${RC_NUM}"
58+
echo "RELEASE_CANDIDATE_NOTES=${release_notes}" >> ${GITHUB_ENV}
59+
- name: Create Release tarball
60+
run: |
61+
cd dev/release/ && ./utils-create-release-tarball.sh ${VERSION} ${RC_NUM}
62+
echo "RELEASE_TARBALL=apache-arrow-${VERSION}.tar.gz" >> ${GITHUB_ENV}
63+
- name: Create GitHub Release
64+
run: |
65+
gh release create ${GITHUB_REF_NAME} \
66+
--verify-tag \
67+
--prerelease \
68+
--title "${RELEASE_CANDIDATE_TITLE}" \
69+
--notes "Release Notes: ${RELEASE_CANDIDATE_NOTES}" \
70+
dev/release/${RELEASE_TARBALL}

‎dev/release/01-prepare.sh

+6-6
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ next_version=$2
3333
next_version_snapshot="${next_version}-SNAPSHOT"
3434
rc_number=$3
3535

36-
release_tag="apache-arrow-${version}"
36+
release_candidate_tag="apache-arrow-${version}-rc${rc_number}"
3737
release_branch="release-${version}"
3838
release_candidate_branch="release-${version}-rc${rc_number}"
3939

@@ -46,9 +46,9 @@ release_candidate_branch="release-${version}-rc${rc_number}"
4646
: ${PREPARE_TAG:=${PREPARE_DEFAULT}}
4747

4848
if [ ${PREPARE_TAG} -gt 0 ]; then
49-
if [ $(git tag -l "${release_tag}") ]; then
50-
echo "Delete existing git tag $release_tag"
51-
git tag -d "${release_tag}"
49+
if [ $(git tag -l "${release_candidate_tag}") ]; then
50+
echo "Delete existing git tag $release_candidate_tag"
51+
git tag -d "${release_candidate_tag}"
5252
fi
5353
fi
5454

@@ -88,7 +88,7 @@ if [ ${PREPARE_LINUX_PACKAGES} -gt 0 ]; then
8888
fi
8989

9090
if [ ${PREPARE_VERSION_PRE_TAG} -gt 0 ]; then
91-
echo "Prepare release ${version} on tag ${release_tag} then reset to version ${next_version_snapshot}"
91+
echo "Prepare release ${version} on tag ${release_candidate_tag} then reset to version ${next_version_snapshot}"
9292

9393
update_versions "${version}" "${next_version}" "release"
9494
git commit -m "MINOR: [Release] Update versions for ${version}"
@@ -97,5 +97,5 @@ fi
9797
############################## Tag the Release ##############################
9898

9999
if [ ${PREPARE_TAG} -gt 0 ]; then
100-
git tag -a "${release_tag}" -m "[Release] Apache Arrow Release ${version}"
100+
git tag -a "${release_candidate_tag}" -m "[Release] Apache Arrow Release ${version} RC${rc_number}"
101101
fi

‎dev/release/02-source-test.rb

+5-2
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,10 @@ class SourceTest < Test::Unit::TestCase
2222
def setup
2323
@current_commit = git_current_commit
2424
detect_versions
25-
@tag_name = "apache-arrow-#{@release_version}"
25+
@tag_name = "apache-arrow-#{@release_version}-rc0"
26+
@archive_name = "apache-arrow-#{@release_version}.tar.gz"
2627
@script = File.expand_path("dev/release/02-source.sh")
28+
@tarball_script = File.expand_path("dev/release/utils-create-release-tarball.sh")
2729

2830
Dir.mktmpdir do |dir|
2931
Dir.chdir(dir) do
@@ -40,8 +42,9 @@ def source(*targets)
4042
targets.each do |target|
4143
env["SOURCE_#{target}"] = "1"
4244
end
45+
sh(env, @tarball_script, @release_version, "0")
4346
output = sh(env, @script, @release_version, "0")
44-
sh("tar", "xf", "#{@tag_name}.tar.gz")
47+
sh("tar", "xf", @archive_name)
4548
output
4649
end
4750

‎dev/release/02-source.sh

+23-36
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
set -eu
2222

2323
: ${SOURCE_DEFAULT:=1}
24+
: ${SOURCE_DOWNLOAD:=${SOURCE_DEFAULT}}
2425
: ${SOURCE_RAT:=${SOURCE_DEFAULT}}
2526
: ${SOURCE_UPLOAD:=${SOURCE_DEFAULT}}
2627
: ${SOURCE_PR:=${SOURCE_DEFAULT}}
@@ -37,11 +38,10 @@ fi
3738
version=$1
3839
rc=$2
3940

40-
tag=apache-arrow-${version}
41+
tag=apache-arrow-${version}-rc${rc}
4142
maint_branch=maint-${version}
4243
rc_branch="release-${version}-rc${rc}"
43-
tagrc=${tag}-rc${rc}
44-
rc_url="https://dist.apache.org/repos/dist/dev/arrow/${tagrc}"
44+
rc_url="https://dist.apache.org/repos/dist/dev/arrow/${tag}"
4545

4646
echo "Preparing source for tag ${tag}"
4747

@@ -56,35 +56,19 @@ fi
5656

5757
echo "Using commit $release_hash"
5858

59-
tarball=${tag}.tar.gz
60-
61-
rm -rf ${tag}
62-
# be conservative and use the release hash, even though git produces the same
63-
# archive (identical hashes) using the scm tag
64-
(cd "${SOURCE_TOP_DIR}" && \
65-
git archive ${release_hash} --prefix ${tag}/) | \
66-
tar xf -
67-
68-
# Resolve all hard and symbolic links.
69-
# If we change this, we must change ArrowSources.archive in
70-
# dev/archery/archery/utils/source.py too.
71-
rm -rf ${tag}.tmp
72-
mv ${tag} ${tag}.tmp
73-
cp -R -L ${tag}.tmp ${tag}
74-
rm -rf ${tag}.tmp
75-
76-
# Create a dummy .git/ directory to download the source files from GitHub with Source Link in C#.
77-
dummy_git=${tag}/csharp/dummy.git
78-
mkdir ${dummy_git}
79-
pushd ${dummy_git}
80-
echo ${release_hash} > HEAD
81-
echo '[remote "origin"] url = https://github.com/apache/arrow.git' >> config
82-
mkdir objects refs
83-
popd
84-
85-
# Create new tarball from modified source directory
86-
tar czf ${tarball} ${tag}
87-
rm -rf ${tag}
59+
tarball=apache-arrow-${version}.tar.gz
60+
61+
if [ ${SOURCE_DOWNLOAD} -gt 0 ]; then
62+
# Wait for the release candidate workflow to finish before attempting
63+
# to download the tarball from the GitHub Release.
64+
. $SOURCE_DIR/utils-watch-gh-workflow.sh ${tag} "release_candidate.yml"
65+
rm -f ${tarball}
66+
gh release download \
67+
${tag} \
68+
--repo apache/arrow \
69+
--dir . \
70+
--pattern "${tarball}"
71+
fi
8872

8973
if [ ${SOURCE_RAT} -gt 0 ]; then
9074
"${SOURCE_DIR}/run-rat.sh" ${tarball}
@@ -105,18 +89,21 @@ if [ ${SOURCE_UPLOAD} -gt 0 ]; then
10589
${sha256_generate} $tarball > ${tarball}.sha256
10690
${sha512_generate} $tarball > ${tarball}.sha512
10791

92+
# Upload signed tarballs to GitHub Release
93+
gh release upload ${tag} ${tarball}.sha256 ${tarball}.sha512
94+
10895
# check out the arrow RC folder
10996
svn co --depth=empty https://dist.apache.org/repos/dist/dev/arrow tmp
11097

11198
# add the release candidate for the tag
112-
mkdir -p tmp/${tagrc}
99+
mkdir -p tmp/${tag}
113100

114101
# copy the rc tarball into the tmp dir
115-
cp ${tarball}* tmp/${tagrc}
102+
cp ${tarball}* tmp/${tag}
116103

117104
# commit to svn
118-
svn add tmp/${tagrc}
119-
svn ci -m "Apache Arrow ${version} RC${rc}" tmp/${tagrc}
105+
svn add tmp/${tag}
106+
svn ci -m "Apache Arrow ${version} RC${rc}" tmp/${tag}
120107

121108
# clean up
122109
rm -rf tmp

‎dev/release/03-binary-submit.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ version=$1
2828
rc=$2
2929
version_with_rc="${version}-rc${rc}"
3030
crossbow_job_prefix="release-${version_with_rc}"
31-
release_tag="apache-arrow-${version}"
31+
release_tag="apache-arrow-${version}-rc${rc}"
3232
rc_branch="release-${version_with_rc}"
3333

3434
: ${ARROW_REPOSITORY:="apache/arrow"}

0 commit comments

Comments
 (0)