Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(dev/release): Update verification instructions for forthcoming release #652

Merged
merged 3 commits into from
Oct 7, 2024
Merged

chore(dev/release): Update verification instructions for forthcoming release #652

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1ec7ae2
update versions in docs
paleolimbot Oct 7, 2024
2714f1c
better job at finding Python in dev mode
paleolimbot Oct 7, 2024
0e5334d
fix instructions
paleolimbot Oct 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 48 additions & 89 deletions dev/release/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,12 @@ Release candidates for nanoarrow are uploaded to https://dist.apache.org/repos/d
prior to a release vote being called on the
[Apache Arrow developer mailing list](https://lists.apache.org/[email protected]).
A script (`verify-release-candidate.sh`) is provided to verify such a release candidate.
For example, to verify nanoarrow 0.5.0-rc0, one could run:
For example, to verify nanoarrow 0.6.0-rc0, one could run:

```bash
git clone https://github.com/apache/arrow-nanoarrow.git arrow-nanoarrow
cd arrow-nanoarrow/dev/release
./verify-release-candidate.sh 0.5.0 0
./verify-release-candidate.sh 0.6.0 0
```

Full verification requires [CMake](https://cmake.org/download/) to build and run the test
Expand All @@ -56,19 +56,19 @@ manager except on Windows (see below).
To run only C library verification (requires CMake and Arrow C++ but not R or Python):

```bash
TEST_DEFAULT=0 TEST_C=1 TEST_C_BUNDLED=1 ./verify-release-candidate.sh 0.5.0 0
TEST_DEFAULT=0 TEST_C=1 TEST_C_BUNDLED=1 ./verify-release-candidate.sh 0.6.0 0
```

To run only R package verification (requires R but not CMake or Arrow C++):

```bash
TEST_DEFAULT=0 TEST_R=1 ./verify-release-candidate.sh 0.5.0 0
TEST_DEFAULT=0 TEST_R=1 ./verify-release-candidate.sh 0.6.0 0
```

To run only Python verification (requires Python but not CMake or Arrow C++):

```bash
TEST_DEFAULT=0 TEST_PYTHON=1 ./verify-release-candidate.sh 0.5.0 0
TEST_DEFAULT=0 TEST_PYTHON=1 ./verify-release-candidate.sh 0.6.0 0
```

### MacOS
Expand Down Expand Up @@ -120,7 +120,7 @@ conda create --name nanoarrow-verify-rc
conda activate nanoarrow-verify-rc
conda config --set channel_priority strict

conda install -c conda-forge compilers git cmake arrow-cpp
conda install -c conda-forge compilers git cmake libarrow
# For R (see below about potential interactions with system R
# before installing via conda on MacOS)
conda install -c conda-forge r-testthat r-hms r-blob r-pkgbuild r-bit64
Expand Down Expand Up @@ -148,17 +148,17 @@ the verification script.

```bash
# Build Arrow C++ from source
curl https://github.com/apache/arrow/archive/refs/tags/apache-arrow-14.0.2.tar.gz | \
curl -L https://github.com/apache/arrow/archive/refs/tags/apache-arrow-17.0.0.tar.gz | \
tar -zxf -
mkdir arrow-build && cd arrow-build
cmake ../apache-arrow-14.0.2/cpp -DCMAKE_INSTALL_PREFIX=../arrow
cmake ../apache-arrow-17.0.0/cpp -DCMAKE_INSTALL_PREFIX=../arrow-minimal
cmake --build .
cmake --install . --prefix=../arrow --config=Debug
cmake --install . --prefix=../arrow-minimal --config=Debug
cd ..

# Pass location of Arrow and R to the verification script
export NANOARROW_CMAKE_OPTIONS="-DArrow_DIR=$(pwd -W)/arrow/lib/cmake/Arrow -Dgtest_force_shared_crt=ON -DNANOARROW_ARROW_STATIC=ON"
export R_HOME="/c/Program Files/R/R-4.2.2"
export NANOARROW_CMAKE_OPTIONS="-DCMAKE_PREFIX_PATH=$(pwd -W)/arrow-minimal -Dgtest_force_shared_crt=ON -DNANOARROW_ARROW_STATIC=ON"
export R_HOME="/c/Program Files/R/R-4.4.1"
```

### Debian/Ubuntu
Expand Down Expand Up @@ -196,7 +196,7 @@ On Arch Linux (e.g., `docker run --rm -it archlinux:latest`, you can install all
using `pacman`):

```bash
pacman -S git gcc make cmake r-base gnupg curl arrow
pacman -Sy git gcc make cmake r-base gnupg curl arrow python
```

### Alpine Linux
Expand All @@ -220,47 +220,6 @@ apk add bash linux-headers git cmake R R-dev g++ gnupg curl apache-arrow-dev \
python3-dev
```

### Centos7

On Centos7 (e.g., `docker run --rm -it centos:7`), most prerequisites are
available via `yum install` except Arrow C++, which must be built from
source. Arrow C++ 9.0.0 was the last version to support the default system
compiler (gcc 4.8).

```bash
yum install epel-release # needed to install R
yum install git gnupg curl R gcc-c++ cmake3

# Needed to get a warning-free R CMD check if the en_US.UTF-8 locale is not defined
# (e.g., in the centos:7 docker image)
# localedef -c -f UTF-8 -i en_US en_US.UTF-8
# export LC_ALL=en_US.UTF-8

# Build Arrow C++ 9.0.0 from source
curl -L https://github.com/apache/arrow/archive/refs/tags/apache-arrow-9.0.0.tar.gz | tar -zxf - && \
mkdir /arrow-build && \
cd /arrow-build && \
cmake3 ../arrow-apache-arrow-9.0.0/cpp \
-DARROW_JEMALLOC=OFF \
-DARROW_SIMD_LEVEL=NONE \
-DCMAKE_INSTALL_PREFIX=../arrow && \
cmake3 --build . && \
make install

# Pass location of Arrow, cmake, and ctest to the verification script
export NANOARROW_CMAKE_OPTIONS="-DArrow_DIR=$(pwd)/arrow/lib64/cmake/arrow"
export CMAKE_BIN=cmake3
export CTEST_BIN=ctest3

# gpg on centos7 errors for some keys in the Arrow KEYS file. This does
# not skip verifying signatures, just allows errors for unsupported entries in
# the global Arrow KEYS file.
export NANOARROW_ACCEPT_IMPORT_GPG_KEYS_ERROR=1

# System Python on centos7 is not new enough to support the Python package
export TEST_PYTHON=0
```

### Big endian

One can verify a nanoarrow release candidate on big endian by setting
Expand All @@ -270,7 +229,7 @@ One can verify a nanoarrow release candidate on big endian by setting
## Creating a release candidate

The first step to creating a nanoarrow release is to create a `maint-VERSION` branch
(e.g., `usethis::pr_init("maint-0.5.0")`) and push the branch to `upstream`. This is
(e.g., `usethis::pr_init("maint-0.6.0")`) and push the branch to `upstream`. This is
a good opportunity to run though the above instructions to make sure the verification
script and instructions are up-to-date.
targeting the maint-XX branch that was just pushed.
Expand All @@ -290,11 +249,11 @@ When these steps are complete, run
```bash
# from the repository root
# 01-prepare.sh <nanoarrow-dir> <prev_veresion> <version> <next_version> <rc-num>
dev/release/01-prepare.sh . 0.4.0 0.5.0 0.6.0 0
dev/release/01-prepare.sh . 0.5.0 0.6.0 0.7.0 0
```

This will update version numbers, the changelong, and create the git tag
`apache-arrow-nanoarrow-0.5.0-rc0`. Check to make sure that the changelog
`apache-arrow-nanoarrow-0.6.0-rc0`. Check to make sure that the changelog
and versions are what you expect them to be before pushing the tag (you
may wish to do this by opening a dummy PR to run CI and look at the diff
from the main branch).
Expand All @@ -318,7 +277,7 @@ file to exist setting the appropriate `GPG_KEY_ID` environment variable.

```bash
# 02-sign.sh <version> <rc-num>
dev/release/02-sign.sh 0.5.0 0
dev/release/02-sign.sh 0.6.0 0
```

Finally, run
Expand All @@ -330,7 +289,7 @@ file to exist setting the appropriate `APACHE_USERNAME` environment variable.

```
# 03-source.sh $0 <version> <rc-num>
dev/release/03-source.sh 0.5.0 0
dev/release/03-source.sh 0.6.0 0
```

You should check that the release verification runs locally and/or
Expand All @@ -340,11 +299,11 @@ start a
At this point the release candidate is suitable for a vote on the Apache Arrow developer mailing list.

```
[VOTE] Release nanoarrow 0.5.0
[VOTE] Release nanoarrow 0.6.0

Hello,

I would like to propose the following release candidate (rc0) of Apache Arrow nanoarrow [0] version 0.5.0. This is an initial release consisting of 44 resolved GitHub issues from 5 contributors [1].
I would like to propose the following release candidate (rc0) of Apache Arrow nanoarrow [0] version 0.6.0. This is an initial release consisting of 44 resolved GitHub issues from 5 contributors [1].

This release candidate is based on commit: {rc_commit} [2]

Expand All @@ -355,15 +314,15 @@ Please download, verify checksums and signatures, run the unit tests, and vote o

The vote will be open for at least 72 hours.

[ ] +1 Release this as Apache Arrow nanoarrow 0.5.0
[ ] +1 Release this as Apache Arrow nanoarrow 0.6.0
[ ] +0
[ ] -1 Do not release this as Apache Arrow nanoarrow 0.5.0 because...
[ ] -1 Do not release this as Apache Arrow nanoarrow 0.6.0 because...

[0] https://github.com/apache/arrow-nanoarrow
[1] https://github.com/apache/arrow-nanoarrow/milestone/4?closed=1
[2] https://github.com/apache/arrow-nanoarrow/tree/apache-arrow-nanoarrow-0.5.0-rc0
[3] https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-nanoarrow-0.5.0-rc0/
[4] https://github.com/apache/arrow-nanoarrow/blob/apache-arrow-nanoarrow-0.5.0-rc0/CHANGELOG.md
[2] https://github.com/apache/arrow-nanoarrow/tree/apache-arrow-nanoarrow-0.6.0-rc0
[3] https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-nanoarrow-0.6.0-rc0/
[4] https://github.com/apache/arrow-nanoarrow/blob/apache-arrow-nanoarrow-0.6.0-rc0/CHANGELOG.md
[5] https://github.com/apache/arrow-nanoarrow/blob/main/dev/release/README.md
```

Expand Down Expand Up @@ -397,7 +356,7 @@ and mark it as closed.
The reporter system for Arrow can be found at
<https://reporter.apache.org/addrelease.html?arrow>. To add a release, a
PMC member must log in with their Apache username/password. The release
names are in the form `NANOARROW-0.5.0`.
names are in the form `NANOARROW-0.6.0`.

### Upload artifacts to Subversion / Create GitHub Release

Expand All @@ -407,7 +366,7 @@ This script must be run by a PMC member whose `APACHE_USERNAME` environment vari
has been set in `.env`.

```bash
dev/release/post-01-upload.sh 0.5.0 0
dev/release/post-01-upload.sh 0.6.0 0
```

### Submit R package to CRAN
Expand All @@ -419,8 +378,8 @@ Before a release candidate is created, the first section of
`usethis::use_release_issue()` should all be completed (i.e., any changes
after release should be minor tweaks). The steps are:

- Ensure you are on the release branch (i.e., `git switch maint-0.5.0`)
- Run `usethis::pr_init("r-cran-maint-0.5.0")` and push the branch to your
- Ensure you are on the release branch (i.e., `git switch maint-0.6.0`)
- Run `usethis::pr_init("r-cran-maint-0.6.0")` and push the branch to your
fork.
- Ensure `cran_comments.md` is up-to-date.
- Run `devtools::check()` locally and verify that the package version is correct
Expand All @@ -432,12 +391,12 @@ after release should be minor tweaks). The steps are:
Any changes required at this stage should be made as a PR into `main` and
cherry-picked into the `r-cran-maint-XXX` packaging branch. (i.e.,
`git cherry-pick 01234abcdef`). If any changes
to the source are required, bump the "tweak" version (e.g., `Version: 0.5.0.1`
to the source are required, bump the "tweak" version (e.g., `Version: 0.6.0.1`
in `DESCRIPTION`).

### Submit Python package to PyPI

The Python package source distribution and wheels are built using the [Build Python Wheels](https://github.com/apache/arrow-nanoarrow/actions/workflows/python-wheels.yaml) action on the `maint-0.5.0` branch after cutting the release candidate.
The Python package source distribution and wheels are built using the [Build Python Wheels](https://github.com/apache/arrow-nanoarrow/actions/workflows/python-wheels.yaml) action on the `maint-0.6.0` branch after cutting the release candidate.

To submit these to PyPI, download all assets from the run into a folder (e.g., `python/dist`) and run:

Expand Down Expand Up @@ -467,18 +426,18 @@ git clone -b asf-site --single-branch https://github.com/apache/arrow-nanoarrow.
cd arrow-nanoarrow
```

Download the [0.5.0 documentation](https://github.com/apache/arrow-nanoarrow/releases/download/apache-arrow-nanoarrow-0.5.0/docs.tgz):
Download the [0.6.0 documentation](https://github.com/apache/arrow-nanoarrow/releases/download/apache-arrow-nanoarrow-0.6.0/docs.tgz):

```shell
curl -L https://github.com/apache/arrow-nanoarrow/releases/download/apache-arrow-nanoarrow-0.5.0/docs.tgz \
curl -L https://github.com/apache/arrow-nanoarrow/releases/download/apache-arrow-nanoarrow-0.6.0/docs.tgz \
-o docs.tgz
```

Extract the documentation and rename the directory to `0.5.0`:
Extract the documentation and rename the directory to `0.6.0`:

```shell
tar -xvzf docs.tgz
mv nanoarrow-docs 0.5.0
mv nanoarrow-docs 0.6.0
```

Then remove the existing `latest` directory and run the extraction again, renaming to `latest` instead:
Expand All @@ -489,7 +448,7 @@ tar -xvzf docs.tgz
mv nanoarrow-docs latest
```

Finally, update `switcher.json` with entries pointing `/latest/` and `/0.5.0/` to `"version": "0.5.0"`:
Finally, update `switcher.json` with entries pointing `/latest/` and `/0.6.0/` to `"version": "0.6.0"`:

```json
[
Expand All @@ -498,16 +457,16 @@ Finally, update `switcher.json` with entries pointing `/latest/` and `/0.5.0/` t
"url": "https://arrow.apache.org/nanoarrow/main/"
},
{
"version": "0.5.0",
"version": "0.6.0",
"url": "https://arrow.apache.org/nanoarrow/latest/"
},
{
"version": "0.5.0",
"url": "https://arrow.apache.org/nanoarrow/0.5.0/"
"version": "0.6.0",
"url": "https://arrow.apache.org/nanoarrow/0.6.0/"
},
{
"version": "0.4.0",
"url": "https://arrow.apache.org/nanoarrow/0.4.0/"
"version": "0.5.0",
"url": "https://arrow.apache.org/nanoarrow/0.5.0/"
},
...
]
Expand All @@ -529,9 +488,9 @@ the `mail-relay.apache.org` outgoing server.
Email template:

```
[ANNOUNCE] Apache Arrow nanoarrow 0.5.0 Released
[ANNOUNCE] Apache Arrow nanoarrow 0.6.0 Released

The Apache Arrow community is pleased to announce the 0.5.0 release of
The Apache Arrow community is pleased to announce the 0.6.0 release of
Apache Arrow nanoarrow. This initial release covers 79 resolved issues
from 9 contributors[1].

Expand Down Expand Up @@ -567,10 +526,10 @@ Please report any feedback to the mailing lists ([6], [7]).
Regards,
The Apache Arrow Community

[1] https://github.com/apache/arrow-nanoarrow/issues?q=milestone%3A%22nanoarrow+0.5.0%22+is%3Aclosed
[2] https://www.apache.org/dyn/closer.cgi/arrow/apache-arrow-nanoarrow-0.5.0
[3] https://github.com/apache/arrow-nanoarrow/blob/apache-arrow-nanoarrow-0.5.0/CHANGELOG.md
[4] https://arrow.apache.org/blog/2024/05/27/nanoarrow-0.5.0-release/
[1] https://github.com/apache/arrow-nanoarrow/issues?q=milestone%3A%22nanoarrow+0.6.0%22+is%3Aclosed
[2] https://www.apache.org/dyn/closer.cgi/arrow/apache-arrow-nanoarrow-0.6.0
[3] https://github.com/apache/arrow-nanoarrow/blob/apache-arrow-nanoarrow-0.6.0/CHANGELOG.md
[4] https://arrow.apache.org/blog/2024/05/27/nanoarrow-0.6.0-release/
[5] https://arrow.apache.org/nanoarrow/
[6] https://lists.apache.org/[email protected]
[7] https://lists.apache.org/[email protected]
Expand All @@ -586,7 +545,7 @@ using:
# Once
export APACHE_USERNAME=xxx
# Once for every release candidate
svn rm --username=$APACHE_USERNAME -m "Clean up svn artifacts" https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-nanoarrow-0.5.0-rc0/
svn rm --username=$APACHE_USERNAME -m "Clean up svn artifacts" https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-nanoarrow-0.6.0-rc0/
```

### Bumped versions on main
Expand All @@ -595,5 +554,5 @@ This is handled by
[post-02-bump-versions.sh](https://github.com/apache/arrow-nanoarrow/blob/main/dev/release/post-02-bump-versions.sh). Create a branch and then run:

```bash
dev/release/post-02-bump-versions.sh . 0.5.0 0.6.0
dev/release/post-02-bump-versions.sh . 0.6.0 0.7.0
```
15 changes: 15 additions & 0 deletions r/bootstrap.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,21 @@
# Use ci/scripts/bundle.py to generate source files/headers that are
# easier to use with the default R package build system

python <- function() {
from_env <- Sys.getenv("PYTHON_BIN", "python3")
has_from_env <- system(
paste(from_env, "--version"),
ignore.stdout = TRUE,
ignore.stderr = TRUE
)

if (has_from_env == 0) {
from_env
} else {
"python"
}
}

run_bundler <- function() {
args <- c(
"--symbol-namespace=RPkg",
Expand Down
Loading