Skip to content

Commit 72665fc

Browse files
ScottToddmarbre
ScottTodd
and
marbre
authored
[libshortfin] Add workflow to build nightly packages. (nod-ai#238)
Progress on nod-ai#130, follow-up to nod-ai#230. ## Overview This gets us the general structure of a release pipeline that: 1. Computes some version information metadata based on build id and date 2. Calls `shortfin/build_tools/build_linux_package.sh`, which runs `python -m pip wheel` under a manylinux Docker container 3. Uploads wheels to both GitHub artifacts (for the workflow run itself) and GitHub releases (for archival and developer usage) This follows how stablehlo (https://github.com/openxla/stablehlo/releases/tag/dev-wheels) and torch-mlir (https://github.com/llvm/torch-mlir-release/releases/tag/dev-wheels) both publish to a constantly growing "dev-wheels" release rather than separate releases like IREE does. That makes it harder to directly "promote" a candidate release, but it avoids polluting the release history with nightly builds. ## Testing * Sample run: https://github.com/ScottTodd/SHARK-Platform/actions/runs/11078636913 * Dev wheels release: https://github.com/ScottTodd/SHARK-Platform/releases/tag/dev-wheels * Install with `python -m pip install shortfin -f https://github.com/ScottTodd/SHARK-Platform/releases/expanded_assets/dev-wheels` ## What's next In no particular order, - [ ] Verify that the Tracy build enabled by `SHORTFIN_ENABLE_TRACING` is functional - [ ] Set up dependencies and optional components such that a user can just `pip install shortfin` and have it bring along a compatible version of packages like `iree-runtime` and `iree-compiler` - [ ] Update the dockerfile version from manylinux2014 (that is independent of this workflow) - [x] Build for Python 3.13 free threaded and test that - [ ] Build for other platforms/architectures (macOS, Windows, arm64, etc.) - [ ] Set up a workflow like IREE's pkgci that installs the packages and runs some tests on them - [ ] Document how to install the developer packages --------- Co-authored-by: Marius Brehler <[email protected]>
1 parent 084adc5 commit 72665fc

File tree

3 files changed

+154
-6
lines changed

3 files changed

+154
-6
lines changed

‎.github/workflows/build_packages.yml

+111
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Copyright 2024 Advanced Micro Devices, Inc.
2+
#
3+
# Licensed under the Apache License v2.0 with LLVM Exceptions.
4+
# See https://llvm.org/LICENSE.txt for license information.
5+
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
6+
7+
name: Build packages
8+
9+
on:
10+
workflow_dispatch:
11+
schedule:
12+
# Runs at 11:00 AM UTC, which is 3:00 AM PST (UTC-8)
13+
- cron: '0 11 * * *'
14+
15+
jobs:
16+
# Note: metadata generation could happen in a separate trigger/schedule
17+
# workflow. For cross platform builds, it's useful to just generate the
18+
# metadata on Linux and pass that to later jobs using artifacts.
19+
setup_metadata:
20+
runs-on: ubuntu-24.04
21+
outputs:
22+
shark_package_version: ${{ steps.version.outputs.shark_package_version }}
23+
steps:
24+
# For now the version is just a calendar date + an automatically
25+
# incrementing value. We may want different versions for nightly/dev
26+
# builds and stable releases published to official places like pypi.
27+
- name: Compute version
28+
id: version
29+
run: |
30+
shark_package_version="$(printf '%(%Y%m%d)T.${{ github.run_number }}')"
31+
echo "shark_package_version=${shark_package_version}" >> $GITHUB_OUTPUT
32+
cat << EOF > ./version_info.json
33+
{
34+
"package-version": "${shark_package_version}"
35+
}
36+
EOF
37+
cat ./version_info.json
38+
- name: Upload version_info.json
39+
uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4.4.0
40+
with:
41+
name: version_info
42+
path: version_info.json
43+
44+
build_packages:
45+
name: "${{ matrix.package }} :: ${{ matrix.platform }} :: ${{ matrix.python-version }}"
46+
runs-on: ${{ matrix.runs-on }}
47+
needs: [setup_metadata]
48+
strategy:
49+
fail-fast: false
50+
matrix:
51+
include:
52+
# Ubuntu packages.
53+
- runs-on: ubuntu-24.04
54+
platform: linux-x86_64
55+
package: shortfin
56+
python-version: cp312-cp312
57+
- runs-on: ubuntu-24.04
58+
platform: linux-x86_64
59+
package: shortfin
60+
python-version: cp313-cp313
61+
- runs-on: ubuntu-24.04
62+
platform: linux-x86_64
63+
package: shortfin
64+
python-version: cp313-cp313t
65+
66+
# TODO(#130): macOS platform
67+
# TODO(#130): Windows platform
68+
# TODO(#130): sharktank packages
69+
70+
steps:
71+
- name: Checkout repository
72+
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
73+
with:
74+
path: "c" # Windows can hit path length limits, so use a short path.
75+
submodules: false
76+
77+
- name: Download version_info.json
78+
uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
79+
with:
80+
name: version_info
81+
path: ./c/shortfin/
82+
merge-multiple: true
83+
84+
- name: Build shortfin (Linux x86_64, ${{ matrix.python-version }})
85+
if: "matrix.package == 'shortfin' && matrix.platform == 'linux-x86_64'"
86+
env:
87+
OUTPUT_DIR: "${{ github.workspace }}/bindist"
88+
OVERRIDE_PYTHON_VERSIONS: "${{ matrix.python-version }}"
89+
run: |
90+
[ -e ./bindist/* ] && rm ./bindist/*
91+
./c/shortfin/build_tools/build_linux_package.sh
92+
93+
- name: Upload python wheels
94+
uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4.4.0
95+
with:
96+
if-no-files-found: error
97+
name: snapshot-${{ matrix.package }}-${{ matrix.platform }}-${{ matrix.python-version }}
98+
path: bindist
99+
100+
- name: Release python wheels
101+
uses: ncipollo/release-action@2c591bcc8ecdcd2db72b97d6147f871fcd833ba5 # v1.14.0
102+
with:
103+
artifacts: bindist/*.whl
104+
token: "${{ secrets.GITHUB_TOKEN }}"
105+
tag: "dev-wheels"
106+
name: "dev-wheels"
107+
body: "Automatic snapshot release of SHARK-Platform python wheels."
108+
removeArtifacts: false
109+
allowUpdates: true
110+
replacesArtifacts: true
111+
makeLatest: false

‎shortfin/.gitignore

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Local-only config options
2+
version_info.json

‎shortfin/setup.py

+41-6
Original file line numberDiff line numberDiff line change
@@ -4,16 +4,17 @@
44
# See https://llvm.org/LICENSE.txt for license information.
55
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
66

7-
from distutils.core import setup, Extension
8-
import sys
7+
import json
8+
import os
99
import shutil
1010
import subprocess
11-
import os
12-
from pathlib import Path
11+
import sys
1312
from distutils.command.build import build as _build
13+
from distutils.core import setup, Extension
14+
from pathlib import Path
1415
from setuptools import find_namespace_packages
15-
from setuptools.command.build_ext import build_ext as _build_ext
1616
from setuptools.command.build_py import build_py as _build_py
17+
from setuptools.command.build_ext import build_ext as _build_ext
1718

1819

1920
def get_env_boolean(name: str, default_value: bool = False) -> bool:
@@ -133,6 +134,40 @@ def copy_extensions_to_source(self, *args, **kwargs):
133134
...
134135

135136

137+
# Setup and get version information.
138+
VERSION_INFO_FILE = os.path.join(SOURCE_DIR, "version_info.json")
139+
140+
141+
def load_version_info():
142+
with open(VERSION_INFO_FILE, "rt") as f:
143+
return json.load(f)
144+
145+
146+
def find_git_version():
147+
try:
148+
return (
149+
subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=SOURCE_DIR)
150+
.decode("utf-8")
151+
.strip()
152+
)
153+
except subprocess.SubprocessError as e:
154+
print(f"ERROR: Could not get git revision: {e}", file=sys.stderr)
155+
return None
156+
157+
158+
try:
159+
version_info = load_version_info()
160+
except FileNotFoundError:
161+
print("version_info.json not found. Using defaults", file=sys.stderr)
162+
version_info = {}
163+
git_version = find_git_version()
164+
165+
PACKAGE_VERSION = version_info.get("package-version")
166+
if not PACKAGE_VERSION:
167+
PACKAGE_VERSION = f"0.dev0+{git_version or '0'}"
168+
print(f"Using PACKAGE_VERSION: '{PACKAGE_VERSION}'")
169+
170+
136171
def maybe_nuke_cmake_cache(cmake_build_dir):
137172
# From run to run under pip, we can end up with different paths to ninja,
138173
# which isn't great and will confuse cmake. Detect if the location of
@@ -324,7 +359,7 @@ def populate_built_package(abs_dir):
324359

325360
setup(
326361
name="shortfin",
327-
version="0.9",
362+
version=f"{PACKAGE_VERSION}",
328363
description="Shortfin native library implementation",
329364
author="SHARK Authors",
330365
packages=packages,

0 commit comments

Comments
 (0)