Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch presubmit CI workflows to use pinned IREE versions. #774

Merged
merged 6 commits into from
Jan 7, 2025
Merged

Switch presubmit CI workflows to use pinned IREE versions. #774

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3ed9483
Switch presubmit CI workflows to use pinned IREE versions.
ScottTodd Jan 7, 2025
672fe07
Add pytest-xdist to sharktank/requirements-tests.txt.
ScottTodd Jan 7, 2025
5e350a4
Include iree-turbine in IREE requirements files.
ScottTodd Jan 7, 2025
04ef073
Bump version pins to 20250107.
ScottTodd Jan 7, 2025
2c158c4
Cherry-pick fix for mobilenet numerics that caused unit tests to hang.
ScottTodd Jan 7, 2025
335df0e
Merge branch 'main' into ci-pinned
ScottTodd Jan 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 5 additions & 10 deletions .github/workflows/ci-llama-quick-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,18 +55,13 @@ jobs:
python -m pip install --no-compile --upgrade pip

# Note: We install in three steps in order to satisfy requirements
# from non default locations first. Installing the PyTorch CPU
# wheels saves multiple minutes and a lot of bandwidth on runner setup.
# from non default locations first.
pip install --no-compile -r pytorch-cpu-requirements.txt
pip install -r requirements-iree-pinned.txt
pip install --no-compile \
-r sharktank/requirements-tests.txt \
-e sharktank/

# Install nightly IREE packages.
# We could also pin to a known working or stable version.
pip install -f https://iree.dev/pip-release-links.html --pre --upgrade \
iree-base-compiler \
iree-base-runtime \
iree-turbine

pip install --no-compile -r requirements.txt -r sharktank/requirements-tests.txt -e sharktank/
pip freeze

- name: Run llama 8b f16 decomposed test
Expand Down
16 changes: 5 additions & 11 deletions .github/workflows/ci-shark-ai.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,18 +51,12 @@ jobs:
python -m pip install --no-compile --upgrade pip

# Note: We install in three steps in order to satisfy requirements
# from non default locations first. Installing the PyTorch CPU
# wheels saves multiple minutes and a lot of bandwidth on runner setup.
# from non default locations first.
pip install --no-compile -r pytorch-cpu-requirements.txt

# Install nightly IREE packages.
# We could also pin to a known working or stable version.
pip install -f https://iree.dev/pip-release-links.html --pre --upgrade \
iree-base-compiler \
iree-base-runtime \
iree-turbine

pip install --no-compile -r requirements.txt -e sharktank/ shortfin/
pip install -r requirements-iree-pinned.txt
pip install --no-compile \
-r requirements.txt \
-e sharktank/ shortfin/

pip freeze

Expand Down
52 changes: 21 additions & 31 deletions .github/workflows/ci-sharktank.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,18 +72,15 @@ jobs:
python -m pip install --no-compile --upgrade pip

# Note: We install in three steps in order to satisfy requirements
# from non default locations first. Installing the PyTorch CPU
# wheels saves multiple minutes and a lot of bandwidth on runner setup.
pip install --no-compile --index-url https://download.pytorch.org/whl/cpu torch==${{matrix.torch-version}}+cpu
# from non default locations first.
pip install --no-compile \
--index-url https://download.pytorch.org/whl/cpu torch==${{matrix.torch-version}}+cpu
pip install -r requirements-iree-pinned.txt
pip install --no-compile \
-r sharktank/requirements-tests.txt \
-e sharktank/

# Install nightly IREE packages.
# We could also pin to a known working or stable version.
pip install -f https://iree.dev/pip-release-links.html --pre \
iree-base-compiler \
iree-base-runtime \
iree-turbine

pip install --no-compile -r requirements.txt -r sharktank/requirements-tests.txt -e sharktank/
pip freeze

- name: Run sharktank tests
if: ${{ !cancelled() }}
Expand Down Expand Up @@ -121,19 +118,16 @@ jobs:
run: |
source ${VENV_DIR}/bin/activate
python -m pip install --no-compile --upgrade pip

# Note: We install in three steps in order to satisfy requirements
# from non default locations first. Installing the PyTorch CPU
# wheels saves multiple minutes and a lot of bandwidth on runner setup.
# from non default locations first.
pip install --no-compile -r pytorch-cpu-requirements.txt
pip install -r requirements-iree-pinned.txt
pip install --no-compile \
-r sharktank/requirements-tests.txt \
-e sharktank/

# Install nightly IREE packages.
# We could also pin to a known working or stable version.
pip install -f https://iree.dev/pip-release-links.html --pre --upgrade \
iree-base-compiler \
iree-base-runtime \
iree-turbine

pip install --no-compile -r requirements.txt -r sharktank/requirements-tests.txt -e sharktank/
pip freeze

- name: Run tests
# TODO: unify with-*-data flags into a single flag and make it possible to run
Expand Down Expand Up @@ -180,18 +174,14 @@ jobs:
python -m pip install --no-compile --upgrade pip

# Note: We install in three steps in order to satisfy requirements
# from non default locations first. Installing the PyTorch CPU
# wheels saves multiple minutes and a lot of bandwidth on runner setup.
# from non default locations first.
pip install --no-compile -r pytorch-cpu-requirements.txt
pip install -r requirements-iree-pinned.txt
pip install --no-compile \
-r sharktank/requirements-tests.txt \
-e sharktank/

# Install nightly IREE packages.
# We could also pin to a known working or stable version.
pip install -f https://iree.dev/pip-release-links.html --pre \
iree-base-compiler \
iree-base-runtime \
iree-turbine

pip install --no-compile -r requirements.txt -r sharktank/requirements-tests.txt -e sharktank/
pip freeze

- name: Run punet tests
run: |
Expand Down
15 changes: 5 additions & 10 deletions .github/workflows/ci_eval_short.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,18 +52,13 @@ jobs:
python -m pip install --no-compile --upgrade pip

# Note: We install in three steps in order to satisfy requirements
# from non default locations first. Installing the PyTorch CPU
# wheels saves multiple minutes and a lot of bandwidth on runner setup.
# from non default locations first.
pip install --no-compile -r pytorch-cpu-requirements.txt
pip install -r requirements-iree-pinned.txt
pip install --no-compile \
-r sharktank/requirements-tests.txt \
-e sharktank/

# Install nightly IREE packages.
# We could also pin to a known working or stable version.
pip install -f https://iree.dev/pip-release-links.html --pre --upgrade \
iree-base-compiler \
iree-base-runtime \
iree-turbine

pip install --no-compile -r requirements.txt -r sharktank/requirements-tests.txt -e sharktank/
pip freeze

- name: Run perplexity test with vmfb
Expand Down
4 changes: 1 addition & 3 deletions requirements-iree-pinned.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,4 @@
--find-links https://iree.dev/pip-release-links.html
iree-base-compiler==3.1.0rc20241204
iree-base-runtime==3.1.0rc20241204

# TODO(#760): include iree-turbine in this requirements file too?
# iree-turbine==3.1.0rc20241205
iree-turbine==3.1.0rc20241205
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All Llama sharktank/shortfin pre-submits pass on latest IREE release, so shouldn't be an issue updating to latest release. (Not sure about sdxl)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They pass on the latest IREE release today. They might not in the future.

We'll catch issues as part of version pin update PRs, like #773.

Copy link
Collaborator

@archana-ramalingam archana-ramalingam Jan 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to know we have a PR to bump to latest version.
shark-ai CI is passing in main but failing here. @stbaione @renxida Did we see any regression on this IREE version? Not an issue, if we are bumping the version soon.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear with this PR: this isn't a temporary stabilizing measure, this is rolling out a methodical way of managing versions. At the moment we only have manually created PRs to update the versions. I'm looking into tools like dependabot to automate that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Via, conversation in previous PR, this version should be fine!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or maybe not actually... shark-ai is currently passing in main, but seems to be failing at the compilation step in this PR

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, good catch. I can update the pin in this PR too then 🤔

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, looks good now

4 changes: 1 addition & 3 deletions requirements-iree-unpinned.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,4 @@
--find-links https://iree.dev/pip-release-links.html
iree-base-compiler
iree-base-runtime

# TODO(#760): include iree-turbine in this requirements file too?
# iree-turbine
iree-turbine
3 changes: 2 additions & 1 deletion sharktank/requirements-tests.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
datasets==3.0.0
diffusers
parameterized
pytest==8.0.0
pytest-html
diffusers
pytest-xdist==3.5.0
Loading