Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add [pinned, unpinnned] matrix to ci_eval.yaml. #767

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Add [pinned, unpinnned] matrix to ci_eval.yaml. #767

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fa7ab97
Add `[pinned, unpinnned]` matrix to `ci_val.yaml`.
ScottTodd Jan 7, 2025
d4f56bf
Merge remote-tracking branch 'upstream/main' into users/scotttodd/req…
ScottTodd Jan 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 15 additions & 27 deletions .github/workflows/ci_eval.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,11 @@ jobs:
test_perplexity_iree:
if: ${{ github.repository_owner == 'nod-ai' || github.event_name != 'schedule' }}
timeout-minutes: 1000
name: "IREE Perplexity"
name: "IREE Perplexity :: ${{ matrix.version }} :: ${{ matrix.requirements }}"
strategy:
matrix:
version: [3.11]
requirements: [pinned, unpinned]
runs-on: [llama-mi300x-3]
fail-fast: false
runs-on: ${{matrix.runs-on}}
Expand All @@ -47,24 +48,16 @@ jobs:
- name: Create Python venv
run: python -m venv ${VENV_DIR}

- name: Install sharktank deps
- name: Install sharktank deps (${{ matrix.requirements }} IREE)
run: |
source ${VENV_DIR}/bin/activate
python -m pip install --no-compile --upgrade pip

# Note: We install in three steps in order to satisfy requirements
# from non default locations first. Installing the PyTorch CPU
# wheels saves multiple minutes and a lot of bandwidth on runner setup.
pip install --no-compile -r pytorch-cpu-requirements.txt

# Install nightly IREE packages.
# We could also pin to a known working or stable version.
pip install -f https://iree.dev/pip-release-links.html --pre --upgrade \
iree-base-compiler \
iree-base-runtime \
iree-turbine

pip install --no-compile -r requirements.txt -r sharktank/requirements-tests.txt -e sharktank/
pip install -r requirements-iree-${{ matrix.requirements }}.txt
pip install --no-compile \
-r sharktank/requirements.txt \
-r sharktank/requirements-tests.txt \
-e sharktank/
pip freeze

- name: Run perplexity test with IREE
Expand Down Expand Up @@ -106,22 +99,17 @@ jobs:
- name: Create Python venv
run: python -m venv ${VENV_DIR}

- name: Install sharktank deps
- name: Install sharktank deps (unpinned IREE)
run: |
source ${VENV_DIR}/bin/activate
python -m pip install --no-compile --upgrade pip

# Note: We install in three steps in order to satisfy requirements
# from non default locations first. Installing the PyTorch CPU
# wheels saves multiple minutes and a lot of bandwidth on runner setup.
pip install --no-compile -r pytorch-cpu-requirements.txt

# Install nightly iree-turbine.
# We could also pin to a known working or stable version.
pip install -f https://iree.dev/pip-release-links.html --pre --upgrade \
iree-turbine

pip install --no-compile -r requirements.txt -r sharktank/requirements-tests.txt -e sharktank/
pip install -r requirements-iree-unpinned.txt
pip install --no-compile \
-r sharktank/requirements.txt \
-r sharktank/requirements-tests.txt \
-e sharktank/
pip freeze

- name: Run perplexity test with Torch
run: |
Expand Down
Loading