Skip to content

Use uv tool for isolate cicd env #2597

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: development
Choose a base branch
from
Open

Use uv tool for isolate cicd env #2597

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
3b2414b
Use uv tool for isolate cicd env
hexu33 Mar 3, 2025
283801d
refactor uv, use Getting-Started-Samples/IntelPython_XGBoost_GettingS…
bopeng1234 Mar 7, 2025
8cc42c2
Getting-Started-Samples/IntelPython_XGBoost_GettingStarted ready
bopeng1234 Mar 7, 2025
8ffc083
uv refactor add pyproject.toml
bopeng1234 Mar 7, 2025
99d6abd
add kernel name for getstart sample
bopeng1234 Mar 7, 2025
080bc26
uv refactor add pyproject.toml
bopeng1234 Mar 7, 2025
14a36af
uv refactor add pyproject.toml
troy818 Mar 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AI-and-Analytics/End-to-end-Workloads/JobRecommendationSystem/JobRecommendationSystem.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -904,6 +904,7 @@
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import intel_extension_for_tensorflow as itex\n",
"\n",
"itex.experimental_ops_override()"
Expand Down
1 change: 1 addition & 0 deletions AI-and-Analytics/End-to-end-Workloads/JobRecommendationSystem/JobRecommendationSystem.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -428,6 +428,7 @@ def lemmatize_text(text):
# Now, let us import Intel Extension for TensorFlow*. We are using Python API `itex.experimental_ops_override()`. It automatically replace some TensorFlow operators by Custom Operators under `itex.ops` namespace, as well as to be compatible with existing trained parameters.

# %%
import numpy as np
import intel_extension_for_tensorflow as itex

itex.experimental_ops_override()
Expand Down
16 changes: 10 additions & 6 deletions AI-and-Analytics/End-to-end-Workloads/JobRecommendationSystem/sample.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,16 @@
"id": "JobRecommendationSystem_py",
"steps": [
"source /intel/oneapi/intelpython/bin/activate",
"conda env remove -n user_tensorflow-gpu",
"conda create --name user_tensorflow-gpu --clone tensorflow-gpu",
"conda activate user_tensorflow-gpu",
"pip install -r requirements.txt",
"python -m ipykernel install --user --name=user_tensorflow-gpu",
"python JobRecommendationSystem.py"
"conda activate tensorflow-gpu",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any particular reason to activate conda environment? As far as I know, UV will create an isolated environment.

Copy link
Contributor Author

@bopeng1234 bopeng1234 Mar 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reply at below

"pip install uv",
"uv init",
"uv python pin $(which python)",
"uv venv --system-site-packages",
"uv add -r requirements.txt",
"uv add seaborn",
"uv add numpy==1.26.4",
"uv run python -m spacy download en_core_web_sm",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

super nit but can have them in a single uv add command?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

refactored

"uv run python JobRecommendationSystem.py"
]
}
]
Expand Down
9 changes: 7 additions & 2 deletions ...s/Features-and-Functionality/INC_QuantizationAwareTraining_TextClassification/sample.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,16 @@
"env": [
"source /intel/oneapi/intelpython/bin/activate",
"conda activate pytorch",
"pip install -r requirements.txt"
"pip install uv",
"uv init",
"uv python pin $(which python)",
"uv venv --system-site-packages",
"uv add -r requirements.txt",
"uv add torch==2.5.0 torchvision==0.20.0"
],
"id": "inc_text_classification_quantization_py",
"steps": [
"python INC_QuantizationAwareTraining_TextClassification.py"
"uv run INC_QuantizationAwareTraining_TextClassification.py"
]
}
]
Expand Down
15 changes: 10 additions & 5 deletions ...cs/Features-and-Functionality/IntelPyTorch_GPU_InferenceOptimization_with_AMP/sample.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,16 @@
"id": "intel pytorch gpu inference optimization with amp",
"steps": [
"source /intel/oneapi/intelpython/bin/activate",
"source activate pytorch-gpu",
"pip install -r requirements.txt",
"pip install jupyter ipykernel",
"python -m ipykernel install --user --name=pytorch-gpu",
"python IntelPyTorch_GPU_InferenceOptimization_with_AMP.py"
"conda activate pytorch-gpu",
"pip install uv notebook",
"uv init",
"uv python pin $(which python)",
"uv venv --system-site-packages",
"uv add -r requirements.txt",
"uv add numpy==1.26.4",
"uv add --dev ipykernel",
"uv run ipython kernel install --user --name=pytorch-gpu",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't have to install ipykernel if we not using it. I know its there in current JSON file, but its good opportunity to clean it up.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

"uv run IntelPyTorch_GPU_InferenceOptimization_with_AMP.py"
]
}
]
Expand Down
15 changes: 10 additions & 5 deletions ...lytics/Features-and-Functionality/IntelPyTorch_TrainingOptimizations_AMX_BF16/sample.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,16 @@
"steps": [
"source /intel/oneapi/intelpython/bin/activate",
"conda activate pytorch",
"python -m pip install -r requirements.txt",
"python -m ipykernel install --user --name=pytorch",
"python pytorch_training_avx512_bf16.py",
"python pytorch_training_amx_bf16.py",
"jupyter nbconvert --ExecutePreprocessor.enabled=True --ExecutePreprocessor.kernel_name=pytorch --to notebook IntelPyTorch_TrainingOptimizations_AMX_BF16.ipynb"
"pip install uv",
"uv init",
"uv python pin $(which python)",
"uv venv --system-site-packages",
"uv add -r requirements.txt",
"uv add --dev ipykernel notebook",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please install ipykernel in standard dependency? No need for --dev

Copy link
Contributor Author

@bopeng1234 bopeng1234 Mar 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed --dev

"uv run ipython kernel install --user --name pytorch",
"uv run python pytorch_training_avx512_bf16.py",
"uv run python pytorch_training_amx_bf16.py",
"uv run jupyter nbconvert --ExecutePreprocessor.enabled=True --ExecutePreprocessor.kernel_name=pytorch --to notebook IntelPyTorch_TrainingOptimizations_AMX_BF16.ipynb"
]
}
]
Expand Down
20 changes: 12 additions & 8 deletions ...d-Analytics/Features-and-Functionality/IntelPython_GPU_dpnp_Genetic_Algorithm/sample.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,22 @@
"os":["linux"],
"targetDevice": ["GPU"],
"ciTests": {
"linux": [
"linux": [
{
"env": [
"env": [
"source /intel/oneapi/intelpython/bin/activate",
"conda activate base",
"pip install -r requirements.txt"
"pip install uv",
"uv init",
"uv python pin $(which python)",
"uv venv --system-site-packages",
"uv add -r requirements.txt"
],
"id": "idp_ga_dpnp_py",
"steps": [
"python IntelPython_GPU_dpnp_Genetic_Algorithm.py"
]
}
"id": "idp_ga_dpnp_py",
"steps": [
"uv run python IntelPython_GPU_dpnp_Genetic_Algorithm.py"
]
}
]
},
"expertise": "Code Optimization"
Expand Down
16 changes: 10 additions & 6 deletions AI-and-Analytics/Features-and-Functionality/IntelPython_Numpy_Numba_dpnp_kNN/sample.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,17 @@
"env": [
"source /intel/oneapi/intelpython/bin/activate",
"conda activate base",
"pip install numba"
"pip install uv",
"uv init",
"uv python pin $(which python)",
"uv venv --system-site-packages",
"uv add numba"
],
"id": "idp_numpy_numba_dpnp_gs_py",
"steps": [
"python IntelPython_Numpy_Numba_dpnp_kNN.py"
]
}
"id": "idp_numpy_numba_dpnp_gs_py",
"steps": [
"uv run python IntelPython_Numpy_Numba_dpnp_kNN.py"
]
}
]
},
"expertise": "Code Optimization"
Expand Down
2 changes: 1 addition & 1 deletion AI-and-Analytics/Features-and-Functionality/IntelPython_XGBoost_Performance/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
###### Requirements ######
notebook
matplot
xgboost==0.81
xgboost==0.82
scikit-learn
pandas
27 changes: 17 additions & 10 deletions AI-and-Analytics/Features-and-Functionality/IntelPython_XGBoost_Performance/sample.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,24 @@
"os":["linux"],
"targetDevice": ["CPU"],
"ciTests": {
"linux": [
"linux": [
{
"id": "idp_xgb_performance",
"steps": [
"source /intel/oneapi/intelpython/bin/activate",
"python IntelPython_XGBoost_Performance.py",
"conda remove xgboost --y",
"pip install -r requirements.txt",
"python IntelPython_XGBoost_Performance.py"
]
}
"env": [
"source /intel/oneapi/intelpython/bin/activate",
"conda activate base"
],
"id": "idp_xgb_performance",
"steps": [
"python IntelPython_XGBoost_Performance.py",
"pip install uv",
"uv init",
"uv python pin $(which python)",
"uv venv --system-site-packages",
"uv add -r requirements.txt",
"uv add numpy==1.26.4",
"uv run python IntelPython_XGBoost_Performance.py"
]
}
]
},
"expertise": "Code Optimization"
Expand Down
9 changes: 7 additions & 2 deletions ...nd-Analytics/Features-and-Functionality/IntelPython_daal4py_DistributedKMeans/sample.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,16 @@
"env": [
"source /intel/oneapi/intelpython/bin/activate",
"conda activate base",
"pip install -r requirements.txt"
"pip install uv",
"uv init",
"uv python pin $(which python)",
"uv venv --system-site-packages",
"uv add -r requirements.txt",
"uv add numpy==1.26.4"
],
"id": "idp_d4p_KM_Dist",
"steps": [
"mpirun -n 4 python ./IntelPython_daal4py_Distributed_Kmeans.py"
"uv run mpirun -n 4 python ./IntelPython_daal4py_Distributed_Kmeans.py"
]
}]
},
Expand Down
14 changes: 10 additions & 4 deletions ...cs/Features-and-Functionality/IntelPython_daal4py_DistributedLinearRegression/sample.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,19 @@
"ciTests": {
"linux": [{
"env": [
"source /intel/oneapi/intelpython/bin/activate",
"pip install -r requirements.txt"
"source /intel/oneapi/intelpython/bin/activate",
"conda activate base",
"pip install uv",
"uv init",
"uv python pin $(which python)",
"uv venv --system-site-packages",
"uv add -r requirements.txt",
"uv add numpy==1.26.4"
],
"id": "idp_d4p_Linear_Regression_Dist",
"steps": [
"python download_data.py",
"mpirun -n 4 python ./IntelPython_daal4py_Distributed_LinearRegression.py"
"uv run python download_data.py",
"uv run mpirun -n 4 python ./IntelPython_daal4py_Distributed_LinearRegression.py"
]
}]
},
Expand Down
15 changes: 10 additions & 5 deletions AI-and-Analytics/Features-and-Functionality/IntelTensorFlow_AMX_BF16_Inference/sample.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,16 @@
"steps": [
"source /intel/oneapi/intelpython/bin/activate",
"conda activate tensorflow",
"pip install -r requirements.txt --no-deps",
"pip install ipykernel jupyter",
"python Intel_TensorFlow_AMX_BF16_Inference.py",
"python -m ipykernel install --user --name=tensorflow",
"jupyter nbconvert --ExecutePreprocessor.enabled=True --ExecutePreprocessor.kernel_name=tensorflow --to notebook IntelTensorFlow_AMX_BF16_Inference.ipynb"
"pip install uv",
"uv init",
"uv python pin $(which python)",
"uv venv --system-site-packages",
"uv add -r requirements.txt",
"uv add numpy==1.26.4",
"uv add ipykernel jupyter notebook",
"uv run python Intel_TensorFlow_AMX_BF16_Inference.py",
"uv run python -m ipykernel install --user --name=tensorflow",
"uv run jupyter nbconvert --ExecutePreprocessor.enabled=True --ExecutePreprocessor.kernel_name=tensorflow2 --to notebook IntelTensorFlow_AMX_BF16_Inference.ipynb"
]
}
]
Expand Down
3 changes: 3 additions & 0 deletions ...d-Functionality/IntelTensorFlow_AMX_BF16_Training/IntelTensorFlow_AMX_BF16_Training.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,10 @@
"is_tune_model = True # or False\n",
"log_dir = \"logs\"\n",
"profiling_needed = False\n",
"execution_mode_param = os.getenv('execution_mode')\n",
"execution_mode = \"graph\"\n",
"if execution_mode_param == \"eager\":\n",
" execution_mode = \"eager\"\n",
"load_weights_dir = \"weights\"\n",
"save_weights_dir = \"weights\""
]
Expand Down
17 changes: 12 additions & 5 deletions AI-and-Analytics/Features-and-Functionality/IntelTensorFlow_AMX_BF16_Training/sample.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,18 @@
"steps": [
"source /intel/oneapi/intelpython/bin/activate",
"conda activate tensorflow",
"pip install -r requirements.txt",
"pip install jupyter ipykernel",
"python Intel_TensorFlow_AMX_BF16_Training.py",
"python -m ipykernel install --user --name=tensorflow",
"jupyter nbconvert --ExecutePreprocessor.enabled=True --ExecutePreprocessor.kernel_name=tensorflow --to notebook IntelTensorFlow_AMX_BF16_Training.ipynb"
"pip install uv",
"uv init",
"uv python pin $(which python)",
"uv venv --system-site-packages",
"uv add -r requirements.txt",
"uv add --dev ipykernel notebook",
"uv add numpy==1.26.4",
"uv add keras==2.15.0",
"export execution_mode=eager",
"uv run python Intel_TensorFlow_AMX_BF16_Training.py -m eager",
"uv run ipython kernel install --user --name tensorflow",
"uv run jupyter nbconvert --ExecutePreprocessor.enabled=True --ExecutePreprocessor.kernel_name=tensorflow --to notebook IntelTensorFlow_AMX_BF16_Training.ipynb"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't think we need kernel_name argument any more

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure, if the .ipynb file not specify the kernel name and command line also not include it, we might meet issue like jupyter_client.kernelspec.NoSuchKernel: No such kernel named tf_amx_bf16_2, maybe keep the arg for now.

]
}]
},
Expand Down
13 changes: 9 additions & 4 deletions ...ctionality/IntelTensorFlow_Enabling_Auto_Mixed_Precision_for_TransferLearning/sample.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,15 @@
"apt-get update && apt-get install tensorflow-model-server",
"source /intel/oneapi/intelpython/bin/activate",
"conda activate tensorflow",
"pip install -r requirements.txt --no-deps",
"pip install tensorflow==2.15.0.post1",
"pip install jupyter ipykernel",
"python -m ipykernel install --user --name=tensorflow"
"pip install uv",
"uv init",
"uv python pin $(which python)",
"uv venv --system-site-packages",
"uv add -r requirements.txt",
"uv add numpy==1.26.4",
"uv add --dev ipykernel",
"uv run ipython kernel install --user --name tensorflow",
"pip install notebook"
],
"id": "tensorflow_AMP_transfer_learning",
"steps": [
Expand Down
15 changes: 9 additions & 6 deletions ...Analytics/Features-and-Functionality/IntelTensorFlow_TextGeneration_with_LSTM/sample.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,19 @@
{
"env": [
"source /intel/oneapi/intelpython/bin/activate",
"conda env remove -n user_tensorflow-gpu",
"conda create --name user_tensorflow-gpu --clone tensorflow-gpu",
"conda activate user_tensorflow-gpu",
"pip install -r requirements.txt",
"python -m ipykernel install --user --name=user_tensorflow-gpu"
"conda activate tensorflow-gpu",
"pip install uv notebook",
"uv init",
"uv python pin $(which python)",
"uv venv --system-site-packages",
"uv add -r requirements.txt",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here, notebook is install separately. Both notebook and ipykernel are not used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, removed

"uv add --dev ipykernel",
"uv run ipython kernel install --user --name=tensorflow-gpu"
],
"id": "inc_text_generation_lstm_py",
"steps": [
"export ITEX_ENABLE_NEXTPLUGGABLE_DEVICE=0",
"ITEX_NUM_EPOCHS=5 KERAS_NUM_EPOCHS=5 python TextGenerationModelTraining.py"
"ITEX_NUM_EPOCHS=5 KERAS_NUM_EPOCHS=5 uv run TextGenerationModelTraining.py"
]
}
]
Expand Down
21 changes: 15 additions & 6 deletions AI-and-Analytics/Features-and-Functionality/IntelTensorFlow_for_LLMs/sample.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,23 @@
"cpuInstructionSets": ["AVX512", "AMX"],
"ciTests": {
"linux": [{
"env": [
"source /intel/oneapi/intelpython/bin/activate",
"conda activate tensorflow",
"pip install uv",
"uv init",
"uv python pin $(which python)",
"uv venv --system-site-packages",
"uv add -r requirements.txt",
"uv add py-cpuinfo nbformat nbconvert",
"uv add --dev ipykernel",
"uv run ipython kernel install --user --name tensorflow",
"uv add notebook nbconvert"
],
"id": "intel llm bf16 infrence and fine-tuning",
"steps": [
"source activate tensorflow",
"pip install -r requirements.txt",
"python -m pip install py-cpuinfo",
"python GPTJ_finetuning.py",
"/opt/intel/oneapi/intelpython/latest/envs/tensorflow/bin/python -m ipykernel install --user --name=tensorflow",
"python ci_test.py"
"uv run python GPTJ_finetuning.py",
"uv run python ci_test.py"
]
}]
},
Expand Down
12 changes: 6 additions & 6 deletions AI-and-Analytics/Features-and-Functionality/IntelTransformers_Quantization/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
accelerate==0.29.3
datasets==2.09.0
intel-extension-for-transformers==1.4.1
neural-speed==1.0
peft==0.10.0
accelerate
datasets
intel-extension-for-transformers
neural-speed
peft
sentencepiece
transformers==4.38.0
transformers
Loading