Skip to content

Commit fac0d13

Browse files
dsrobertsdsroberts
authored
Merge pull request #72 from coecms/python_venvs
Python venvs
2 parents ac48c2c + b9b298d commit fac0d13

6 files changed

+194
-5
lines changed

‎images/ARE-recommended.PNG

149 KB
Loading

‎images/dashboard_w40.PNG

52.3 KB
Loading

‎images/notebook_virtualenv_launch.gif

229 KB
Loading

‎post_meta.yaml

+5-1
Original file line numberDiff line numberDiff line change
@@ -289,4 +289,8 @@
289289

290290
2024-05-20-pytorch_intro:
291291
title: Introduction to Pytorch
292-
tags: [ python ML ]
292+
tags: [ python ML ]
293+
294+
2024-06-05-mixing-python-envs:
295+
title: Building custom python environments on top of conda/analysis3
296+
tags: [ python ]

‎posts/2022-04-26-storage-where-what-why-how.md

+2-4
Original file line numberDiff line numberDiff line change
@@ -37,11 +37,9 @@ To query usage for an on-disk file system, NCI provides commands which report on
3737
nci-files-report --group project_id
3838
```
3939

40-
CMS has put in place a `Grafana` server for visualising a range of accounting statistics for CLEX. You can access [this server](https://accessdev.nci.org.au/grafana/login:) using your NCI credentials.
40+
CMS has put in place a dashboard for visualising a range of accounting statistics for CLEX. Please contact CMS for the passowrd to access [this server](http://climate-cms.org/dashboard/)
4141

42-
You can access a range of useful statistics by user and group via the [User Report dashboard](https://accessdev.nci.org.au/grafana/d/toeLAYDWz/user-report?orgId=1)
43-
44-
![Grafana User Report](../images/grafana_example.png "Grafana User Report")
42+
![Dashboard](../images/dashboard_w40.PNG "Dashboard")
4543

4644
More details on how to use these and other accounting tools are available from the [CMS wiki](http://climate-cms.wikis.unsw.edu.au/Accounting_at_NCI).
4745

‎posts/2024-06-05-mixing-python-envs.md

+187
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
---
2+
layout: post
3+
title: Building custom python environments on top of the hh5 conda analysis environments
4+
author: Dale Roberts
5+
categories: python
6+
---
7+
<script src="https://cdn.jsdelivr.net/gh/atteggiani/animated-terminal/animated-terminal.min.js" defer></script>
8+
9+
# Building custom python environments on top of `conda/analysis3`
10+
11+
## Introduction
12+
13+
The conda analysis environments in the `hh5` project are a community-led approach to create a central
14+
python environment specifically for the analysis of climate and weather data. As such, these environments
15+
contain a wide range of packages curated by the CLEX CMS team that can fulfil the requirements of most
16+
analysis workflows.
17+
18+
CMS uses the following guidelines to determine whether a package can go into the `analysis3-unstable`
19+
environment:
20+
* Is the package available from `conda-forge` or some other reputable conda channel?
21+
* Is the package in line with the scope of the environment (i.e. climate data analysis)?
22+
* Is the package actively maintained?
23+
* Is the package and its dependencies compatible with the existing environment?
24+
25+
If the answer to all of these questions is yes, then CMS will install the package into the environment on request. If, however, these conditions aren't met, that doesn't mean that you need to create a new conda environment
26+
from scratch to use a new package.
27+
28+
## `pip install --user`
29+
30+
Lets use the [xWMB](https://github.com/hdrake/xwmb) package as an example.
31+
There is no conda distribution for this package, it can only be installed by `pip` from a git repository so it does not meet our criteria above. This
32+
package also has an extensive list of dependencies, most of which are available in the `conda/analysis3` environment, so using the `analysis3`
33+
environment as a base will save quite a lot of disk space.
34+
35+
The simplest approach is to load the `conda/analysis3` environment and `pip install --user` the xWMB package.
36+
<terminal-window typingDelay=30 lineDelay=200>
37+
<terminal-line data="input">module use /g/data/hh5/public/modules</terminal-line>
38+
<terminal-line data="input">module load conda/analysis3</terminal-line>
39+
<terminal-line data="input">pip install -\\\-user git+https://github.com/hdrake/xwmb.git@main</terminal-line>
40+
<terminal-line>Collecting git+https://github.com/hdrake/xwmb.git@main</terminal-line>
41+
<terminal-line> Cloning https://github.com/hdrake/xwmb.git (to revision main) to /scratch/v45/dr4292/tmp/pip-req-build-qox565pe</terminal-line>
42+
<terminal-line> Running command git clone -\\\-filter=blob:none -\\\-quiet https://github.com/hdrake/xwmb.git /scratch/v45/dr4292/tmp/pip-req-build-qox565pe</terminal-line>
43+
<terminal-line> ...</terminal-line>
44+
<terminal-line>Successfully installed regionate-0.0.1 sectionate-0.2.1 xbudget-0.1.0 xgcm-0.8.2.dev15+g7492277 xwmb-0.1.1 xwmt-0.1.1</terminal-line>
45+
</terminal-window>
46+
47+
This is sufficient in certain cases, for instance, quickly testing user code, or when long-term
48+
reproducibility is not necessary. The downside with this approach is that it can lead to problems in the future. First of all, everything will be installed in your `/home` directory, which has a small quota and can easily be filled by local `pip` installations. Another consideration is that the `analysis3` module
49+
is updated to every 3 months as new environments are released. For example, if you load `conda/analysis3` today,
50+
you'll get the `conda/analysis3-24.01` environment. If you were to load the same module a couple of months from
51+
now, you'll get the `conda/analysis3-24.04` environment. Our policies around updating the conda environments
52+
can be found [here](https://climate-cms.org/cms-wiki/services/services-conda.html). As time goes on, your
53+
locally installed package will get further and further out of date, and could potentially conflict with newer
54+
packages installed in updated analysis environments.
55+
56+
In our advice to users, we recommend the following Advanced Settings for ARE environments:
57+
58+
![ARE recommended settings](../images/ARE-recommended.PNG "ARE recommended JupyterLab settings")
59+
60+
Note the `PYTHONNOUSERSITE=1` environment variable. This will have the effect of preventing anything installed with `pip install --user` from loading at all. This setting also works in the terminal and can be used as a first step to diagnosing any issues with python package imports.
61+
62+
## Virtual environments
63+
What we recommend is creating a [virtual environment](https://docs.python.org/3/library/venv.html). A virtual environment is a self-contained Python
64+
environment built on top of an existing Python installation. The self-contained
65+
nature of a virtual environment means that any issues due to conflicting
66+
dependencies can be avoided entirely. Being able to base the environment on top of
67+
an existing `conda/analysis3` environment means that the installation can be kept
68+
small, as most of the dependencies will have already been satisfied.
69+
70+
There are only two additional commands required to install `xWMB` into a virtual environment, as shown below:
71+
<terminal-window typingDelay=30 lineDelay=200>
72+
<terminal-line data="input">module use /g/data/hh5/public/modules</terminal-line>
73+
<terminal-line data="input">module load conda/analysis3</terminal-line>
74+
<terminal-line data="input">python3 -m venv xwmb_venv -\\\-system-site-packages</terminal-line>
75+
<terminal-line data="input">source xwmb_venv/bin/activate</terminal-line>
76+
<terminal-line data="input" inputChar="(xwmb_venv) $">pip install git+https://github.com/hdrake/xwmb.git@main</terminal-line>
77+
<terminal-line>Collecting git+https://github.com/hdrake/xwmb.git@main</terminal-line>
78+
<terminal-line> Cloning https://github.com/hdrake/xwmb.git (to revision main) to /scratch/v45/dr4292/tmp/pip-req-build-qox565pe</terminal-line>
79+
<terminal-line> Running command git clone -\\\-filter=blob:none -\\\-quiet https://github.com/hdrake/xwmb.git /scratch/v45/dr4292/tmp/pip-req-build-qox565pe</terminal-line>
80+
<terminal-line> ...</terminal-line>
81+
<terminal-line>Successfully installed regionate-0.0.1 sectionate-0.2.1 xbudget-0.1.0 xgcm-0.8.2.dev15+g7492277 xwmb-0.1.1 xwmt-0.1.1</terminal-line>
82+
</terminal-window>
83+
```{warning}
84+
Do not use `pip install --user` here, as this will result in the packages being installed in your `~/.local` directory, leaving the virtual environment empty
85+
```
86+
87+
And that's it. We now have a fully self-contained environment with a custom
88+
package built on top of the `conda/analysis3` environment. It is important to note
89+
that this virtual environment is tied to the version of `conda/analysis3` that was loaded
90+
at the time of its creation, so it avoids the out-of-date dependencies issue mentioned earlier.
91+
<terminal-window typingDelay=30 lineDelay=200>
92+
<terminal-line data="input" inputChar="(xwmb_venv) $">which python3</terminal-line>
93+
<terminal-line>~/xwmb_venv/bin/python3</terminal-line>
94+
<terminal-line data="input" inputChar="(xwmb_venv) $">ls -l ~/xwmb_venv/bin/python3</terminal-line>
95+
<terminal-line>lrwxrwxrwx 1 dr4292 v45 67 Jun 5 14:43 /home/563/dr4292/xwmb_venv/bin/python3 -> /g/data/hh5/public/apps/miniconda3/envs/analysis3-24.01/bin/python3</terminal-line>
96+
</terminal-window>
97+
98+
Immediately after loading the `conda/analysis3` module, you can run
99+
`source xwmb_venv/bin/activate` and any `python` command or script will now run under the new virtual environment.
100+
```
101+
#!/usr/bin/env bash
102+
#PBS -l walltime=1:00:00,mem=4GB,ncpus=1,storage=gdata/hh5
103+
104+
module use /g/data/hh5/public/modules
105+
module load conda/analysis3
106+
source xwmb_venv/bin/activate
107+
108+
python3 my_script.py
109+
```
110+
111+
If you're running a script directly, you can
112+
change the path in the [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) line to the path to the `python` symlink in the virtual environment:
113+
```
114+
#!/home/563/dr4292/xwmb_venv/bin/python3
115+
116+
import xwmb
117+
...
118+
```
119+
120+
## Troubleshooting
121+
Virtual environments are designed to be disposable, so if something goes wrong, instead of trying
122+
to fix the environment in place, it is safe to simply delete the environment and start again.
123+
There are many reasons this might need to happen, for example, you're installing an old
124+
package that is not compatible with e.g. `numpy` installed in the `conda/analysis3` environment.
125+
In that case, you could simply delete the environment, load an older `conda/analysis3` environment
126+
and re-create it.
127+
128+
<terminal-window typingDelay=30 lineDelay=200>
129+
<terminal-line data="input" inputChar="(xwmb_venv) $">deactivate</terminal-line>
130+
<terminal-line data="input">rm -rf xwmb_venv</terminal-line>
131+
<terminal-line data="input">module unload conda</terminal-line>
132+
<terminal-line data="input">module load conda/analysis3-23.01</terminal-line>
133+
<terminal-line data="input">python3 -m venv xwmb_venv --system-site-packages</terminal-line>
134+
<terminal-line data="input">source xwmb_venv/bin/activate</terminal-line>
135+
<terminal-line data="input" inputChar="(xwmb_venv) $">pip install git+https://github.com/hdrake/xwmb.git@main</terminal-line>
136+
</terminal-window>
137+
138+
In a similar vein, if you're installing more packages into a virtual environment and they
139+
begin to conflict with each other, the best course of action is to create a new virtual environment.
140+
Virtual environments are isolated from each other, so rather than creating large, monolithic
141+
environments, it is common practice to create virtual environments for a single package, and have
142+
multiple virtual environments depending on your immediate requirements.
143+
144+
In general, if anything in the environment doesn't work for whatever reason, the best course of action
145+
is to delete it and try again. You may need to try different `conda/analysis3` environments, or, if none of those
146+
work, use one of the NCI installed Python modules as your base.
147+
```{note}
148+
NCI's Python modules do not have `dask`, `xarray`, etc. installed, so the resultant virtual environments could be quite large.
149+
```
150+
151+
152+
## Jupyter Kernel
153+
There is one more step necessary to have your new virtual environment usable as a kernel in an ARE JupyterLab session.
154+
You'll need to call `ipykernel install` while the virtual environment is activated.
155+
<terminal-window typingDelay=30 lineDelay=200>
156+
<terminal-line data="input">module use /g/data/hh5/public/modules</terminal-line>
157+
<terminal-line data="input">module load conda/analysis3</terminal-line>
158+
<terminal-line data="input">source xwmb_venv/bin/activate</terminal-line>
159+
<terminal-line data="input" inputChar="(xwmb_venv) $">python3 -m ipykernel install -\\\-user -\\\-name xwmb-venv -\\\-display-name "xWMB + conda/analysis3"</terminal-line>
160+
</terminal-window>
161+
162+
```{note}
163+
Since this virtual environment is built on top of `conda/analysis3`, you do not need to install `ipykernel` in it, as this is already present in the conda environment.
164+
```
165+
166+
Now, when
167+
you launch a new ARE JupyterLab instance, you will be able to see your new environment,
168+
launch a notebook and import the `xWMB` package
169+
170+
![Jupyterlab launch](../images/notebook_virtualenv_launch.gif)
171+
172+
You only have to do this once. If you delete the environment and recreate it at the same path,
173+
Jupyterlab will still be able to use it.
174+
175+
## Summary
176+
177+
While CMS tries to make the `hh5` conda analysis environments as comprehensive as possible, we can't
178+
always install the packages you need. By creating a virtual environment using the `conda/analysis3`
179+
environment as your base,
180+
its possible to add in packages CMS can't install, without having the overhead of maintaining
181+
your own large conda environment. Furthermore, using a virtual environment is more stable and easier to maintain and reuse than installing software using `pip install --user`. Virtual environments are simple to
182+
install, and simple to delete and re-create as necessary. Virtual environments can also be seamlessly integrated into
183+
ARE JupyterLab sessions, allowing you to use it anywhere you would use an `analysis3` conda environment.
184+
185+
186+
## Acknowledgements
187+
Thanks to Davide Marchegiani for [animated-terminal.js](https://github.com/atteggiani/animated-terminal.js).

0 commit comments

Comments
 (0)