Skip to content

Simplify python environment setup #310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Feb 21, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .binder/environment.yml

This file was deleted.

42 changes: 42 additions & 0 deletions .binder/environment.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
name: default
channels:
- conda-forge
- nodefaults
dependencies:
- jupyter-book >=1.0.3,<2
- pre-commit >=4.1.0,<5
- dask-labextension >=7.0.0,<8
- jupyterlab >=4.3.5,<5
- jupyter_bokeh >=4.0.5,<5
- jupyterlab-myst >=2.4.2,<3
- jupyter-resource-usage >=1.1.1,<2
- cartopy >=0.24.0,<0.25
- cf_xarray >=0.10.0,<0.11
- dask >=2025.2.0,<2026
- datashader >=0.17.0,<0.18
- distributed >=2025.2.0,<2026
- gcsfs >=2025.2.0,<2026
- geoviews-core >=1.14.0,<2
- gsw >=3.6.19,<4
- hvplot >=0.11.2,<0.12
- h5netcdf >=1.5.0,<2
- ipykernel >=6.29.5,<7
- matplotlib-base >=3.10.0,<4
- netcdf4 >=1.7.2,<2
- numpy >=2.1.3,<3
- pint-xarray >=0.4,<0.5
- pydap >=3.5.3,<4
- python-graphviz >=0.20.3,<0.21
- pooch >=1.8.2,<2
- rioxarray >=0.18.2,<0.19
- scipy >=1.15.2,<2
- sphinx-codeautolink >=0.17.0,<0.18
- sphinxcontrib-mermaid >=1.0.0,<2
- sphinx-notfound-page >=1.0.4,<2
- sphinxext-rediraffe >=0.2.7,<0.3
- s3fs >=2025.2.0,<2026
- xarray >=2025.1.2,<2026
- zarr >=3.0.3,<4
- flox >=0.10.0,<0.11
- numbagg >=0.9.0,<0.10
- python >=3.10
1 change: 0 additions & 1 deletion .devcontainer/Dockerfile

This file was deleted.

5 changes: 1 addition & 4 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,5 @@
{
"build": {
"dockerfile": "./Dockerfile",
"context": "../conda"
},
"image": "quay.io/pangeo/pangeo-notebook:latest",
"postCreateCommand": {
"vscode": "mkdir ${containerWorkspaceFolder}/.vscode && cp ${containerWorkspaceFolder}/.devcontainer/tasks.json ${containerWorkspaceFolder}/.vscode/tasks.json"
},
Expand Down
5 changes: 1 addition & 4 deletions .devcontainer/scipy2023/devcontainer.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,5 @@
{
"build": {
"dockerfile": "../Dockerfile",
"context": "../../conda"
},
"image": "quay.io/pangeo/pangeo-notebook:2023.07.05",
"postCreateCommand": {
"jupyterlab": "mkdir /home/jovyan/.jupyter && cp ${containerWorkspaceFolder}/.devcontainer/scipy2023/jupyter_lab_config.py /home/jovyan/.jupyter/jupyter_lab_config.py",
"vscode": "mkdir ${containerWorkspaceFolder}/.vscode && cp ${containerWorkspaceFolder}/.devcontainer/scipy2023/tasks.json ${containerWorkspaceFolder}/.vscode/tasks.json"
Expand Down
12 changes: 0 additions & 12 deletions .devcontainer/scipy2023/jupyter_lab_config.py
Original file line number Diff line number Diff line change
@@ -1,15 +1,3 @@
# Configuration file for lab.

c = get_config() # noqa

## The default URL to redirect to from `/`
# Default: '/lab'
c.LabApp.default_url = '/lab/tree/workshops/scipy2023/index.ipynb'

## Set the Access-Control-Allow-Origin header
#
# Use '*' to allow any origin to access your server.
#
# Takes precedence over allow_origin_pat.
# Default: ''
c.ServerApp.allow_origin = '*'
5 changes: 1 addition & 4 deletions .devcontainer/scipy2024/devcontainer.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,5 @@
{
"build": {
"dockerfile": "../Dockerfile",
"context": "../../conda"
},
"image": "quay.io/pangeo/pangeo-notebook:2024.07.08",
"postCreateCommand": {
"jupyterlab": "mkdir /home/jovyan/.jupyter && cp ${containerWorkspaceFolder}/.devcontainer/scipy2024/jupyter_lab_config.py /home/jovyan/.jupyter/jupyter_lab_config.py",
"vscode": "mkdir ${containerWorkspaceFolder}/.vscode && cp ${containerWorkspaceFolder}/.devcontainer/scipy2024/tasks.json ${containerWorkspaceFolder}/.vscode/tasks.json"
Expand Down
12 changes: 0 additions & 12 deletions .devcontainer/scipy2024/jupyter_lab_config.py
Original file line number Diff line number Diff line change
@@ -1,15 +1,3 @@
# Configuration file for lab.

c = get_config() # noqa

## The default URL to redirect to from `/`
# Default: '/lab'
c.LabApp.default_url = '/lab/tree/workshops/scipy2024/index.ipynb'

## Set the Access-Control-Allow-Origin header
#
# Use '*' to allow any origin to access your server.
#
# Takes precedence over allow_origin_pat.
# Default: ''
c.ServerApp.allow_origin = '*'
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# SCM syntax highlighting & preventing 3-way merges
pixi.lock merge=binary linguist-language=YAML linguist-generated=true
4 changes: 0 additions & 4 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
# Regularly update Docker tags and Actions steps
version: 2
updates:
- package-ecosystem: "docker"
directory: "/.devcontainer"
schedule:
interval: "daily"
- package-ecosystem: "github-actions"
directory: "/.github"
schedule:
Expand Down
13 changes: 5 additions & 8 deletions .github/workflows/main.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,18 +30,15 @@ jobs:
with:
path: _build
# NOTE: change key to "jupyterbook-DATE" to force rebuilding cache
key: jupyterbook-20240517
key: jupyterbook-20250221

- name: Install Conda environment with Micromamba
uses: mamba-org/setup-micromamba@v1
- uses: prefix-dev/[email protected]
with:
environment-file: conda/conda-lock.yml
environment-name: xarray-tutorial
cache-environment: true
manifest-path: pyproject.toml
cache: true
activate-environment: true

- name: Build JupyterBook
# NOTE: login shell activates conda environment
shell: bash -el {0}
run: |
jupyter-book build ./ --warningiserror --keep-going

Expand Down
37 changes: 37 additions & 0 deletions .github/workflows/nocache.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
name: Rebuild Entire Jupyter Book on all Platforms

on:
workflow_dispatch:

# Allow one concurrent deployment
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
build:
name: Build on ${{ matrix.runs-on }}
runs-on: ${{ matrix.runs-on }}
strategy:
fail-fast: false
matrix:
runs-on: [ubuntu-latest, macos-latest, windows-latest]

steps:
- name: Checkout repository
uses: actions/checkout@v4

- uses: prefix-dev/[email protected]
with:
manifest-path: pyproject.toml
cache: true
activate-environment: true

- name: Build JupyterBook
run: |
jupyter-book build ./ --warningiserror --keep-going

- name: Dump Build Logs
if: always()
run: |
if (test -a _build/html/reports/*log); then cat _build/html/reports/*log ; fi
15 changes: 5 additions & 10 deletions .github/workflows/pull_request.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,6 @@ concurrency:
jobs:
preview:
runs-on: ubuntu-latest
defaults:
run:
shell: bash -el {0}
steps:
- name: Checkout repository
if: github.event.action != 'closed'
Expand All @@ -27,15 +24,13 @@ jobs:
with:
path: _build
# NOTE: change key to "jupyterbook-DATE" to force rebuilding cache
key: jupyterbook-20240517
key: jupyterbook-20250221

- name: Install Conda environment with Micromamba
if: github.event.action != 'closed'
uses: mamba-org/setup-micromamba@v1
- uses: prefix-dev/[email protected]
with:
environment-file: conda/conda-lock.yml
environment-name: xarray-tutorial
cache-environment: true
manifest-path: pyproject.toml
cache: true
activate-environment: true

- name: Build JupyterBook
if: github.event.action != 'closed'
Expand Down
12 changes: 4 additions & 8 deletions .github/workflows/qaqc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,19 +14,15 @@ concurrency:
jobs:
quality-control:
runs-on: ubuntu-latest
defaults:
run:
shell: bash -el {0}

steps:
- uses: actions/checkout@v4

- name: Install Conda environment with Micromamba
uses: mamba-org/setup-micromamba@v1
- uses: prefix-dev/[email protected]
with:
environment-file: conda/conda-lock.yml
environment-name: xarray-tutorial
cache-environment: true
manifest-path: pyproject.toml
cache: true
activate-environment: true

# NOTE: this isn't a comprehensive spellcheck, just common typos
- name: Spellcheck
Expand Down
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,7 @@ venv.bak/
# misc
.DS_Store
.vscode/

# pixi environments
.pixi
*.egg-info
5 changes: 5 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,11 @@ repos:
- id: check-yaml
- id: double-quote-string-fixer

- repo: https://github.com/codespell-project/codespell
rev: "v2.3.0"
hooks:
- id: codespell

- repo: https://github.com/psf/black
rev: 25.1.0
hooks:
Expand Down
1 change: 0 additions & 1 deletion .prettierignore

This file was deleted.

57 changes: 45 additions & 12 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,9 @@ guidelines:
1. Be concise and limited to one or two topics, such that a reader can get through the example within a few minutes of reading
1. Be of general relevance to Xarray users, and so not too specific on a particular problem or use case.

## Fork this repository
## Contribution process

### Fork this repository

We recommend first forking this repository and creating a local copy:

Expand All @@ -31,24 +33,28 @@ git clone https://github.com/YOURACCOUNT/xarray-tutorial.git
cd xarray-tutorial
```

## Create a Python environment

You'll need `conda` or `mamba`, which can be installed from https://github.com/conda-forge/miniforge
### Create a Python environment

We also use [conda-lock](https://conda.github.io/conda-lock/) to ensure we have reproducible environments across different operating systems
You'll need `pixi` or `conda` or `mamba`, which can be installed from https://github.com/conda-forge/miniforge

We also use [pre-commit hooks](https://pre-commit.com) to run styling and other checks before committing code.

#### Using pixi (recommended)

```
pixi install
pixi shell # exit
```
conda-lock install -f conda/conda-lock.yml
# Or latest package versions: `mamba env create -f conda/environment-unpinned.yml`

conda activate xarray-tutorial
#### Using conda

```
mamba env create -f .binder/environment.yml -n xarray-tutorial
conda activate xarray-tutorial # conda deactivate
pre-commit install
```

## Add content
### Add content

Develop your new content on a branch. See [JupyterBook Docs](https://jupyterbook.org/en/stable/intro.html) for guides on adding `.md`, `.ipynb` and other content.

Expand All @@ -58,18 +64,45 @@ git add .
git commit -m "added pages x,y and improved z"
```

## Preview your changes
### Preview your changes

Running jupyterbook will execute notebooks and render HTML pages for the website. Be sure to fix any execution errors and preview the website in your web browser to make sure everything looks good!

```
jb build .
jupyter-book build ./ --warningiserror --keep-going
# Or "pixi run build"
```

## Open a pull request
### Open a pull request

```
git push
```

Follow the link reported in a terminal to open a pull request!

## Instructions for environment management

[`pixi`](https://pixi.sh) can be used to create and update a multi-platform lockfile, so a reproducible set of package versions is installed across different operating systems.

Dependencies (with optional pins) are specified in the `pyproject.toml` file, and specific locked versions for all platforms are kept in `pixi.lock`.

Install environment from the lockfile

```
pixi install
pixi shell # activate environment, "exit" to deactivate
```

Upgrade all packages to latest versions:

```
pixi upgrade
```

## Render conda/mamba environment files

```
pixi project export conda-environment -p linux-64 .binder/environment.yml
pixi project export conda-explicit-spec -p linux-64 /tmp
```
13 changes: 3 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,12 @@ All notebooks can be run via the Mybinder.org 'Launch Binder' badge at the top o

#### Github Codespaces

This tutorial is available to run within [Github Codespaces](https://github.com/features/codespaces) - "a development environment that's hosted in the cloud" - with the conda environment specification in the [`conda-lock.yml`](conda/conda-lock.yml) file.
This tutorial is available to run within [Github Codespaces](https://github.com/features/codespaces) - "a development environment that's hosted in the cloud"

[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/xarray-contrib/xarray-tutorial/tree/main)

☝️ Click the button above to go to options window to launch a Github codespace.

A codespace is a development environment that's hosted in the cloud.
GitHub currently gives every user [120 vCPU hours per month for free](https://docs.github.com/en/billing/managing-billing-for-github-codespaces/about-billing-for-github-codespaces#monthly-included-storage-and-core-hours-for-personal-accounts), beyond that you must pay. **So be sure to explicitly stop or shut down your codespace when you are done by going to this page (https://github.com/codespaces).**

Once your codespace is launched, the following happens:
Expand All @@ -34,18 +33,12 @@ Once your codespace is launched, the following happens:

#### Locally

You can also run these notebooks on your own computer! We recommend using [`micromamba`](https://mamba.readthedocs.io/en/latest/installation.html#micromamba) or [`conda-lock`](https://conda-incubator.github.io/conda-lock/) to ensure a fully reproducible Python environment:
You can also run these notebooks on your own computer! We recommend using [`pixi`](https://pixi.sh/latest/#installation) to ensure a fully reproducible Python environment:

```bash
git clone https://github.com/xarray-contrib/xarray-tutorial.git
cd xarray-tutorial

conda-lock install conda/conda-lock.yml --name xarray-tutorial
# Or `micromamba create -n xarray-tutorial -f conda-lock.yml`
# Or latest package versions: `mamba env create -f conda/environment-unpinned.yml`

conda activate xarray-tutorial
jupyter lab
pixi run tutorial
```

## Contributing
Expand Down
Loading
Loading