Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update ondemand #4

Open
wants to merge 101 commits into
base: gh-pages
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
450fc54
Revert "Merge pull request #410 from carpentries-incubator/revert-407…
tkphd Jun 23, 2022
11ad96c
Revert "Merge pull request #410 from carpentries-incubator/revert-407…
tkphd Jun 23, 2022
8a6f91b
Substitute Amdahl tarball for lesson files
tkphd Jan 19, 2023
5403b6a
Merge branch 'amdahl-code-redux' of github:tkphd/hpc-intro into amdah…
tkphd Jan 19, 2023
2309e99
pip install amdahl in parallel episode
tkphd Jan 19, 2023
971442d
fix missing blockquote tags
tkphd Jan 19, 2023
64e034a
see amdahl not cpi in history
tkphd Jan 19, 2023
0eb4195
update Python version
tkphd Jan 19, 2023
fb93453
updated server name
tkphd Jan 19, 2023
115f82d
a few readability fixes
tkphd Jan 19, 2023
decb121
remove unused tarball, consolidate pi code in new tarball
tkphd Jan 31, 2023
2155be8
Ask learners to think about why a transfer may not work
tkphd Jan 31, 2023
b2baaa7
Expand on the remote_destination for scp
tkphd Jan 31, 2023
44dee4f
Clarify that remote_path may not exist
tkphd Jan 31, 2023
4e305be
discuss the tarball before transferring it
tkphd Jan 31, 2023
11c0f81
Merge branch 'amdahl-code-redux' of github:tkphd/hpc-intro into amdah…
tkphd Jan 31, 2023
de82b5d
Ask learners to think about why a transfer may not work
tkphd Jan 31, 2023
4181a65
Expand on the remote_destination for scp
tkphd Jan 31, 2023
5eafc00
Clarify that remote_path may not exist
tkphd Jan 31, 2023
2851268
remove unused tarball, consolidate pi code in new tarball
tkphd Jan 31, 2023
fac84f2
discuss the tarball before transferring it
tkphd Jan 31, 2023
1d489ba
Merge branch 'amdahl-bad-merge' into amdahl-code-redux
tkphd Feb 3, 2023
7dfc411
clean up & clarify a thing or two
tkphd Feb 10, 2023
7b9c78e
revised discussion of tar and scp flags & fields
tkphd Feb 15, 2023
9b656ad
use nano to edit bashrc
tkphd Feb 16, 2023
470c4c5
substitute previously-seen example job ID
tkphd Feb 16, 2023
62fd20d
resolve build failure
tkphd Feb 16, 2023
160e626
Update _episodes/16-parallel.md
ocaisa Feb 16, 2023
3117db8
Merge pull request #411 from tkphd/amdahl-code-redux
ocaisa Feb 16, 2023
f63f2ca
Update 13-scheduler.md
eckhrd Mar 20, 2023
742c396
Merge pull request #431 from eckhrd/patch-1
tkphd Mar 20, 2023
d479704
Update 17-resources.md
eckhrd Apr 20, 2023
05f26f8
Merge pull request #434 from eckhrd/patch-2
tkphd Apr 20, 2023
0646b2a
Generate HPCC Cluster library, fix some random lesson deficiencies
tkphd Jul 6, 2023
fe69a7b
Update using-nodes-interactively.snip
ocaisa Sep 21, 2023
b369a9a
Merge pull request #436 from carpentries-incubator/cluster.hpc-carpen…
ocaisa Sep 21, 2023
790b9cb
Switch default config to HPCC cluster
ocaisa Sep 21, 2023
a8bb0c1
Retain other lesson info
ocaisa Sep 21, 2023
a67d53d
Retain other useful info
ocaisa Sep 21, 2023
5f56875
Update _config.yml
ocaisa Sep 21, 2023
3b80a22
Update _config.yml
ocaisa Sep 21, 2023
a38df44
Update _config.yml
ocaisa Sep 21, 2023
d2f3d2e
Update _config.yml
ocaisa Sep 21, 2023
c356c28
Update _config.yml
ocaisa Sep 21, 2023
f2c5e57
Update _config.yml
ocaisa Sep 21, 2023
072536a
Merge pull request #440 from carpentries-incubator/ocaisa-patch-1
tkphd Sep 21, 2023
dda9cbe
Update Gemfile and fix some typos
tkphd Jan 10, 2024
2783320
codespell args are hard
tkphd Jan 10, 2024
39b9ad4
cleanup crufty text from another lib
tkphd Jan 10, 2024
24d8195
Use packaged codespell instead of pip
tkphd Jan 10, 2024
f4671be
Create missing file
tkphd Jan 10, 2024
292704b
cache bundle
tkphd Jan 10, 2024
f02f317
missing file
tkphd Jan 10, 2024
30a2417
Overhaul markdownlint using config file
tkphd Jan 10, 2024
5683b94
Missed a file
tkphd Jan 10, 2024
2edc1a9
Figuring out how GH Actions work.
tkphd Jan 10, 2024
b725003
Long URLs should maybe be ref-links?
tkphd Jan 10, 2024
23e931f
Complete listing of mdl rules
tkphd Jan 10, 2024
5f19e45
Merge pull request #444 from tkphd/update-workflows
ocaisa Jan 10, 2024
8525205
Added all changes associated with EPCC_Cirrus_slurm.
mbareford Feb 15, 2022
386f6b9
Removed references to Cirrus PBS.
mbareford Feb 15, 2022
bb98847
Update episode list
tkphd Jan 10, 2024
3f08d99
Repair some Markdown, prune some files
tkphd Jan 10, 2024
0172a81
Restore a couple missing files
tkphd Jan 10, 2024
cc4f5f2
Fold templated stuff into core lesson
tkphd Jan 10, 2024
642c4b5
Restore rule: blanks around headings
tkphd Jan 10, 2024
5296cd7
Populate missing config flag, fix broken link
tkphd Jan 10, 2024
b28d74e
Merge pull request #445 from tkphd/epcc-cirrus-slurm
ocaisa Jan 10, 2024
7588e52
Add episode on environment variables
ocaisa Jan 26, 2024
2c78920
Pass linter
ocaisa Jan 26, 2024
d39a26e
Fix code block naming
ocaisa Jan 27, 2024
29e5aa1
Fix linting again
ocaisa Jan 27, 2024
0416de9
Some tweaks
ocaisa Jan 27, 2024
7ab14d5
Further tweaks
ocaisa Jan 27, 2024
e91145b
Make sure environment variables are not referenced before we cover them
ocaisa Jan 28, 2024
198e575
Update snippet lib episode lists
tkphd Feb 1, 2024
5f272f8
Teeny little typo
tkphd Feb 1, 2024
fd61fb6
Merge pull request #446 from ocaisa/runtime_env
tkphd Feb 1, 2024
d275660
Create filezilla-remote-instructions.snip
heatherkellyucl Jun 5, 2023
561becf
Reference a site snippet for Filezilla in the transferring-files lesson
cdkharris Jul 25, 2024
bd6a82c
Untangle comments about rsync on directories
cdkharris Jul 25, 2024
ac2a996
Removing a redundant comment about scp -r
cdkharris Jul 25, 2024
61c0cb3
Merge pull request #454 from UCL-ARC/rsync
reid-a Jul 25, 2024
2c447a6
fix markdown and relocate UCL snippet
cdkharris Jul 25, 2024
c3b35c1
New snippet files for using Fliezilla with SSH tunnels
cdkharris Jul 25, 2024
14cd11e
Merge branch 'gh-pages' into filezilla
cdkharris Jul 25, 2024
20aa87c
Merge pull request #453 from UCL-ARC/filezilla
reid-a Jul 25, 2024
b47837d
Add `-L` option to `curl` command
ocaisa Aug 7, 2024
dfa22e6
Try again
ocaisa Aug 7, 2024
ff7e430
Lint
ocaisa Aug 7, 2024
0a7320a
Whoops
ocaisa Aug 7, 2024
6fef636
Merge pull request #456 from carpentries-incubator/ocaisa-patch-1
reid-a Aug 7, 2024
2eb4a6e
Update login and compute node
cdkharris Sep 23, 2024
a209d45
Update new python scripts with Myriad env setup
cdkharris Sep 23, 2024
c7b43fc
Update option flags for Myriad SGE
cdkharris Sep 23, 2024
39981f1
Merge pull request #458 from UCL-ARC/cdkharris/UCL-update-site-snippets
reid-a Sep 23, 2024
aff84ce
Merge branch 'gh-pages' of https://github.com/carpentries-incubator/h…
jennyfothergill Nov 22, 2024
9ace241
Merging changes to allow tutorial to build locally
jennyfothergill Jan 28, 2025
bbd81ed
Update to match Borah output
jennyfothergill Jan 28, 2025
5703385
Show using module avail with argument
jennyfothergill Jan 29, 2025
08833dc
Remove login node comparison
jennyfothergill Feb 10, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .codespell-ignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Fram
dropse
figurestyle
hist
namd
rouge
40 changes: 24 additions & 16 deletions .github/workflows/test_and_build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,28 +8,35 @@ on:
- gh-pages

jobs:
spellcheck:
lint_common_files:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.7
- name: Install codespell
run: |
pip3 install codespell
sudo apt-get install codespell
- name: Check spelling
run: |
make spellcheck
- name: Check top-level Markdown syntax
uses: DavidAnson/markdownlint-cli2-action@v9
- name: Check episode Markdown syntax
uses: DavidAnson/markdownlint-cli2-action@v9
with:
globs: _episodes/*.md
- name: Check extra Markdown syntax
uses: DavidAnson/markdownlint-cli2-action@v9
with:
globs: _extras/*.md

check_lesson_and_build_default:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: ruby/setup-ruby@v1
with:
ruby-version: '2.7'
ruby-version: '3.0.4'
bundler-cache: true
- name: Install basic requirements
run: |
# Need this library for nokogiri
Expand All @@ -38,9 +45,6 @@ jobs:
bundle config set path '.vendor/bundle'
bundle config build.nokogiri --use-system-libraries
bundle install
- name: "Lint episode markdown"
run: |
find _episodes -name \*.md -exec bundle exec mdl -r MD001,MD003,MD005,MD006,MD007,MD008,MD009,MD010,MD011,MD012,MD015,MD016,MD017,MD018,MD019,MD020,MD021,MD022,MD023,MD025,MD035,MD036,MD037,MD038,MD039,MD046 {} \;
- name: "Check lesson for warnings"
run: |
make lesson-check-all
Expand All @@ -56,18 +60,21 @@ jobs:
strategy:
matrix:
HPC_JEKYLL_CONFIG:
- Birmingham_Baskerville_slurm
- ComputeCanada_Graham_slurm
- EPCC_Cirrus_pbs
- EPCC_Cirrus_slurm
- HPCC_MagicCastle_slurm
- Magic_Castle_EESSI_slurm
- NIST_CTCMS_slurm
- Norway_SIGMA2_SAGA_slurm
- UCL_Myriad_sge
- Magic_Castle_EESSI_slurm
- BSU_Borah_slurm
steps:
- uses: actions/checkout@v2
- uses: ruby/setup-ruby@v1
with:
ruby-version: '2.7'
ruby-version: '3.0.4'
bundler-cache: true
- name: Install basic requirements
run: |
# Need this library for nokogiri
Expand All @@ -79,6 +86,7 @@ jobs:
- name: Check build ${{matrix.HPC_JEKYLL_CONFIG}}
run: |
make --always-make site HPC_JEKYLL_CONFIG=_includes/snippets_library/${{matrix.HPC_JEKYLL_CONFIG}}/_config_options.yml
- name: "Lint snippet markdown"
run: |
find _includes/snippets_library/${{matrix.HPC_JEKYLL_CONFIG}} -name \*.snip -exec bundle exec mdl -r MD001,MD003,MD005,MD006,MD007,MD008,MD009,MD010,MD011,MD012,MD015,MD016,MD017,MD018,MD019,MD020,MD021,MD022,MD023,MD025,MD035,MD036,MD037,MD038,MD039,MD046 {} \;
- name: Lint snippet markdown
uses: DavidAnson/markdownlint-cli2-action@v9
with:
globs: _includes/snippets_library/${{matrix.HPC_JEKYLL_CONFIG}}/**/*.snip
73 changes: 73 additions & 0 deletions .markdownlint-cli2.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# markdownlint-cli2:
# - Example: <https://github.com/DavidAnson/markdownlint-cli2/blob/main/test/markdownlint-cli2-yaml-example/.markdownlint-cli2.yaml>
# - Rules: <https://github.com/DavidAnson/markdownlint/blob/main/README.md#rules--aliases>
---
config:
MD001: false # heading-increment
MD002: false # mdl rule undefined
MD003: true # heading-style
MD004: true # ul-style
MD005: true # list-indent
MD006: false # mdl rule undefined
MD007: true # ul-indent
MD008: false # mdl rule undefined
MD009: true # no-trailing-spaces
MD010: true # no-hard-tabs
MD011: true # no-reversed-links
MD012: true # no-multiple-blanks
MD013: false # line-length
MD014: false # commands-show-output
MD015: false # mdl rule undefined
MD016: false # mdl rule undefined
MD017: false # mdl rule undefined
MD018: true # no-missing-space-atx
MD019: true # no-multiple-space-atx
MD020: true # no-missing-space-closed-atx
MD021: true # no-multiple-space-closed-atx
MD022: true # blanks-around-headings
MD023: true # heading-start-left
MD024: false # no-duplicate-heading
MD025: true # single-h1
MD026: false # no-trailing-punctuation
MD027: false # no-multiple-space-blockquote
MD028: false # no-blanks-blockquote
MD029: true # ol-prefix
MD030: true # list-marker-space
MD031: false # blanks-around-fences
MD032: true # blanks-around-lists
MD033: false # no-inline-html
MD034: true # no-bare-urls
MD035: true # hr-style
MD036: true # no-emphasis-as-heading
MD037: true # no-space-in-emphasis
MD038: true # no-space-in-code
MD039: true # no-space-in-links
MD040: false # fenced-code-language
MD041: false # first-line-h1
MD042: true # no-empty-links
MD043: false # required-headings
MD044: true # proper-names
MD045: true # no-alt-text
MD046: true # code-block-style
MD047: true # single-trailing-newline
MD048: true # code-fence-style
MD049: false # emphasis-style
MD050: false # strong-style
MD051: true # link-fragments
MD052: true # reference-links-images
MD053: false # link-image-reference-definitions
MD054: false # link-image-style
MD055: true # table-pipe-style
MD056: true # table-column-count

ignores:
- .vendor
- .bundle
- _extras/figures.md # really HTML
- _includes/links.md # just a list of links
- "**/*.html"
- "**/*.sh"
- "**/*.yaml"
- "**/*.yml"

# noProgress: true
1 change: 0 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,4 +130,3 @@ which everyone is welcome to join. You can also [reach us by email][email].
[swc-issues]: https://github.com/issues?q=user%3Aswcarpentry
[swc-lessons]: https://software-carpentry.org/lessons/
[swc-site]: https://software-carpentry.org/

18 changes: 16 additions & 2 deletions Gemfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,18 @@
source "https://rubygems.org"
gem "github-pages", group: :jekyll_plugins
# frozen_string_literal: true

source 'https://rubygems.org'

git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }

# Synchronize with https://pages.github.com/versions
ruby '3.3.4'

gem 'github-pages', group: :jekyll_plugins

if Gem::Version.new(RUBY_VERSION) >= Gem::Version.new('3.0.0')
gem 'webrick', '>= 1.6.1'
end

gem "kramdown-parser-gfm"

gem "mdl"
4 changes: 1 addition & 3 deletions LICENSE.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
---
title: "Licenses"
title: "License: CC BY 4.0"
---

# CC BY 4.0

## Instructional Material

All High Performance Computing Carpentry instructional material is
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ lesson-fixme :

## spellcheck
spellcheck:
codespell --skip="assets,*.svg,.vendor" --quiet-level=2 -L "dropse,figurestyle,hist,namd,rouge"
codespell --skip=".bundle,.vendor,assets,*.svg" --quiet-level=2 --ignore-words=".codespell-ignore"

##
## IV. Auxililary (plumbing) commands
Expand Down
20 changes: 11 additions & 9 deletions _config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,14 @@
#------------------------------------------------------------

# Cluster host and scheduler options: the defaults come from
# Graham at Compute Canada, running Slurm. Other options can
# be found in the library of snippets,
# the HPC Carpentry tutorial cluster, running Slurm. Other
# options can be found in the library of snippets,
# `_includes/snippets_library`. To use one, replace options
# below with those in `_config_options.yml` from the
# library. E.g, to customise for Cirrus at EPCC, running
# PBS, we could replace the options below with those from
# Slurm, we could replace the options below with those from
#
# _includes/snippets_library/EPCC_Cirrus_pbs/_config_options.yml
# _includes/snippets_library/EPCC_Cirrus_slurm/_config_options.yml
#
# If your cluster is not represented in the library, please
# copy an existing folder, rename it, and customize for your
Expand All @@ -23,7 +23,7 @@
snippets: "/snippets_library/BSU_Borah_slurm"

local:
prompt: "[user@laptop ~]$"
prompt: "[you@laptop:~]$"
bash_shebang: "#!/usr/bin/env bash"

remote:
Expand All @@ -35,6 +35,7 @@ remote:
homedir: "/bsuhome"
user: "yourUsername"
prompt: "[yourUsername@borah-login ~]$"
module_python3: "python/3.9.7"
bash_shebang: "#!/usr/bin/env bash"

sched:
Expand All @@ -60,16 +61,17 @@ sched:
info: "sinfo"
comment: "#SBATCH"
hist: "sacct"
hist_filter: ""

episode_order:
- 10-hpc-intro
- 11-connecting
- 12-cluster
- 13-scheduler
- 01-break
- 14-modules
- 15-transferring-files
- 18-responsibility
- 15-modules
- 16-transferring-files
- 19-responsibility

#------------------------------------------------------------
# Values for this lesson
Expand All @@ -79,7 +81,7 @@ episode_order:
# "swc": Software Carpentry
# "dc": Data Carpentry
# "lc": Library Carpentry
# "cp": Carpentries (e.g., instructor traning)
# "cp": Carpentries (e.g., instructor training)
carpentry: "incubator"

# Overall title for pages.
Expand Down
42 changes: 21 additions & 21 deletions _episodes/11-connecting.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,17 +19,17 @@ keypoints:
## Secure Connections

The first step in using a cluster is to establish a connection from our laptop
to the cluster. When we are sitting at a computer, we have come to expect a
visual display with icons, widgets, and perhaps some windows or applications:
a _graphical user interface_, or GUI. Since computer clusters are remote
resources that we connect to over slow or intermittent interfaces (WiFi
and VPNs especially), it is more practical to use a _command-line interface_,
or CLI, to send commands as plain-text. If a command returns output, it is
printed as plain text as well. The commands we run today will not open a window
to the cluster. When we are sitting at a computer, we have come to expect a
visual display with icons, widgets, and perhaps some windows or applications:
a _graphical user interface_, or GUI. Since computer clusters are remote
resources that we connect to over slow or intermittent interfaces (WiFi
and VPNs especially), it is more practical to use a _command-line interface_,
or CLI, to send commands as plain-text. If a command returns output, it is
printed as plain text as well. The commands we run today will not open a window
to show graphical results.

If you have already taken The Carpentries' courses on the UNIX Shell or
Version Control, you have used the CLI on your _local machine_ extensively.
If you have already taken The Carpentries' courses on the UNIX Shell or
Version Control, you have used the CLI on your _local machine_ extensively.
The only leap to be made here is to open a CLI on a _remote machine_,
while taking some precautions so that other folks on the network can't see (or
change) the commands you're running or the results the remote machine sends
Expand Down Expand Up @@ -70,12 +70,12 @@ the password prompt are not displayed on the screen. Normal output will resume
once you press `Enter`.

You may have noticed that the prompt changed when you logged into the remote
system using the terminal. This change is important because it can help you
distinguish on which system the commands you type will be run when you pass
them into the terminal. This change is also a small complication that we will
need to navigate throughout the workshop. Exactly what is displayed as the
prompt (which conventionally ends in `$`) in the terminal when it is connected
to the local system and the remote system will typically be different for
system using the terminal. This change is important because it can help you
distinguish on which system the commands you type will be run when you pass
them into the terminal. This change is also a small complication that we will
need to navigate throughout the workshop. Exactly what is displayed as the
prompt (which conventionally ends in `$`) in the terminal when it is connected
to the local system and the remote system will typically be different for
every user. We still need to indicate which system we are entering commands
on though so we will adopt the following convention:

Expand All @@ -87,7 +87,7 @@ on though so we will adopt the following convention:

## Changing Your Password

When your account is created, Research Computing assigns you a password. The
When your account is created, Research Computing assigns you a password. The
first thing you should do upon logging in is change it!

You can change your password by entering the `passwd` command as shown below:
Expand All @@ -105,12 +105,12 @@ the new password, and finally confirmation of the new password.
> When prompted, enter a strong password that you will remember. There are two
> common approaches to this:
>
> 1. Create a memorable passphrase with some punctuation, mixed-case and
> number-for-letter substitutions, 32 characters or longer. Please note
> 1. Create a memorable passphrase with some punctuation, mixed-case and
> number-for-letter substitutions, 32 characters or longer. Please note
> that passwords are case sensitive.
> 2. Use a password manager and its built-in password generator with all
> character classes, 25 characters or longer. [KeePass][keepass] and
> [BitWarden][bitwarden] are two good options. This is also a good option
> [BitWarden][bitwarden] are two good options. This is also a good option
> for storing passwords.
{: .callout}

Expand Down Expand Up @@ -157,8 +157,8 @@ scratch
```
{: .output}

The system administrators have configured your home directory with a link
(a shortcut) to a scratch space reserved for you. You can also include
The system administrators have configured your home directory with a link
(a shortcut) to a scratch space reserved for you. You can also include
hidden files in your directory listing:

```
Expand Down
Loading
Loading