Skip to content

Commit

Permalink
update episode 05
Browse files Browse the repository at this point in the history
  • Loading branch information
@DininduSenanayake
DininduSenanayake committed Jul 28, 2024
1 parent 48aa62c commit 3c0c41d
Showing 1 changed file with 66 additions and 210 deletions.
276 changes: 66 additions & 210 deletions docs/05-writing-scripts.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,39 +13,20 @@
- How can we automate a commonly used set of commands?


<script language="javascript" type="text/javascript">
function set_page_view_defaults() {
document.getElementById('div_win').style.display = 'block';
document.getElementById('div_unix').style.display = 'none';
};

function change_content_by_platform(form_control){
if (!form_control || document.getElementById(form_control).value == 'win') {
set_page_view_defaults();
} else if (document.getElementById(form_control).value == 'unix') {
document.getElementById('div_win').style.display = 'none';
document.getElementById('div_unix').style.display = 'block';
} else {
alert("Error: Missing platform value for 'change_content_by_platform()' script!");
}
}

window.onload = set_page_view_defaults;
</script>

## Writing files

We've been able to do a lot of work with files that already exist, but what if we want to write our own files? We're not going to type in a FASTA file, but we'll see as we go through other tutorials, there are a lot of reasons we'll want to write a file, or edit an existing file.

To add text to files, we're going to use a text editor called Nano. We're going to create a file to take notes about what we've been doing with the data files in `~/obss_2023/commandline/shell_data/untrimmed_fastq`.
To add text to files, we're going to use a text editor called Nano. We're going to create a file to take notes about what we've been doing with the data files in `~/shell_data/untrimmed_fastq`.

This is good practice when working in bioinformatics. We can create a file called `README.txt` that describes the data files in the directory or documents how the files in that directory were generated. As the name suggests, it's a file that we or others should read to understand the information in that directory.

Let's change our working directory to `~/obss_2023/commandline/shell_data/untrimmed_fastq` using `cd`,
Let's change our working directory to `~/shell_data/untrimmed_fastq` using `cd`,
then run `nano` to create a file called `README.txt`:

```bash
$ cd ~/obss_2023/commandline/shell_data/untrimmed_fastq
$ cd ~/shell_data/untrimmed_fastq
$ nano README.txt
```

Expand All @@ -55,7 +36,7 @@ You should see something like this:

The text at the bottom of the screen shows the keyboard shortcuts for performing various tasks in `nano`. We will talk more about how to interpret this information soon.

::::::::::::::::::::::::::::::::::::::::: callout


## Which Editor?

Expand All @@ -80,7 +61,7 @@ your computer's start menu, the editor may want to save files in your desktop or
documents directory instead. You can change this by navigating to
another directory the first time you "Save As..."

::::::::::::::::::::::::::::::::::::::::::::::::::


Let's type in a few lines of text. Describe what the files in this
directory are or what you've been doing with them.
Expand All @@ -91,7 +72,7 @@ press <kbd>Return</kbd> to accept the suggested default of `README.txt`.
Once our file is saved, we can use <kbd>Ctrl</kbd>\-<kbd>X</kbd> to quit the `nano` editor and
return to the shell.

::::::::::::::::::::::::::::::::::::::::: callout


## Control, Ctrl, or ^ Key

Expand All @@ -111,26 +92,19 @@ In `nano`, along the bottom of the screen you'll see `^G Get Help ^O WriteOut`.
This means that you can use <kbd>Ctrl</kbd>\-<kbd>G</kbd> to get help and <kbd>Ctrl</kbd>\-<kbd>O</kbd> to save your
file.

::::::::::::::::::::::::::::::::::::::::::::::::::

Now you've written a file. You can take a look at it with `less` or `cat`, or open it up again and edit it with `nano`.

::::::::::::::::::::::::::::::::::::::: challenge

## Exercise

Open `README.txt` and add the date to the top of the file and save the file.
Now you've written a file. You can take a look at it with `less` or `cat`, or open it up again and edit it with `nano`.

::::::::::::::: solution

## Solution
!!! dumbbell "Exercise"

Use `nano README.txt` to open the file.
Add today's date and then use <kbd>Ctrl</kbd>\-<kbd>X</kbd> followed by `y` and <kbd>Enter</kbd> to save.
Open `README.txt` and add the date to the top of the file and save the file.

:::::::::::::::::::::::::
??? success "Solution"

::::::::::::::::::::::::::::::::::::::::::::::::::
Use `nano README.txt` to open the file.
Add today's date and then use <kbd>Ctrl</kbd>\-<kbd>X</kbd> followed by `y` and <kbd>Enter</kbd> to save.

## Writing scripts

Expand All @@ -140,92 +114,97 @@ One thing we will commonly want to do with sequencing results is pull out bad re

We're going to create a new file to put this command in. We'll call it `bad-reads-script.sh`. The `sh` isn't required, but using that extension tells us that it's a shell script.

```bash
$ nano bad-reads-script.sh
```
!!! terminal "code"

```bash
$ nano bad-reads-script.sh
```

Bad reads have a lot of N's, so we're going to look for `NNNNNNNNNN` with `grep`. We want the whole FASTQ record, so we're also going to get the one line above the sequence and the two lines below. We also want to look in all the files that end with `.fastq`, so we're going to use the `*` wildcard.

```bash
grep -B1 -A2 -h NNNNNNNNNN *.fastq | grep -v '^--' > scripted_bad_reads.txt
```
!!! terminal "code"

```bash
grep -B1 -A2 -h NNNNNNNNNN *.fastq | grep -v '^--' > scripted_bad_reads.txt
```


::::::::::::::::::::::::::::::::::::::::: callout

## Custom `grep` control

We introduced the `-v` option in [the previous episode](04-redirection.md), now we
are using `-h` to "Suppress the prefixing of file names on output" according to the documentation shown by `man grep`.

::::::::::::::::::::::::::::::::::::::::::::::::::


Type your `grep` command into the file and save it as before. Be careful that you did not add the `$` at the beginning of the line.

Now comes the neat part. We can run this script. Type:

```bash
$ bash bad-reads-script.sh
```

It will look like nothing happened, but now if you look at `scripted_bad_reads.txt`, you can see that there are now reads in the file.
!!! terminal "Code"

::::::::::::::::::::::::::::::::::::::: challenge
```bash
$ bash bad-reads-script.sh
```

## Exercise
It will look like nothing happened, but now if you look at `scripted_bad_reads.txt`, you can see that there are now reads in the file.

We want the script to tell us when it's done.

1. Open `bad-reads-script.sh` and add the line `echo "Script finished!"` after the `grep` command and save the file.
2. Run the updated script.
!!! dumbbell "Exercise"

::::::::::::::: solution
We want the script to tell us when it's done.

1. Open `bad-reads-script.sh` and add the line `echo "Script finished!"` after the `grep` command and save the file.
2. Run the updated script.

## Solution
??? success "Solution"

```
$ bash bad-reads-script.sh
Script finished!
```
```
$ bash bad-reads-script.sh
Script finished!
```

:::::::::::::::::::::::::

::::::::::::::::::::::::::::::::::::::::::::::::::

## Making the script into a program

We had to type `bash` because we needed to tell the computer what program to use to run this script. Instead, we can turn this script into its own program. We need to tell the computer that this script is a program by making the script file executable. We can do this by changing the file permissions. We talked about permissions in [an earlier episode](03-working-with-files.md).

First, let's look at the current permissions.
!!! terminal-2 "First, let's look at the current permissions."

```bash
$ ls -l bad-reads-script.sh
```
```bash
$ ls -l bad-reads-script.sh
```

```output
-rw-rw-r-- 1 dcuser dcuser 0 Oct 25 21:46 bad-reads-script.sh
```
```output
-rw-rw-r-- 1 dcuser dcuser 0 Oct 25 21:46 bad-reads-script.sh
```

We see that it says `-rw-r--r--`. This shows that the file can be read by any user and written to by the file owner (you). We want to change these permissions so that the file can be executed as a program. We use the command `chmod` like we did earlier when we removed write permissions. Here we are adding (`+`) executable permissions (`+x`).

```bash
$ chmod +x bad-reads-script.sh
```
!!! terminal "code"

Now let's look at the permissions again.
```bash
$ chmod +x bad-reads-script.sh
```

```bash
$ ls -l bad-reads-script.sh
```
!!! terminal-2 "Now let's look at the permissions again."

```output
-rwxrwxr-x 1 dcuser dcuser 0 Oct 25 21:46 bad-reads-script.sh
```
```bash
$ ls -l bad-reads-script.sh
```

```output
-rwxrwxr-x 1 dcuser dcuser 0 Oct 25 21:46 bad-reads-script.sh
```

Now we see that it says `-rwxr-xr-x`. The `x`'s that are there now tell us we can run it as a program. So, let's try it! We'll need to put `./` at the beginning so the computer knows to look here in this directory for the program.

```bash
$ ./bad-reads-script.sh
```
!!! terminal "code"

```bash
$ ./bad-reads-script.sh
```

The script should run the same way as before, but now we've created our very own computer program!

Expand Down Expand Up @@ -314,133 +293,10 @@ command line belongs to. So, if you are logged into AWS on the command line and
the `curl` command above in the AWS terminal, the file will be downloaded to your AWS
machine, not your local one.

### Moving files between your laptop and NeSI with Jupyterhub

With Jupyterhub on NeSI, one of the easiest way to move small-medium sized files is to use the upload option on the file explorer panel

![](fig/nesi_images/upload.png)

And to download a file, you can right click on it in the explorer panel and select "Download"

![](fig/nesi_images/download.png)

::::::::::::::::::::::::::::::::::::::::: callout

**Original instructions for downloading data from AWS**

### Moving files between your laptop and your instance - AWS

What if the data you need is on your local computer, but you need to get it _into_ the
cloud? There are also several ways to do this, but it's _always_ easier
to start the transfer locally. **This means if you're typing into a terminal, the terminal
should not be logged into your instance, it should be showing your local computer. If you're
using a transfer program, it needs to be installed on your local machine, not your instance.**

## Transferring Data Between your Local Machine and the Cloud

If you're using Windows with PuTTY instead of Git Bash, please select the alternative option here:
<select id="id_platform" name="platformlist" onchange="change_content_by_platform('id_platform');return false;">

<option value="unix" id="id_unix" selected> Linux, Mac OS, Git Bash </option>
<option value="win" id="id_win"> PuTTY </option>
</select>

<div id="div_unix" style="display:block" markdown="1">

### Uploading Data to your Virtual Machine with scp

`scp` stands for 'secure copy protocol', and is a widely used UNIX tool for moving files
between computers. The simplest way to use `scp` is to run it in your local terminal,
and use it to copy a single file:

```bash
scp <file I want to move> <where I want to move it>
```

Note that you are always running `scp` locally, but that _doesn't_ mean that
you can only move files from your local computer. In order to move a file from your local computer to an AWS instance, the command would look like this:

```bash
$ scp <local file> <AWS instance>
```

To move it back to your local computer, you re-order the `to` and `from` fields:

```bash
$ scp <AWS instance> <local file>
```

#### Uploading Data to your Virtual Machine with scp

Open the terminal and use the `scp` command to upload a file (e.g. local_file.txt) to the dcuser home directory:

```bash
$ scp local_file.txt [email protected]:/home/dcuser/
```

#### Downloading Data from your Virtual Machine with scp

Let's download a text file from our remote machine. You should have a file that contains bad reads called ~/shell_data/scripted_bad_reads.txt.

**Tip:** If you are looking for another (or any really) text file in your home directory to use instead, try:

```bash
$ find ~ -name *.txt
```

Download the bad reads file in ~/shell_data/scripted_bad_reads.txt to your home ~/Download directory using the following command **(make sure you substitute [[email protected]](mailto:[email protected]) with your remote login credentials)**:

```bash
$ scp [email protected]:/home/dcuser/shell_data/untrimmed_fastq/scripted_bad_reads.txt ~/Downloads
```

Remember that in both instances, the command is run from your local machine, we've just flipped the order of the to and from parts of the command.

</div>

<div id="div_win" style="display:block" markdown="1">

### Uploading Data to your Virtual Machine with PSCP

If you're using a Windows PC without Git Bash, we recommend you use the _PSCP_ program.
This program is from the same suite of tools as the PuTTY program we have been using to connect.

1. If you haven't done so, download pscp from [http://the.earth.li/~sgtatham/putty/latest/x86/pscp.exe](https://the.earth.li/~sgtatham/putty/latest/x86/pscp.exe)
2. Make sure the _PSCP_ program is somewhere you know on your computer. In this case,
your Downloads folder is appropriate.
3. Open the windows [PowerShell](https://en.wikipedia.org/wiki/Windows_PowerShell);
go to your start menu/search enter the term **'cmd'**; you will be able to start the shell
(the shell should start from C:\\Users\\your-pc-username>).
4. Change to the Downloads directory:

```bash
> cd Downloads
```

5. Locate a file on your computer that you wish to upload (be sure you know the path). Then upload it to your remote machine **(you will need to know your AMI instance address (which starts with ec2), and login credentials)**. You will be prompted to enter a password, and then your upload will begin. **(make sure you substitute 'your-pc-username' for your actual pc username and 'ec2-54-88-126-85.compute-1.amazonaws.com' with your AMI instance address)**

```bash
C:\User\your-pc-username\Downloads> pscp.exe local_file.txt [email protected]:/home/dcuser/
```

### Downloading Data from your Virtual Machine with PSCP

1. Follow the instructions in the Upload section to download (if needed) and access the _PSCP_ program (steps 1-3)
2. Download the text file to your current working directory (represented by a .) using the following command **(make sure you substitute 'your-pc-username' for your actual pc username and 'ec2-54-88-126-85.compute-1.amazonaws.com' with your AMI instance address)**

```bash
C:\User\your-pc-username\Downloads> pscp.exe [email protected]:/home/dcuser/shell_data/untrimmed_fastq/scripted_bad_reads.txt .

C:\User\your-pc-username\Downloads
```

</div>

::::::::::::::::::::::::::::::::::::::::::::::::::
!!! graduation-cap "keypoints"

:::::::::::::::::::::::::::::::::::::::: keypoints
- Scripts are a collection of commands executed together.
- Transferring information to and from virtual and local computers.

- Scripts are a collection of commands executed together.
- Transferring information to and from virtual and local computers.

::::::::::::::::::::::::::::::::::::::::::::::::::

0 comments on commit 3c0c41d

Please sign in to comment.