-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #124 from seqeralabs/quickstart-demo
Getting started content: Platform demo (RNASeq) and other updates
- Loading branch information
Showing
155 changed files
with
2,459 additions
and
81 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
112 changes: 112 additions & 0 deletions
112
platform_versioned_docs/version-23.4/getting-started/quickstart-demo/add-data.mdx
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,112 @@ | ||
--- | ||
title: "Add data" | ||
description: "An introduction to adding pipeline input data in Seqera Platform" | ||
date: "21 Jul 2024" | ||
tags: [platform, data, data explorer, datasets] | ||
--- | ||
|
||
import Tabs from '@theme/Tabs'; | ||
import TabItem from '@theme/TabItem'; | ||
|
||
Most bioinformatics pipelines require an input of some sort. This is typically a samplesheet where each row consists of a sample, the location of files for that sample (such as FASTQ files), and other sample details. Reliable shared access to pipeline input data is crucial to simplify data management, minimize user data-input errors, and facilitate reproducible workflows. | ||
|
||
In Platform, samplesheets and other data can be made easily accessible in one of two ways: | ||
- Use **Data Explorer** to browse and interact with remote data from AWS S3, Azure Blob Storage, and Google Cloud Storage repositories, directly in your organization workspace. | ||
- Use **Datasets** to upload structured data to your workspace in CSV (Comma-Separated Values) or TSV (Tab-Separated Values) format. | ||
|
||
## Data Explorer | ||
|
||
For pipeline runs in the cloud, users typically need access to buckets or blob storage to upload files (such as samplesheets and reference data) for analysis and to view pipeline results. Managing credentials and permissions for multiple users and training users to navigate cloud consoles and CLIs can be complicated. Data Explorer provides the simplified alternative of viewing your data directly in Platform. | ||
|
||
### Add a cloud bucket | ||
|
||
Private cloud storage buckets accessible by the [credentials](../../credentials/overview.mdx) in your workspace are added to Data Explorer automatically by default. However, you can also add custom directory paths within buckets to your workspace to simplify direct access. | ||
|
||
To add individual buckets (or directory paths within buckets): | ||
|
||
1. From the **Data Explorer** tab, select **Add cloud bucket**. | ||
1. Specify the bucket details: | ||
- The cloud **Provider**. | ||
- An existing cloud **Bucket path**. | ||
- A unique **Name** for the bucket. | ||
- The **Credentials** used to access the bucket. For public cloud buckets, select **Public** from the dropdown menu. | ||
- An optional bucket **Description**. | ||
1. Select **Add**. | ||
|
||
 | ||
|
||
You can now use this data in your analysis without the need to interact with cloud consoles or CLI tools. | ||
|
||
#### Public data sources | ||
|
||
Select **Public** from the credentials dropdown menu to add public cloud storage buckets from resources such as: | ||
|
||
- [The Cancer Genome Atlas (TCGA)](https://registry.opendata.aws/tcga/) | ||
- [1000 Genomes Project](https://registry.opendata.aws/1000-genomes/) | ||
- [NCBI SRA](https://registry.opendata.aws/ncbi-sra/) | ||
- [Genome in a Bottle Consortium](https://docs.opendata.aws/giab/readme.html) | ||
- [MSSNG Database](https://cloud.google.com/life-sciences/docs/resources/public-datasets/mssng) | ||
- [Genome Aggregation Database (gnomAD)](https://cloud.google.com/life-sciences/docs/resources/public-datasets/gnomad) | ||
|
||
### View pipeline outputs | ||
|
||
In Data Explorer, you can: | ||
|
||
- **View bucket details**: | ||
Select the information icon next to a bucket in the list to view the cloud provider, bucket address, and credentials. | ||
|
||
 | ||
|
||
- **View bucket contents**: | ||
Select a bucket name from the list to view the bucket contents. The file type, size, and path of objects are displayed in columns next to the object name. For example, view the outputs of your [nf-core/rnaseq](./comm-showcase.mdx#launch-the-nf-corernaseq-pipeline) run: | ||
|
||
 | ||
|
||
- **Preview files**: | ||
Select a file to open a preview window that includes a **Download** button. For example, view the resultant gene counts of the salmon quantification step of your [nf-core/rnaseq](./comm-showcase.mdx#launch-the-nf-corernaseq-pipeline) run: | ||
|
||
 | ||
|
||
## Datasets | ||
|
||
Datasets in Platform are CSV (comma-separated values) and TSV (tab-separated values) files stored in a workspace. You can select stored datasets as input data when launching a pipeline. | ||
|
||
<details> | ||
<summary>**Example: nf-core/rnaseq test samplesheet**</summary> | ||
|
||
The [nf-core/rnaseq](https://github.com/nf-core/rnaseq) pipeline works with input datasets (samplesheets) containing sample names, FASTQ file locations, and indications of strandedness. The Seqera Community Showcase sample dataset for nf-core/rnaseq specifies the paths to 7 small sub-sampled FASTQ files from a yeast RNAseq dataset: | ||
|
||
**Example nf-core/rnaseq dataset** | ||
|
||
| sample | fastq_1 | fastq_2 | strandedness | | ||
| ------------------- | ------------------------------------ | ------------------------------------ | ------------ | | ||
| WT_REP1 | s3://nf-core-awsmegatests/rnaseq/... | s3://nf-core-awsmegatests/rnaseq/... | reverse | | ||
| WT_REP1 | s3://nf-core-awsmegatests/rnaseq/... | s3://nf-core-awsmegatests/rnaseq/... | reverse | | ||
| WT_REP2 | s3://nf-core-awsmegatests/rnaseq/... | s3://nf-core-awsmegatests/rnaseq/... | reverse | | ||
| RAP1_UNINDUCED_REP1 | s3://nf-core-awsmegatests/rnaseq/... | | reverse | | ||
| RAP1_UNINDUCED_REP2 | s3://nf-core-awsmegatests/rnaseq/... | | reverse | | ||
| RAP1_UNINDUCED_REP2 | s3://nf-core-awsmegatests/rnaseq/... | | reverse | | ||
| RAP1_IAA_30M_REP1 | s3://nf-core-awsmegatests/rnaseq/... | s3://nf-core-awsmegatests/rnaseq/... | reverse | | ||
|
||
</details> | ||
|
||
Download the nf-core/rnaseq [samplesheet_test.csv](samplesheet_test.csv). | ||
|
||
### Add a dataset | ||
|
||
From the **Datasets** tab, select **Add Dataset**. | ||
|
||
 | ||
|
||
Specify the following dataset details: | ||
|
||
- A **Name** for the dataset, such as `nf-core-rnaseq-test-dataset`. | ||
- A **Description** for the dataset. | ||
- Select the **First row as header** option to prevent Platform from parsing the header row of the samplesheet as sample data. | ||
- Select **Upload file** and browse to your CSV or TSV file in local storage, or simply drag and drop it into the box. | ||
|
||
Notice the location of the files in the nf-core/rnaseq example dataset point to a path on S3. This could also be a path to a shared filesystem, if using an HPC compute environment. Nextflow will use these paths to stage the files into the task working directory. | ||
|
||
:::info | ||
Platform does not store the data used for analysis in pipelines. The datasets must provide the locations of data that is stored on your own infrastructure. | ||
::: |
81 changes: 81 additions & 0 deletions
81
...m_versioned_docs/version-23.4/getting-started/quickstart-demo/add-pipelines.mdx
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,81 @@ | ||
--- | ||
title: "Add pipelines" | ||
description: "An introduction to adding pipelines to Seqera Platform workspaces" | ||
date: "12 Jul 2024" | ||
tags: [platform, launch, pipelines, launchpad] | ||
--- | ||
|
||
import Tabs from '@theme/Tabs'; | ||
import TabItem from '@theme/TabItem'; | ||
|
||
The Launchpad lists the preconfigured Nextflow pipelines that can be executed on the [compute environments](../../compute-envs/overview.mdx) in your workspace. | ||
|
||
Platform offers two methods to import pipelines to your workspace Launchpad — directly from Seqera Pipelines, or manually via **Add pipeline** in Platform. This page illustrates each method, using [nf-core/rnaseq](https://github.com/nf-core/rnaseq) as an example. | ||
|
||
### Import nf-core/rnaseq from Seqera Pipelines | ||
|
||
[Seqera Pipelines](https://seqera.io/pipelines) is an open-source resource that provides a curated collection of high-quality, open-source pipelines. Each pipeline includes a curated test dataset to use in a test run in just a few steps. | ||
|
||
 | ||
|
||
To import the `nf-core/rnaseq` pipeline: | ||
1. Select **Launch** next to the pipeline name in the list. In the **Add pipeline** tab, select **Cloud** or **Enterprise** depending on your Platform account type, then provide the information needed for Seqera Pipelines to access your Platform instance: | ||
- **Seqera Cloud**: Paste your Platform **Access token** and select **Next**. | ||
- **Seqera Enterprise**: Specify the **Seqera Platform URL** (hostname) and **Base API URL** for your Enterprise instance, then paste your Platform **Access token** and select **Next**. | ||
:::note | ||
If you do not have a Platform access token, select **Get your access token from Seqera Platform** to open the Access tokens page in a new browser window. | ||
::: | ||
1. Select the Platform **Organization**, **Workspace**, and **Compute environment** for the imported pipeline. | ||
1. (Optional) Customize the **Pipeline Name** and **Pipeline Description**. | ||
1. Select **Add Pipeline**. | ||
|
||
 | ||
|
||
:::tip | ||
To launch pipelines directly with CLI tools, select the **Launch Pipeline** tab to grab commands for Nextflow, [Seqera Platform CLI](./automation.mdx), and [nf-core/tools](https://nf-co.re/docs/nf-core-tools): | ||
|
||
 | ||
::: | ||
|
||
### Add nf-core/rnaseq from the Launchpad | ||
|
||
 | ||
|
||
From your workspace Launchpad, select **Add Pipeline** and specify the following pipeline details: | ||
|
||
- **Name**: `nf-core/rnaseq`, or a custom name of your choice. | ||
- (*Optional*) **Description**: A summary of the pipeline or any information that may be useful to workspace participants when selecting a pipeline to launch. | ||
- (*Optional*) **Labels**: Categorize the pipeline according to arbitrary criteria (such as reference genome version) that may help workspace participants to select the appropriate pipeline for their analysis. | ||
- **Compute environment**: Select an existing workspace [compute environment](../../compute-envs/overview.mdx). | ||
- **Pipeline to launch**: `https://github.com/nf-core/rnaseq` | ||
- Platform allows you to select any public or private Git repository that contains Nextflow source code. | ||
- **Revision number**: Platform will search all of the available tags and branches in the provided pipeline repository and render a dropdown to select the appropriate version. | ||
:::tip | ||
Selecting a specific pipeline version is important for reproducibility as this ensures that each run with the same input data will generate the same results. | ||
::: | ||
- (*Optional*) **Config profiles**: `test` | ||
- Select a predefined profile for the Nextflow pipeline. | ||
:::info | ||
All nf-core pipelines include a `test` profile that is associated with a minimal test dataset. This profile runs the pipeline with heavily sub-sampled input data for the purposes of [CI/CD](https://resources.github.com/devops/ci-cd/) and to quickly confirm that the pipeline runs on your infrastructure. | ||
::: | ||
- (*Optional*) **Pipeline parameters**: | ||
- Set any custom pipeline parameters that will be prepopulated when users launch the pipeline from the Launchpad. For example, set the path to local reference genomes so users don't have to worry about locating these files when launching the pipeline. | ||
 | ||
- (*Optional*) **Pre-run script**: | ||
- Define Bash code that executes before the pipeline launches in the same environment where Nextflow runs. | ||
:::info | ||
Pre-run scripts are useful for defining executor settings, troubleshooting, and defining a specific version of Nextflow with the `NXF_VER` environment variable. | ||
::: | ||
 | ||
|
||
:::note | ||
Pre-filled pipeline settings (such as compute environment, config profiles, and pipeline parameters) can be overridden during pipeline launch by workspace participants with the necessary [permissions](../../orgs-and-teams/roles.mdx). | ||
::: | ||
|
||
After you have populated the appropriate fields, select **Add**. `nf-core/rnaseq` is now available for workspace participants to launch in the preconfigured compute environment. | ||
|
||
|
||
|
||
|
||
|
||
|
Binary file added
BIN
+9.43 MB
...oned_docs/version-23.4/getting-started/quickstart-demo/assets/all_runs_view.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+39.7 KB
..._docs/version-23.4/getting-started/quickstart-demo/assets/connect-to-studio.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+111 KB
...docs/version-23.4/getting-started/quickstart-demo/assets/create-a-data-link.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+3.94 MB
...docs/version-23.4/getting-started/quickstart-demo/assets/create-data-studio.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+4.94 MB
...ned_docs/version-23.4/getting-started/quickstart-demo/assets/dashboard_view.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+4.48 MB
...ersion-23.4/getting-started/quickstart-demo/assets/data-explorer-add-bucket.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+2.21 MB
...ion-23.4/getting-started/quickstart-demo/assets/data-explorer-preview-files.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+1.47 MB
...sion-23.4/getting-started/quickstart-demo/assets/data-explorer-view-details.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+309 KB
...version-23.4/getting-started/quickstart-demo/assets/data-studio-checkpoints.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+5.14 MB
...sion-23.4/getting-started/quickstart-demo/assets/data-studio-create-jupyter.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+130 KB
...getting-started/quickstart-demo/assets/data-studio-jupyter-notebook-example.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+112 KB
...s/version-23.4/getting-started/quickstart-demo/assets/data-studios-overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+1.28 MB
...ocs/version-23.4/getting-started/quickstart-demo/assets/general-run-details.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+2.14 MB
...s/version-23.4/getting-started/quickstart-demo/assets/generate-access-token.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+3.14 MB
...ed_docs/version-23.4/getting-started/quickstart-demo/assets/go-to-workspace.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+468 KB
...ioned_docs/version-23.4/getting-started/quickstart-demo/assets/landing_page.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.