Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting started content: Platform demo (RNASeq) and other updates #124

Merged
merged 59 commits into from
Aug 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
e46ea9b
Add Showcase demo content
llewellyn-sl Jul 12, 2024
a525c73
WIP demo content
llewellyn-sl Jul 12, 2024
a39bf8e
WIP
llewellyn-sl Jul 16, 2024
7c7b9d7
WIP
llewellyn-sl Jul 18, 2024
2b35ae5
WIP
llewellyn-sl Jul 19, 2024
57f8ef6
Fix build issues
llewellyn-sl Jul 19, 2024
0f5af24
Fix build issues
llewellyn-sl Jul 19, 2024
2796ac6
Launch pipelines page + tutorial intro
llewellyn-sl Jul 19, 2024
3678921
Update add-datasets.mdx
llewellyn-sl Jul 19, 2024
7613c2b
Refinement
llewellyn-sl Jul 20, 2024
3c88572
Still going
llewellyn-sl Jul 22, 2024
cb165ad
Cleanup
llewellyn-sl Jul 22, 2024
9e72cf6
Merge branch 'master' into quickstart-demo
llewellyn-sl Jul 22, 2024
0293a27
Update automation.mdx
llewellyn-sl Jul 22, 2024
584e6aa
Merge branch 'quickstart-demo' of https://github.com/seqeralabs/docs …
llewellyn-sl Jul 22, 2024
6238b48
Convert to 1-pager
llewellyn-sl Jul 22, 2024
50e9fd8
Update version-24.1.1-sidebars.json
llewellyn-sl Jul 22, 2024
b9ddcf6
Update comm-showcase.mdx
llewellyn-sl Jul 23, 2024
8ac700f
Update automation.mdx
llewellyn-sl Jul 23, 2024
c44072f
Update add-pipelines.mdx
llewellyn-sl Jul 23, 2024
4f23ca2
Add workspace setup page to get started section
llewellyn-sl Jul 24, 2024
06d878d
Merge branch 'master' into quickstart-demo
llewellyn-sl Jul 24, 2024
8d548c6
Update workspace-setup.mdx
llewellyn-sl Jul 24, 2024
7d141e9
Merge branch 'quickstart-demo' of https://github.com/seqeralabs/docs …
llewellyn-sl Jul 24, 2024
865831e
Update workspace-setup.mdx
llewellyn-sl Jul 24, 2024
7fc6287
Merge branch 'master' into quickstart-demo
llewellyn-sl Jul 26, 2024
68df8ba
Platform demo formatting
llewellyn-sl Jul 26, 2024
bb02f0d
Merge branch 'quickstart-demo' of https://github.com/seqeralabs/docs …
llewellyn-sl Jul 26, 2024
13722b4
Update index.mdx
llewellyn-sl Jul 26, 2024
2146d47
Link fix
llewellyn-sl Jul 26, 2024
519fe19
Update platform_versioned_docs/version-24.1.1/getting-started/quickst…
llewellyn-sl Jul 29, 2024
6b14e11
Update platform_versioned_docs/version-24.1.1/getting-started/quickst…
llewellyn-sl Jul 29, 2024
dff2fc7
Update platform_versioned_docs/version-24.1.1/getting-started/quickst…
llewellyn-sl Jul 29, 2024
dea97cc
Update platform_versioned_docs/version-24.1.1/getting-started/quickst…
llewellyn-sl Jul 29, 2024
9992020
Update platform_versioned_docs/version-24.1.1/getting-started/quickst…
llewellyn-sl Jul 29, 2024
8183270
Update platform_versioned_docs/version-24.1.1/getting-started/quickst…
llewellyn-sl Jul 29, 2024
7da68df
Update platform_versioned_docs/version-24.1.1/getting-started/quickst…
llewellyn-sl Jul 29, 2024
72a1f9e
Update platform_versioned_docs/version-24.1.1/getting-started/quickst…
llewellyn-sl Jul 29, 2024
9b59fb5
Update platform_versioned_docs/version-24.1.1/getting-started/quickst…
llewellyn-sl Jul 29, 2024
3645550
Update platform_versioned_docs/version-24.1.1/getting-started/quickst…
llewellyn-sl Jul 29, 2024
1e24891
Update platform_versioned_docs/version-24.1.1/getting-started/quickst…
llewellyn-sl Jul 29, 2024
657a80f
Update platform_versioned_docs/version-24.1.1/getting-started/quickst…
llewellyn-sl Jul 29, 2024
5176e63
Update platform_versioned_docs/version-24.1.1/getting-started/quickst…
llewellyn-sl Jul 29, 2024
fceb78b
Update platform_versioned_docs/version-24.1.1/getting-started/quickst…
llewellyn-sl Jul 29, 2024
52f3409
Update platform_versioned_docs/version-24.1.1/getting-started/quickst…
llewellyn-sl Jul 29, 2024
7ad21a1
Update platform_versioned_docs/version-24.1.1/getting-started/quickst…
llewellyn-sl Jul 29, 2024
6974b32
Code review fixes
llewellyn-sl Jul 29, 2024
124a8c3
Update platform_versioned_docs/version-24.1.1/getting-started/quickst…
llewellyn-sl Jul 30, 2024
3ad3a3b
Biological replicates
llewellyn-sl Jul 30, 2024
c8dca66
Merge branch 'quickstart-demo' of https://github.com/seqeralabs/docs …
llewellyn-sl Jul 30, 2024
1413c75
Update launch-pipelines.mdx
llewellyn-sl Jul 30, 2024
7999f16
MultiQC docs links
llewellyn-sl Jul 30, 2024
adc926f
Update platform_versioned_docs/version-24.1.1/index.mdx
justinegeffen Aug 1, 2024
3bb3475
Merge branch 'master' into quickstart-demo
llewellyn-sl Aug 7, 2024
fa662da
Harmonize with master
llewellyn-sl Aug 7, 2024
e6164fe
Platform index and 23.4 sidebar
llewellyn-sl Aug 7, 2024
3a0f72f
Formatting fixes
llewellyn-sl Aug 7, 2024
b62e798
Code formatting
llewellyn-sl Aug 7, 2024
926b08c
Last ones, STG
llewellyn-sl Aug 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
---
title: "Deploy Seqera Platform"
title: "Deploy Platform"
description: "An overview of deployment versions and ways to run Seqera Platform."
date: "21 Apr 2023"
tags: [deployment]
---

Seqera Platform is available in two deployment editions and can be accessed via the web-based user interface (UI), [API](../api/overview.mdx), [CLI](../cli/cli.mdx), or in Nextflow directly using a flag.

## Seqera Platform editions
## Platform editions

### Seqera Platform Cloud
### Seqera Cloud

The hosted Seqera Cloud edition is recommended for users who are new to Seqera. It's an ideal choice for individuals and organizations looking to set up quickly. Seqera Cloud has a limit of five concurrent workflow runs per user. It's available free of charge at [cloud.seqera.io](https://cloud.seqera.io).
The hosted Seqera Cloud edition is recommended for users who are new to Platform. It's an ideal choice for individuals and organizations looking to set up quickly. [Seqera Cloud](https://cloud.seqera.io) has a free tier with a limit of five concurrent workflow runs per user. Seqera Cloud Pro offers unlimited runs, dedicated support, and more — [contact us](https://cloud.seqera.io/demo/) for a demo to discuss your requirements.

### Seqera Platform Enterprise
### Seqera Enterprise

[Seqera Platform Enterprise](../enterprise/index.mdx) is installed in an organization's own cloud or on-premises infrastructure. It includes:
[Seqera Enterprise](../enterprise/index.mdx) is installed in an organization's own cloud or on-premises infrastructure. It includes:

- Monitoring, logging, and observability
- Pipeline execution Launchpad
Expand All @@ -26,15 +26,15 @@ The hosted Seqera Cloud edition is recommended for users who are new to Seqera.
- Full-featured API
- Dedicated support for Nextflow and Seqera Platform

To install Seqera Platform in your organization's infrastructure, [contact us](https://cloud.seqera.io/demo/) for a demo to discuss your requirements.
To install Platform in your organization's infrastructure, [contact us](https://cloud.seqera.io/demo/) for a demo to discuss your requirements.

## How to use Seqera Platform
## How to use Platform

You can access your Seqera instance through the UI, the [API](../api/overview.mdx), the [CLI](../cli/cli.mdx), or in Nextflow directly using the `-with-tower` option.

### Seqera Platform web-based UI
### Platform web-based UI

1. Create an account and log in to Seqera at [cloud.seqera.io](https://cloud.seqera.io).
1. Create an account and log in to Seqera Cloud at [cloud.seqera.io](https://cloud.seqera.io).

:::note
Platform login sessions remain active as long as the application browser window remains open and active. When the browser window is terminated, automatic logout occurs within 6 hours by default.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
---
title: "Add data"
description: "An introduction to adding pipeline input data in Seqera Platform"
date: "21 Jul 2024"
tags: [platform, data, data explorer, datasets]
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

Most bioinformatics pipelines require an input of some sort. This is typically a samplesheet where each row consists of a sample, the location of files for that sample (such as FASTQ files), and other sample details. Reliable shared access to pipeline input data is crucial to simplify data management, minimize user data-input errors, and facilitate reproducible workflows.

In Platform, samplesheets and other data can be made easily accessible in one of two ways:
- Use **Data Explorer** to browse and interact with remote data from AWS S3, Azure Blob Storage, and Google Cloud Storage repositories, directly in your organization workspace.
- Use **Datasets** to upload structured data to your workspace in CSV (Comma-Separated Values) or TSV (Tab-Separated Values) format.

## Data Explorer

For pipeline runs in the cloud, users typically need access to buckets or blob storage to upload files (such as samplesheets and reference data) for analysis and to view pipeline results. Managing credentials and permissions for multiple users and training users to navigate cloud consoles and CLIs can be complicated. Data Explorer provides the simplified alternative of viewing your data directly in Platform.

### Add a cloud bucket

Private cloud storage buckets accessible by the [credentials](../../credentials/overview.mdx) in your workspace are added to Data Explorer automatically by default. However, you can also add custom directory paths within buckets to your workspace to simplify direct access.

To add individual buckets (or directory paths within buckets):

1. From the **Data Explorer** tab, select **Add cloud bucket**.
1. Specify the bucket details:
- The cloud **Provider**.
- An existing cloud **Bucket path**.
- A unique **Name** for the bucket.
- The **Credentials** used to access the bucket. For public cloud buckets, select **Public** from the dropdown menu.
- An optional bucket **Description**.
1. Select **Add**.

![Add public bucket](assets/data-explorer-add-bucket.gif)

You can now use this data in your analysis without the need to interact with cloud consoles or CLI tools.

#### Public data sources

Select **Public** from the credentials dropdown menu to add public cloud storage buckets from resources such as:

- [The Cancer Genome Atlas (TCGA)](https://registry.opendata.aws/tcga/)
- [1000 Genomes Project](https://registry.opendata.aws/1000-genomes/)
- [NCBI SRA](https://registry.opendata.aws/ncbi-sra/)
- [Genome in a Bottle Consortium](https://docs.opendata.aws/giab/readme.html)
- [MSSNG Database](https://cloud.google.com/life-sciences/docs/resources/public-datasets/mssng)
- [Genome Aggregation Database (gnomAD)](https://cloud.google.com/life-sciences/docs/resources/public-datasets/gnomad)

### View pipeline outputs

In Data Explorer, you can:

- **View bucket details**:
Select the information icon next to a bucket in the list to view the cloud provider, bucket address, and credentials.

![Bucket details](assets/data-explorer-view-details.gif)

- **View bucket contents**:
Select a bucket name from the list to view the bucket contents. The file type, size, and path of objects are displayed in columns next to the object name. For example, view the outputs of your [nf-core/rnaseq](./comm-showcase.mdx#launch-the-nf-corernaseq-pipeline) run:

![Data Explorer bucket](assets/sp-cloud-data-explorer.gif)

- **Preview files**:
Select a file to open a preview window that includes a **Download** button. For example, view the resultant gene counts of the salmon quantification step of your [nf-core/rnaseq](./comm-showcase.mdx#launch-the-nf-corernaseq-pipeline) run:

![Preview pipeline results](assets/data-explorer-preview-files.gif)

## Datasets

Datasets in Platform are CSV (comma-separated values) and TSV (tab-separated values) files stored in a workspace. You can select stored datasets as input data when launching a pipeline.

<details>
<summary>**Example: nf-core/rnaseq test samplesheet**</summary>

The [nf-core/rnaseq](https://github.com/nf-core/rnaseq) pipeline works with input datasets (samplesheets) containing sample names, FASTQ file locations, and indications of strandedness. The Seqera Community Showcase sample dataset for nf-core/rnaseq specifies the paths to 7 small sub-sampled FASTQ files from a yeast RNAseq dataset:

**Example nf-core/rnaseq dataset**

| sample | fastq_1 | fastq_2 | strandedness |
| ------------------- | ------------------------------------ | ------------------------------------ | ------------ |
| WT_REP1 | s3://nf-core-awsmegatests/rnaseq/... | s3://nf-core-awsmegatests/rnaseq/... | reverse |
| WT_REP1 | s3://nf-core-awsmegatests/rnaseq/... | s3://nf-core-awsmegatests/rnaseq/... | reverse |
| WT_REP2 | s3://nf-core-awsmegatests/rnaseq/... | s3://nf-core-awsmegatests/rnaseq/... | reverse |
| RAP1_UNINDUCED_REP1 | s3://nf-core-awsmegatests/rnaseq/... | | reverse |
| RAP1_UNINDUCED_REP2 | s3://nf-core-awsmegatests/rnaseq/... | | reverse |
| RAP1_UNINDUCED_REP2 | s3://nf-core-awsmegatests/rnaseq/... | | reverse |
| RAP1_IAA_30M_REP1 | s3://nf-core-awsmegatests/rnaseq/... | s3://nf-core-awsmegatests/rnaseq/... | reverse |

</details>

Download the nf-core/rnaseq [samplesheet_test.csv](samplesheet_test.csv).

### Add a dataset

From the **Datasets** tab, select **Add Dataset**.

![Add a dataset](assets/sp-cloud-add-a-dataset.gif)

Specify the following dataset details:

- A **Name** for the dataset, such as `nf-core-rnaseq-test-dataset`.
- A **Description** for the dataset.
- Select the **First row as header** option to prevent Platform from parsing the header row of the samplesheet as sample data.
- Select **Upload file** and browse to your CSV or TSV file in local storage, or simply drag and drop it into the box.

Notice the location of the files in the nf-core/rnaseq example dataset point to a path on S3. This could also be a path to a shared filesystem, if using an HPC compute environment. Nextflow will use these paths to stage the files into the task working directory.

:::info
Platform does not store the data used for analysis in pipelines. The datasets must provide the locations of data that is stored on your own infrastructure.
:::
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
---
title: "Add pipelines"
description: "An introduction to adding pipelines to Seqera Platform workspaces"
date: "12 Jul 2024"
tags: [platform, launch, pipelines, launchpad]
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

The Launchpad lists the preconfigured Nextflow pipelines that can be executed on the [compute environments](../../compute-envs/overview.mdx) in your workspace.

Platform offers two methods to import pipelines to your workspace Launchpad — directly from Seqera Pipelines, or manually via **Add pipeline** in Platform. This page illustrates each method, using [nf-core/rnaseq](https://github.com/nf-core/rnaseq) as an example.

### Import nf-core/rnaseq from Seqera Pipelines

[Seqera Pipelines](https://seqera.io/pipelines) is an open-source resource that provides a curated collection of high-quality, open-source pipelines. Each pipeline includes a curated test dataset to use in a test run in just a few steps.

![Seqera Pipelines overview](assets/seqera-pipelines-overview.gif)

To import the `nf-core/rnaseq` pipeline:
1. Select **Launch** next to the pipeline name in the list. In the **Add pipeline** tab, select **Cloud** or **Enterprise** depending on your Platform account type, then provide the information needed for Seqera Pipelines to access your Platform instance:
- **Seqera Cloud**: Paste your Platform **Access token** and select **Next**.
- **Seqera Enterprise**: Specify the **Seqera Platform URL** (hostname) and **Base API URL** for your Enterprise instance, then paste your Platform **Access token** and select **Next**.
:::note
If you do not have a Platform access token, select **Get your access token from Seqera Platform** to open the Access tokens page in a new browser window.
:::
1. Select the Platform **Organization**, **Workspace**, and **Compute environment** for the imported pipeline.
1. (Optional) Customize the **Pipeline Name** and **Pipeline Description**.
1. Select **Add Pipeline**.

![Seqera Pipelines add to Launchpad](assets/seqera-pipelines-add-pipeline.gif)

:::tip
To launch pipelines directly with CLI tools, select the **Launch Pipeline** tab to grab commands for Nextflow, [Seqera Platform CLI](./automation.mdx), and [nf-core/tools](https://nf-co.re/docs/nf-core-tools):

![Launch Seqera Pipeline](assets/seqera-pipelines-launch-cli.png)
:::

### Add nf-core/rnaseq from the Launchpad

![Add nf-core/rnaseq pipeline](assets/sp-cloud-add-rnaseq.gif)

From your workspace Launchpad, select **Add Pipeline** and specify the following pipeline details:

- **Name**: `nf-core/rnaseq`, or a custom name of your choice.
- (*Optional*) **Description**: A summary of the pipeline or any information that may be useful to workspace participants when selecting a pipeline to launch.
- (*Optional*) **Labels**: Categorize the pipeline according to arbitrary criteria (such as reference genome version) that may help workspace participants to select the appropriate pipeline for their analysis.
- **Compute environment**: Select an existing workspace [compute environment](../../compute-envs/overview.mdx).
- **Pipeline to launch**: `https://github.com/nf-core/rnaseq`
- Platform allows you to select any public or private Git repository that contains Nextflow source code.
- **Revision number**: Platform will search all of the available tags and branches in the provided pipeline repository and render a dropdown to select the appropriate version.
:::tip
Selecting a specific pipeline version is important for reproducibility as this ensures that each run with the same input data will generate the same results.
:::
- (*Optional*) **Config profiles**: `test`
- Select a predefined profile for the Nextflow pipeline.
:::info
All nf-core pipelines include a `test` profile that is associated with a minimal test dataset. This profile runs the pipeline with heavily sub-sampled input data for the purposes of [CI/CD](https://resources.github.com/devops/ci-cd/) and to quickly confirm that the pipeline runs on your infrastructure.
:::
- (*Optional*) **Pipeline parameters**:
- Set any custom pipeline parameters that will be prepopulated when users launch the pipeline from the Launchpad. For example, set the path to local reference genomes so users don't have to worry about locating these files when launching the pipeline.
![Add pipeline parameters](assets/sp-cloud-pipeline-params.gif)
- (*Optional*) **Pre-run script**:
- Define Bash code that executes before the pipeline launches in the same environment where Nextflow runs.
:::info
Pre-run scripts are useful for defining executor settings, troubleshooting, and defining a specific version of Nextflow with the `NXF_VER` environment variable.
:::
![Specify NF version in pre-run script](assets/sp-cloud-pre-run-options.gif)

:::note
Pre-filled pipeline settings (such as compute environment, config profiles, and pipeline parameters) can be overridden during pipeline launch by workspace participants with the necessary [permissions](../../orgs-and-teams/roles.mdx).
:::

After you have populated the appropriate fields, select **Add**. `nf-core/rnaseq` is now available for workspace participants to launch in the preconfigured compute environment.






Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading