Skip to content

Commit

Permalink
Merge branch 'master' into changelog-wave-v1.18.1
Browse files Browse the repository at this point in the history
  • Loading branch information
justinegeffen authored Feb 26, 2025
2 parents 7cccb0e + c2cfa7d commit 5b6b964
Show file tree
Hide file tree
Showing 17 changed files with 784 additions and 23 deletions.
2 changes: 1 addition & 1 deletion fusion_docs/guide.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Fusion is a virtual, lightweight, distributed file system designed to optimise t
Fusion enables seamless filesystem I/O to cloud object stores via a standard POSIX interface resulting in simpler pipeline logic and faster, more efficient pipeline execution.

:::note
Fusion requires a license for use beyond limited testing and validation within Seqera Platform or directly within Nextflow. [Contact Seqera](https://seqera.io/contact-us/) for more details.
Fusion requires a license for use in Seqera Platform compute environments or directly in Nextflow. See [Fusion licensing](https://seqera.io/contact-us) for more information.
:::

## Getting started
Expand Down
35 changes: 28 additions & 7 deletions fusion_docs/guide/snapshots.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ More specifically, the first use case for this feature is for Seqera Platform us
Fusion Snapshots v1.0.0 requires the following [Seqera compute environment](https://docs.seqera.io/platform/latest/compute-envs/aws-batch) configuration:

- **Provider**: AWS Batch
- **Pipeline work directory**: An S3 bucket in the same region as the compute environment
- **Pipeline work directory**: An S3 bucket located in the same region as your AWS Batch compute resources
- **Enable Wave containers**
- **Enable Fusion v2**
- **Enable fast instance storage**
Expand All @@ -43,18 +43,39 @@ fusion.containerConfigUrl = '<CUSTOM_CONTAINER_URL>'

`maxSpotAttempts` must be a value higher than `0`.

### Recommended instance sizes
### EC2 instance selection guidelines

Fusion Snapshots require EC2 Spot instances with enough memory and network bandwidth to dump the cache of task intermediate files to S3 storage before AWS terminates an instance. When AWS issues a Spot instance reclamation notice, Fusion has two minutes to complete this transfer.
- Choose EC2 Spot instances with sufficient memory and network bandwidth to dump the cache of task intermediate files to S3 storage before AWS terminates an instance.
- Select instances with guaranteed network bandwidth (not instances with bandwidth "up to" a maximum value).
- Maintain a 5:1 ratio between memory (GiB) and network bandwidth (Gbps).
- Recommended instance families: `c6id`, `r6id`, or `m6id` series instances work optimally with Fusion fast instance storage.

It is recommended to select instances with guaranteed network bandwidth (as opposed to bandwidth _up to_ a maximum value) and maintain a ratio of 5:1 between memory and network bandwidth.
:::info Example
A c6id.8xlarge instance provides 64 GiB memory and 12.5 Gbps guaranteed network bandwidth. This configuration can transfer the entire memory contents to S3 in approximately 70 seconds, well within the 2-minute reclamation window.

For example, taking into account the bandwidth and compute necessary to create a snapshot, a `c6i.8xlarge` instance with 64 GIB memory and a guaranteed network bandwidth of 12.5 Gbps can take approximately 70 seconds to dump the entire instance to S3 storage before instance reclamation occurs.
Instances with memory:bandwitdth ratios over 5:1 may not complete transfers before termination, potentially resulting in task failures.
:::

#### Recommended instance types

| Instance type | Memory (GiB) | Network bandwidth (Gbps) | Memory:bandwidth ratio | Est. Snapshot time|
|----------------|--------------|--------------------------|------------------------|-------------------|
| c6id.4xlarge | 32 | 12.5 | 2.56:1 | ~45 seconds |
| c6id.8xlarge | 64 | 12.5 | 5.12:1 | ~70 seconds |
| r6id.2xlarge | 64 | 12.5 | 5.12:1 | ~70 seconds |
| m6id.4xlarge | 64 | 12.5 | 5.12:1 | ~70 seconds |
| c6id.12xlarge | 96 | 18.75 | 5.12:1 | ~70 seconds |
| r6id.4xlarge | 128 | 12.5 | 10.24:1 | ~105 seconds |
| m6id.8xlarge | 128 | 25 | 5.12:1 | ~70 seconds |

### Amazon Linux 2023 ECS-optimized AMI
### (Seqera Enterprise only) Select an Amazon Linux 2023 ECS-optimized AMI

To obtain sufficient performance, Fusion Snapshots require instances with Amazon Linux 2023 (which ships with Linux Kernel 6.1), with an ECS Container-optimized AMI.

:::note
Selecting a custom Amazon Linux 2023 ECS-optimized AMI is only required for compute environments in Seqera Enterprise deployments. Seqera Cloud AWS Batch compute environments use Amazon Linux 2023 AMIs by default.
:::

To find the recommended AL2023 ECS-optimized AMI for your region, run the following (replace `eu-central-1` with your AWS region):

```bash
Expand All @@ -77,4 +98,4 @@ The result for the `eu-central-1` region is similar to the following:
}
```
Note the `image_id` in your result (in this example, `ami-0281c9a5cd9de63bd`). Specify this ID in the **AMI ID** field under **Advanced options** when you create your Seqera compute environment.
Note the `image_id` in your result (in this example, `ami-0281c9a5cd9de63bd`). Specify this ID in the **AMI ID** field under **Advanced options** when you create your Seqera compute environment.
4 changes: 2 additions & 2 deletions fusion_docs/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ storage. Fusion enables seamless filesystem I/O to cloud object stores via a sta
simpler pipeline logic and faster, more efficient pipeline execution.

:::note
Fusion requires a license for use in Seqera Platform compute environments or directly in Nextflow. Fusion can be trialed at no cost. [Contact Seqera](https://seqera.io/contact-us/) for more details.
Fusion requires a license for use in Seqera Platform compute environments or directly in Nextflow. See [Fusion licensing](https://seqera.io/contact-us) for more information.
:::

### Transparent, automated installation
Expand Down Expand Up @@ -43,4 +43,4 @@ The result is significant overhead for every task. Fusion enables direct file ac

While some open-source projects provide a POSIX interface over cloud storage, they require developers to install and configure additional software and add packages to containers or VMs.

Unlike third-party solutions, Fusion is optimized for Nextflow and handles these tasks automatically. Fusion delivers fast, seamless access to cloud object storage.
Unlike third-party solutions, Fusion is optimized for Nextflow and handles these tasks automatically. Fusion delivers fast, seamless access to cloud object storage.
27 changes: 27 additions & 0 deletions fusion_docs/licensing.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
title: Fusion licensing
description: "Understand how Fusion licensing works"
date: "20 Feb 2025"
tags: [fusion, storage, compute, file system, license, quotas]
---

:::info
Fusion licensing requirements and free tier quotas will be enforced starting June 30th, 2025. [Contact Seqera](https://seqera.io/contact-us) with any questions or licensing requests.
:::

A Fusion license sets a specific quota for Fusion use, measured by total throughput per month (defined by [Nextflow’s I/O metrics](https://www.nextflow.io/docs/latest/metrics.html#i-o-usage)).

### For Seqera Cloud

- Seqera Platform will automatically add and manage a Fusion license on your behalf, up to the allowed usage quota. Seqera will track Fusion use for all pipelines run in Platform.
- The default quota for free use within Seqera Cloud is 100TB.
- Once your quota is met or exceeded, you will be prevented from running additional Nextflow pipelines with Fusion.

### For Seqera Enterprise instances with connectivity to Seqera’s license server

- Seqera Platform will automatically add and manage a Fusion license on your behalf, up to the allowed usage quota. Seqera will track Fusion use for all pipelines run in Platform.
- Once your quota is met or exceeded, you will be prevented from running additional Nextflow pipelines with Fusion.

### For Seqera Enterprise instances without network connectivity, and direct Fusion use with Nextflow

[Contact Seqera](https://seqera.io/contact-us) for a license that can be integrated into your workflow, or to set up an evaluation.
1 change: 1 addition & 0 deletions fusion_docs/sidebar.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
}
]
},
"licensing",
"reference",
"troubleshooting",
"faq",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,37 @@ $ python3

### DNS allowlist

In order for you to access resources such as Fusion tarballs, `nf-xpack` files, Wave cloud containers and other services, you'll need to add `*.seqera.io.cdn.cloudflare.net` to the allowlist in your network firewall. If DNS wildcards aren't supported by your firewall, you can use the following:
In order for you to access resources such as Fusion tarballs, `nf-xpack` files, Wave cloud containers and other services provided by Seqera, you'll need to add `*.seqera.io.cdn.cloudflare.net` to the allowlist in your network firewall. If DNS wildcards aren't supported by your firewall, you can use the following:

- `cloud.seqera.io`
- `api.cloud.seqera.io`
- `user-data.cloud.seqera.io`
- `tower.nf`
- `api.tower.nf`
- `connect.cloud.seqera.io` and its subdomains `*.connect.cloud.seqera.io`
- `hub.seqera.io`
- `intern.seqera.io`
- `wave.seqera.io`
- `community.wave.seqera.io`
- `cerbero.seqera.io`
- `public.cr.seqera.io`
- `auth.cr.seqera.io`
- `cr.seqera.io`
- `licenses.seqera.io`
- `api.multiqc.info`
- `fusionfs.seqera.io`
- `nf-xpack.seqera.io`
- `community-cr-prod.seqera.io`
- `fusionfs.seqera.io`
- `nf-xpack.seqera.io`
- `public-cr-prod.seqera.io`
- `wave-cache-prod-cloudflare.seqera.io`
- `fusionfs.seqera.io.cdn.cloudflare.net`
- `nf-xpack.seqera.io.cdn.cloudflare.net`
- `community-cr-prod.seqera.io.cdn.cloudflare.net`
- `fusionfs.seqera.io.cdn.cloudflare.net`
- `nf-xpack.seqera.io.cdn.cloudflare.net`
- `public-cr-prod.seqera.io.cdn.cloudflare.net`
- `wave-cache-prod-cloudflare.seqera.io.cdn.cloudflare.net`

If you chose to filter by specific DNS records, please note that new services may be added in the future.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,37 @@ $ python3

### DNS allowlist

In order for you to access resources such as Fusion tarballs, `nf-xpack` files, Wave cloud containers and other services, you'll need to add `*.seqera.io.cdn.cloudflare.net` to the allowlist in your network firewall. If DNS wildcards aren't supported by your firewall, you can use the following:
In order for you to access resources such as Fusion tarballs, `nf-xpack` files, Wave cloud containers and other services provided by Seqera, you'll need to add `*.seqera.io.cdn.cloudflare.net` to the allowlist in your network firewall. If DNS wildcards aren't supported by your firewall, you can use the following:

- `cloud.seqera.io`
- `api.cloud.seqera.io`
- `user-data.cloud.seqera.io`
- `tower.nf`
- `api.tower.nf`
- `connect.cloud.seqera.io` and its subdomains `*.connect.cloud.seqera.io`
- `hub.seqera.io`
- `intern.seqera.io`
- `wave.seqera.io`
- `community.wave.seqera.io`
- `cerbero.seqera.io`
- `public.cr.seqera.io`
- `auth.cr.seqera.io`
- `cr.seqera.io`
- `licenses.seqera.io`
- `api.multiqc.info`
- `fusionfs.seqera.io`
- `nf-xpack.seqera.io`
- `community-cr-prod.seqera.io`
- `fusionfs.seqera.io`
- `nf-xpack.seqera.io`
- `public-cr-prod.seqera.io`
- `wave-cache-prod-cloudflare.seqera.io`
- `fusionfs.seqera.io.cdn.cloudflare.net`
- `nf-xpack.seqera.io.cdn.cloudflare.net`
- `community-cr-prod.seqera.io.cdn.cloudflare.net`
- `fusionfs.seqera.io.cdn.cloudflare.net`
- `nf-xpack.seqera.io.cdn.cloudflare.net`
- `public-cr-prod.seqera.io.cdn.cloudflare.net`
- `wave-cache-prod-cloudflare.seqera.io.cdn.cloudflare.net`

If you chose to filter by specific DNS records, please note that new services may be added in the future.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,37 @@ $ python3

### DNS allowlist

In order for you to access resources such as Fusion tarballs, `nf-xpack` files, Wave cloud containers and other services, you'll need to add `*.seqera.io.cdn.cloudflare.net` to the allowlist in your network firewall. If DNS wildcards aren't supported by your firewall, you can use the following:
In order for you to access resources such as Fusion tarballs, `nf-xpack` files, Wave cloud containers and other services provided by Seqera, you'll need to add `*.seqera.io.cdn.cloudflare.net` to the allowlist in your network firewall. If DNS wildcards aren't supported by your firewall, you can use the following:

- `cloud.seqera.io`
- `api.cloud.seqera.io`
- `user-data.cloud.seqera.io`
- `tower.nf`
- `api.tower.nf`
- `connect.cloud.seqera.io` and its subdomains `*.connect.cloud.seqera.io`
- `hub.seqera.io`
- `intern.seqera.io`
- `wave.seqera.io`
- `community.wave.seqera.io`
- `cerbero.seqera.io`
- `public.cr.seqera.io`
- `auth.cr.seqera.io`
- `cr.seqera.io`
- `licenses.seqera.io`
- `api.multiqc.info`
- `fusionfs.seqera.io`
- `nf-xpack.seqera.io`
- `community-cr-prod.seqera.io`
- `fusionfs.seqera.io`
- `nf-xpack.seqera.io`
- `public-cr-prod.seqera.io`
- `wave-cache-prod-cloudflare.seqera.io`
- `fusionfs.seqera.io.cdn.cloudflare.net`
- `nf-xpack.seqera.io.cdn.cloudflare.net`
- `community-cr-prod.seqera.io.cdn.cloudflare.net`
- `fusionfs.seqera.io.cdn.cloudflare.net`
- `nf-xpack.seqera.io.cdn.cloudflare.net`
- `public-cr-prod.seqera.io.cdn.cloudflare.net`
- `wave-cache-prod-cloudflare.seqera.io.cdn.cloudflare.net`

If you chose to filter by specific DNS records, please note that new services may be added in the future.
Expand Down
8 changes: 4 additions & 4 deletions platform_versioned_docs/version-24.2/data_studios/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -89,27 +89,27 @@ The default user is the `root` account. The following [conda-forge](https://cond

To install additional Python packages during a running session, execute `!pip install <packagename>` commands in your notebook environment. Additional system-level packages can be installed in a terminal window using `apt install <packagename>`.

To see the list of all JupyterLab image templates available, see [public.cr.seqera.io/repo/platform/data-studio-jupyter][ds-jupyter].
To see the list of all JupyterLab image templates available, including security scan results or to inspect the container specification, see [public.cr.seqera.io/repo/platform/data-studio-jupyter][ds-jupyter].

**RStudio Server 4.4.1**

The default user is the `root` account. To install R packages during a running session, execute `install.packages("<packagename>")` commands in your notebook environment. Additional system-level packages can be installed in a terminal window using `apt install <packagename>`.

To see the list of all RStudio Server image templates available, see [public.cr.seqera.io/repo/platform/data-studio-rstudio][ds-rstudio].
To see the list of all RStudio Server image templates available, including security scan results or to inspect the container specification, see [public.cr.seqera.io/repo/platform/data-studio-rstudio][ds-rstudio].

**Visual Studio Code 1.93.1**

[Visual Studio Code][def-vsc] is an integrated development environment (IDE) that supports many programming languages. The default user is the `root` account. The container template image ships with the latest stable version of [Nextflow] and the [VSCode extension for Nextflow][nf-lang-server] to make troubleshooting Nextflow workflows easier. To install additional extensions during a running session, select **Extensions**. Additional system-level packages can be installed in a terminal window using `apt install <packagename>`.

To see the list of all Visual Studio Code image templates available, see [public.cr.seqera.io/platform/data-studio-vscode][ds-vscode].
To see the list of all Visual Studio Code image templates available, including security scan results or to inspect the container specification, see [public.cr.seqera.io/platform/data-studio-vscode][ds-vscode].

**Xpra 6.2.0**

[Xpra][def-xpra], known as _screen for X_, allows you to run X11 programs by giving you remote access to individual graphical applications. The container template image also installs NVIDIA Linux x64 (AMD64/EM64T) drivers for Ubuntu 22.04 for running GPU-enabled applications. To use these GPU drivers, your compute environment must specify GPU instance families.

The default user is the `root` account. The image is based on `ubuntu:jammy`. Additional system-level packages can be installed during a running session in a terminal window using `apt install <package_name>`.

To see the list of all Xpra image templates available, see [public.cr.seqera.io/repo/platform/data-studio-xpra][ds-xpra].
To see the list of all Xpra image templates available, including security scan results or to inspect the container specification, see [public.cr.seqera.io/repo/platform/data-studio-xpra][ds-xpra].

## Studios statuses

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,37 @@ $ python3

### DNS allowlist

In order for you to access resources such as Fusion tarballs, `nf-xpack` files, Wave cloud containers and other services, you'll need to add `*.seqera.io.cdn.cloudflare.net` to the allowlist in your network firewall. If DNS wildcards aren't supported by your firewall, you can use the following:
In order for you to access resources such as Fusion tarballs, `nf-xpack` files, Wave cloud containers and other services provided by Seqera, you'll need to add `*.seqera.io.cdn.cloudflare.net` to the allowlist in your network firewall. If DNS wildcards aren't supported by your firewall, you can use the following:

- `cloud.seqera.io`
- `api.cloud.seqera.io`
- `user-data.cloud.seqera.io`
- `tower.nf`
- `api.tower.nf`
- `connect.cloud.seqera.io` and its subdomains `*.connect.cloud.seqera.io`
- `hub.seqera.io`
- `intern.seqera.io`
- `wave.seqera.io`
- `community.wave.seqera.io`
- `cerbero.seqera.io`
- `public.cr.seqera.io`
- `auth.cr.seqera.io`
- `cr.seqera.io`
- `licenses.seqera.io`
- `api.multiqc.info`
- `fusionfs.seqera.io`
- `nf-xpack.seqera.io`
- `community-cr-prod.seqera.io`
- `fusionfs.seqera.io`
- `nf-xpack.seqera.io`
- `public-cr-prod.seqera.io`
- `wave-cache-prod-cloudflare.seqera.io`
- `fusionfs.seqera.io.cdn.cloudflare.net`
- `nf-xpack.seqera.io.cdn.cloudflare.net`
- `community-cr-prod.seqera.io.cdn.cloudflare.net`
- `fusionfs.seqera.io.cdn.cloudflare.net`
- `nf-xpack.seqera.io.cdn.cloudflare.net`
- `public-cr-prod.seqera.io.cdn.cloudflare.net`
- `wave-cache-prod-cloudflare.seqera.io.cdn.cloudflare.net`

If you chose to filter by specific DNS records, please note that new services may be added in the future.
Expand Down
Loading

0 comments on commit 5b6b964

Please sign in to comment.