Issue loading IndexedFastaAdapter from Azure Blob Storage using SAS links

[JasperBoomBejo]

Greetings fellow JBrowsers,

I’m encountering an issue while attempting to load a reference sequence directly from Azure Blob Storage using SAS links in JBrowse2.

Current setup

We are running JBrowse 2 in a Docker container using the following setup:

FROM node:alpine3.21
WORKDIR /

RUN apk update && \
    apk add bash && \
    apk add nodejs npm && \
    npm install -g npm@latest
RUN npm cache clean -f && \
    npm install -g n && \
    n stable && \
    npm install -g http-server && \
    npm install -g @jbrowse/cli && \
    jbrowse --version
RUN jbrowse create jbrowse2
WORKDIR /jbrowse2

We are running this in Docker Compose:

services:
  jbrowse2:
    build: .
    image: jbrowse2
    command: npx http-server --cors
    volumes:
      - ./config.json:/jbrowse2/config.json
    ports:
      - "8004:8080"
    stdin_open: true
    tty: true

Configuration

We are trying to add a ReferenceSequenceTrack using direct links (with SAS tokens) to files stored in Azure Blob Storage. Here's the relevant section from our config.json:

{
  "assemblies": [
    {
      "name": "SomeReference",
      "sequence": {
        "type": "ReferenceSequenceTrack",
        "trackId": "SomeReference-ReferenceSequenceTrack",
        "adapter": {
          "type": "IndexedFastaAdapter",
          "fastaLocation": {
            "uri": "https://storageacount.blob.core.windows.net/path/to/the/folder/SomeReference.fa?sv=ThisWillBeOurSASKeyToTheBlobContainer",
            "locationType": "UriLocation"
          },
          "faiLocation": {
            "uri": "https://storageacount.blob.core.windows.net/path/to/the/folder/SomeReference.fai?sv=ThisWillBeOurSASKeyToTheBlobContainer",
            "locationType": "UriLocation"
          }
        }
      },
      "displayName": "SomeReference, v1"
    },
    {
      "name": "hg19",
      "aliases": [
        "GRCh37"
      ],
      "sequence": {
        "type": "ReferenceSequenceTrack",
        "trackId": "hg19_config",
        "adapter": {
          "type": "BgzipFastaAdapter",
          "fastaLocation": {
            "uri": "https://jbrowse.org/genomes/hg19/fasta/hg19.fa.gz",
            "locationType": "UriLocation"
          },
          "faiLocation": {
            "uri": "https://jbrowse.org/genomes/hg19/fasta/hg19.fa.gz.fai",
            "locationType": "UriLocation"
          },
          "gziLocation": {
            "uri": "https://jbrowse.org/genomes/hg19/fasta/hg19.fa.gz.gzi",
            "locationType": "UriLocation"
          }
        },
        "rendering": {
          "type": "DivSequenceRenderer"
        }
      }
    },

The hg19 assembly (loaded from a public URI) displays correctly. Also, tracks based on files on the local file system worked fine. However, our SomeReference track fails to load.
We based our config.json input on this example: https://jbrowse.org/jb2/docs/config_guides/assemblies/

Error encountered

We see the following error in the browser:
Error: Failed to fetch fetching https://storageacount.blob.core.windows.net/path/to/the/folder/SomeReference.fai?sv=ThisWillBeOurSASKeyToTheBlobContainer

Notes and observations

• The .fai file in Azure was originally named SomeReference.fa.fai. Since the error message only mentioned SomeReference.fai, we renamed the file accordingly, assuming JBrowse2 might not recognise .fa.fai as a valid name. This was reflected in both cases in the URI we used in the config.json.
• The URLs (with SAS tokens) work when accessed directly in the browser, so the links themselves seem functional.
• We enabled CORS on both (but maybe we did something wrong here):
  • the JBrowse2 server (via http-server --cors), and
  • the Azure Storage container (to allow cross-origin requests).
• We tried both HTTP and HTTPS-allowed SAS keys (no change in behaviour).
• We also attempted to add the assembly via CLI (instead of manual):

jbrowse add-assembly \
    --type indexedFasta "https://storageacount.blob.core.windows.net/path/to/the/folder/SomeReference.fa?sv=ThisWillBeOurSASKeyToTheBlobContainer" \
    --faiLocation "https://storageacount.blob.core.windows.net/path/to/the/folder/SomeReference.fai?sv=ThisWillBeOurSASKeyToTheBlobContainer" \
    --out /jbrowse2/

…but the result was the same.

Question

Has anyone successfully used Azure Blob Storage with SAS links to load an ReferenceSequenceTrack in JBrowse2? Are there specific CORS settings we might be missing, or is the issue something else? Any insight would be greatly appreciated!

I did not find anything mentioning Azure in other discussions, but maybe I missed those...

Thank you for any input you might have!

[cmdcolin]

there is sort of a lot going on in this question, so it's hard to address all aspects of this but i can say

1. docker config is irrelevant
2. http-server --cors is irrelevant, the server you are fetching files from is where the cors config matters, so it is the azure blob storage cors config that matters
3. jbrowse config is basically irrelevant, there's nothing inherently wrong with it
4. "The .fai file in Azure was originally named SomeReference.fa.fai. Since the error message only mentioned SomeReference.fai, we renamed the file accordingly, assuming JBrowse2 might not recognise .fa.fai as a valid name. This was reflected in both cases in the URI we used in the config.json."<-- there is nothing wrong with .fa.fai. probably what you saw was that it created the URL "https://storageacount.blob.core.windows.net/path/to/the/folder/SomeReference.fa?sv=ThisWillBeOurSASKeyToTheBlobContainer.fai" because it just appended .fai to the URL, but that is improper due to the query param

Troubleshooting

Error: Failed to fetch fetching https://storageacount.blob.core.windows.net/path/to/the/folder/SomeReference.fai?sv=ThisWillBeOurSASKeyToTheBlobContainer

this is the most important aspect. it more than likely indicates cors error. you will need to fix the cors configuration on azure blob storage. you can look in your browsers network log to see what the cors error is. the error is opaque (unreadable) by javascript for security concerns, making it hard for jbrowse to properly report any better error message.

Extra note

The notion of using "temporary SAS tokens" in a persistent config.json may not be best practice. if they expire, there is no notion of refreshing them. for specialized cases like this, it may be worth it to instead use our embedded components, or a specialized 'internet account', which will retrieve the sas token dynamically using client side javascript rather than hardcoding it in the config

it is more work, but these kinds of specialized authentications often are.

[cmdcolin]

you can try cors-tester https://cors-test.codehappy.dev/ to see if your azure blob URLs work with cors. here is an example with a file from jbrowse.org, that confirms our volvox fai file is successfully requested via cors https://cors-test.codehappy.dev/?url=https%3A%2F%2Fjbrowse.org%2Fcode%2Fjb2%2Fmain%2Ftest_data%2Fvolvox%2Fvolvox.fa.fai&origin=https%3A%2F%2Fcors-test.codehappy.dev%2F&method=get

if you get a success, but it still doesn't work in jbrowse, please paste the response headers

[JasperBoomBejo]

Hey @cmdcolin,
Thank you very much for your quick reply!
This was just what I needed, https://cors-test.codehappy.dev/ really helped me check things and setting the correct origin in Azure CORS fixed the problem 👍.