Adding support for [email protected]

[llk578496]

Is your feature related to a problem?

The latest supporting model for --basecaller_cfg is [email protected].

Describe the solution you'd like

Is it possible to add the model [email protected] in the --basecaller_cfg? Thanks a lot.

Describe alternatives you've considered

Additional context

No response

[cjw85]

Hi @llk578496,

This is on our TODO list.

[ammaraziz]

I am getting a related issue. @cjw85 could please explain where the models are being retrieved from? The docker image of medaka or somewhere else? I am recieving this #104 error again. Trying to debug where/what is happening.

The model I am trying to run is [email protected] and also dna_r10.4.1_e8.2_400bps_hac

[cjw85]

@mattdmem

[ammaraziz]

@cjw85 @mattdmem Sorry to bug you guys, we've switched over to r10 but I'm struggling to get the pipeline to use the medaka r10 models. We're running the docker images in such a way that they can't access the internet.

I think I have identified the issue, the wf-artic docker image comes with only r941_min_hac_g507_model.tar.gz model.

Could the docker image please be updated to include r10 which is standard now?

Using the latest docker image:

docker run -it ontresearch/wf-artic:latest medaka tools resolve_model
/home/epi2melabs/conda/lib/python3.8/site-packages/medaka/data/r941_min_hac_g507_model.tar.gz

using the 7month old one specified in the nextflow config:

docker run -it ontresearch/wf-artic:sha6e8c02f120faf92b4e61e1d0797d71210aaec20b medaka tools resolve_model
/home/epi2melabs/conda/lib/python3.8/site-packages/medaka/data/r941_min_hac_g507_model.tar.gz

[julibeg]

Hi @llk578496 and @ammaraziz, the workflow and image have been updated. Could you please confirm if this fixes the issue? Thanks!

[ammaraziz]

Will test out this week or next. Thanks Julibeg!

[ammaraziz]

I am sorry for not replying as promised.

The new updated docker image has newer models but not the old one:

docker run -it ontresearch/wf-artic:sha21d0682f5f67b0378de70765c50f06f0aa32ecee ls /opt/custflow/epi2meuser/conda/lib/python3.8/site-packages/medaka/data/

output:

r1041_e82_400bps_hac_v5.0.0_model.tar.gz
r1041_e82_400bps_hac_variant_v5.0.0_model.tar.gz
r1041_e82_400bps_sup_v5.0.0_model.tar.gz
r1041_e82_400bps_sup_variant_v5.0.0_model.tar.gz

This causes a whole set of new problems: there is no r9 models but the default medaka model for this pipeline is the r9 model.

pinging @cjw85 @julibeg @mattdmem imo this is a big problem

[julibeg]

@ammaraziz I'm not involved with the epi2me workflows anymore, but a quick hint: Medaka will download models if necessary:

$ docker run -it ontresearch/wf-artic:sha21d0682f5f67b0378de70765c50f06f0aa32ecee bash -c 'medaka tools resolve_model --model [email protected]:consensus && echo "-------------" && ls /opt/custflow/epi2meuser/conda/lib/python3.8/site-packages/medaka/data/'
Cannot import pyabpoa, some features may not be available.
/opt/custflow/epi2meuser/conda/lib/python3.8/site-packages/medaka/data/r941_min_hac_g507_model.tar.gz
-------------
r1041_e82_400bps_hac_v5.0.0_model.tar.gz  r1041_e82_400bps_hac_variant_v5.0.0_model.tar.gz  r1041_e82_400bps_sup_v5.0.0_model.tar.gz  r1041_e82_400bps_sup_variant_v5.0.0_model.tar.gz  r941_min_hac_g507_model.tar.gz

Does this not work on your end?

[ammaraziz]

Unfortunately, there's no Internet connection on my system and medaka doesn't allow specifying path to models.

I flagged this previously that the docker image itself needs the models to be available.

[julibeg]

I see. If you need a solution quickly you could make a custom image for the time being I suppose. On a machine with internet connection create a Dockerfile

FROM ontresearch/wf-artic:sha21d0682f5f67b0378de70765c50f06f0aa32ecee

RUN medaka tools download_models

and run

docker build -t my-artic .

Then, export as tar and transfer to the other machine. There, import with docker load.

You will also need a custom config file to tell nextflow which image to use

process {
    withLabel:artic {
        container = "my-artic"
    }
}

Finally, run the command with nextflow run ... -c local.conf.

I haven't tested the above, but with some luck it should work.

[ammaraziz]

@julibeg I can not thank you enough. My knowledge of docker is abysmal (I have to learn it one of these days), so I had no idea you can build upon docker images like this.

Your instructions were excellent, though they were missing how to export:

docker save --output="my-artic.tar" my-artic

I also pushed the docker image to my personal account https://hub.docker.com/r/ammaraziz/my-artic.

theres a few more tweaks to the nextflow config to make so it will accept the models I want, but overall I think this is a decent solution (for me at least!).