← All case studies ◆ Genomics · DiagAI validation

Migrating variant classifier pipelines into the Genomics England airlock

SeqOne's DiagAI pipeline is a large network of interlinked containers which classify germline variants against ACMG criteria. We rebuilt each container required by the Nextflow validation pipeline to be compatible with running inside the Genomics England Research Environment, a secure, offline system with no internet access.

Illustration of software containers passing through an airlock into a secure environment, where they connect as a pipeline

The challenge

SeqOne builds software for genomic diagnostics. Its DiagAI system uses machine learning to classify germline variants against ACMG criteria and to rank the variants most likely to explain a patient's condition. SeqOne needed to validate DiagAI on whole genome data held in the Genomics England Research Environment.

The validation pipeline consists of Docker containers orchestrated by Nextflow, which in a normal environment install Python libraries and custom functions while running. The Research Environment works very differently. It has no internet access, supports Singularity rather than Docker, does not allow root paths, and schedules jobs on the Double Helix HPC cluster through LSF. Every container, Python dependency and reference resource had to be rebuilt, including its dependency wheels and Python environment, so it could run there without downloading anything at runtime, and without the possibility of directly testing the new design in situ.

What we set out to do

  • Convert the pipeline's containers from Docker images to Singularity images that run offline
  • Adapt the Nextflow configuration for Singularity, LSF and the Research Environment's file system
  • Transition the pipeline to use a new, updated reference sequence
  • Get the machine learning stack, including the ACMG classifier, running on the Double Helix cluster
  • Validate the migrated pipeline end to end and document it so SeqOne's team can maintain it

Our approach

Assessment and planning

We reviewed the pipeline architecture, 17 process modules and 10 containers spanning annotation (vcfanno, VEP, SpliceAI) through to ML classification and validation metrics. We identified the hardest component early, the acmg-ml container, which combines Python 3.11, Java 8, PySpark, XGBoost and MLflow and needs 80GB of RAM, and agreed reference data, test data, image storage and the overall approach with SeqOne before work began.

Offline container builds

We built an offline wheelhouse of every Python dependency, exported from Poetry and matched to each container's Python version, then built the containers in Docker and converted them to Singularity images. We removed runtime downloads and unresolved dependency versions, restructured non-standard tools into installable packages with a structured CLI, and settled on a Debian base image that met every tool's dependency requirements.

Configuration, testing and handover

The Nextflow configuration was rewritten for Singularity, with absolute paths, scratch storage routing and no reliance on root. We set bind mounts at build time, ran stub-mode checks of the orchestration followed by test runs on real VCF samples, and recorded memory use on the cluster so resource requests could be set accurately.

Results and impact

DiagAI's validation pipeline was packaged as Singularity images ready to transfer into the Research Environment, with an offline wheelhouse and a repeatable build order, a Nextflow configuration suited to Genomics England, and documentation covering known issues, version conflicts, mounts, permissions and dependency fixes. The builds are deterministic, so the same images can be rebuilt and checked when tools are updated, and SeqOne's team has what it needs to maintain the pipeline and add new modules.

In their words

“Complex problem, executed well.”

Lead Engineer, SeqOne

Timeline

Seven weeks.

Looking forward

The same build process can bring other SeqOne pipelines into the Research Environment, or update DiagAI's models and annotation resources as new versions are released.