Hidden GPU pipeline failures cost genomics teams up to 25% more per sample, Nebius warns

Genomics teams moving short-read sequencing pipelines to GPU-accelerated cloud infrastructure may be paying 25 percent more per completed sample than their metrics show, according to Nebius, a cloud provider specializing in AI and life sciences workloads. The hidden cost stems from pipeline failures that force full restarts, and the waste is invisible in standard cost-per-sample reporting.

The typical whole genome germline variant calling pipeline runs for roughly two hours on an Nvidia H200 GPU and costs about $9 at current on-demand rates. But Nebius reports that among its genomics customers, between 15 and 40 percent of pipeline runs hit at least one failure and restart before completion. At a 25 percent failure rate, the real cost per completed sample climbs to $11.25, a markup that never shows up in the budget.

Waste accumulates in config failures and spot instance interruptions

Nebius senior product manager Anastasia Raskolova, writing for The Register, attributed the problem to a mismatch between CPU-era assumptions and GPU cloud reality. Neither workflow managers nor checkpointing systems are the root cause, she argued. Instead, misconfigured persistent disks for cache directories cause resume flags to fail, restarting pipelines from step one. Large tasks that fail mid-execution also require a full rerun because checkpointing happens only at step boundaries.

Key factors driving the waste include:

• Workflow managers like Nextflow require properly configured persistent disk space for their cache directory; without it, resumption fails automatically.
• Spot instances, often used to cut GPU costs, interrupt runs more frequently, pushing failure rates toward the 40 percent upper bound.
• Standard cost-per-sample accounting adds the retry cost into the completed sample, hiding the waste from finance and operations teams.
• For a team processing 2,000 samples monthly, the hidden cost reaches $4,500 per month, or $54,000 annually.
• Storage costs per genome are manageable at roughly 30 GB compressed, but retrieval costs can spike if data is cold and must be read back for retries.

Cloud providers and tool makers face pressure to surface retry metrics

The findings highlight a broader gap in cloud cost observability for genomics. Most cloud cost management tools track per-resource spend but do not attribute dollars to failed runs. Nebius suggested that platform leads and ML infrastructure engineers should shift their primary metric from cost per sample to cost per attempt, which includes failed runs as a separate line item. Short-read sequencing workflows remain the dominant data type in production genomics, so the problem is widespread. Nebius did not offer a timeline for releasing tooling to make retry waste transparent, but noted that better configuration documentation and defaults could cut failure rates among teams newer to GPU cloud. The company expects more genomics teams to adopt GPU acceleration for AI-driven variant calling tools such as Parabricks, which will make the cost mismatch more acute if left unmeasured.