Checking outputs for re-runs¶

fiberhmm-check tells you whether an existing FiberHMM output was made before a fix or default change that alters it, from the provenance the output records. It reads BAMs (their header and a bounded sample of records, plus the QC report fiberhmm-call writes beside them), fiberhmm-qc reports, fiberhmm-posteriors files, fiberhmm-consensus result directories and fiberhmm-pipeline output directories (the called BAM and QC report their outputs.json records).

fiberhmm-check calls.bam other.bam qc/sample.qc.json consensus_out/ pipeline_out/

ont_hia5.bam: RE-RUN REQUIRED
  - hia5-nanopore-gt-table (rerun-required; affected, medium confidence)
    Nanopore Hia5 calls made with the 2.x emission table [calls]
    evidence:
      @PG fiberhmm-call.3 command line runs code from a tree named after commit 1ca7d0a, which
      predates the fix (dc7abba)
    why:
      The Nanopore Hia5 emission table has been updated in 3.0. Calls made with the earlier table,
      shipped in every 2.x release, differ: nucleosome, MSP and TF calls all used it. Against 3.0
      calls, about 75-83% keep both edges within 5 bp; call totals are about unchanged.
    fix:
      Re-call from the aligned reads with FiberHMM >= 3.0: fiberhmm-call -i aligned.bam -o calls.bam
      --enzyme hia5 --seq nanopore (or fiberhmm-pipeline). fiberhmm-recall-tfs alone is not enough:
      the footprint pass used the same table. Re-run QC and consensus made from the old calls.
calls_v3.bam: clean

Exit status	Meaning
0	nothing to re-run: every path is clean, has only `info` advisories, or is not a FiberHMM output (for example an uncalled BAM)
3	at least one path has a `rerun-required` or `rerun-recommended` advisory (affected or possibly affected)
4	no re-run advisory, but at least one path holds calls without FiberHMM provenance (`unverifiable`): they cannot be checked, so re-run them if they came from FiberHMM < 3.0
2	a path could not be read or is not an output `fiberhmm-check` knows

When paths fall in several rows, the exit status is the first that applies in the order 2, 3, 4, 0. Only 2 means the check itself failed; 3 and 4 are recommendations.

A file whose reads carry footprint calls but whose header records no FiberHMM @PG (a 2.x file whose header was replaced, or calls made by an old release that wrote none) is reported as

calls.bam: CANNOT VERIFY: calls without FiberHMM provenance; re-run recommended if they came from FiberHMM < 3.0

never as clean: nothing in it shows which version or tables made the calls.

--json prints one document for scripts and GUIs (see JSON); --list lists every known advisory (with --json, as a JSON array of each advisory's id, severity, artifact, title, reason, fix, fixed_in and changelog); --scan-records N sets how many BAM records are scanned for read-level evidence (default 5000; 0 reads only the header); --no-sidecars skips the QC report beside a BAM. The full option table is in the command-line reference.

Severity and status¶

Each advisory has a severity, fixed per advisory:

Severity	Meaning
`rerun-required`	made by code or tables that 3.0 replaces (for example the 2.x Nanopore Hia5 table); re-run it
`rerun-recommended`	made under rules 3.0 corrects (for example DAF dedup by orientation); re-run where those rules matter
`unverifiable`	FiberHMM results without FiberHMM provenance: no fix can be ruled out; re-run if made by FiberHMM < 3.0 (exit 4)
`info`	a default changed; the output is not wrong, but a 3.0 run with defaults would differ

and a status with a confidence for this output:

affected: the evidence shows the output predates the fix;
possibly_affected: the evidence cannot tell (for example a bare version string that builds on both sides of the fix reported).

Outputs the evidence clears are not listed.

What counts as evidence¶

Strongest first:

Table digests. Since 3.0 every run's FIBERHMM-CHEMISTRY declaration records the sha256 of the emission tables it read (apply_sha256, recall_sha256; see Header declarations). FiberHMM ships the digest of every historical Nanopore Hia5 and DddB table, so table advisories are decided exactly, whatever the version string.
Commit. The declaration's fiberhmm_commit (a git checkout, or a wheel built from one), or a commit named in the @PG CL path of the program that ran (a source tree such as frozen_dc7abba/), checked against the commits that contain each fix.
Version. The @PG VN of the program. Development builds between 2.16.8 and 3.0.0 reported 2.16.8 while some already contained 3.0 fixes, so 2.16.8 alone gives possibly_affected (low confidence); earlier versions give affected.
Options. @PG CL/DS decide whether an advisory applies at all (and, for ddda-hmm-cpg-mask, a DS token every run since the change writes): --enzyme/--seq/mode= pick the chemistry, -m without --enzyme marks a custom-table run, an explicit --prob-threshold or --primary/--alignments silences the matching default-change note.
Records (BAMs, when scanned): di/ds duplicate-cluster tags, pairing tags and calls without any FiberHMM @PG.

Only the calls a file holds count: on each @PG history (the PP chains), the last fiberhmm-call/-apply run and the recalls after it. A fiberhmm-dedup run that cleared earlier marks (prior_marks=cleared, 3.0) likewise supersedes the dedup runs in its history. A BAM re-called with 3.0 is clean even if its header still lists a 2.x call. A samtools merge keeps every input's chain (clashing @PG IDs get a -XXXXXXXX suffix) and joins them with its own records; the calls of every merged branch are in the file, so if any branch is affected the file is affected. A FiberHMM run after the merge re-calls every read and supersedes them all. When the header cannot settle it -- @PG records without PP links (a history written without links, or a merge), or chemistry declarations whose pg= names an ID that the merge renamed -- every plausible reading is checked: an advisory that holds in all of them is reported as found, one that holds in only some as possibly_affected (low confidence); never clean. PP links that form a cycle or name an ID several records carry never count as proof that a call was replaced, and when the pairings of declarations and runs are too many to check one by one, each run is checked with every declaration it could carry (at most possibly affected). samtools cat of several files (and Picard GatherBamFiles) keeps only one input's header, so the other inputs' calls have no recorded history: unless a later FiberHMM call re-called every read, such a file is at least possibly affected, never clean. (fiberhmm-call's own samtools cat of its region files is recognised and does not count.)

Advisories¶

fiberhmm-check --list prints the current list. For 3.0.2:

ID	Severity	Output	Applies to
`hia5-nanopore-gt-table`	rerun-required	calls	Nanopore Hia5 calls with the 2.x emission table
`dddb-gt-table`	rerun-required	calls	DddB calls with the pre-3.0 emission table
`hia5-nanopore-called-as-pacbio`	rerun-required	calls	Hia5 run without `--seq` before 3.0 on reads the header shows are Nanopore (`@RG PL`, `map-ont`, basecaller `@PG`)
`custom-table-without-enzyme-defaults`	rerun-recommended	calls	`-m` without `--enzyme` on DddA/DddB input, before 3.0
`daf-dedup-orientation`	rerun-recommended	dedup flags	DAF duplicates grouped by alignment orientation, before 3.0
`pair-paired-duplicates`	rerun-recommended	calls	`fiberhmm-pair` before 3.0 on duplicate-flagged input
`tag-m5c-missing-ucg`	rerun-recommended	calls	`fiberhmm-tag-m5c` before 3.0 (no `ddda_ucg`)
`posteriors-reverse-frame`	rerun-required	posteriors	`fiberhmm-posteriors` files before 3.0
`qc-nanopore-opportunities`	rerun-recommended	QC	Nanopore `fiberhmm-qc` reports before 3.0
`qc-assay-misdetected`	rerun-required	QC	QC reports graded under an assay mode that is not one (`mode=flag`/`collapse` taken from `fiberhmm-dedup`'s `@PG`)
`recaller-tier-double-count`	rerun-recommended	consensus tiers	lattice-recaller results with edge/loose tiers before the fix
`daf-unaligned-evidence`	rerun-recommended	calls	DAF calls before 3.0 counted insertion, soft-clip and SNP-masked bases as protected
`hia5-nanopore-ml-threshold`	info	calls	ONT Hia5 called at the 2.x default ML threshold
`primary-only-default`	info	calls	secondary alignments called (2.x default)
`ddda-cpg-mask-default`	info	calls	DddA recall with CpG observations (2.x default)
`ddda-hmm-cpg-mask`	rerun-recommended	calls	DddA calls whose first-pass HMM (call, apply, or the pair/merge joint HMM, which records a `fiberhmm-merge` `@PG`) decoded CpG observations, before 3.0; a run without `hmm_cpg_mask=` in its `@PG DS` and no indexed commit predates the mask. DddA posteriors files without `cpg_mask_policy` in their metadata likewise (reported with artifact `posteriors`)
`untracked-calls`	unverifiable	calls	calls without any FiberHMM `@PG` (cannot be checked)

QC reports, posteriors files and fiberhmm-dedup/fiberhmm-merge BAMs record their FiberHMM version since 3.0; older ones without it are reported as possibly_affected where the advisory applies to them.

JSON output¶

fiberhmm-check --json prints {"schema": "fiberhmm.advisory_check.v1", "reports": [...]} with one report per path, the same object fiberhmm.advisories.report(path) returns:

{
  "schema": "fiberhmm.advisory_report.v1",
  "path": "ont_hia5.bam",
  "checked_with": {
    "fiberhmm_version": "3.0.2",
    "advisories_revision": 3
  },
  "kind": "bam",
  "status": "rerun-required",
  "needs_rerun": true,
  "confirmed": true,
  "unverifiable": false,
  "error": null,
  "advisories": [
    {
      "id": "hia5-nanopore-gt-table",
      "severity": "rerun-required",
      "status": "affected",
      "confidence": "medium",
      "title": "Nanopore Hia5 calls made with the 2.x emission table",
      "reason": "The Nanopore Hia5 emission table has been updated in 3.0. Calls made with the earlier table, shipped in every 2.x release, differ: nucleosome, MSP and TF calls all used it. Against 3.0 calls, about 75-83% keep both edges within 5 bp; call totals are about unchanged.",
      "artifact": "calls",
      "fix": "Re-call from the aligned reads with FiberHMM >= 3.0: fiberhmm-call -i aligned.bam -o calls.bam --enzyme hia5 --seq nanopore (or fiberhmm-pipeline). fiberhmm-recall-tfs alone is not enough: the footprint pass used the same table. Re-run QC and consensus made from the old calls.",
      "fixed_in": "3.0.0",
      "evidence": [
        "@PG fiberhmm-call.3 command line runs code from a tree named after commit 1ca7d0a, which predates the fix (dc7abba)"
      ],
      "program": "fiberhmm-call.3",
      "path": "ont_hia5.bam",
      "needs_rerun": true
    }
  ]
}

kind: bam, qc, posteriors, consensus, pipeline, or null on error.
status: clean, info, unverifiable, rerun-recommended, rerun-required (the worst severity present), not-fiberhmm (a BAM with no FiberHMM @PG, MA-TYPES or calls in the scanned records), or error (then error holds the message and advisories is empty).
needs_rerun: any rerun-required or rerun-recommended advisory; confirmed: any such advisory has status affected (a GUI can show "needs re-run" vs "may need re-run"); unverifiable: any unverifiable advisory (results that cannot be checked).
advisories[].artifact: calls, dedup flags, QC, posteriors or consensus tiers; program is the @PG ID matched (null for file-level checks); path is the file concerned (a QC report beside a BAM has its own path).

Python¶

from fiberhmm.advisories import check_path, check_bam, check_header, report

report("calls.bam")                        # dict above; never raises for bad paths
check_bam("calls.bam", scan_records=0)     # header only -> list[Advisory]
check_header(bam.header)                   # pysam header, header dict or SAM text
check_path("consensus_out/")               # BAM, QC JSON, posteriors, consensus dir

Advisory is a frozen dataclass with the fields of the JSON object above (evidence is a tuple) and .to_dict(). check_* raise fiberhmm.advisories.AdvisoryInputError for unreadable or unknown paths; report returns status: "error" instead. The advisory list ships in the package as fiberhmm/advisories.json (schema fiberhmm.advisories.v1), so a newer FiberHMM knows about more fixes: check with the newest version you have.