Concepts
2 Sequencing Data and File Formats: Concepts
2.1 The problem
A file format is a contract between two programs that will never meet. The aligner that writes a BAM in 2019 and the variant caller that reads it in 2027 share no code, no authors and no assumptions except the ones the specification wrote down. Everything the format does not say explicitly is a place where the two can disagree while both believe they are correct.
That is the whole subject. Formats look like a clerical topic — a matter of extensions and conversions — and they are in fact where a large share of silent, wrong results come from. A misread coordinate does not crash anything. It shifts a feature by one base and produces a plausible number.
The useful mental model is not “a file holds data” but “a file makes claims”. A sorted BAM claims its records are in coordinate order. An indexed VCF claims that a byte offset exists for every contig. A reference-compressed CRAM claims you still have the exact reference it was written against. Reading a file is accepting those claims, and almost none of them are checked at read time.
2.2 Four families, four assertions
Nearly everything in this book is one of four kinds of file, and each asserts something different about the world.
Reads — FASTQ. A name, a sequence, and a per-base quality string. It asserts almost nothing beyond “these bases were observed with these confidences”. There are no coordinates because nothing has been located yet. Its weakness is its looseness: quality encodings have historically varied, and nothing in the file records which convention it uses.
Alignments — SAM, and its binary and reference-compressed forms (Li et al. 2009). These assert placement: this read came from about here, matching like this. That is a far stronger claim than FASTQ makes, and Section 6.6 covers the specific traps.
Variants — VCF, and its binary form BCF (Danecek et al. 2011). These assert difference from a reference at a position, across some number of samples. The format is deceptively simple and its subtleties are mostly about representation: the same biological event can be written several non-identical, individually valid ways, which is why comparing two VCFs is a harder problem than comparing two files.
Intervals and annotations — BED, GFF, GTF, WIG and the indexed binary forms. These assert only that a region is interesting, and carry whatever attributes the producer chose. They are the least constrained family and the most casually generated, which is exactly why they are where coordinate errors concentrate.
2.3 Coordinates are the sharpest edge
Two conventions are in use, and both are entirely reasonable in isolation.
A 1-based, closed system numbers the first base 1 and includes both endpoints, so bases three through seven inclusive are \([3, 7]\). A 0-based, half-open system numbers the first base 0 and excludes the endpoint, so the same bases are \([2, 7)\).
The half-open convention is not perversity. It makes length a subtraction, makes an empty interval expressible, and makes adjacent intervals share an endpoint instead of requiring an off-by-one. It is the better system for computation. The 1-based convention is the better one for a human reading a genome browser. The field uses both, and the SAM specification is explicit about which is which:
| Convention | Formats |
|---|---|
| 1-based, closed | SAM, VCF, GFF, Wiggle |
| 0-based, half-open | BAM, BCFv2, BED, PSL |
WarningRead that table again
SAM and BAM are on different rows. So are VCF and BCF. A format and its own binary encoding use opposite coordinate conventions, and a library converting between them is doing arithmetic on your behalf every time. This is deliberate — each encoding uses the convention that suits it — but it means “the position in the file” is not a well-defined quantity until you say which file.
The practical consequence is narrow and important: a coordinate that has been through a hand-written parser, a spreadsheet, or a quick conversion script is suspect until proven otherwise. Off-by-one errors of this kind do not announce themselves. They shift every feature by one base, which is invisible in aggregate statistics and wrong in every individual case.
2.4 Compression and indexing are one problem
A naive gzipped file cannot be read from the middle. Decompression is sequential, so answering “give me chromosome 7, position 500,000” means decompressing everything before it. For a file of hundreds of gigabytes that is not a slow answer, it is no answer.
The resolution is to give up a little compression to buy addressability. BGZF is block compression built on top of ordinary gzip: the file is a series of concatenated gzip blocks, each no larger than 64 KiB before or after compression. Because each block stands alone, an index can record the offset of the block containing a given coordinate, and a query decompresses only the blocks it needs.
The design has a property worth appreciating: a BGZF file is still a valid gzip file. Ordinary gunzip reads it correctly, having no idea it is structured. Compatibility was preserved rather than broken, which is a large part of why the format was adopted.
Indexing on top of that generalizes beyond alignments. Any position-sorted, tab-delimited file compressed this way can carry a coordinate index (Li 2011), which is why BED, GFF and VCF all acquired fast random access without any of them needing a new format.
An index is not part of the file. It is a second file that becomes wrong if the first one changes, and nothing forces you to regenerate it. A stale index is among the nastier failure modes in this area precisely because the data file is perfectly fine — the queries just quietly return the wrong records.
2.5 Reference-based compression, and what “lossless” means
Storing aligned reads as sequence is redundant: most bases agree with the reference, and the reference is a file you already have. Reference-based compression stores the differences instead (Hsi-Yang Fritz et al. 2011), which is the idea behind CRAM.
This introduces a dependency that the earlier formats did not have. A CRAM is not self-contained — decoding requires the exact reference it was written against, identified by checksum. Losing that reference does not degrade the file; it makes it unreadable. In exchange the file is substantially smaller, and the trade is usually worth it in an archive where the reference is guaranteed to persist.
The deeper point is that “lossless” is a claim about what you decided to keep. Reference-based compression of the sequence can be exactly lossless. But these formats also allow quality scores to be binned or dropped, and read names to be discarded or regenerated — and quality scores dominate the file size. A file described as compressed losslessly may have had its qualities binned, which is lossless for the bases and lossy for everything downstream that reads qualities. The question to ask of an archive is never “is it lossless” but “which fields survived”.
2.6 Streaming and the shape of a pipeline
Formats divide into those you can process as a stream and those you cannot, and this quietly determines what your pipeline looks like.
A sorted, indexed file supports random access, which is what lets a tool ask for one region. An unsorted stream supports only one pass in order, which is what lets a tool run without ever holding the data. The two are in tension: sorting requires seeing everything, so a sort is a barrier in a pipeline where every other stage might have been a pipe.
This is why so much of practical bioinformatics is written as a chain of processes connected by pipes with exactly one sort in the middle. It is not stylistic. It is the shape the formats impose: stream while you can, sort once where you must, and index afterwards so the next stage can seek.
2.7 The specification is not the implementation
Formats are defined by documents, but in practice they are defined by whatever the dominant library accepts. The two diverge. A specification permits things no file contains; implementations accept things the specification forbids; and a file that round-trips through one library is not thereby valid.
This has a direct consequence for anyone tempted to write a parser. The format is simple. Reading it correctly is not, because correctness means agreeing with the ecosystem on every case the specification left ambiguous and every case it addressed but nobody implemented. The next chapter’s central recommendation follows from this and nothing else: read formats through a library that already lost the arguments, rather than one that has not yet had them.
NoteWhy the specifications are worth reading anyway
The specifications for these formats are unusually good — short, precise, and maintained alongside the reference implementation. When a question about behavior comes up, the specification usually answers it in a paragraph, and that is much faster than inferring the answer from a tool’s output. Knowing they exist and are readable is most of the benefit.
2.8 What to take forward
- A format is a set of claims about the data, and almost none of them are verified when you open the file.
- Two coordinate conventions are in use, and a format’s binary form may not share the convention of its text form. Coordinates that passed through hand-written code are suspect.
- Block compression exists to make random access possible; the index is a separate file that can silently go stale.
- Reference-based compression makes the reference part of the data, and “lossless” describes the fields you chose to keep, not the file as a whole.
- Sorting is the barrier in an otherwise streaming pipeline, which is why pipelines have the shape they do.
- Reading a format correctly means agreeing with the ecosystem, not just with the specification.
Section 3.1 names what to run.