Hacker News Reader

Bioinformatics file formats are broken, but it's better to try to understand why they are broken before rushing to fix them.

One key problem is that technology changes quickly. There are always new instruments generating new kinds of data with new properties and new features. People are using that data in new applications.

Software comes many years behind the state of the art. First you need to figure out what is the exact problem the software is supposed to solve. Then you have to solve the problem and turn the prototype into a useful tool. This work is mostly done by researchers who may know a little about software engineering. By the time the situation is stable enough that software engineers who are not active researchers in the field could be useful, it's often too late to change the file formats. There is already too much legacy data and too many tools supporting the established formats.

Another key problem is that the "broken" file formats are often good enough. When you have tabular data where the fields can be reasonably understood as text, a simple TSV-based format often gets the job done. Especially if individual datasets are only tens of gigabytes. By using a custom format, you avoid having to choose from many existing formats that all have their own issues. And that often guarantee you version conflicts and breaking changes in the future.

Also, when it comes to parallelization, it's hard to beat running many independent jobs in parallel. While computers are getting bigger, individual problems are often not, as the underlying biological problems remain the same. In the work I do, a reasonable target system had 32 cores and 256 GB memory in 2015. That's still a reasonable target in 2022. The computers I use have become cheaper and faster, but they have not really changed.

> When you have tabular data where the fields can be reasonably understood as text, a simple TSV-based format often gets the job done. Especially if individual datasets are only tens of gigabytes.

In what other industry is less than 0.5% utilisation accepted as "gets the job done?"

TSV is a terrible format for multi-gigabyte files, because it uses line breaks.

Technically it's possible to parse them in parallel, but if the format enables quoted strings with newlines, then it isn't possible to do this safely.

A simple binary row-based or columnar format could be read orders of magnitude faster. No character-by-character processing needed, just "memcpy" and go.

You did say yourself that overall utilization was more like 50 percent.

Interesting discussion though. I write some academic code in which I anticipate file load will be a bottleneck when we scale up one day, so I appreciate your thoughts

What I think jltsiren is trying to say is that if you're trying to parallelize within one file, you've already lost. Or your job is tiny, but then who cares.

This matches my experience a decade ago as a rare scientist who could program. When I had to run a job on the giant cluster against our ~100TB dataset, I did not put hundreds of threads to work against one file. We had everything broken up into hundreds of files, so I could run one thread against each file (with a tiny bit of boundary patch-up), and it all just flew. Trying to get it "all in one" would have been an exercise in misery and pain.

This can also be done by simply running many jobs at once. Many (but not all! never all!) scientific analyses naturally work well with that model, so don't fight it on technical purity grounds.

This was all done rather straightforwardly with the horrid piece of radioactive software garbage that is (was? please say was? I dare not check) CERN's ROOT. It had few redeeming characteristics... except for being fast and efficient on massive datasets, once you got it running at all. And that counts for something!

The glory of ROOT prevails!

But yes, it is better now than back in the 4.X days or whatever ungodly version it sounds like you used.

> What I think jltsiren is trying to say is that if you're trying to parallelize within one file, you've already lost.

I disagree, you don't need to split up a file to parallelize things if you just use a moderately recent format.

Put it in parquet, have sensible row groups, and turn on zstd compression. Split it into multiple files if you want but fast access to subsets of files which are neatly compressed is very easy to get now.

You also get the ability to store things losslessly, which you don't get so much without custom work with TSV (main example here is floats) and things like schemas.

I've just tested out moving from one of my zstd parquet test files that's 190M, and it's 4.8G as a CSV file for a single datapoint.

What we're trying to say is that that approach does not work so well if your individual files are 1-2 orders of magnitude larger and your whole analysis dataset (which must be processed in its entirety to answer any question!) is 4-5 orders of magnitude larger.

Yes, not everyone has such large data sets. (And not everyone has such small data sets!) But it is imperative that scientific computing infrastructure developers understand what classes of experiments they are serving, and what classes they can not or should not serve.

I'm not sure I follow. Splitting the file is fine, and very common, but you can absolutely get parallelisation within a file now using common standards. Even your single threaded code can be faster.

Currently working for a bioinformatics data processing company. I've seen, heard and disproved this "many files is better than one big file" argument before... But there is a trick to it...

---

Starting with an example parquet file first, something similar to bioinformatics data i've been working on recently. This is the "fully read into memory as a table / data frame" view of the file

    | File ID        | spectrum attributes                                    | groupings       | numbers and stuff | more numbers as a list | File as raw string |
    | -------------- | ------------------------------------------------------ | --------------- | ----------------- | ---------------------- | ------------------ |
    | 2984704        | ["other_thing", "another_thing", "different_things"]   | group       1   | 329021854.0935902 | [0, 2344, 22, 74, 745] | iw c3lyultrc3l.... |
    | 2984705        | ["other_thing", "another_thing", "more_things"]        | group       2   | 329021854.0934522 | [231, 09, 123, 15, 5]  | sdalkjfh2cn232.... |
    | 2984707        | ["other_thing", "other_thing2", "some_thing", "thing"] | group       2   | 3232518.032532    | [892342, 52, 252, 525] | cnm3247cmo27xm.... |

The "File as raw string" column is the magic one here. It's going to let us do what you did, but without having to manually manage thousands, hundreds of thousands or millions of files.

---

### PROCESSING

It sounds like you distributed your computation (smaller pieces of data, executing on many threads). But it reads like you did it manually (it reads like you wrote the orchestration code yourself, rather than sitting there submitting one file at a time). By bunching everything together in one file you can get the tools to do the work for you (at least with current tech you can, no idea about CERN's ROOT).

For modern business, the data engineering tech stack often uses Apache Spark for the distributed data processing engine / cluster engine [0].

Spark distributes data across the cluster nodes by partitioning your data frames/tables/tabular data. Rows 1-500 are loaded on cluster node 1, rows 501-100 on cluster node 2, etc. Spark then executes processing in threads on each node. Each partitioned row on a node is passed to each available thread for that node and processed. The results are stored in memory on the node and can be accessed later on for "other stuff"^{TM}.

Remember that I stored the raw file contents in the "rows" of the "table"? Now I can just "load" that file as part of the processing in a single thread [1]. Et Voila! I'm doing exactly what you did, many files being passed to many threads, but I haven't had to orchestrate anything myself.

Spark has done it for me! So the PROCESSING element can become easier, from the human operator orchestrating the processing of many files perspective, when you have one big file...

What about LOADING the data?

---

### LOADING

> What I think jltsiren is trying to say is that if you're trying to parallelize within one file, you've already lost

I can store this example data as parquet format because it is tabular. As part of the parquet standard, I can:

- partition the data by column values -- a file per partition based on the "grouping" column

- use row groups -- a file per partition of 10,000 row subsets

The above two mechanisms mean you can parallelise the LOAD of the data as well.

When you have a 100TB data set, this becomes *really* important to get right to minimise network data shuffles -- where data is being transferred around the cluster nodes because Spark needs to repartition the data across the cluster. If the data is already partitioned as a parquet file then Spark can load in 1x partition of parquet data onto 1x cluster node.

In an ideal case, where there is 1x cluster node for every 1x data partition, your full dataset will load onto the cluster in the same time it takes to load 1x parquet partition.

Network transfer times for 1TB vs 100TB are significantly different, and this approach can significantly reduce loading time when needing to execute different variants of the processing code on the same source data.

---

In summary, I get where you are coming from. But there are tools that do a bunch of magic things these days so we don't need to worry about stuff.

From jltsiren's original post

> Also, when it comes to parallelization, it's hard to beat running many independent jobs in parallel.

Embarrassingly parallel computation is what everyone is talking about here. Me, you, everyone. Spark does it very well. Python's multiprocessing library does it quite well.

The problem is that no-one thinks about loading and/or storing the data in a convenient format to do embarrassingly parallel computation ... they just stick it in a CSV/TSV file.

---

> This was all done rather straightforwardly with the horrid piece of radioactive software garbage that is CERN's ROOT

https://root.cern/releases/release-62606/

It is still being updated .... I agree with your sentiment, I would not use this out of choice after a brief glance at the docs.

---

[0]: The same principles here apply to something simpler like python's multiprocessing library, which is what I applied to gain an 8x speed up in processing times (they were running it single threaded before).

[1]: See the comment about pymzml about why it's not usually "just" that simple...

> The problem is that no-one thinks about loading and/or storing the data in a convenient format to do embarrassingly parallel computation ... they just stick it in a CSV/TSV file.

Independent jobs go beyond what is usually understood as embarrassingly parallel. In a typical bioinformatics workflow, you download the data, process it locally for hours, and send the results back. The ratio of computation to data is high enough that data transfers (and reading/writing data) rarely become a bottleneck. Meanwhile, the natural size of problems is both large enough that handling them takes hours and small enough that they can be handled on commodity servers.

In work like that, distributed systems tend to be overengineered solutions that make everything harder and more expensive. They make it harder to find developers who understand both the technology and the business needs – the biological problems in question. And they make it harder to install the solution in a new environment that may be fundamentally different from the one it was developed in. Especially in the fairly typical case where the person trying to install it does not have administrator rights.

> Independent jobs go beyond what is usually understood as embarrassingly parallel.

Independent = embarrassingly parallel, i.e. CPU bound. You want to apply some f on x to get y ==> y = f(x). You have many cases of x, all with f applied independently to get all the independent y outputs. There is no difference between "independent" and "embarrassingly parallel" in this case.

> In a typical bioinformatics workflow, you download the data, process it locally for hours, and send the results back.

And I'm saying that you do not need to be waiting for hours, if you do the data storage and data processing using sane and modern methods (i.e. not with CERN's ROOT tool).

This is not theory BTW -- I've been doing it for the last month. This is a very real, evidence based observation.

---

I was writing out a bunch of other point by point replies here, but I get the sense that it's more likely going to cause you to dig in to your currently held beliefs rather than open you up to the magic methods used by data engineering teams, so I decided to give up and go eat some pizza.

I enjoy pizza.

I think you are missing the context here.

We are talking about the kind of bioinformatics tools that establish new popular file formats. They are typically developed by researchers rather than software engineers. These tools are intended for end users to install and run on a wide variety of systems. And they are often released before the first high-profile papers on the topic are published. At that point, it's rare to find a software engineer who is familiar enough with the topic to be able to contribute.

Waiting a few hours for the results is not a big deal with these tools, because you are probably going to submit a large number of jobs anyway. Getting the full results will likely take days, regardless of whether you are using naive or modern methods. Because you are probably not allowed to use unlimited compute, your primary constraint is throughput rather than latency. Naive file formats survive, because tools that solve independent problems locally provide similar throughput to state-of-the-art distributed systems.

Bioinformatics, and particularly the subfield that focuses on genomics, is noteworthy because it relies more on freely available open source software than most other academic fields. That may be because the state of the art is changing so rapidly.

Companies sometimes take over old well-established problems. By providing services instead of tools, they can use whatever technology they prefer internally. But you rarely see these companies attacking state-of-the-art problems.

Bioinformatics tools are often used in batch processing pipelines, where the expected running time is hours. The ratio of users to developers is usually low enough that trade-offs between developer time and running time remain meaningful.

If you have a 50-gigabyte TSV file, you can memory-map, index, and validate it in a few minutes. Overhead like that is usually negligible. And if you encounter something weird like quoted strings, you can just print an error message and abort. It's your file format, so you don't have to make your life difficult by supporting unnecessary features.

That approach still kind of works with hundreds of gigabytes of data. Once you have terabytes, text files become too cumbersome. (Here I'm unfortunately speaking from experience.)

> TSV is a terrible format for multi-gigabyte files, because it uses line breaks.

Put it into a database if you care about these things. But let's not reinvent the wheel, and let's not bikeshed binary file formats, of all things. It's a non-starter. Plain text is perfectly fine for long-term storage and interop, converting to a more effective representation is just a one-time cost.

A lot of data set sizes are best measured in terabytes and petabytes. Many of these file formats must target that mentality, because small things grow up and get big whether anyone planned for that or not. People who can get away with plaintext for their needs probably aren't even reading this.

I ran into this problem while building my new data management system. I create relational tables by parsing CSV, TSV, or other 'character separated' files to get the row data and then use separate threads to insert each row. Once I had successfully parsed out a row, I could multi-thread the insertion code but only one parser thread could work with the actual file since embedded line breaks could be enclosed within quotes. It only takes one line to break any kind of parallel processing on the file.

The big issue is how many tools would need to be rewritten to accommodate the updated file format. For better or worse there is a lot of inertia to the terrible file formats, it's not as simple as "Hey, change".

As an example, consider clinical use cases. It can be near impossible to update the software in the pipelines for these situations. How do they move on from the old tools? Ok, now suppose they need to interact with data from non-clinical sources?

Things like you're proposing can be done. As others have intimated in this thread, solving the technical issue is the easiest part of the chain. More than one Tech Person has walked into the fray assuming the only thing holding stuff back is that no one ever thought to apply Good Tech Solutions. Instead what's necessary are people who can understand why things are the way they are, and the human dynamics at play. Building a better mouse trap requires taking these into account from the start.

That's why we need more cooperation between researchers and actual software engineers and less "I can also code!" thinking of researchers. Yes they can, but please in their free time or publicly, so that actual software engineers can chime in and stop bad design from happening. Well coding in public is of course also no guarantee and neither is hiring a software engineer, but at least the chances are better for something useful to come out of it.

yes, we need replace all scientists with software developer.

You do not seem to grasp the meaning of "cooperation". Or were you trying to comment on another comment?

sorry for this comment, I just frustrated.

not all coding is software engineering man

So what prevented the use HDF or netcdf in bioinformatics? I mean these are not new format by any definition of new and I disagree the broken formats are often not good enough, it seems more that often the bio/med fields (probably/hopefully not bioinformatics) are still used to process data using excel (speaking from my limited interaction with mainly med researchers).

Relative obscurity, most likely. If the formats are not used in bioinformatics, people developing bioinformatics tools are usually not familiar with them. And if developers are not familiar with the formats, they can't make informed decisions about using them. Yet another consequence of researcher-driven software development.

HDF has been around in some capacity in bioinformatics for at least 20 years. For instance the Bioconductor project had a package for it in the early aughts. But as to why it never took off, I suspect the reasons you cite are correct.

We presented using Parquet formats for bioinformatics 2012/13-ish at the Bioinformatics Open Source Conference (BOSC) and got laughed out of the place.

While using Apache Spark for bioinformatics [0] never really took off, I still think Parquet formats for bioinformatics [1] is a good idea, especially with DuckDB, Apache Arrow, etc. supporting Parquet out of the box.

0 - https://github.com/bigdatagenomics/adam

1 - https://github.com/bigdatagenomics/bdg-formats

Maybe column-oriented formats like Parquet never became popular in bioinformatics because new file formats usually come from people developing tools for upstream tasks such as read mapping, variant calling, and genome assembly. They are the ones who work with new kinds of data first.

Those upstream tasks tend to be row-oriented. You often iterate over all rows, do something with them, and output new rows in another format. Alternatively, you read the entire input into in-memory data structures, do something, and later serialize the data structures. Using column-oriented formats for such tasks does not feel natural.

HDF is used in single cell genomics

>One key problem is that technology changes quickly.

Each of your plateaus of stability often need to become recognized before the next step can be taken.

With scientific software at the end of the train, the file-type/file-system needs to be well established and more stable than any software could be, and for a lot longer than the whole scientific project itself takes.

A scientific filetype needs to be well-documented (better than ordinary software) and unchanged for long enough so that all agree no further changes are intended. It needs to have already been virtually perfectly stable, for more years than most research projects are likely to have their data remain useful in the future.

netcdf is in this category while still being extensible and it is very old (well established) if not well known. Public domain government "codec" which basically decompresses netcdf to structured text, and in reverse.

Now this new ASDF filetype looks like it does have useful features of its own except one thing:

>ASDF is under active development

Which can still be a drawback in this situation.

It's probably a useful reminder that biologists do all kinds of other things besides loading sequence files. For instance, they generate the sequence files and grow the organisms that are getting sequenced.

These files are all about persistent storage for many many decades, not sub second processing.

It's not that hard to pre process prior to feeding into whatever the latest processing architecture of the decade is .. but it's hell week when you're tasked with decoding some decades old non standard "only used for 18 months" "best practice at the time" obscure compression format data.

Field seperated line orientated ASCII data doesn't require extensive out of band notes to understand or decipher years after you've passed on.

> It's not that hard to pre process prior to feeding into whatever the latest processing architecture of the decade is .. but it's hell week when you're tasked with decoding some decades old non standard "only used for 18 months" "best practice at the time" obscure compression format data.

That's quite disingenuous, there are very well documented data formats that have been around for decades (netCDF, HDF) those are not "obscure" formats and are infinitesimally better than text.

> Field seperated line orientated ASCII data doesn't require extensive out of band notes to understand or decipher years after you've passed on.

Field separated ACII data is terrible because it does not contain relevant metadata (except for column names), you quite possibly loose precision (also what even was the precision?), are slow to read, need to be additionally compressed ...

> (netCDF, HDF) those are not "obscure" formats and are infinitesimally better than text.

so, not much better?

> infinitesimally better

Sounds like they better stick with the TSV format then ;)

"bonus" points:

* text/TSV data files made public as a requirement from the published articles may have spaces or dots in column names, missing line ends, non-unique row IDs, etc.

* there is no limit as how many records can be stored in a single file. Latest dbSNP has more than 1000 millions of rows.

* bunch of formats (GFF, GTF, VCF) has several TSV delimited "proper" columns (as: 1 value) and then special column where optional fields are piled in a different format, with another separator, field names etc. Real fun to parse...

I work in proteomics, where the field decided to standardize on XML based file formats. It can take longer to parse the data out of the XML container than it does to actually run the analysis.

Have you tried pigz? It’s an implementation of gzip that enables parallel processing.

For tabular data, parquet with zstd compression is very straight forward and quick. Columnar and split into row groups with stats on the columns within each, easily processed by many tools as standard. You can go further into arrow but parquet alone solves the majority of key issues IMO for tabular data.

Parquet looks good but HDF5 is good just because it's hierarchical and can group together multiple related datasets. Maybe data taken in the same session on completely different axes/coordinates this is what I've missed with parquet.

I had started a little bit of work towards that recently: https://github.com/celtera/uvfs

It's very optimized towards my specific needs but could be a basis for what you mention

> CRC style checksums are automatically a no-go. They're inherently sequential.

AFAIK the linearity of CRCs allows them to be split, parallelized, and combined. This is also the case for polynomial MACs, like GHash or Poly1305.

Yep, the only downside for CRC being that the recombination involves some expensive multiplications that are proportional to the total input length.

For Poly1305 you can use square-and-multiply to compute the exponentiation required for recombination, which has logarithmic cost, not linear. Are you sure a similar algorithm can't be applied to CRCs?

Sorry, I didn't phrase it well. It's also log(n) for CRCs via the same algorithm.

The "bioinformatics formats" might be terrible, but they work. In fact, they are meant to be Excel-readable which keeps my collaborators (and me) happy. Coming from a CS /programming background, it is natural to feel the urge to "fix" the formats (<insert relevant XKCD>), until you realise that there are a libraries that easily handle serialization/parsing.

Besides, "bioinformatic formats" is a meaningless word anyway. FASTQ, VCFs, BCL, AIRR-seq -- all different and it just works.

"Just works" and "I've been waiting 15 minutes now for this file to un-gzip" aren't compatible in my book, especially on a computer that should be able to process that file in seconds.

Also, I'd love to see someone open a 75 GB FASTQ file in Excel.

VCF at least typically uses bgzip which is essentially gzipped sections concatenated, but parallel unzipable for random access, cram is also parallelisable in the same way. Maybe you just dont know the formats and tooling so well? Im not sure anyone opens a fastq directly for viewing anymore, but they will want pile ups from a bam. The problem with bio formats isnt that they're text its that they are shit text formats too.

CRAM is a great example for some of the other people in the thread who say "just get a better format". There's been slow uptake in the larger community despite the benefits. For anyone looking to Solve Bioinformatics File Formats, it's important to understand why this is the case.

FASTQ is not for Excel, obviously - although you can still explore it in the shell. Nonetheless operating directly on FASTA/FASTQ files is often a "one-time" preprocessing task. You then serialize the preprocessed data and continue on from there.

FASTA (and its various incantations) are not going anywhere anytime soon.

Nearly all bioinfo tools operate in streaming mode which means line based gzipped formats work great as you can parallelise the processing with reading the file. Nobody ever unzips the whole file before starting to process it.

Excel-readable is a bad thing. I wonder how much data is ignored or misinterpreted because Excel misinterpreted the input to be something else.

(I actually know biologists who have run into this problem.)

https://pubmed.ncbi.nlm.nih.gov/27552985/ estimates that about one fifth of papers with supplementary Excel lists of genes contain mangled gene names. I remember talking about this problem back in 2003. The HGNC has been quietly going around changing the names of some of these genes to try and stop this from being a problem.

Thanks for the pointer. Indeed, a more recent paper (cited below) estimates an even higher error rate (30.9%), but the fact that we are not talking of 0.001% tells me that excel is simply a non-starter for this kind of work. (Actually, this is just one of many reasons why I discourage my students from using excel for any dataset.)

Abeysooriya, Mandhri, Megan Soria, Mary Sravya Kasu, and Mark Ziemann. “Gene Name Errors: Lessons Not Learned.” PLoS Computational Biology 17, no. 7 (July 30, 2021): e1008984. https://doi.org/10.1371/journal.pcbi.1008984.

> The "bioinformatics formats" might be terrible, but they work. In fact, they are meant to be Excel-readable which keeps my collaborators (and me) happy. Coming from a CS /programming background, it is natural to feel the urge to "fix" the formats (<insert relevant XKCD>), until you realise that there are a libraries that easily handle serialization/parsing.

And this is the crux of the issue, people still think excel processing is acceptable practice in 2022. If you are required to publish your analysis code (if you are not yet, it will come, the writing is on the wall), are you just publishing the excel sheets?

#927 is a fun quip, but as an actual critique of engineering practices, I'd actually much rather see folks attempt new standards and innovate when they feel like they have an idea which could work better, rather than be discouraged by the perennial "great now we have N+1 standards".

Every standard nowadays aside from the very first one, are an N+1. Heck, even IFF and ASN.1, the absolute old timers of file/serialization formats, are improvements on "just mmap to disk" application formats.

I do feel that there is a "missing" simple/basic text format which is somewhere in between csv and json/yaml. csv has no hierarchy while json/yaml are horribly unreadable for tabular data as it has no concept of columns (see https://csvjson.com/csv2json as an example). Maybe such a thing exists but I haven't seen it.

>Simply uncompressing a file can be a significant bottleneck, as this can inherently use only 1 CPU core.

Look this up: gnu parallel

Also, with building on what the top reply to what you said, if "uncompressing is a bottleneck" and you didn't know you could uncompress multiple files at once suggests you should spend time learning about tools that exist before you jump and try to force your colleagues (who likely have much more experience than you) to adopt "new practices" that the js community will move on from in 6 months.

TAR files can't be uncompressed in parallel. ZIP archives containing a single large file can't be uncompressed in parallel.

The resulting CSV file can't be processed in parallel, as there might be quoted record delimiters.

There are compressed file formats where a single file can be processed in parallel. Apache Avro and Parquet are examples I'm familiar with, but these handle columnar data (i.e. replacing CSV), not large numeric matrices etc.

Of course...I'm assuming (perhaps incorrectly) that they are decompressing many files sequentially without knowing that you can easily run gunzip in parallel using tools like gnu parallel.

The OP says they get to "ten gigabytes in size." It does not take 15 minutes to decompress a 10GB file on a modern workstation, as they complain in a child comment, unless you decompress multiple files sequentially. I've routinely compressed and decompressed 100GB+ files on 5 year old workstation class machines and it takes at most 5-ish minutes for one direction, (stress on "at most").

> If you find yourself in 2022 or later designing a file format intended for bulk data and you use any of the words "stream", "serialization",

Nit: I think I know what you are getting at, with blocks streams vs byte streams, but it's kinda hard to design a file format without serialization or byte streams. Not sure how that would work.

> I think it's high time that the industry standardised on a generic "container" format to replace legacy archive file formats.

I have a side project chipping away at just such a thing. It's quite daunting, so if this at all interests anyone, please comment/reach out. I'd love more of an excuse to work on this.

SITO in a nutshell:

- It's all based on msgpack, which does most of the heavy lift for serialization and datatype encoding

- a sito stream comprises blocks, each block is a self-contained, independently decodeable msgpack array object.

- each block an array of the form (type: smallint, header: optional(hashmap), data: any)

- the type is either a single-byte int, or a packed int indicating a substream id

- the sito primary stream comprises multiple independent substreams

- there's no raw plaintext fields, but there is a plain unicode block which can be used to embed whatever plaintext metadata

- substreams can each have whatever compression/codec they want

- since each stream is a block stream, it's trivial to de/interleave

- for data integrity, I want to do something like block-level CRC/FEC along with per-stream merkle trees but I haven't worked out the details yet

- there are periodic "sync-blocks" which have a magic 8-byte sequence for starting a file, but also throughout the stream, to facilitate re-alignment of read heads

- I've also been toying with the idea of using sqlite as stream indexes and as a general glue to keep track of what's going on (right now, you can arbitrarily start a new substream at any point in the primary stream, so it's hard to tell at the start of a file what's in it, sqlite pre-allocates pages so write heads can go back and update a prior index block)

- nd-arrays are a particularly interesting datatype so there's an emphasis on ergonomics around handling them

I plan on doing a simple PoC at some point soon showcasing SITAR, the sito archive format, with a python tarfile-like interface.

I just updated the github page with my current state of notes, in case folks are curious to some of the details.

https://github.com/xkortex/sito

Echoing my frustration... A few months ago I went through a similar search. IIRC my needs were: binary data storage (not ascii), index-based access on arrays without loading the whole file in memory, being able to attach metadata with the file contents, with interfaces for multiple scientific programming languages (Python, Julia, etc).

I recall going through ASDF, BSDF and a handful of other formats, finally ending up with HDFS -- which was okay, but not fully satisfactory (I don't recall all my gripes right now).

you're looking for zarr.

I'll argue the corner of bioinformatic file formats here. The main ones are FASTQ, SAM, BAM, VCF, and that format that GATK's DepthOfCoverage spews out.

FASTQ is a plain text format, and is usually gzipped. This is usually not a problem, as the only thing that's going to happen to a FASTQ file is you're going to shove it through an aligner, and a single thread can un-gzip that file fast enough to keep a lot of cores busy doing the alignment.

SAM is basically used for nothing, except very briefly as an output from the aligner before it is promptly converted into BAM.

The BAM format is actually sensible. It's compressed and indexed. There's an alternative format out there called CRAM, which can be a little more efficient, but you need to ensure that some external files are still available in order to decompress it.

VCF (and gVCF) files are text, and they are usually gzipped. Whether they are gzipped or not, they usually have an accompanying index file that allows any section to be accessed without having to sequentially read through. This is possible with the gzipped version because a variant of gzip called bgzip is used that compresses blocks of data, and locations of the start positions of those blocks are stored in the index.

The DepthOfCoverage file format - OK, I don't have any defence of it. It's just huge. If you're storing the read depth for a single sample, it uses about 24 bytes per location, to store a single number that's usually less than 256. So, a typical file for a whole genome sequencing sample would be around 75GB. It also has no index. A few months ago I decided to write an alternative file format, which delta-encodes then huffman-encodes the number in blocks, and uses about 1.3 bits per location, and includes an index, so that 75GB file is now 0.5GB and is a heck of a lot faster to read.

As an aside the GATK DepthOfCoverage is a fairly dire example of slow software. About 7 years ago I wrote my own DepthOfCoverage, which produces the same results but runs about 50 times faster. It wasn't hard. And because the BAM file format is sensible, yes it does access the files in parallel as you suggest.

There are advantages of text file formats. The format is unlikely to be non-readable in 10 years. You can just load it up in less and have a read. The text format doesn't stop it being compressed and indexed and accessed in parallel. But yes, the data could often be stored in a more efficient manner.

Finally, if you're finding that your server is regularly under-utilised, then you aren't doing load-management properly. You should use a queuing system that knows for each job how much RAM and how many CPU threads are used, and therefore how many can be run simultaneously.

> CRC or SHA style checksums are automatically a no-go

CRC is 100% parallelizable FYI. Both in SIMD and also on a block level which can be merged.