Open-source could finally get the world’s microscopes speaking the same language

>flexible enough that it's also becoming widely used in climate science.

For scientific instruments just the opposite took place in the 1990's with chromatography.

They adapted a climate file type for laboratory instrument use.

It was quite a saga but in a nutshell it was the manufacturers who well understood the benefit of a common file type once computers had taken enough of a hold. Most lab analyses didn't need a computer except for the most data-intensive, which could never wait and were being handled by mainframes or minis before the PC appeared. Only by those that could afford such costly investments like oil companies and drug companies. Which are some of the most desirable customers for instrumentation.

At the time HP's lab instrument division (spun off decades ago as part of Agilent) had the most agressive sales approach but if a potential customer was already entrenched with one of the few other vendors there was no efficient way to carry their past proprietary data forward into a new system.

So the vendors got together, formed the Analytical Instrument Association (AIA) and began work in earnest toward a common file type they could all agree on for data interchange, without having to abandon each of the file types they had developed earlier which remained in everyday use within their proprietary software.

Once this had been accomplished it would be more of a battle of the salespeople with one less (big) technical obstacle to overcome.

They ended up choosing a well-established extensible file type, netCDF, which was in wide use at NOAA without encumbrance. Tailored it to the task and after a number of years the new standard was almost complete. Major vendors were already supporting it.

Well, by that time HP had already become dominant so they stealthily lost interest for a while before pulling out of AIA completely.

Fortunately with no further progress in sight the whole thing was dropped in the lap of ASTM, where it has remained unchanged for over two decades. That is even more fortunate because hindsight has shown the difference between an ideal standard and a nonideal standard is that an ideal standard never changes, and this is one of the best examples. These are often referred to as CDF files even though that does not stand for Chromatography Data File, they were just named netCDF by the weather service during the mainframe days.

So for more than 20 years each PC chromatography data system has had the capability of handling AIA files for either input or output. The underlying proprietary files which each software package utilizes under the hood for real-time data acquisition may have some unique features, but the common netCDF interchange covers all the essential bases needed for post-processing. Anything less is sub-standard.

Only in the last few years has a major software package appeared which for the first time does not fully support CDF's and that is the latest offering from Agilent of all places.

It can only create optional CDF files from the data acquired by their own instruments, so far there is no allowance for inputting a standard CDF file for processing by their software any more.

It's almost the same in the field of ELN/LIMs. Researchers are required to use one, but there's no wider strategy mandating the systems to be interoperable

I'm in a field that would have to change significantly if funding agencies demanded open access. I would love it.

Right not there's basically zero funding for people who work on making our data open. It's kind of a hobby project that you do if you have free time and no one has found a real job for you.

Labs should not be allowed to spend grant money on this bullshit.

:/ [tell me about it...](https://paulbourke.net/dataformats/svs/)

bioformats2raw https://github.com/glencoesoftware/bioformats2raw handles a wide range of obfuscated formats.

I'm wondering if decent standards are created, we'll be able to buy an electron microscope on amazon, just like 3d printers.

> Some inventing custom compression codecs.

I'm probably not being cynical enough here, but this doesn't necessarily imply a desire for obfuscation.

Yes, it's really frustrating that Zarr is promoting these exploded file/object trees and not providing a viable standard serialization for an entire image.

It's ironic that they mention FAIR but don't see how much better it would be to have a serialized byte-stream you can cite and share with checksums etc.

Absolutely. I spent an entire summer writing translation layers for a bunch of different microscopes so that they could interface with the detector software of the company I was interning for. It was tedious work!

As far as I know, not technically (although I haven't kept up in the area), but you can definitely sweep through a volumetric sample; there are microscopes that can for example illuminate a thin z-plane of a transparent sample and collect the image or those that can reject out-of-focus (off-z) light for a particular z-plane, then move to another z-plane, etc. and then generate a volume on the software side.

There is a lot of specialized microscopes out there. Confocal is pretty widespread.

https://en.wikipedia.org/wiki/Confocal_microscopy

> Each pixel? Why?

I use the Zarr format (for climate science data rather than microscope data), and I think this is just poor wording in the article. In the Zarr specification the metadata is stored separately from the actual chunks of compressed array data. So the metadata applies at the array level, not the pixel level.

> Wait - are there optical microscopes that can create 3D images?

I think so - they do it by scanning lots of images at different focal lengths to create a 3D section (I think?). There are whole projects just for visualizing the multi-TeraByte 3D image files produced - Napari is an open-source image viewer which opens OME-Zarr data.

Probably not all these things need per-pixel metadata, but anisotropy exists and that means many of the variables you'd think are per-exposure are actually dependent on where in the exposure the individual pixel is. For instance, illumination level isn't uniform across a field of view for all cameras, and may need to be normalized.

They don't really "label every pixel" in the sense that I think about it.

Instead, they have a collection of dense arrays representing the image data itself, then have metadata at the per-array, or overall level.

A typical dataset I work with is multidimensional, it starts as: 1) 2D planes of multiple channel image intensities, typically 5K x 5K pixels, each covering just part of an overall field of view. These are like patches when you do panoramas- take 20 partly overlapping shots. Each plane contains multiple channels- that could be "red green and blue" or more complicated spectral distributions.

2) 3D information- the microscope takes photos at various depths, only the "in-focus" (within the volume of view) information. These can be stacked (like depth stacking) or turned into a 3D "volume".

3) Maybe the data was collected over multiple time points, so (1) and (2) repeat every hour. Other parameters- like temperature, etc, could also represent an entire dimension.

4) Every 2D plane has its own key-value metadata, such as "what color channels were used", "what objective was used" (magnification), and lots of other high-dimensional attributes (that's what they mean by "each pixel must be labelled with metadata"- the 3D position is the same for every pixel in a 2D plane.

5)

Generally all of this is modelled as structures, arrays, and structures of arrays/arrays of structures. In the case of OME-zarr, it's modelled as an n-Dimensional array with dimensions expressed in a filesystem hierarchy (first dimension typicall the outermost directory, innermost dimension usually a flat file containing a block of scalar points using some compressed numpy storage. Then at each level of the directory you have additional .json files which contain attributes at that level of the data.

Those partly overlapping 2D planes are often assembled into panoramas, which can be a lot more convenient to work with. There are various tools for working with this- I've used navigation map javascript, but napari is a desktop app wiht full suport for sectioned viewing of high-dimensional (7d) data.

OME-zarr is nice because it sort of uses the same underlying tech that the machine learning folks use, and it's ostensibly "optimized for object storage", but I still have lots of complaints with the implementation details, but it's important for me not to distract the OME-zarr team from making the standard successful.

There's real overhead. For instance, one spec that comes to mind the vendor starts the 0,0 pixel in the lower right. The spec calls for 0,0 to be the lower left. That leads to an inverted handedness for all images. Not a problem in projection, but as soon as you start in 3D, it creates a real mess. Some software packages try to detect when the camera was this vendor, and correct for it (silently). Other software packages refuse to correct for it, because they don't have a reliable way to detect it. A pipeline might have 3-4 of these software packages involved, eventually you have no idea if you handedness is correct and no real way to tell without being at a high enough resolution that you can use biological landmarks of known handedness.

Don't get me started on Euler angles.

Caveat being I'm more used to electron microscopy, maybe these things aren't as important with light microscopy because the resolutions are lower?

I work with microscopy data and we have to convert all the proprietary-but-still-readable-by-some-random-package image data generated by microscopy companies to ome-tiff/ome-zarr for it to be in a manageable format. I think it's great!

> People can share data in whatever format they have and it wouldn’t be hard for me or others to import it one way or another.

It's one thing for a highly-skilled user with a decade of experience to be able to import it eventually, another for an unskilled user to just have off-the-shelf tooling do it the same way for everyone automatically. This is more about meta-studies and unlocking new use cases. It sucks that the state-of-the-art for sharing academic microscopy data right now is basically looking at raster images embedded in PDFs, or email the authors, and then write a custom Scikit-Image script. Imagine if you had to read a PDF catalog and then email someone to order something off Amazon, or if your favorite CRUD app consisted instead of having an expert read a PDF and email a screenshots to you. What if sending those very emails to different recipients required implementing each users custom IMAP-like mail client. That sounds absurd, but it's kind of the way academic data sharing works now, lots of people are re-inventing the wheel and creating custom file formats.

Consider, for example, the work of Dr. Bik (example at [1]) who identifies cloned sections from microscopy data. Or what if, instead of each researcher having to generate their own images, or get lucky and remember a particular image there was a Getty Images/AP Newsroom platform where you could just filter for your particular subject and imaging parameters and share your data. A collection of proprietary RAW files with randomly-formatted Excel documents for metadata would allow individual researchers to get their work done, but would be pretty worthless in comparison.

[1] https://scienceintegritydigest.com/2023/06/27/concerns-about...

I write a slide management platform that doesn't convert images to a single format and it mostly works. From time to time you do get issues though (cause you see millions of slides). Sometimes vendor's own libraries fail to open the slides (cough Phillips cough). Or there is a new version of firmware that sets some flag that wasn't used before.

We support DICOM supp 145 too, but it's no panacea. There are still vendor specific quirks. The "surface" is larger (cause you expect all the metadata to be there in the standard format) so you still sometimes see differences.

The OME-zarr creators are pretty experienced in this field and have already done the work you're describing. Let's not create another standard after OME-zarr.

ome-zarr puts metadata in json not XML. There was an older standard that put the metadata in XML. I've already written converters (converted the XML schema to JSON schema), it wasn't really an issue. The raw data is in blocked compressed numpy ararys in a directory structure on disk.

It'll convert the image to something that has the same overall statistics as the real data but may not bear any resemblance to reality in the details