How does lossless compression in Fuji RAF files work? (2020) | Hacker News Reader

How does lossless compression in Fuji RAF files work? (2020)

capnfabs.net

110 pointsdsego72 comments

I wonder if something about this explains why Fuji RAF files are the only RAW files that make Lightroom struggle.

I never heard the fans on my M1 mac until I started processing those RAF files, they're not even that large but something about them makes Lightroom struggle. And judging by posts on photography forums, I'm not the only one either

I moved away from Adobe recently and ended up trying out Affinity Photo. Opening a RAF file would regularly seize my entire computer.

I always wondered if it was a deadlock situation, because it doesn't spit out any overflow errors.

Lightroom still is terrible for fuji raws from a worm standpoint. Almost looks like they use bilinear debayering. Dcraw is night and day better.

Lightroom struggles with Fuji's unconventional sensor pattern. Other programs like Darktable, GraphicConverter, Luminar Neo or FastRawViewer don't make fans of my M1 audible.

I'm not sure what causes the terrible RAF handling in Lightroom, but it's a fairly well-known issue. Lightroom produces so-called "worming" and apparently this is related to the X-Trans sensor which requires a different decoding algorithm. On my end, I can recommend Capture One - the handling here is perfect, though you do have to get used to the UI unfortunately

As someone who shoots on Fujifilm (XT-3), this was an intensely fascinating read. Thanks, now I have half a mind to sit down and re-implement this code, just to get a feel of how it works.

I've been thinking of buying that camera for a while, do you recommend it? Do you have anything to say that will finally push me over the edge to actually but it?

I have the same one and I can definitely recommend it. It depends what your camera experience is, but if you have had one that collected dust on a shelf in the past, I can guarantee you that this one is more fun to use and has a much lower risk of dust collection

I recently upgraded from an XT-3 to an XT-5, but loved my XT-3 and would still recommend it as a good purchase if you can find a decent deal on one in good condition. Fuji’s AF is not the best in the business, so I wouldn’t recommend one if you’re planning on using it for e.g. sports photography, but apart from that the XT series has no real downsides. The physical dials for ISO+exposure+shutter speed are fantastic and Fuji’s color processing makes images that I just enjoy looking at, even if they’re not as strictly neutral and accurate as what you’d get from someone else.

Fujifilm's whole X-mount series is wonderful and while I shoot "full-frame" M mount to remain interoperable between digital and film, there is no doubt in my mind that I would have a Fujifilm X-mount series if I only shot digital based on how much fun they have been when I have borrowed/tested them. Great "enthusiast level" cameras, great glass, solid build, everything has a button/dial, does not break the bank, and I actually know more than one professional photographer that shoots them and one of them even shooting sports!

I have an XT-1 from 2015 (still working!) and recently started considering upgrading to an XT-5 but I'm a little hesitant to buy a "new" camera first released in 2023 that still retails for almost the same price as two years ago. I'm so torn between just going for it and waiting (who knows how long) for the X-T6 to come out. Perhaps I should just try to find a good deal on an X-T4.

If it helps, I pay reasonably close attention to Fuji rumors because I'm deep in the ecosystem, and at present there appears to be no indication that an XT-6 is coming any time soon. They just released the GFX100RF and XE-5, plus there are rumors of an X-T30 III soon, and with all that in the pipeline I doubt they are also finishing up an XT-6. The -4 and -5 are still great cameras, I would just go for whichever of those you think is a better deal.

> an XT-6 is coming any time soon

Besides, it's still near impossible to get an X100 VI. B&H's backlog must be over a year at this point.

Innovation is very slow in photography world these days, X-T5 made a big jump in MP count compared to X-T4, but resolution aside image quality is pretty much the same, and other improvements were marginal.

I still use X-T2, and it has not really aged, even when compared to my X100V. Infamous Fuji AF is where they progress slowly but steadily, so that's the primary feature that I'd look into when choosing between generations.

See the trick is not to buy it new!

https://www.mpb.com/en-uk/product/fujifilm-x-t5

Or, as I have done myself and would recommend:

https://www.mpb.com/en-uk/product/fujifilm-x-t50

(Smaller, lighter)

The X-T4 is fantastic. See my other comment in this thread.

The "new release premium" is just too high, in my opinion. Cameras aren't getting better so fast that you aren't better off with the previous model.

Another happy X-T3 owner here (I had in my hand a Nikon D40X, D300s, D810 before getting a X-T1 and then upgrading to X-T3 ; thanks dad).

Yes, this is a very good camera. I love UI of Fujifilm cameras; and by that I do not mean the menu system (which is... serviceable) but the physical dial for each of the main setting. Putting them in "A" for automatic just make sense compared to the usual PSAM modes.

I own 4 Fujifilm cameras and personally, I'd recommend being VERY careful and thinking hard about this purchase. This isn't the same Fujifilm as it used to be. The company was once known for its "Kaizen" approach, which has long since disappeared. Prices are now inflated because they're riding on popularity. Autofocus in Fuji is simply weak.

The question is whether you actually need such a camera for anything. With a new smartphone that has multiple lenses, out-of-the-box photos will turn out MUCH NICER than from a camera, because initial processing is built into the software. Digital cameras don't have this. You need to take RAW and work pretty hard on it to make the photo look as good as what a smartphone delivers right away.

In tourist destinations, you can often find middle-aged guys running around with huge cameras when in reality most of their photos are quite poor. Because they don't realize that with a regular phone, their pictures would be much nicer.

> The question is whether you actually need such a camera for anything. With a new smartphone that has multiple lenses, out-of-the-box photos will turn out MUCH NICER than from a camera, because initial processing is built into the software. Digital cameras don't have this. You need to take RAW and work pretty hard on it to make the photo look as good as what a smartphone delivers right away.

You’re completely neglecting to highlight Fuji’s film simulations. I use Fuji’s specifically because they produce excellent jpgs out of camera. Not really sure where your take is coming from, an xt3 on auto will blow any smartphone picture FAR out of the water.

For those who love the Fuji film simulation looks but can't or don't want to buy an overpriced-because-influencers camera, there are now apps that do great Fuji’s film simulation: https://apps.apple.com/us/app/rni-films-photo-raw-editor/id1...

This is not true. Yes, these are characteristic color grading profiles, but if you want your photo to actually look proper, you still need to process the RAW file and you can add the Fujifilm profile as an extra on top of that.

There's NOTHING special about these profiles. It's a matter of taste. If you're buying a mirrorless camera, it means you have ambitions to take photos at a reasonably high level. Nobody who wants to be at a high level will shoot JPGs.

This aligns with my experience as well. The bigger sensor does generate pictures that look more crisp in big prints or zoomed in. In theory it should gather more light, but in reality, phones stitch together multiple exposures, and frequently produce nicer low light images without much noise. For sharing on social media, it's hard to notice a difference. For me its event worse with the x100 since the wide lens doesn't have that signature compression and depth of field, so the photos don't really stand out that much, no wonder most x100 photographers rely on color filters (film sims) and high contrast to draw attention.

I know of no phone camera that can produce the portraits of an X100s 23mm lens at f/2.

Not sure what you mean by produce, it depends on lighting and photographer skill. Not like the 23mm is really a portrait lens either and f/2 isn't spectacular.

Here you're talking about shallow depth of field which is desirable for portraits. But show me a camera that will have in JPG the dynamic range that you have in a smartphone by default? Show me a camera that will have as LARGE depth of field as smartphones have thanks to their small sensor.

These are all pros and cons depending on the scenario, but a phone has one advantage - it's small and you have it always with you.

It’s true that phones cameras are miracles of technology, especially considering their size. But I take a modern Fuji traveling because the modern phone camera look is so over-processed and distinct. There’s no faking the real optics a large aperture and sensor gives, the portrait mode on phones is still a poor imitation of the real thing.

Fuji then has the whole film simulation system with all their colour science from the last century. It’s a ton of fun, and the jpgs it produces are distinct and beautiful, and I believe better than 99% of people could achieve from post processing the raws, myself included.

The middle-age guy part is accurate though, I got it as a thirtieth present.

Even if phone cameras were twice as good, for me its simply more fun to take pictures with my camera.

I don’t find this at all, even compared to my (now rather old) X-T1.

For quick shots to remember an event or night out, modern phone cameras are fine.

For anything that I’d call photography and actually want to print, display, etc. I rarely if ever get results I’m really happy with from a phone camera.

If you’re in any way interested in photography beyond taking a few snaps at parties and on holidays, I highly recommend getting a real camera. I’ve found the Fuji system to be great, from the lenses to the out of camera JPEGs and film simulations that mean you can pretty much avoid doing any significant editing or post-processing if, like me, you find that all quite tedious.

Yes, if someone's goal is to learn photography and they're also interested in it from a technical point of view, then these are definitely cameras worth considering. My main point is that if someone just wants to "take nice photos" they should seriously think about whether to buy a good phone instead.

Gonna be honest: if you have to frequently use RAW to make Fuji photos look good, it may be a skill issue.

Wrong. If you have to frequently use film simulations to make camera photos look good, it may be a skill issue.

This comment exhibits the normal sort of hideous gatekeeping attitude that is common in photography.

I have an X-T3 and I love it. I went from an X-E2, to a Sony set up, and then quickly went back to Fuji. There's just something about Fuji that made it more enjoyable to shoot, for me (mostly travel photos).

I will say the only thing that gives me FOMO is the lack of the Classic Negative film sim, as a lot of recipes that I see online that I really like uses that film sim as the base.

If what appeals to you about Fuji's are the recipes and film sims, I'd make sure to research which ones you like, and then work out which model has the film sim you need to recreate it.

You can get the X-T4 relatively cheaply. Unlike the T3, it has a fully articulated screen and in-body image stabilization.

I have the X-T4 and X-E3, both of which purchased used for much below the price of the newest models (about $800 each). No regrets, and I love both equally.

The E3 is my stripped-down pocketable camera; with the Fuji 27mm pancake lens, I can fit it in a jacket pocket or shoulder strap bag, and it weighs almost nothing, less than my iPhone. This combo is pretty much equivalent to the immensely popular X100IV, but much better value for money.

The T4 is the bigger camera I use for nature and macro shooting. Tons of settings, more advanced features (focus bracketing and "picture in picture" focus closeup are important to me), more advanced dials. It's heavier and bulkier, but also more solid (IBIS, weather sealing).

For some reason Fuji appears to consider yellow focus peaking (which IMHO is the best colourbfor it) to be a high-end feature reserved for the T4, which is annoying.

Why yellow and not red? I find red much easier to see. Also I tend to agree about the X-E line but it’s been refreshing to use the X100IV with the inbuilt ND filter and not worry about changing lenses.

I don't know, but I recommend trying it, you might be surprised.

The X100IV is awesome, of course, and if I could afford it, I'd probably own one. But it's more than 2x what I paid for my X-E3.

A fixed-lens camera is built around the limitation of having just that lens. To me, if I only bring the 27mm with me when shooting, then that is exactly like a fixed-lens camera. But it also means I have the option to take it on a bird-watching trip using my Fujifilm 70-300mm lens — something you just wouldn't be able to do with an X100. That flexibility is worth something, which in my opinion makes the lower price of the X-E range even more of a bargain compared to the X100.

Will give it a shot!

Definitely agree with you, I think if Fuji made the X-E range contain an ND filter, then it would be the ultimate every day camera. Whilst the 27mm F2 on the X100IV is nice, being able to go to an even lower aperture can be priceless in some situations.

Apologies, didn't check HN for a while. I recommend it if you can get it around ~500-ish USD. I paid $750 (for the body) + $150 (for a 23mmF2 lens) in Jul 2024 used with a bunch of accessories including 4 batteries.

The biggest annoyance I've found is the horrendous battery life on the X-T3. For a long day outside on a trip, I end up going through at least 3 batteries.

The XT-4 is identical to the X-T3 (well, more so than any other x-tn -> x-t(n+1) camera) but fixes a few of the flaws in the X-T3 with massively improved battery life + IBIS which I'd recommend just because a lot of acclaimed lenses these days forgo OIS (ref: many Sigmas for instance), which could be worth it over the long term.

If you are very price sensitive then the X-T3 is still a really good purchase, with nifty features like dual SD slots which make it great to have backups/RAW+JPEG on two cards. Compared to an average photo from a phone, there just isn't much computationally going on in mirrorless cameras so even an x-t1 would be a good purchase.

If you want to shoot photos for the experience rather than getting clinically perfect images, and do not want absolute performance wrt focusing etc., it's definitely at the top IMO; analog with every control having a dedicated physical control (ISO, Shutter Speed and Exposure Compensation and aperture on Fuji lenses). I love it because it's the equivalent of driving an air-cooled Porsche, warts and all.

There are always these crazy discussions in photo forums where people are arguing that uncompressed RAW files are better than lossless compressed.

Glad to hear that audiophiles have some competition in who can invent the most pointless things to split hairs over.

The photo guys need to start taping magic rocks to their cameras to really keep up though.

They weren't just rocks. They were brilliant pebbles.

https://web.archive.org/web/20150220012844/http://www.machin...

The worst part is its often in areas people improperly understand the benefits and downsides.

Biggest problem with good raw compression is you have a linear DNG, half processed essentially. Great, the file size is smaller, but now you miss data that processes like AI denoise can benefit from as the image is already debayered.

On the flip side, good compression like DNG 1.7 spec's jpeg-xl compression is borderline magic. Lossless is actually lossless. The lossy flavour is so good even at 105 megapixels in 16 bit (per color channel) I would challenge anyone to spot a noticeable difference compared to the original, a file possibly 20x it's size.

On a tangent, bits per channel is yet another part people split hairs over. 14 vs 16 has almost no difference, no the colours are not 'better' even in a full 16 bit workflow, the only real world perceivable difference is your darkest darks are more precise and under extreme editing conditions do look a little better if being raised extensibly in post. I digress 16 is bigger than 14 and yay marketing.

Looping back to compression, 14 bit raws without compression are padded to 16 bit lengths due to word sizes and file constraints. This bit throws off the less technically minded who make all sorts of assumptions about file sizes and being 'more lightweight to edit'.

I'm kind of surprised we haven't seen things like 16 bit luminence, 12 bit chromanance. (I guess to do that before debayering would require RGBW pixels or something like that)

The problem is camera marketing makes it really unclear what you're actually getting. Really lossless? Does it trigger more on-body processing? Does it throw away bits to be "visually lossless"?

If you shoot a few thousand photos and then find you can't fix exposure as well as you'd hoped and the whole batch is worse, it's a pretty big disappointment, so it's smart to be risk adverse and skeptical.

>The problem is camera marketing [...]

IMHO marketing is almost always (part of) the problem. They shouldn't drink or smoke that stuff …

Equating "lossless" with "visually lossless" or some other phrase is newspeak. We could call a JPEG of quality >= 95 (or 96 .. 99) visually lossless too then.

Losslessness is easy to define: compress something, then uncompress it again and both the original and the uncompressed file should compute to the same (cryptographic) checksum.

There's probably an untapped market for sd cards with "special" holographic stickers on them retailing for $100 each.

Maybe an audiophile SD card would also produce better photos in a camera? You never know until you try!

https://www.whathifi.com/news/sony-claims-high-end-sd-card-o...

Those discussions I feel are fuelled by manufacturers like Sony saying [0] things like:

  Lossless compressed RAW:
  ...
  This is a popular format that occupies less space with minimal quality loss.

"minimal" "loss"? That's not "no loss", so what exactly is it?

[0] https://www.sony.co.uk/electronics/support/articles/00257081

I actually asked Sony Support about that. Their reply: "We can confirm that with Lossless compressed RAW there is a minimal quality loss. To have no impact on image quality I suggest using Uncompressed RAW files." Lossless isn't what it used to be.

They read it lessloss I guess

That's insane. I honestly thought that lossless basically means to run zip over the file and not more.

Gotta hate companies these days with their dishonesty. "Lossless" means "lossy". "Unlimited" means "limited to 50GB".

Seems you just got 1st line robo-reply repeating what public resources state. Does not say much about actual compression algorithm Sony uses.

Their first reply was "we have passed your question to a higher technical team", then they came back four days later with the above reply. I was enquiring about the A7R mark V, which introduced the much needed "lossless" option. I think I asked because I wondered why they kept the uncompressed option and because experts warned that Sony did that before with "lossless" formats.

It is a shame that Sony has such an obsession with weird proprietary formats.

A lot of cameras write "lossless" to mean "perceptually lossless". This is easy to do because ~12-bit ADCs have lots of noise in the low order bits.

"lossless" has always referred to compression, not sampling - but it seems camera manufacturers want to change that for marketing reasons.

Similarly (without starting an audiophile thread): Recording a vinyl record and compressing to a MP3 is "perceptually lossless" but will be different to compressing to a FLAC, never mind that the sampling output will always have random noise.

It’s because manufacturers keep inventing their own formats instead of using plain DNG, and you can only figure out the difference reverse engineering the file. Even for the same manufacturer and file type, they may change encoding details between models or firmware revisions.

There are situations where they may decrease the bit depth of the final image if there’s not enough dynamic range, there are situations where even though the file is “uncompressed” the camera already does noise reduction and essentially compresses detail in the image, and so on…

Manufacturers not supporting DNG is so annoying. But of course they do it. Allows them to sell licenses to software vendors like Adobe or Capture One.

How is DNG tooling these days? It's been ages since I checked, but it used have near zero support in (FLOSS) tools (including ffmpeg/imagemagick). You even had to use Adobe's Windows converter.

DNG is well supported by the typical raw libraries in my experience. My Ricoh records DNG natively, and I work with the results in rawtherapee and darktable. I am less sure about ffmpeg or imagemagick as I don't really automate raw processing.

I see however that DNG support is still read-only-ish, e.g. libraw still can't convert to DNG [1]. Most camera's still output raw in their own formats, which may not and probably should not be what you store them as, long term.

[1] https://github.com/ImageMagick/ImageMagick/issues/6344

DNG license might include some legal stuff that probably they don't want in their devices.

https://helpx.adobe.com/camera-raw/digital-negative.html#dng

What is the problem there? It essential provides an unlimited license any Adobe patents needed to implement DNG, and revocation if you assert a patent against Adobe related to DNG.

It provides an unlimited license specifically for reading and writing spec-compliant files, which could conceivably open you up to issues if you have implementation errors or attempt to extend the spec.

Yep, or doing something like a forensics software, that could be considered out of the 'correct implementation'.

Honestly the best argument for uncompressed is actually nothing to do with file quality or loss - it's that Apple only supports uncompressed Fuji RAWs.

You cannot preview or process lossless compressed Fuji RAWs on iOS natively but the uncompressed files are equal to Apple's own RAWs in support. On the field, it is sadly worth every byte to be able to grab a file directly off the camera and tweak it or send it to an editor. :/

The problem is that the size difference between compressed and uncompressed is enormous. This adds up quickly if you shoot sports or wildlife with 20fps bursts.

> Honestly the best argument for uncompressed is actually nothing to do with file quality or loss - it's that Apple only supports uncompressed Fuji RAWs.

Nitro¹ (macOS, xOS), the spiritual successor to Aperture², does support Fuji compressed RAW: https://www.gentlemencoders.com/extended-raw/

(I left Lightroom for Nitro + Apple Photos a couple years ago, and can strongly recommend Nitro to fellow photo takers.)

¹ https://www.gentlemencoders.com/nitro-for-macos/ ² https://www.gentlemencoders.com/about/

These forum discussions seem unconnected to reality IMO. Sometimes funny, more often strange, like audiophile discussions about oxygen free copper cables. Why?

The Fuji X-T4 and X-S20 here produce images of 6240x4160 pixels, but I almost never look at images in 1:1. My 4K monitors, mostly set to 2K, display 2560 x 1440 pixels. And even when switch them to their full 4K resolution I don't view my images in 1:1 obviously. And the tablet l'm typing this comment on offers "meager" 1800 × 2880 pixels. Most family members look at images on their "smart" devices nowadays, where 2K or 4K aren't present. So even decompressed images are fine for them.

I have my cameras configured to take both JPEG (fine) + RAW (lossless) of course. Fuji JPEGs (and Canon ... etc too) are fine for most casual viewers. And if I want to crop certain parts, or adjust certain details (esp exposure), I have my RAW images as a fallback.

Storage? My SD cards are 256GB and my disks are definitely not a problem either.

I almost never look at full images in 1:1, either, but I find very often that I have the wrong lens for an application and/or the wrong exposure.

You don't have to go very far before you've cropped a 20 MP source image into a 4K or 2K image, and if that part of the image that you've wanted to highlight is not well-lit, well, exposure is logarithmic and I want all of that RAW color depth that the camera can find if I'm going to turn black or white into perceptually accurate colors.

It's true that when my framing and exposure are great out of the box, I probably wouldn't notice or care if JPEG compression cut my file size by a factor of 4...but that's not always the case.

The main reason my wife shoots lossless is touch-up editing in light room. You can rescue some pretty crappy photos when all the sensor data is present.

> My 4K monitors, mostly set to 2K, display 2560 x 1440 pixels. And even when switch them to their full 4K resolution I don't view my images in 1:1 obviously.

macOS, but my understanding of Retina is that even if your effective resolution is lower (e.g., my 6K ProDisplay has a resolution of 3008x1692), applications can designate regions to be "original resolution," so I'm getting the image's original resolution in the editing window or region.

Reply on news.ycombinator.com