Hacker News Reader

Comment by ryandamm | Hacker News Reader

I haven't pulled the patents, but they probably have to do with compressing the pre-debayered image. Apple sends Red a big check every year for the same patents; I believe they're part of ProRes but not sure.

Essentially, by compressing the raw RGB data coming off the sensor (where each pixel only has R, or G, or B) you avoid the data explosion of interpolating color values at each pixel. It's a technique for compressing raw files, and Red was early to the game.

Perhaps someone on here has more detailed technical info, I haven't actually read the patents, just talked to engineers close to the project.

View on HN

For those who may not be aware: digital cameras capture a grayscale image with a single value per pixel. The image sensor is covered by an array of colour filters (red, green, blue) in a pattern called the Bayer pattern. Typically, one row of filters will alternate RGRGRGRGRG and the next row will alternate GBGBGBGBGB. "Debayering" involves interpolating the colour-filtered grayscale values of neighbouring pixels into each pixel, and then often applying some software filtering to remove dead pixels, noise, and sharpen the image.

Aren't there still plenty of cameras using 3 separate sensors and an optical beam splitter, instead of a Bayer filter array?

I don't know of any consumer cameras that do this. Some manufacturers use a different filter on some models, Fuji uses an X-Trans filter on some models which in the past can cause issues with some software demosaicing.

X-trans is still a mosaic and the underlying tech is mostly the same (color filter array with monochrome "buckets"). Instead of a 2x2 pattern, it uses a 6x6 pattern.

Goes something like: GRBGBR BGGRGG RGGBGG

Instead of: RG GB

Consumers camera use a Bayer colour filter array because it is cheaper and more compact. Demosaicing is performant enough for most consumer applications that it is hard to justify for anything than scientific applications. As a matter of fact, even Motion Picture cameras, e.g. ARRI LF, are using a Bayer CFA.

If they do, that has to be limited to "broadcast" cameras. Digital cinema cameras all pretty prominently use a single sensor.

There is one model still sold by sony using these! https://pro.sony/en_GB/products/shoulder-camcorders/pxw-z750

All the others have now switched to bayer sensors.

Only $3k from Adorama; are pro cameras really that cheap these days?

[edit] That was google shopping popping up the Z150, which is presumably less expensive. The Z450 is 20k, body only, and the Z750 is "in store purchase only"

Yes, it's everything else that's expensive — a set of cinema lenses will set you back many multiples of that. Then you need to pay the people to operate them.

Here's a deep dive with an expert from Arri (big movie glass company) here: https://www.youtube.com/watch?v=q1n2DR6H7mk

There’s also a stacked sensor version.

https://en.m.wikipedia.org/wiki/Foveon_X3_sensor

Very surprised to see them still kicking, and preparing a 5th generation sensor by themselves.

https://m.dpreview.com/news/1885231343/sigma-releases-update...

They are really dinosaurs of electronic imaging. Sans Sony, I can say ALL sensor makers from the dawn of the era died off, or were sold/resold a million times.

Canon still makes their own sensors, don't they?

Canon is a long story, they been in, and out of the sensor game over the years

They eventually made a CMOS sensor development from scratch

Sigma had their Foveon sensor in APS sized version several years ago but they're currently working on the 35mm version. https://www.dpreview.com/news/1885231343/sigma-releases-upda...

It's taking forever though, they are encountering many issues to scale to 35mm.

Not plenty, “3CCD” cameras exist but are rare and mainly used for industrial inspections and TV broadcasts. Some old home video cameras did use it but not anymore.

What is the advantage in these last applications?

"Image quality" seems to be the most common answer... It eliminates the need for color filter deposited on the sensor, and also seem to require cares taken in lens design to have optical parameters converged for all three image planes in respective wavelengths(can't have one color being magnified or out of focus than others).

I suppose whether those points are pros or cons depends on design/manufacturing capability, and considering rarity of modern 3CCD designs, likely decisions are difficulties outweigh advantages today.

Nope, Bayer is the norm, but there's still some Foveon cameras that are typically rather slow to produce images.

No, nobody uses a 3-strip/3-sensor approach anymore. It was used in the early Technicolor process, and was revived during the digital era with cameras like the Minolta RD-175, and it lingered on in cine cameras for a while longer, but nowadays almost everyone uses a bayer-filter approach (or a very similar one - there are several proprietary alternatives for the mosaic layout that claim various quality improvements).

The only similar thing nowadays is Foveon sensors where you have 3 color sites stacked on top of each other, but Foveon didn't really have the money to keep up with the arms race and nowadays the modern sensors are far better even though they do have to apply a low-pass filter to avoid mosaicing. When you have many, many, many nodes of improvement past the Foveon sensor, that stops being a big deal.

It was also never particularly widely adopted, because none of these things are new - it was a tough sell to market a "really high color-resolution!" 5mp camera going up against 20mp filtered sensors and 50mp unfiltered sensors even 10 years ago.

Interestingly though, the Foveon approach is similar to film. Film has multiple layers and all are exposed at once. There's no denying bayer is sort of weird. It's just very expensive to be the odd man out doing your own sensors. "Good enough" usually wins, Foveon may be betamax but bayer is VHS - and I use that analogy knowingly, in the sense of Betamax also doing key features worse in areas that consumers care a lot about.

> It was also never particularly widely adopted

I think you must be ignoring television and video here? Basically everything that was shot on digital video for broadcast for decades was shot on 3-sensor cameras, and every "serious" video operation from sports to news all the way down to wedding photographers and high school videographers with any scratch at all were using 3-ccd setups. Sony's current (and only) 8K TV studio camera uses 3 sensors.

They meant Foveon was never widely adopted...

This means that the true resolution of digital cameras is roughly one third of the advertised megapixel value, depending on the exact Bayer matrix design.

Well that’s sort of true but the algorithms that decide on which pixel has which colour are better than just the number of underlying RGB pixels…

It is roughly half of the pixel count, as the Bayer interpolation gives better resolution than a naive conversion. Also, a lot of detail is in the green channel, which is registered by half of the total pixels.

Only if you consider raw pixel count in a single channel as the only "true" measure.

The true resolution of digital cameras isn't measured in pixels, it's measured in cycles per pixel and then related to the sensor size and pixel pitch[0]. Camera sensor pixels are not squares like the pixels in a digital image, they are point samples. You can then interpolate from the point samples onto an arbitrary number of digital pixels, so what matters is basically how close you can still see edges between lines, which depends on more things than the sensor size. When megapixels are quoted for a digital camera, they are either quoting sensor size (resolution) (like on real engineering documentation) or they are just quoting the highest resolution digital image they think they can get away with the software producing (marketing, advertisements), which is arbitrary, which is part of why you do hear not to just select a camera with the most megapixels.

It's a pretty deep topic and people do get degrees in basically just digital imaging. Not me though! I don't really understand most of this stuff...

[0] https://www.image-engineering.de/library/technotes/742-units...

It's only an approximation that cameras do point sampling, as the sensors are obviously not points.

That camera sensors mean a different thing for a pixel than what's typical for an image or an electronic display is a matter of convention.

True that interpolation is possible, but that's no reason to use different conventions as noted above.

I have seen a display manufacturer manipulate the same to artificially boost their resolution specifications.

Doesn't that mean green is over represented per area? Why not something like

    RGBRGBRGB  
    GBRGBRGBR  
    BRGBRGBRG

The human eye is particularly sensitive to green. This is also why the 5:6:5 encoding of RGB has its focus on green.

Also why Pentile OLEDs exist

Green is in the middle of the spectrum so it is the best for detecting amount of light. Color is determined by the G-R and G-B ratios.

There are sensors with more esoteric Bayer filters, like to better capture in low light there are sensors which replace one of the G's with an N (neutral light / "no" color filter)

I believe there are two things to keep in mind here:

1. There is the idea of compressing each Bayer color.

2. There is the compression method.

I believe this issue is about point 2, the compression. But it seems that Nikon is using a different compression so I am not sure Red will win this.

Edit: from the legal document, I believe it is about this part:

"The camera can be configured to transform blue and red image data in a manner that enhances the compressibility of the data. The data can then be compressed and stored in this form. This allows a user to reconstruct the red and blue data to obtain the original raw data for a modified version of the original raw data that is visually lossless when demosaiced. Additionally, the data can be processed so the green image elements are demosaiced first and then the red and blue elements are reconstructed based on values of the demosaiced green image elements."

That quote sounds a lot like 1. (the idea of compressing each Bayer color), but I might misunderstand it.

Sounds a lot like converting "left-right" audio into "mid-side" audio for the same benefits.

2 is RED obfuscating ordinary JPEG2000

Edit 2: Nikon is using the TicoRAW compression from intoPIX.

So I wonder why Red isn't suing intoPIX.

I have a guess, but I haven't actually researched this specific case to be sure.

intoPIX seems to be based in Belgium. I'm not sure exactly what type of patents RED has, but there are types like software patents which don't exist in the EU. Or perhaps it isn't about the type and RED just straight up doesn't have EU patents even if they theoretically could. Combine that with intoPIX (and I haven't verified this) only operating inside the EU, or at least outside of the US. You end up in a situation where RED would have difficulty suing intoPIX, because intoPIX doesn't operate in a jurisdiction where RED holds the relevant patents.

Once Nikon enters the picture, things change. Nikon clearly has US presence.

Of course it could also just be that RED doesn't care about small fish and only started caring now because Nikon has the potential for a lot of revenue.

Mindblowing that this technique is patentable

Until tested in court, many, many things are patentable...

It's been tested in court.

https://www.engadget.com/2019-11-11-apple-prores-raw-red-pat...

Major patent reform is required.

We actually have had major patent reform! Look into IPR review, the Alice Supreme Court decision, and the patent reform act of ... was it 2012? 2013?

The IP landscape is wildly different than it was when these patents were first filed. That they've held up under scrutiny implies they're not _total_ junk.

Also, it probably still needs reform. But my understanding from talking to lawyers (who I was paying, to be fair), it's better than it was.

Apple is really good at losing patent cases in court, even when someone sues using a patent copied verbatim from Apple itself (S3TC - road pizza video codec)

Keep in mind that video cameras of that era were mostly not bayer pattern sensors, at least for professional television cameras. They were three, full raster sensors, typically with an interference-based prism block shunting blue, green, and red light to different sensors. These were so-called "three chip" cameras, the optics of which dates back to analog cameras that used tubes.

Bayer pattern sensors were more common in DSLRs, because they were an extension of 35mm film, and you had to have a single, flat imaging device to work with film optics.

In the film industry, the only digital cameras were the Sony F900 series (iirc) which I believe had the three-chip design, the Thomson Viper (again, memory foggy), and maybe the Panasonic Varicam... I don't know if the Viper or Varicam were single sensor, but they evidently weren't Bayer pattern or didn't compress the Bayer pattern. (I think at least one had RGB stripe layout, which is just inherently inferior.)

So yeah, it's obvious now. But it was patentable then, because nobody (or very few) had ever made a Bayer pattern sensor for a video camera. It's easy to forget how long ago 2007 was, at least until I try to get out of bed in the mornings.

There's a good wikipedia page for anyone interested in how the 3 sensors were implemented: https://en.wikipedia.org/wiki/Three-CCD_camera

There's a fascinating detail about 3CCD/3CMOS that's not mentioned in that article, namely that the sensors aren't actually in the same plane, but slightly offset in depth (probably because of the filter films between the prisms). Broadcast lenses are specifically corrected for being either 3CCD or 1CCD, the former having a back-focal distance that varies with wavelength, so that red is focused in the red's sensor plane and so on. You cannot reasonably use one kind of lens on the other kind of system. Almost all broadcast lenses that shore up on eBay are for 3CCD - so they're pretty useless for use with "normal" small-format cameras, even if their specs are otherwise enticing.

Yeah, they used telecentric lenses; for the prism block to work properly, you have to have each principal ray across the lens perpendicular to the filter. This puts very rough optical constraints on the lens; a typical video camera lens would have dozens of elements arranged in many groups, while an equivalent 'cine' lens might have half or fewer elements.

And on top of that, the lenses produced ugly images. Going to single chip designs was a big upgrade in image quality, mostly for optical reasons.

> Bayer pattern sensors were more common in DSLRs, because they were an extension of 35mm film, and you had to have a single, flat imaging device to work with film optics.

A lot of people were shooting video with their DSLRs before more dedicated devices became available (myself included). Is there actually a fundamental difference between a DSLR and video camera at that point?

>A lot of people were shooting video with their DSLRs before more dedicated devices became available

I doubt it. DSLRs didn't have video shooting capabilities till quite later on (late 00's?) compared to phones and point and shoot digital cameras which did much earlier.

Yeah you're right - I was off by a few years. Seems like it wasn't common until 2009

There was a Bayer-filter camcorder on the market in 1997: the Canon Optura (or MV1, outside America).

That was such a great camera! That and my Nikon 990 were my transition to a full digital video and still image workflow.

But if you had a raw image from a camera with a bayer pattern, and you were tasked to compress it, anyone with decent knowledge about compression and the technology they're working with should be able to come up with compressing before de-bayering. Is something really patentable just because it applies to non-mainstream technology?

From the sounds of it, the patent literally seems to be about... compressing the data you get off of the image sensor. And that's it. Or are there important details I'm missing?

> But if you had a raw image from a camera with a bayer pattern, and you were tasked to compress it, anyone with decent knowledge about compression and the technology they're working with should be able to come up with compressing before de-bayering. Is something really patentable just because it applies to non-mainstream technology?

Actually this exact reasoning is textbook grounds for rejecting a patent application. If a “person having ordinary skill in the art” would trivially come up with the claimed invention, a patent application can be rejected without a reference of prior art by citing obviousness of the invention. If what you’re saying is true, the examiner that originally granted the parent was probably not familiar with the subject matter(which is common).

Source: was a patent examiner a good while back

Yes that's the theory, but in practice patent offices massively err on the side of "eh it's probably novel, we'll let the courts sort it out later, oh and by the way here's our fee".

Absolutely. Well, first "let me issue a vague rejection and force a slight amendment to show due diligence, then issue an allowance on the amended claims". And then let the courts sort it out, as you said.

A lot of great inventions can seem obvious in retrospect once it's been explained to you, so it's a tricky razor to apply.

Perhaps, but this line is more often used to pretend that obvious inventions are more novel than they really are.

While I intuitively agree with you, how do you tell?

And more importantly, how do you tell in the limited time window the patent examiner has ([1] claims it's as low as 19 hours.)

edit: [2] has a more in-depth look at the problem

[1] https://www.oecd.org/site/stipatents/5_4_Frakes_Wasserman.pd... - "on average, an examiner spends only nineteen hours reviewing an application, including reading the patent application, searching for prior art, comparing the prior art with the patent application, writing a rejection, responding to the patent applicant’s arguments, and often conducting an interview with the applicant’s attorney"

[2] https://www.patentprogress.org/2018/03/06/granted-19-hours/

The problem is that, as you say, examiners usually lack ordinary skill in the art. Judges likewise. And there is never a shortage of "experts" who will testify (for a fee, of course) to how hard this was in retrospect.

I see a lot of people say that software shouldn't be patentable, but I actually think the real problem is the "ordinary skill in the art" test seems to be entirely broken / almost absent in that domain.

A lot of high value software patents seem to be things that would be in the realm of a solid candidate to come up with on the spot in a 30 minute interview for a programming job, and not because the technique is "known". Compressing individual channels is definitely something that sounds in the realm of good intern candidate to be able to come up with the idea on the spot in an interview.

So the problem isn't really software parents intrinsically, it's that they are so absurdly out of line with the "invention costs" such that "something an average programming intern would be able to come up with independently has a high chance of actually being patented". If the parents were only actually for "nonobvious to someone who hasn't been exposed to it before" things then they wouldn't really be a problem.

It's not either/or but rather both. Software (like all other math) shouldn't be patentable, and the "ordinary skill in the art" test is entirely broken / almost absent in that domain.

Sure the idea of compressing before de-bayering might be obvious. I'm not so sure the actual technique for doing the compression in a manner that also made it easy to de-compress while de-bayering is quite so obvious.

It probably took quite a bit of experimentation to come up with a number of different ways of separating the colours and compressing the individual colours, and identifying the approach that played best with the de-bayering process. So the basic "compress before de-bayering" idea might not be patentable, but the actual algorithm and specific approach(s) sounds reasonably difficult to refine, and thus patentable.

My understanding is that all lossless on-camera raw compression is patent encumbered, and so companies use fancy proprietary mostly-lossless compression eg. blackmagic raw.

> So yeah, it's obvious now. But it was patentable then, because nobody (or very few) had ever made a Bayer pattern sensor for a video camera. It's easy to forget how long ago 2007 was, at least until I try to get out of bed in the mornings.

yeah really this is a commentary on just how poorly-suited a 28-year exclusivity period is given the rate at which modern technology advances.

I fully agree, this is something that definitely would have been novel at the time, it's a good patent, but even 14 years later it's grating that fairly basic/obvious stuff to contemporary eyes is still patent-protected.

There were Bayer sensors before Red.

If I remember correctly, three-chip cameras became impractical at higher resolutions because of the focusing problems that resulted from refraction. Red, green, and blue wavelengths refract at different rates... which was a problem when they're all coming through the same lens. I don't remember details beyond that.

I think it had more to do with the requirement that three chip camera lenses be telecentric (see my earlier comment). Tends to make worse lenses that are more expensive.

Seriously. I zip my raw files for storage on my NAS. Would that be an infringement?

> Essentially, by compressing the raw RGB data coming off the sensor (where each pixel only has R, or G, or B) you avoid the data explosion of interpolating color values at each pixel. It's a technique for compressing raw files, and Red was early to the game.

So the novelty is that they separate R,G and B planes before compression?

There are other compression algorithms that work with other colorspaces, giving even better compression. E.g. splitting luminance and chromatic components. I think these algorithms might also be older.

Digital camera image sensors only have one image plane. Each pixel is recorded as a grayscale value where the light has passed through a red, green, or blue filter. To get a complete pixel (with values for all 3 colours) requires software to interpolate a pixel with its neighbours.

Well... yeah, that's the crux of it.

But the patented part is treating each channel separately; the actual compression algorithm isn't at issue (which, in Red's case, is JPEG2000).

That said, as soon as you get into colorspace conversions or color sampling conversions (like YUV, etc), you're baking in choices and dealing with quantization and roundoff errors. If you want the most pristine image possible for dealing with it in post, you want the raw photosite information. And if you want that, without deBayering (which introduces those same errors!), you get to pay Red a royalty check.

I don't make the rules, I'm just pointing out what the stakes are here. Yeah, it's pretty fundamental and obvious now, but at the time no one was selling a professional video camera with a single chip for that price. It was a different world.

> So the novelty is that they separate R,G and B planes before compression?

> Yeah, it's pretty fundamental and obvious now, but at the time no one was selling a professional video camera with a single chip for that price. It was a different world.

I don't see how no such cameras being on the market makes the software any less obvious, unless we are totally abdicating the meaning of "obvious" to reduce it to "non-novel". But if that were the law's intent, it wouldn't require non-obviousness and novelty - just one or the other would suffice.

The obviousness test seems to vary by jurisdiction, but typically requires that someone with ordinary skill in the field would not likely come upon that solution[0]. It must also be novel[1].

[0]https://en.wikipedia.org/wiki/Inventive_step_and_non-obvious...

[1] https://en.wikipedia.org/wiki/Novelty_(patent)

> someone with ordinary skill in the field would not likely come upon that solution

This requirement often appears to be effectively ignored, as in this case. Compressing the raw data (i.e., before transformations like de-bayering) is a pretty obvious thing to try out. This sort of run-of-the-mill development does not deserve a 20-year monopoly.

Amazon's one-click patent is probably the most famous example of this. Anyone with any database programming experience can figure out "store the customer's payment details and shipping address and use them later to complete purchases". That was also true in 1999.

The patent office seems to evaluate the requirements in a manner very favorable to patent applicants/holders. I think that's a bad thing; the criteria for what counts as an invention should be stricter.

It was obvious then too. Way back when I was taking the raw grayscale data before debayering and I immediately saw I'd need to treat each color's pixels separately. No background in compression or signal processing or anything, just a tinker's mindset and some experience faffing around in Photoshop/GIMP

On top of that the patent requires the Jpeg2000 compression algorithm to be useful.

If the choice is between the current state of affairs and no software patents, then no software patents is the way to go.

> So the novelty is that they separate R,G and B planes before compression?

No. The novelty is that they compress before de-mosaicing (aka debayering).

In other words, rather rebuilding the image from the signals and compressing that image, it compresses the signals themselves (not even seperated I think).

I'm just guessing it makes more sense to separate them. In my naive no knowledge version. if it was

   RGRGRGRG
   GBGBGBGB
   RGRGRGRG
   GBGBGBGB

Then, if I compress losslessly just grayscale I know at the other end I just interpret the first pixel as Red, next as green, red, green, etc., Then on the next row I interpret the first pixels as green, next as blue, next as green etc...

BUT, if I compress lossily then the unrelated colors next to each other are going to bleed into each other while compressing. VS if I separate them out so

   RGRGRGRG
   GBGBGBGB
   RGRGRGRG
   GBGBGBGB

becomes

   RRRR BBBB  
   RRRR BBBB

   GGGG
   GGGG
   GGGG
   GGGG

The compression would make more sense and less bleeding of the signals from each color into other colors.

It would also seem to compress better. Imagine you're filming something pure green. Exaggerating but you'd get this

Which after separation is

That's just a guess though.

A different comment suggested they compress the green channel, and then compress the red and blue channel as differences against the green channel.

This makes a lot of sense to me, since green is sort of 'luminance'. It is like a poor-mans LAB color space that exists before demosaicing.

This is a very good guess, and I believe it's precisely how it's done. Doing anything diff-ing the color channels still requires a lossy transformation step, so it's not quite as "raw."

And you're right, that treating it as a single monochrome image makes it less compressible, because you get lots of high frequency information between adjacent pixels with different color. Which is sort of the opposite of what you want.

Note also that, despite calling it "raw," Red's implementation is lossily compressed. You can even tune the compression ratio of their "raw" codec, and back in the very early days, there was certain subject matter that just looked terrible coming out of the camera, even as raw -- I remember one particular tree that an early user would film to test new firmware updates; in the earliest firmware builds, it actually crashed the camera (!). The images it did produce showed tons of wavelet compression artifacts.

for people accustomed to looking at highly compressed video data (like mpeg2, H264 or H265/HEVC file, VP9, AV1, etc), what we're looking at here is a whole other category of barely-compressed high bitrate video used by professional cameras. It saves a lot of SSD recording storage space vs truly raw video but also leaves the video in a state to be further edited and cut together, then compressed into a final output product much later in the work flow.

The ordinary consumer will never see these video file formats, unless they go buy at least a $2500 blackmagic pocket 6k pro and add another $2500 of lenses and accessories to make it useful.

Apple actually tried (and failed) to break RAW from the RED royalties in 2019, so I believe you are correct.

https://www.engadget.com/2019-11-11-apple-prores-raw-red-pat...

Apple is paying RED for #9,230,299. A quick look at the claim doesn't seem to have this patent listed.

This is really a clinic on the dereliction of the USPTO and erroneous court rulings that have led to software patents.

But... the fact is that Red ripped off JPEG2000 and are now suing over it. And why not, when they're abetted by a corrupt patent regime.

https://patents.justia.com/inventor/thomas-graeme-nattress

Reply on news.ycombinator.com