The patent on SIFT expired yesterday | Hacker News Reader

The patent on SIFT expired yesterday

patents.google.com

212 pointsgroar67 comments

Does this mean that OpenCV will be able to now include SIFT in the “free” modules. Or are there more roadblocks before including it?

I'm not a lawyer, but as far as I know expired US patents enter the public domain. After their expiration the described invention can be freely used by anyone for any purpose [1].

So I think the answers to your questions are yes and no.

[1] https://www.nolo.com/legal-encyclopedia/how-long-is-my-paten...

Perhaps there are still patents in other regions (?)

As others commented, the only countries where software patents work are the US and Japan. So it'd be just Japan that could have a patent in place? Don't know theh details, and IANAL.

OTOH, I'd really like the patent to be inactive, it'd allow me to work worry free in a lot of things that I've had an interest in.

I really hope so. Including SIFT in OpenCV should be easy since they did have an implementation earlier, which they removed due to the licensing.

https://en.wikipedia.org/wiki/Scale-invariant_feature_transf...

Evolution Robotics (later acquired by iRobot) was the first licensee of this patent for robotics applications. It was very cutting edge for the time and allowed our robots the ability to recognize real world objects. Having the robot follow a book that you carried in front of it was trippy in 2002.

Later we applied it to loss prevention in grocery retail with cameras at ground level watching under the cart. If we recognized anything in the ‘bob’ (bottom of the basket) we would automatically add it to the receipt.

Good memories.

Edit: typo 'tea' -> real

MobileVet, as an ex-Evolution Robotics person, I might have some media you're interested in (they might have come from you, if I'm guessing correctly who you are...).

Email me at jjwiseman@gmail.com.

wait... the concept was to have video cameras on the ground point upward in a supermarket setting?

Did anyone involved in this ever stop to think "what would a person wearing a skirt say to this?" It's not exactly the kind of thing you'd expect to be able to explain away with "oh no but we really don't use it for that" no?

A more charitable reading is that the cameras are at ground level and stay aimed at ground level. I got the impression he's not talking about the main basket of the shopping cart, but the storage right above the wheels where you might forget pet food or bulk paper goods.

okay, that might be my non-native English comprehension. I parsed "bottom of the basket" as the flat underside of the main part of the cart. And you'd have to point the cameras upwards to see that.

If it's horizontal, that's a different topic.

I had the same impression upon first read. And from a programmers perspective of what the ideal conditions for such a software would be, I also thought of a ground placed camera, pointing upwards in a way in which id could see the basket an find items that were "forgotten" in it.

Horizontally aligned, pointing into the space where your cart goes when you check out. Unless we’re worried about blurry pix of well-turned ankles showing up on the web, I don’t think there’s cause for concern.

Until the first horny teen working a station gets an idea into their head.

The cashier can’t see the camera. The camera is watched by object recognition software that can talk to the point of sale.

If object recognition algorithms develop the ability to be aroused by human underwear, we have -other- problems.

How much money did the University of British Columbia actually earn from this patent?

My memory of the story behind getting it patented in the first place was because he was having trouble getting it published.

I suspect that many patents are to stop others from patenting your work. It is a worry.

Open publication (at least in western jurisdictions) serves as prior art. If you don't intend to patent, just write about what you're doing openly and often!

Unfortunately prior art doesn't stop someone else from patenting your work. It just offers an avenue for someone to invalidate that patent. Worse still, the process of doing so is often riskier for small businesses than if they just agree to pay a licence fee.

But your argument applies equally to publishing your work as a patent as it does to publishing by some other means such as a journal article. So your comment doesn't address the parent comment at all.

Only in America and even there that's only true in theory. In practice it is a lot easier to invalidate a patent with an earlier patent than it is to argue prior art with a publication (this is, of course, depending on how the patents are written).

Patents can also be used "defensively" in ways that prior art cannot. So it's not always just about prevention.

Personally I think whole system could use an overall. Not just in America either.

Patent clerks search prior patents far more thoroughly than they do the entire literature of mankind. Having a prior patent is stronger protection from having someone else patent than is simply publishing somewhere.

I remember reading articles in "disclosure journals" published by different companies. My guess is that it was stuff that they didn't really think was worth patenting, but wanted to be safe from others patenting.

Haven't seen one of those in a while, and duck-go-go-ing didn't show anything either. I guess nowadays they just patent everything.

You could be right about just patenting everything. There have been suspiciously similar Chinese patents from an open source biometric projects that I was following. Who is going to check prior art in this avalanche of patents?

Can any publication serve as prior art? Can I use a personal blog or do I need a certain type of publication? I know of people who have sent copies of their work in the mail to themselves to prove prior art.

Any publication is fine as long as you can provide evidence. In fact the only thing special about publication is that it is a convenient way to demonstrate evidence of the prior art. What could be tricky about e.g. a personal blog post is proving that it was published when you say it was and that time stamps are genuine.

For those who are interested and doesn't know already[0] this is one of the places where blockchains and similar, older technoologies are actually useful: to provide irrefutable proof of knowledge at a given time.

Note however that similar services has been done for years[1] before Bitcoins and blockchains became a thing but since they neither need multi-Gigawatt-Proof-of-Work [2] nor any kind of "Coin" that can be pumped (and dumped) it wasn't cool.

[0]: https://xkcd.com/1053/ (and hi to todays lucky 1/10000! : )

[1]: See here for an article that describes it: https://www.vice.com/en_nz/article/j5nzx4/what-was-the-first...

[2]: but, it kind of works anyway since it would take quite some work to find the majority + the archived ones of a certain days New York Times papers and change the classifieds section.

I'm somewhat sure that prior art requires publication, and not just knowledge.

The internet archive has long been used as evidence for such publication-at-time claims, and they even have a paid service allowing you to trigger snapshots IIRC. There's a FAQ regarding the use of their data in court cases that is somewhat interesting to read.

I know the cryptocurrency community has this absurd notion that nothing without cryptographic proof could or should ever be regarded as evidence. That trusting institutions, people, or processes marks you as a gullible fool, and that the Federal Reserve is conspiracy by private banks to keep the gentile from the levers of power[0].

So this is the not entirely novel situation of crypto being proposed to solve a problem that doesn't exist, and failing due to the lack of (usually: active hostility to) subject matter expertise.

[0] last point unrelated to this specific situation and just included for completeness

> I know the cryptocurrency community has this absurd notion that nothing without cryptographic proof could or should ever be regarded as evidence. That trusting institutions, people, or processes marks you as a gullible fool, and that the Federal Reserve is conspiracy by private banks to keep the gentile from the levers of power[0]. So this is the not entirely novel situation of crypto being proposed to solve a problem that doesn't exist, and failing due to the lack of (usually: active hostility to) subject matter expertise.

It reads like you count me in with the cryptocurrency community.

If anyone else does: don't.

I'm pointing out one single place were blockchain can be useful and then immediately pointing out that it isn't strictly necessary.

Yes, you are free to pay a lawyer and sue to attempt to invalidate a patent using your blog as evidence of prior art. Of course whichever side can afford it, will bury the other side in motions and discovery requests until they win.

Not a lawyer, but as I understand it any publicly accessible publication would suffice. Actually I think any sort of open demonstration of the idea that others can learn from is good enough, but I have no idea how edge cases are handled or precisely how open it has to be.

The sibling comment mentions a good point about the falsifiability of time stamps. If you're particularly worried about being challenged, you might point the Internet Archive at your blog or keep your blog content in a repo that you push to GitHub. For the truly paranoid, consult an attorney.

Reddit discussion on this - https://www.reddit.com/r/computervision/comments/fednny/sift...

More info about what this actually is: https://en.wikipedia.org/wiki/Scale-invariant_feature_transf...

---

Also, reading the listing, I see

2020-03-06 - Anticipated expiration

2020-03-08 - Application status is Active

Is "active" some kind of obscure reference with a nonintuitive meaning, or did the patent owner reapply for the patent?

You can't reapply for patents like that, but I don't know what that status means. Patents often get an adjustment to their term and last a bit more than their nominal 20 years (due to delays being issued, etc.). https://en.wikipedia.org/wiki/Term_of_patent

I was also a bit confused, but it turns out that the last line is always dated today (in the local time zone ;-) and displays the last known status, which may be, and in this case definitely is, outdated.

See my other comment https://news.ycombinator.com/item?id=22525533

Anyone know other major patents expiring in the next few years? ( or that have expired recently?)

It could make an interesting hn thread.

SIFT works really well.

We use it for aligning input documents onto templates for OCR.

Super cool.

how did you go about licencing the patent?

Didn’t, was just RND.

One time on caltrain I was amused by some very drunk people trying to explain how SIFT works to each other.

You can still download the autostitch app - have used it several times for one-off panos:

http://matthewalunbrown.com/autostitch/autostitch.html

To what extent were the SIFT patents enforced?

As an undergraduate I worked as an RA for some (IEEE published) research at my uni for some robotics work and I gather that violating the SIFT patent was somewhat common when it wasn't obvious SIFT was being used, e.g. when SIFT was being used as a fungible solution:

Self-driving car demo that recognizes road-signs? No, that's too obvious.

Object recognition primarily using HOG or other approaches but using SIFT as a fallback or for determining a confidence value? Go ahead.

And SIFT was used in a lot of private projects (proof-of-concepts, etc) or lab workers' weekend personal projects. This was in academia, but I'm curious if this attitude extends to industry anywhere.

In UK law you're allowed to work a patent for the purposes of research. You're also allowed to work a patent for personal use, as long as it's not commercial (distributing FOSS is a commercial activity, fwiw, as it could compete with for sale products).

Not allowing research use would be entirely antithetical to the patent system itself.

[This post is my opinion, not legal advice, and is provided for entertainment purposes only.]

The way patents are enforced is the patent holder sues for damages. This obviously means there needs to be some damages, i.e. they claim they lost sales or something. If you aren't using the patent to make products or money, just to do research, this is a hard case to make. I wonder if it actually ever happens in academia.

Note that everything you "invent" that builds on the prior patent, including what you publish, is still blocked by that patent. And even if you ultimately remove the patented part, they can claim it was needed in the research process. So wantonly infringing on a patent in your research doesn't hurt the patent holder anyway.

The research exemption used to be quite broad especially for universities who would often use it as carte blanche, but was narrowed significantly by Madey v. Duke University.

Keep in mind that software patents are only enforceable in the US and Japan. Everywhere else in the world, you can do whatever you want (only copyright applies).

For businesses, it's still a big deal most of the time because you usually can't really neglect the US market, but for academia, there is no such issues.

> Keep in mind that software patents are only enforceable in the US and Japan.

That is sadly untrue. There is a lot of confusion on this front because "software patents" are a specialized term of art that essentially means 'a patent that describes a procedure with no reference to a machine'. There was a period of time in the US when such patents were enforceable, but more recent cases have mostly changed that (Basically Diamond v Diehr in 1981 to CLS v Alice in 2014).

But all this is moot, because patents that apply to the software-implemented functionality of machines are common even without "software patents". I've followed more patents on software cases in _germany_ than anywhere else.

You're absolutely right about businesses. The US alone would be enough of a problem, but it isn't even the US alone that is an issue here.

> a patent that describes a procedure with no reference to a machine

Essentially all software patents describe the von Neumann architecture and make sure to reference machines for this exact reason. Even when it runs on commodity PCs, the patent will still take a ridiculous amount of space to discuss how the RAM is attached to the CPU etc.

> To what extent were the SIFT patents enforced?

I don't know if the patent was ever enforce, but I know for a fact that some companies intentionally adopted SURF and variants derived from SURF due to the patent threat.

There have been many better feature detectors introduced in the mix over the past fifteen years. This change won't make too big a difference for people in the CV community.

Mhhhhh, I'm a researcher in the field and I'd say it's a mostly inaccurate statement. The root-SIFT detectors and descriptors are still really good. If you try to match images in different conditions (day/night), then CNN based approaches are OK.

Which one were you thinking about?

I would argue more that SIFT was valuable in introducing a new paradigm of feature extraction/matching. It is a staple of many computer vision courses. But yes the main algorithm I'm sure is quite (practically) performant as well

  2020-03-09 
  Application status is Active

Not anymore

Google Patent's interface is dumb. This means "Today (on 2020-03-09), the patent application is not in a state of pending to be filed." The fact they put this in the Events section is horrible UI. It's not an event, it's a current status as of the date the webpage was loaded. And it does mean that the patent is expired or not, it's a boolean specifying that the patent is filed (active) or pending filing. A better UI would be the text "Filed: yes" somewhere.

I don't understand what this means. US Patents are 20 years, right? And it was applied for in 1999, so it must be expiring?

Perhaps. I'm just pointing out (badly) that when this was posted (March 8th depending on Timezone) to HN that new status may not have been there. That latest status updated just happened March 9th.

TIL SIFT was patented. I wonder if SURF was as well. Not that it matters nowadays

yes it is

>In computer vision, speeded up robust features (SURF) is a patented local feature detector and descriptor.

https://en.m.wikipedia.org/wiki/Speeded_up_robust_features

Is sift still relevant today?

> Is sift still relevant today?

It is extremely relevant.

The fact that there exist other methods that "consistently surpass SIFT in all reasonable metrics" is inconsequential. You can say the same thing about JPEG compression, but JPEG is not going to disappear anytime soon.

SIFT has very good performance (but there are better methods), it is reasonably easy to compute (but there are easier) and fast (but there are much faster). Its shortcomings are well-known and easy to understand. The combination of SIFT+Ransac is the bread and butter of image matching, and if you work on this problem you cannot seriously propose a new method unless you at least compare to it.

There's a good paper on this:

https://demuc.de/papers/schoenberger2017comparative.pdf

"Our evaluation confirms that, as expected, learned descriptors often surpass SIFT on all evaluation metrics. However, we also observe that advanced versions of hand-crafted descriptors perform on par or better than the state-of-the-art learned feature descriptors, especially in the more complex SFM scenarios. As such, our paper demonstrates that there is still significant room for improvement for learning more powerful feature descriptors."

This paper is a bit "old" by the way. An excellent paper was released a week ago : https://arxiv.org/abs/2003.01587

Findings are mostly the same. For day/day images, with a properly tuned pipeline, SIFT is really good.

As a computer vision practitioner, I would argue that SIFT is still very relevant today. In most real-world scenarios it seems to hold up as well or better than the deep learning approaches I've tried, and it is easier to implement and maintain. Failures are often easy to understand and mitigate. In practice I often end up using FAST or ORB features due to "good enough" accuracy but much faster processing rates, especially on embedded devices. Feature detection and matching is an area where "classical" computer vision is very much alive and well.

Imho a CNN is like a generalization of what SIFT does, so a CNN can be trained to be equivalent to SIFT, but it can also be trained with more specific features for your use case.

Sure, in terms of expressivity, you can obtain much better results with a CNN. But very often, it is done at the cost of computational efficiency: SIFT descriptors are "easy" to compute.

Reply on news.ycombinator.com