Hacker News Reader

Comment by bsenftner | Hacker News Reader

Nice work! But people should be aware that this type of thing is patented left right and sideways. I wrote a patent for automated actor replacement in filmed media in '05, filed it globally, and it began being awarded in '08 in the various regions it was filed. Here's one of the US versions: http://www.google.com/patents/WO2007035558A2?cl=en I no longer own it; now it is part of a broad family of patents collectively forming a "Personalized Media" infrastructure. It is owned by a major media company I'm not allowed to disclose.

Essentially, it is possible to replace people, objects, environments, & audio elements automatically through a series of techniques. Those techniques form the basis of a powerful advertising and message delivery infrastructure. And the collective IP is already owned by a major media conglomerate.

View on HN

This is literally what's wrong with the patent system.

Discouraging people from exploring ideas and building things 'because its patented' is fundamentally wrong.

I bet there are a million patents out there for 'automated face replacement'. I just hope very very much, that somewhere there is some prior art that makes them all completely worthless and bankrupts whomever is making any money from them.

>> Discouraging people from exploring ideas and building things 'because its patented' is fundamentally wrong.

Who's discouraging? Being aware that it's highly patented doesn't mean you shouldn't play around with it, it just means that you probably shouldn't be trying to make money off of someone else's work.

The thing with software patents though is that most of the time they've preventing you from making money off of your own work, just because someone at one point did that same work first. Patents are needed in industries where r&d is expensive enough to warrant protecting inventors for a sizeable time so they can recoup their investment. Software r&d is too cheap for patents on software to do anything but harm.

As a quick rule of thumb, whenever people are being advised never to look at patents, as they are in software, that's when those patents do more harm than good.

http://rationalwiki.org/wiki/Galambosianism

I agree with the first part, and disagree with the second part of your statement.

Just because there are patents doesn't mean a person can't play around and even try to make money off of their work. It just means they need to do a little bit of investigation before trying to monetize their work.

The automated face replacement patents begin in the late 70's. By '05 the process had to be entire in-motion actor replacement to be patentable. To some degree, I'd say patents force people to work in uncharted areas.

You won't believe this one weird trick to save money on a network of local nightly news stations!

face is just a particular pattern, pattern matching and replacing was all done since the sixties.

True that the face is just a pattern, but matching and replacing in a manner that a human viewer is not aware of the manipulation is an entirely different matter. Integrated lighting, media grain, obscuring elements, including any reflections of your subject in the replacement, matching motion blur aliasing and so forth form the realistic problem to solve.

Those are all problems, not solutions. Problems aren't patentable. A method for solving problems is - and even then, only if it's a novel, nonobvious method.

Reading some patents, it seems "use a computer program" is a patentable solution, though.

Patents are for particular methods of doing this. You can't reduce all patents to "moving atoms and molecules around in different configurations" claim prior art, and invalidate all past, present and future patents.

To illustrate this, a patent (as originally intended at least) wouldn't be for something broad like "walking down the street" but maybe for something more like "walking down the street with a particular gait."

While I don't necessarily agree with the patent system, it detracts from the conversation with it's not an informed discussion.

What you said was a little strong, but I do fundamentally agree with you that patent clerks do not perhaps have the benefit of 5-year technology trajectory to make informed decisions.

When you look at the patents that Newegg fights (and wins), clearly something has gone wrong at USPTO.

It's not clear that the linked patent applies to the technique in the original post. https://www.google.com/patents/US7787664 and https://www.google.com/patents/US8712189 both seem more relevant.

Just because something looks like it could be patent encumbered doesn't mean you shouldn't pursue it. Patents need to be fought over in court. It costs millions of dollars to bring a patent lawsuit. In the event that your little side project or startup becomes a target for a patent lawsuit, frivolous or otherwise, you've likely already won.

Even with a semi-broken patent system, the world simply favors active, high-quality execution over passive "intellectual property" generation. Focus 364 days a year on a building a great product that people love and then spend a day to talk to your attorney about patents, IP, and how to protect the valuable company that you've built by focusing on your customers.

Jeff Schox has a great talk on patents and startups that is very relevant to this discussion: https://www.youtube.com/watch?v=hyv75d4JG7c

Be aware that a single technique can infringe on multiple patents, and a patent can extend other patent(s), triggering a situation where a single technique requirements licensing from multiple entities.

It is exactly the issue of fighting patents costing millions of dollars that makes technology development financially risky. When accused of infringing, one has to respond, and that response costs both significant time and significant money.

I agree a side project need not worry. A startup does. I also agree that the world favors active execution over passive intellectual property generation. Our world also forces the active executing organization to protect itself if they need any financing to realize their goals.

> Company I am not allowed to disclose.

Aren't patents supposed to be public information ? One of the (rather hollow in practice) justification for patents is to (also) encourage eventual disclosure of knowledge.

The patents are, but their ownership does not have to be.

> The patents are, but their ownership does not have to be.

I disagree.

It's important that people be able to determine whether the patent is in use or held by an NPE that's obligated to license it to people who want to implement products.

Were I to want to work in motion capture, I have no way to ascertain who owns your patents, if they're an NPE, and what possible licensing options exist.

I think patent owners should be forced to register with the patent office in a look-up table if they want to hold their patents.

I think the GP was just stating the fact, not expressing his opinion about it. But I agree that secret patent ownership sounds a little dodgy. Say I want to apply for a license to use some patented technology? Who am I supposed to turn to?

True. But in the case where the owner is secret, they shouldn't be able to sue infringers (I wish).

> Aren't patents supposed to be public information ?

Private agreements can restrict a person's comment on things without regard to whether the information is already public.

... and people should also be aware those patents might be invalidated by re-examination or an action in federal court. I don't know why you aren't allowed to disclose who now 'owns' the patent you refer to. It is trivial to locate this information with the link you provide and some googling.

Also your statement "the collective IP is already owned by a major media conglomerate" if taken to refer to totality of technology which makes it "possible to replace people, objects, environments, & audio elements automatically through a series of techniques" is false.

I beg to differ, there most definitely is a series of patented technologies that in total enable replacement of people, objects, environments and audio elements in traditional media. That key is traditional, linear media. New forms of media, interactive forms, VR/AR and hybrids are wide open fields. That's a big reason new fields of technology are so exiting - it's not just interesting to us geeks, but valuable to the businesspeople.

Only until it's locked down and killed for 20 years.

Even if it's patented you can still play around with the technology yourself. Just don't sell it or make a product out of it.

I remember that even 'having' the technology is in breach of the patent if there is no license.

Could be wrong though.

One can do anything they want privately if they can afford it.

> It is owned by a major media company I'm not allowed to disclose.

I don't understand this. If the patents are public, why shouldn't the owners be as well?

As far as I can tell, this is just a patent troll's patent. It seems to be a very generic process for replacing people, objects, environments, etc.

It's a patent and you're not allowed to disclose its owner? Is that a thing? Secret patents?

Those were the terms of the sale. I was not in a position to argue.

And it prohibits disclosure even after the sale/reassignment was completed? I don't understand the purpose of that, at least not a legitimate purpose.

Sounds like you were on the losing end of some strong arm tactics. Or did I read your comment wrongly?

http://assignment.uspto.gov/#/search?q=7460731 Here is the most recent public assignment of that patent family. It from Flixor, Inc. to Big Tent Entertainment, LLC. http://www.bigtent.tv/

What's your view on the patent? Pretty broad? Or a sign of very quick progress that in 10 yrs something innovative (hence, a patent) turns into a few open source libraries and few lines of code to replicate. Most responses (including my thought) call 'patent troll' etc, but perhaps there was substantial thought and effort in the techniques below the patent.

The majority of these "Personalized Media" patents rely on existing integrated media - meaning imagery and audio composed of multiple elements - and the ability to isolate one element, remove or modify it, and/or replace it with something else in a manner that the original media's intent and message is retained, only now it is enhanced for a specific audience.

Think product placements that change for every single viewer.

Real time rendering, VR/AR and that entire enchilada of media techniques completely sidestep these patents because there is no original media, it's all created and composited on the fly, dynamically for the single specific consumer of the media.

Just to echo this a bit, I spent 3 years working on image similarity tech and a lot of our research was challenging, not because of the intrinsic complexity of the problem, but because we had to implement it in such a way as to step around existing patents.

I guess you found the math combined with the search for alternative routes interesting? But did you also try to find prior art? My first thought would be that for unreasonably broad patents prior art is the easiest way to be able to use the tech?

I wouldn't say the patents we hit were broad. In some cases we thought they were fairly obvious, but that's much more subjective. We didn't chase prior art, mainly because none of the barriers were unavoidable and we wanted to keep momentum on the build.

I only had one foot in the algo camp to be honest. My main task was making the query engine scale.

Bleurgh.

Would you rather people spend time developing in this area only to learn they can't use their work?

No, he would rather be free to use the code he wrote.

Which is the exact situation I found myself in: I began developing in this area, and learned of patents I needed to sidestep. Realizing a hole in the patent landscape, I filed to protect what I was doing.

So now other people have to sidestep a sidestep?

It's only an issue because the original "sidestepper" sold-off, or was perhaps bullied into selling (theory), the patent to an entity. An entity that will unfortunately just hold it in a "bag of tricks" until they need it, or decide to play with it, or decide to litigate with, or decide to "license" out usage of it to another big player willing to fork over money for it.

Had an innovative tinkerer kept it, then who knows what might have happened. But I'd take a wild guess and say that the odds are more likely that a such an innovator would allow others the use/play/profit off of their novel ideas than would a large entity that sees it as a potential avenue of competition.

I'm not saying that. It's just frustrating.

Try doing this 10 years ago, putting your life savings into it, and then VCs and media companies "not getting it", your money runs out, and then you have to sell your IP simply to pay the rent.

I'm not meaning to criticize you personally (at all). I'm just saying that the idea of being able to "declare ownership" of what is essentially a series of mathematical operations just doesn't feel quite right.

I realise that people have to make money some how, but developing a useful, competitive, monetisable product that people are able to choose to buy is, in my opinion, the way forwards. Patenting the technology so that you can operate without competition rather than developing your product to stay ahead of the competition "organically" seems like the wrong way of doing it to me.

Perhaps I'm an idealist, and need to get real...

I agree with you. But the patent system exists... A startup has to protect its work. Plus, when pitching for financing the first set of questions include "Is this patented?" If you answer no and don't have a valid reason why not, they tell you to get it patented and come back when that is done.

This is a good comment, much better than the "Bleurgh" one, so I transferred you a vote (down there, up here).

So, there are an infinite number of things that are just series of mathematical operations. It's the ones that are useful that matter. Why would we not compensate people for discovering them?

Way to play the victim card, after first playing the everyone-else-was-doing-it card!

"Los Angeles based MBA..."

Makes sense, I guess you learnt this two card trick in MBA 101 :)

If you and people like you hadn't patented these things, they would've been able to use their work.

Thanks for the warning, at least.

The patent system exists. When I filed, there were already dozens of patents in this area. I was simply trying to protect my own work for my startup.

Your argument seems to be "everyone else is doing it, so it's fine". I'm not having a go at you, but that doesn't seem like a very strong argument.

I needed financing for my startup based on this technology. After an initial financing search, it became clear it had to be patented to get financing.

To some degree, patents are a mechanism forcing early technology companies into expensive legal engagements, which drain resources from the company, triggering hefty financing requirements, and a dilution of the inventor's equity.

It is a carefully crafted system that benefits the 1% entirely, and the remaining 99% are merely the crop they farm.

Don't worry about the naysayers. I am in the same position as you right now as we are doing work with AR and computer vision that people/investors etc... don't get yet. Maybe we are too early in the market, but probably not. If we are too early then at least we have something that we can fall back on without losing everything we have put into it.

At the end of the day I have to be able to say that I did everything I could to make our technology "win" in the marketplace.

If we are too early then at least we have something that we can fall back on without losing everything we have put into it.

No, see, what'll happen is your patent will be bought up during your exit event/firesale, and then used to harass other people in the space.

Especially in AR/VR/whatever, there's a huge incentive to patent things which are fairly obvious with just a little applied engineering time, and which will be used to harass a lot of people later.

If you wanted your technology to "win", you should make it open and free (in the software sense) to use--as it is, you've just got some tepid AR entry in the backwards swamp that is AEC.

Ha thanks for all the kind words. Actually our patents don't have to do with AEC and are actually pretty novel in the computer vision realm.

I actually ran the idea of putting our work for "free" by our current investors because I am a fan of FOSS but it's a non-starter when it comes to raising money and it's just a fact of life that we need investment to keep going.

At the end of the day, business is business. Good luck to you and your team!

> If you and people like you hadn't patented these things, they would've been able to use their work.

Someone else would have done it and you would be in the same position you are now.

I'm curious, did you put together an algorithm and\or program which actually did this automated actor replacement or did you just patent the idea? No judgement either way, "don't hate the player..." and all that :)

edit: looking at your bio it seems pretty likely that you had properly implemented it!

Fully implemented, but expensive to use. It's modern visual effects technology, automated. Also, not a single algorithm or program, but a media pipeline of over a dozen custom programs, plus integration with the major VFX software packages.

> It is owned by a major media company I'm not allowed to disclose.

Why would a patent owner want to remain unknown?

Come on guys. Like all patents, the owner must be disclosed to the USPTO. A quick Google search turns up Flixor as the original owner, and "Big Tent Entertainment, LLC" as the current owner of some of the recent patents in this family. A further search of their website indicates it licenses various media technology to companies including "Concept One, Nintendo, Fifth Sun, Parragon Publishing, Harry Abrams, Kids Station, Merch Source and more".

That's one reason for the question! Why would a patent owner want to prevent someone from divulging public information.

Another question being is the owner of record merely a front, and for what purpose?

Likely, this is a standard term in their inventor assignment contract. When bsenftner signed it, the patent applications were probably not public yet (there is an 18 month period between filing and publication in which the application remains "secret"), and during that period it makes sense to have a non-disclosure term.

As for the current owner being a front... if it is, it's not a very good one (real patent troll companies are much better at it, using shell corporations in various jurisdictions, each tied to a single or a small number of patent families).

Not true. The applicant must be disclosed during the application process, but the owner can be legally change without notifying the PTO or updating ownership records. Assignments can be registered, but you don't have to register assignments. So Big Tent could theoretically sell the patents to another party and never register the sale with the PTO.

For a patent troll it makes perfect sense. Rig the line and wait for a bite.

trolling

Noob question: What if the developer who "might" have infringed on the patent, open sources his work, and it's made available on github with multiple forks, and now there's a whole community behind it, and so on?

Certain specific processes may be patented, but "this type of thing" cannot be patented outright. Ideas are not patentable, processes are.

Reply on news.ycombinator.com