Bill Buxton: Multi-Touch Systems that I Have Known (1982) | Hacker News Reader

Bill Buxton: Multi-Touch Systems that I Have Known (1982)

billbuxton.com

115 pointsjohnx123-up23 comments

Iluminating, exhaustive article and illustrated history of multitouch. I wonder how many of these papers, prototypes and devices the patent examiners came across when they were deciding on the prospective multitouch patent now owned by Apple. Money quote:

----

Multi-touch technologies have a long history. To put it in perspective, my group at the University of Toronto was working on multi-touchin 1984 (Lee, Buxton & Smith, 1985), the same year that the first Macintosh computer was released, and we were not the first. Furthermore, during the development of the iPhone, Apple was very much aware of the history of multi-touch, dating at least back to 1982, and the use of the pinch gesture, dating back to 1983. This is clearly demonstrated by the bibliography of the PhD thesis of Wayne Westerman, co-founder of FingerWorks, a company that Apple acquired early in 2005, and now an Apple employee

    Westerman, Wayne (1999). Hand Tracking,Finger Identification, and Chordic Manipulation on a Multi-Touch Surface. U of Delaware PhD Dissertation:  http://www.ee.udel.edu/~westerma/main.pdf

In making this statement about their awareness of past work, I am not criticizing Westerman, the iPhone, or Apple. It is simply good practice and good scholarship to know the literature and do one's homework when embarking on a new product. What I am pointing out, however, is that "new" technologies - like multi-touch - do not grow out of a vacuum. While marketing tends to like the "great invention" story, real innovation rarely works that way. In short, the evolution of multi-touch is a text-book example of what I call "the long-nose of innovation."

----

Interesting examples relating to pinch-to-zoom:

[o] 1983: Video Place / Video Desk (Myron Krueger)

His use of many of the hand gestures that are now starting to emerge can be clearly seen in the following 1988 video, including using the pinch gesture to scale and translate objects: http://youtube.com/watch?v=dmmxVA5xhuo

[o] 1991: Digital Desk(Pierre Wellner, Rank Xerox EuroPARC, Cambridge)

Clearly demonstrated multi-touch concepts such as two finger scaling and translation of graphical objects, using either a pinching gesture or a finger from each hand, among other things.

This page makes it quite clear how improvements in this subfield of tech, like in many other fields, are evolutionary and build upon existing ideas in the literature. Many potential implementations have been proposed and experimented with. What role do patents play in this picture? What is the breadth of Apple's pinch-to-zoom patent - how far does it extend beyond smartphones? Are any of these devices close enough to the patent to legally count as prior art?

Thanks for this great submission. Buxton has been referenced on HN a few times before, but it's my first time reading his page.

In NONE of those cases did anybody use a pinch gesture to zoom anything.

So no. None of those legally count as prior art.

Look in the 1998 video (http://youtube.com/watch?v=dmmxVA5xhuo) around 4:30. You could perhaps argue that's 'scaling' not 'zooming', but that's a very pedantic position, and not one I personally would entertain.

Direct link to the pinch to zoom part.

http://www.youtube.com/watch?feature=player_detailpage&v...

So, tacking on a "BUT ON A CAPACITATIVE SCREEN" got Apple the patent?

Hardly. The screen on Buxton's multi-touch tablet in '85 was capacitive!

The patent system is so broken.

Devil's advocate: It is neigh impossible to judge from the video, but to me, the video shows less of a direct link between hand movement and what happens on the display. It is not as if the hand really grabs the screen. I also got the impression that he had to issue a command, chosen from the top of the screen (outside the video image) to switch from scaling to rotation. Another argument could be that this user does not actually seem to touch the screen (at least, that is my impression). It is likely that all these are due to technological limitations, but if I had to defend this as "not prior art" in court, I would try and work from those observations. There is little else to argue about, as a lot of the interactions are there, in a primitive (technologically) way.

Well, you'll see what you want to see, I guess - but 50 seconds into http://www.youtube.com/watch?v=dmmxVA5xhuo&feature=playe...

He makes a pinch gesture, and the circle on the CRT gets bigger and smaller. If that's not pinch to zoom, or the precursor to it, I don't know what is.

Which is the whole problem with Apple's case. They seem to like to pretend that they invented everything that's used on the iPhone, but this sort of research has been going on for decades.

update: I think the most striking thing about watching that pinch to zoom thing is that he's doing it idly while thinking/talking about something else - it's no big deal for him. I suspect that he's been using gestures like it for quite a while at that point.

You're right about one thing - you see what you want to see. I'd swear if asked that he's using his thumb and finger to draw a rectangle shape on the desk representing the keyboard he describes at the same time, but there's no denying the circle on the screen, even if just a shape bounding his touch points, rather than an object he selected and zoomed.

The later clip with the two fingered multitouch zoom of the person that recoiledsnake links to is more compelling, even if the gesture doesn't quite match that we know as pinch to zoom.

How different is "scaling" an image from "zooming" a browser page? Are they different at all? Because scaling a figure is exactly what Kruger demonstrates in the linked video at 4:30, as the other commenters have pointed out.

Furthermore, it looks like he may be using a touch surface in addition to IR/video cameras, because he can draw strokes on-screen with his finger while his whole silhouette is projected. (A capacitative touchscreen or not, I don't know. Maybe the cameras have good depth perception.) Let's not mince words, "pinching" is equivalent to moving two fingertips together, and "zooming" is scaling.

I also find striking the little crawling/flying sprite his silhouette plays with, starting at 3:34. We're seeing something that should be in a Kinect game, 20+ years before the Kinect. He speaks eloquently about how natural it is for humans to expect behaviours in the real world to translate to the virtual world, when they interact with images on a TV screen.

His exposition is full of so many casually demonstrated possibilities, each of which could probably be patented in our current environment. This guy (probably) didn't - leaving them open to free, public use - and we should be thankful for that.

In addition to the other examples, there's very clear one-handed pinch-to-scale at 6:30 in this video from 1991:

http://video.google.com/videoplay?docid=5772530828816089246

Slightly off-topic:

Does anybody know where the videos of Bill's excellent talk titled What if Leopold didn't have a piano can be found? I saw part of the lecture online several years ago and it was one of the most insightful things I've seen, The story of the sculpting instructor who did an A/B test on his students, grading half of them entirely on the cumulative weight of their assignments was particularly eye-opening.

I've googled many times since and never found the full multi-hour video again.

Why the (1982) in the title? The original article is from 2007, and this is a revision from 2012.

For the context:

  Furthermore, during the development of the iPhone, Apple was 
  very much aware of the history of multi-touch, dating at 
  least back to 1982, and the use of the pinch gesture, dating 
  back to 1983.

It strikes me as odd that this didn't last long on the front page or get upvoted nearly as much as all the samsung vs apple submissions that had a large amount of people defending apple in the comments. Yet this, this extremely valid and informational piece gets pushed down the list like nothing. This is why I generally don't post on these submissions anyways, it feels like they are all one sided in defense of apple all the time, nothing but samsung is a thief that copies apple, with no mention of what apple did to xerox and Steve Jobs openly stating that he did indeed take their idea because they weren't going to do anything with it.

It's a bit disheartening is all.

Thank goodness for companies like Apple that, when needing to rapidly introduce new technologies for users, acquire small businesses already invested in the space like Fingerworks for multi-touch, or Pixo for iPod's interface.

Seems much better for us entrepreneurs than Samsung's approach. In the build versus buy equation, I want big companies to calculate they're better off to license or acquire.

It's curious that Hacker News disagrees with the idea it's better for big businesses to buy small companies than to copy them.

I don't think that was the disagreement, I think it was more about the false comparison. I mean, do you honestly think that Samsung never acquires small businesses and Apple never copies?

I think HN understands that the expense and time involved in having to license every simple idea is a barrier to innovation. Licensing should be reserved for protecting the kind of things that require substantial investment -- implementation details, for example -- rather than simple ideas.

Let's not mix arguments here:

- It's a good thing that Apple acquires smaller companies for their technology/patents instead of building around them, sure. This is an economic calculation that Apple made that turned out well for those companies.

- It's not a good thing that any companies possess patents sufficiently broad to cover most of an idea, instead of a specific implementation.

- It's not a good thing that companies possess patents for which apparent prior art exists, or which are obvious, as seems to be shown by Buxton's page. (The patent office's definition of "obvious" is that the implementation can be created by combining two or more existing patents or pieces of prior art - which are demonstrated on the page.)

- It's not a good thing that companies possessing such patents sue other companies for infringing them, in the sense that it sets off a patent war and makes everyone more concerned about protecting themselves by trying to acquire patents on anything possible. I think this can lead to a chilling effect on competition and innovation in the software market.

I don't know what Samsung would have done if the technology it wanted was under patent by small acquirable companies, but that's a hypothetical, because the point of this page is that the patent in question should perhaps not have been granted in the first place.

If researchers were making multi-touch interfaces on touchscreens a decade or two earlier, and even displaying scaling functionality in their demos, then clearly an idea was 'plundered' from the public domain and privatised with the issuing of this patent.

Why isn't Bill Buxton suing Apple for a billion dollars?

Perhaps he has class.

True, but the schadenfreude in me secretly hopes he sues the bejeezus out of Apple and then donates all the money to people who need to defend themselves against Apple's patent trolling.

Apple, more than any other company, not only exhibits but flaunts the "long-nose of innovation" in everything they say or do. They need a healthy dose of reality.

I think the best illustrative art they could have used in the trial is to hand everyone a rubber band and asked them to make it larger using one hand. I bet you 99% of them would place the band between their index finger and thumb or thumb and all four fingers. They are the two most obvious solutions for any creature with an opposable thumb.

Reply on news.ycombinator.com