Smarter pens, augmented drawing, and more natural user interfaces at TechFest 2011

As always, Microsoft Research has some exciting work in development. Their belle of the ball for TechFest 2011 will be natural user interfaces, and they’ve taken the wraps off several projects at their blog, including 3-D rendering of medical images and objects, computer-assisted freehand drawing, and pens that can recognize different grips.

On the 3-D imaging side, there are two projects. InnerEye is a system to read medical images and interpret them in three dimensions. Coupled with a motion-sensitive input device, like XBox Kinect, it allows a physician to turn and view an organ at different angles with natural hand motions without having to twist a patient’s insides out. A separate project, MirageBlocks, deals with turning real-life objects into virtual ones using 3-D projector technology. Coupled with InnerEye, it may be possible to do such things as setting broken bones virtually before attempting the real thing, reducing the risk of improper positioning.

On the 2-D side, a project called ShadowDraw puts a different spin on computer-aided design (CAD). Instead of telling the computer what you want drawn, you simply begin drawing freehand and let the computer figure out what you’re trying to draw. It then assists by generating “shadows” in the background to help guide you. Since its recognition of a drawing is limited by the accuracy of the initial drawing, its ability to augment is based on an artist’s skill. The more skilled the artist, the better ShadowDraw works.

Finally, there’s an unnamed project designing a smarter stylus with sensors in the shaft that can recognize different grips. These grips can then cause the stylus to be recognized as different types of writing utensils, such pens, pencils, brushes, crayons, etc. It could even be applied to other input devices similar to Nintendo’s Wiimote to allow it to be recognized as different pieces of equipment, such as music instruments. For example, holding it as a flute could make it recognized as a flute.

This last project, of course, should be the most interesting to us. It certainly is to me. As I’ve advocated in the past, I use a variety of grips with my Tablet PC pen. I often switch up grips when writing and drawing, and use different grips for tapping, swiping, and other navigation tasks. I could adopt a pen like this right now. Just hope I don’t have to wait for pen input to come to XBox (or iPad) before seeing this hit the market. Oh, and if we could get those flashlight pens and 3-D rendering from Iron Man, that would be pretty cool too.