Intuitive Touch-Based Operations

Apr 04, 2013– Kawasaki, Japan  (Techreleased) – Fujitsu Laboratories Limited today announced the development of next-generation user interface technology that enables intuitive, fingertip operations. This new technology can accurately detect where the user’s finger is and what it is touching, and uses off-the-shelf cameras and projectors. This eliminates the need for expensive equipment and overcomes imprecise tracking—issues posed by conventional technologies.

The new technology uses the cameras together with a projector to enable a user to trace a finger across a document on a table, copy it as digital data, and display it. This simple interaction with objects makes the technology an interface between people and ICT services, thereby expanding the ways in which ICT is employed.

Background

In recent years, considerable work has been completed in developing intuitive user interfaces with the aim of improving the usability of PCs and smartphones. Commercial applications have been developed for hand gesture-based screen control technologies and speech interaction technologies, and new user interfaces have begun to be employed as replacements for the mouse, keyboard and touchscreen. At the same time, to test the viability of enhanced user interfaces, actually touching and moving objects has been suggested as a potential area of development. Until now, however, this approach has required special sensors to be embedded into objects, which has served as a major roadblock in making the technology practical.

Technological Issues

If it were possible to perform contactless detection of touch operations on real world objects, this would obviate the need to embed objects with special sensors. Current commercial gesture technologies function predominantly on operations performed in the air without touching anything. When a user’s hands draw near to background objects, these technologies have difficulty differentiating between the hands and background. As a result, this approach has been unsuitable for detecting touch operations. In addition, technologies that use infrared and other special devices to measure distance have also begun to be employed in user interfaces. However, these technologies lack the resolution to detect subtle changes in finger movements, in addition to being large and costly.

Newly Developed Technology

Fujitsu Laboratories has developed technology that can accurately and rapidly detect finger operations on actual objects using an ordinary camera. The features of the new technology are as follows.

1) Recognizing and converting coordinates of physical objects and ICT equipment

Fujitsu has developed a technology that uses a camera to automatically measure irregularly shaped objects on a table and then automatically adjust the coordinate systems for the camera, projector and the actual objects. This makes it possible for the software to take the finger movements and touching of objects made by the user, and then match them with the digital display projected onto physical objects.

Intuitive Touch-Based OperationsFigure 1: Recognizing and transforming coordinates of physical objects with ICT equipment

2) Stabilization of finger recognition technology

This Fujitsu technology recognizes the shape of the user’s fingers by extracting the fingers’ colors and contour features. Other technology controls the color and lighting of the camera image, depending on the surrounding ambient light, along with technology that corrects for differences among individual fingers. A stable extraction of the form of the user’s fingers, one that is minimally impacted by the environment and individual variations, can be obtained.

Figure 2: Finger recognition stabilization technology

Figure 2: Finger recognition stabilization technology

3) Accurate and rapid fingertip recognition technology

By enhancing images of the user’s fingertips, the technology is able to obtain a level of precision that is sufficient for touch detection, even using low-resolution images that can be captured on a regular webcam. Moreover, the technology is capable of fingertip tracking speeds of 300 mm per second, thereby enabling it to follow natural finger movements.

Figure 3: Accurate and rapid fingertip recognition technology

Figure 3: Accurate and rapid fingertip recognition technology

Results

A variety of operations are possible using the newly developed technology, one being the ability to copy important parts of a document in digital format by simply tracing one’s finger across the physical document while it is placed on a table. Another is the ability to project copied data onto a tabletop and then expand and shrink the projected image by tracing with a finger. In addition, the technology can capture graphic data from handwritten sticky-notes that have been attached to a table, and enable the user to move around these newly digitalized sticky-notes with a finger over a physical surface. It would then be possible to group and rearrange these digitalized notes or perform other useful operations with them. This new technology makes it possible to easily have interactions with actual objects and serve as an interface between people and ICT services, thereby helping to expand the ways in which ICT is employed.