Touch Screens and Touch Surfaces are Enriched by Haptic Force-Feedback

Tactile feedback can enable more effective use of touch screens, particularly in automotive applications where driver distraction is a problem. The number of technologies used to produce haptic effects continues to increase, providing many options and opportunities for system designers.

by Bruce Banter

THE TERM HAPTICS comes from the Greek word haptikos (from haptesthai, to grasp or touch). For touch user interfaces, the term typically refers to a tactile sensation or force-feedback that users experience when touching the surface. Conventional touch screens do not provide the tactile sensation of mechanical buttons and knobs, resulting in a less satisfying experience. Haptics technology can reproduce the same feel or tactile sensation as mechanical buttons and knobs or generate new sensations not previously possible. Tactile sensation can be more emotional and personal than sight or sound and can therefore enrich the user experience and perception of the interaction.

Haptics have been utilized in automotive applications for some time in rotary controls with joystick types of motion. Conventional rotary knobs are limited with fixed detents ("bumps") and degrees of rotation (end-stops). BMW's iDrive, launched in 2001, was the first haptic interface that provided different feelings (feedback) for different functions with one control. The device operated with rotary and four-axis joystick motions to control the functions on the display. The concept has evolved into the COMAND controller in the Mercedes S-Class, the Multi Media Inter-face (MMI) in most Audi models, and the Remote Touch controller in several Lexus models. Some automotive OEMs believe that these types of rotary/joystick inputs are similar to computer mouse controls and provide more intuitive and comfortable operation than touch screens.

Haptics was presented as a key enabler for touch user interfaces in the article "Tactile-Feedback Solutions for an Enhanced User Experience" in the October 2009 issue of Information Display.¹ This article expands on the information in that piece and presents some new information on additional haptic technologies and the use of haptic force-feedback in automotive touch-screen applications.

Touch Screens and Touch Surfaces

Touch screens and surfaces are increasingly replacing the conventional mechanical buttons and knobs in automotive controls because of their ability to provide a reconfigurable user interface (UI) that blends with the vehicle's theme or styling while improving cost and reliability. Automotive UIs have been changing to meet customer expectations for better interaction. The proliferation of portable devices in vehicles is also driving the need to reduce driver distraction and keep attention on the road. Touch screens are enabling the customization and adaptability of the UI for improved presentation of information and user inputs. The main disadvantages of touch screens have been fingerprints on the screen and the lack of tactile feedback.

Haptics are being utilized to provide unique information to automotive users. Sliding motion inputs such as radio volume or fan speed can be enhanced by increasing the rate and intensity of feedback as the finger moves across the surface to correspond to the loudness of the radio or speed of the fan. Users are able to learn and identify features coded with unique haptic effects. The muscle memory of haptic effects can easily be recognized and quickly understood even without confirmation of sight and sound feedback.

The type and size of information displays impact the need for touch screens and haptics. Automotive OEMs initially began using smaller LCDs and OLED displays for cluster, radio and climate controls without touch interaction. Over time, the "center-stack" area has evolved into a major control, navigation, and communication hub with display sizes continuing to increase, providing increased room for user interaction. Eight-inch LCDs are becoming common in automobiles today. Adaptive and reconfigurable UIs are required to provide an increasing amount of information to the driver and passengers. Multiple screens are being utilized to present different types of information. Touch screens, proximity sensing, and haptics are key parts of providing unique experiences for each screen.

The trend of vehicle-interior styling is toward organic and flowing lines. The center-stack area is being blended into the instrument panel and the center console, creating complex surfaces that leave little room for flat glass panels. Molded surfaces are the trend with decorative finishes. Projected-capacitive touch screens (one of the few touch technologies that can be applied to a curved surface) are being developed for several applications to match the trend. Applying touch, haptics, and motion to a complex surface is challenging, and haptic design considerations must be incorporated during the initial stages of the vehicle-interior design.

Adding Haptics

Optimal UI experiences employ the senses of sight, sound, and touch. The move from mechanical buttons and knobs to touch screens and surfaces can result in the loss of tactile feedback. Users have been programmed to expect tactile confirmation of inputs with mechanical switches. Studies have shown increased input speed, increased accuracy, and less frustration when haptic feedback is part of the UI.² Some current automotive applications of touch screens and surfaces utilize only sight and sound because they are the easiest to implement. Several user clinics and studies have been conducted by auto-motive OEMs to understand user preferences, and they indicate that users prefer the combination of tactile force-feedback and sound.³

The type of haptic effect can vary in complexity from simple vibrations to multifaceted effects driven by complex mathematical models (see Fig. 1). Simple rumble vibrations have been used in mobile phones for several years, but they are evolving to more intricate effects. High-fidelity force-feedback has been shown to produce a more authentic response and engaging user experience. The new gesture-rich user interfaces that are beginning to appear in consumer products will undoubtedly require high-fidelity feedback.

Producing Haptic Effects

Most haptic effects are produced by stimulating the nerve receptors in the finger by motion of the touch surface or vibration of the skin. Movement is generated by pushing or pulling the surface with either a prime mover or magnetic or electrostatic attractive forces. Control of the movement is accomplished by varying the amplitude, frequency and duration of the driving current or voltage. The seven methods currently being utilized to produce haptic effects are shown in Table 1.

The following paragraphs discuss each of these seven methods. (See the aforementioned October article for additional details on inertial, piezo, and surface actuation).

Inertial Actuation

Eccentric rotating mass (ERM) inertial actuators have been used as vibrators in mobile-phone applications for many years and they are starting to be used to produce haptic touch screens. Immersion Corp. has long been a leader in this technology, offering its TS2000 software development kit (SDK) and associated design support.

Piezo Actuation

Piezo actuation is generated by piezo-ceramic elements that deform with applied voltage. The piezo elements are applied to the touch surface, and haptic effects are created by the flexing motion of the elements against another surface. SMK is a leading supplier of resistive touch screens; it has been including piezo force-feedback as an option in its resistive touch screens for a number of years. SMK's first automotive haptic touch screen was introduced with "PulseTouch" in 2004 in Alpine's IVA-D300 aftermarket audio head unit. Development has continued with several auto-motive OEM applications currently in progress.

Fig. 1: The spectrum of haptic feedback ranges from simple vibrations to complex multifaceted events, with the latter being perceived as a more authentic representation of the original tactile feedback it is designed to replicate. Source: Immersion Corp.

SMK's piezo-ceramic actuators are a proprietary material that can be applied in a thickness of 1 mm to resistive film–glass or glass–glass touch screens. Accelerations of 2G can be created with high bandwidth to accomplish high-fidelity effects. SMK has worked with Immersion to provide 25 haptic effects that can be embedded in applications. Their latest generation of products includes a gesture-recognition touch screen with two-finger multi-touch that changes the haptic feedback according to the pressure applied. The change in resistance triggers the change in the haptic effect. Projected-capacitive touch screens that incorporate SMK's piezo actuation are also under development.

Surface Actuation

Surface actuation, developed by Pacinian Corp., is a technology that shows great promise. The October Information Display article explains the operating principle in more detail, but the basic premise is electro-attractive force between two surfaces with a charge differential. System response time is fast from 0 to 500 Hz. The actuation mechanism can be incorporated into the touch-screen components without the need for a separate actuator, which results in a very thin profile. High reliability has been demonstrated with more than 200 million actuations achieved during testing. Pacinian offers hardware and software development support.

Lateral Actuation

Lateral actuation is accomplished by electro-mechanical actuators that move the touch surface in a lateral direction to produce the haptic effect. The small lateral motion (0.2–0.3 mm) stretches the skin of the finger and the tactile receptors. Immersion has developed this technology and provides implementation support with mechanical, hardware, and software design. It has been very active in characterizing and replicating various tactile feelings into haptic effects. An SDK is available to implement an embedded control design and customize haptic effects.

The motion of the touch surface is controlled by the actuator flex frame or by a separate sliding or flexure mechanism with the actuators providing the only force. Multiple actuators can be used to move more mass. Haptic touch screens of 30 in. have been successfully developed. Accelerations of several Gs can be produced to generate high-fidelity haptic effects. Immersion's A110 actuators (Fig. 2) have been tested to auto-motive-grade requirements and have completed 1 million test cycles.⁴ This technology has become the benchmark for haptic touch screens, particularly with regard to the fidelity and strength of effects and the maturity of support and capability data.

Electro-Active Polymer Actuation

Electro-active polymer actuation (EPAM) creates motion by applying a charge to electrodes separated by a dielectric polymer film to create an attraction force that causes the polymer film to contract in thickness and expand in area. The motion is directed in the desired axes by attaching frames and materials. This technology, developed and licensed by Artificial Muscle, is available in several types of actuators; the company's Z-Mode and Reflex HIC actuators are used for touch-screen applications. The Reflex actuator's output is claimed to be directly proportional to the input signal, providing fast response time and the ability to reproduce almost any waveform.

Bending-Wave Actuation

UK-based NXT has adapted its bending-wave technology that was originally developed for loudspeakers to produce both touch-point location and haptic effects (but not simultaneously) in touch screens.⁵ In the former case, NXT licenses its bending-wave technology to 3M Touch Systems for its Dispersive Signal Technology (DST) touch screens; these use piezoelectric sensors at each corner of the screen to convert the mechanical energy from the bending waves produced by a touch into electric signals that are used to calculate the location of the touch point. In the latter case, the same bending-wave technology is licensed to Nissha Printing for use in high-fidelity haptic touch screens. A wide variation of tactile sensations can be produced as a result of the wide bandwidth provided by the system; it can even be extended into the audio range to allow the touch screen to also provide audio feedback. Most haptic effects are produced below 500 Hz; localized feedback is possible with different effects in different locations.

Fig. 2: Immersion's A110 Touch Sense Actuators (upper left), manufactured by Johnson Electric, provide lateral movement of touch surfaces. Lateral (side-to-side) movement of the touch surface stretches the skin and stimulates the nerve receptors to produce tactile sensations (lower left). An exploded diagram of a typical automotive LCD assembly (right) illustrates the combination of a lateral haptic feedback mechanism behind the LCD, a resistive touch-screen on top of the LCD, and capacitive-sensing backlit buttons surrounding the LCD. Source: Immersion Corp.

Capacitive Electrosensory Interface

Capacitive Electrosensory Interface (CEI) or E-Sense is yet another type of haptic technology that was developed by Sweden-based Senseg.⁶ Instead of moving the touch surface, this technology generates electrostatic pressure in the skin of the finger by establishing a charge differential that creates a Coulomb force between the E-sense layer integrated into the touch surface and the finger tissues. The force is modulated in frequencies where the human vibration perception is most sensitive; the oscillating force causes the skin to vibrate and the nerve endings interpret this as touch sensation. Electric-field strengths are below the insulation breakdown so there is no electric arc between the finger and the touch screen. The finger does not have to actually be touching the surface to generate the force. Different areas of the touch screen or surface can be controlled individually to generate small tactile areas that Senseg calls "tixels." The sensations are best felt with light touches or swiping gestures. Virtual surface textures can be created on the touch surface in areas being touched. Senseg provides design and component support for integration of the technology.

Visteon: Contributing to Haptics User Research

Visteon Corp. is a major supplier of auto-motive UI controls (as shown by the example in Fig. 3) and has been a leader in developing touch controls with haptic feedback. The company has developed several touch-screen products that include haptic feedback to identify and solve challenges in implementing the technology and gain user insight. Dr. Michael Tschirhart has been leading the company's user research efforts to gain richer insight into the consumer's perception of haptic technology. In an interview with the author, Dr. Tschirhart said, "This research helps Visteon discover factors that determine the attributes that individuals perceive when using haptics. These findings [many of which are as yet undisclosed] are used to guide the development of haptic solutions that can be tailored to meet the desired objectives of car makers."

When Will Haptics Take Off?

Haptics has gained acceptance in mobile phones but has been slow to take off in touch screens for other applications. Many companies are evaluating haptic implementations and weighing the risk and cost of adding the feature. What are the costs and design challenges to add the additional hardware and software and will consumers see value in the feature? Adding motion to most assemblies requires new or significant re-design. Will some combination of light and sound feedback be sufficient to satisfy consumers? The research seems to indicate that consumers prefer tactile feedback once they are aware of the possibility. How reliable are the technologies? The supporting reliability data is growing, but many of the technologies are very new. Mechanical buttons and knobs have functioned well and are well-understood from a cost point of view, so why replace them? The iPhone triggered the migration to touch screens and away from the fixed UI of mechanical buttons. OEMs are waiting or proceeding cautiously to see if the costs and risks of implementing haptic technologies can be justified.

Automotive products have much longer development cycles than consumer electronics and industrial controls, but automotive OEMs have awakened to the benefits of integrating the latest consumer-electronics technology into their vehicles. They understand that consumers want to interact with their vehicles the same way they do with their electronic devices. Several OEMs are studying haptic touch screens for user interactions and preference. The first major applications will be in 2012-model vehicles.

Fig. 3: A haptic touch screen shown by Visteon at the Consumer Electronics Show in January 2010 shows an automotive "infotainment" panel demonstrating the implementation of an 8-in. multifunction touch screen as part of an integrated control panel housed within the organic and flowing lines of a modern center-stack. Source: Visteon.

Conclusions

Time will tell which haptic technologies provide the best value and the most consumer acceptance. Each technology has advantages and challenges but the application will dictate the best fit. Haptics should grow in acceptance as consumers begin to experience the advantages. The main key will be to use haptics not just to replace the feel of mechanical buttons but to create new holistic and engaging user interfaces. Developers will be able to move in creative directions that are not possible with touch screens alone or mechanical buttons. Implementation costs will decline as volume provides economy of scale and new, more cost-effective methods and components are developed. Haptics is truly an enabler of touch screens and the futures of both these technologies are intertwined.

Acknowledgments

The author would like to thank the following individuals and companies for providing information for this article. Additional information on the technologies and companies is available through them.

• Immersion Corp.: Mike Blicher, Director Business Development, mblicher@immersion.com

• Pacinian Corp: Mike Levin, V.P. Business Development, mlevin@pacinian.com

• SMK: Jeffrey Riggs, JeffreyRiggs@RiggsProductGroup.com

• Visteon Corp.: Dr. Michael Tschirhart, Technical Fellow & Advanced HMI Manager, mtschirh@visteon.com

• NXT: Geoff Boyd, New Business Development Director, g.boyd@nxtsound.com

• Senseg, Ltd.: Ville Makinen, CEO, ville.makinen@senseg.com

References

¹M. Levin and A. Woo, "Tactile-Feedback Solutions for an Enhanced User Experience," Information Display 18-21 (October 2009).

²M. Silfverberg, "Using Mobile Keypads with Limited Visual Feedback: Implications to Handheld and Wearable Devices," Mobile HCI 2003, ed. L. Chittaro, LNCS 2795: 76-90 (2003).

³J. L. Burke, M. S. Prewett, and L. R. Elliot, "Comparing the Effects of Visual-Audio and Tactile Feedback on User Performance: A Meta-Analysis," Proceedings of the 8th International Conference on Multimodal Interfaces, 108-117 (2006).

⁴Immersion Corp., 2009

⁵N. Harris, "Applications of Bending Wave Technology in Human Interface Devices," presented at the 126th Convention of the Audio Engineering Society (2009).

⁶J. Linjam and V. Makinen, "E-Sense Screen: Novel Haptic Display with Capacitive Electrosensory Interface," presented at HAID 09, 4th Workshop for Haptic and Audio Interaction Design, Dresden Germany (Sept. 10–11, 2009). •