A Wearable Sign-to-Speech Translation System

Signed languages are not as pervasive a conversational medium as spoken languages due to the history of institutional suppression of the former. We developed a wearable sign-to-speech translation system to effectively promote the communications between signers and non-signers.

Like Comment
Read the paper

Signed languages are created by the hands, face and body, and perceived primarily visually. Through the signed mode, the language is accessible at the optimal level that remains available through the visual sense. Signed languages are not as pervasive a conversational medium as spoken languages due to the history of institutional suppression of the former and the linguistic hegemony of the latter. Without prior knowledge of sign language, it is difficult for non-signers to receive and understand this conversational medium. This has led to a communication barrier between signers and non-signers that technology-mediated approaches could help with.

In our recent work published in Nature Electronics, highlighted by Science, we show that a wearable sign-to-speech translation system, assisted by machine learning, can accurately translate the hand gestures of American Sign Language into speech. The wearable sign-to-speech translation system consists of yarn based stretchable sensor arrays (YSSA) and a wireless printed circuit board (PCB). Due to its unique structural design and use of soft materials, the yarn based stretchable sensor array can conform to the skin of a human finger under both releasing and stretching states. Analogue triboelectrification and electrostatic induction-based signals generated by sign language components — including hand configurations and motions, and facial expressions-are converted to the digital domain by the wearable sign-to-speech translation system to implement sign-to-speech translation.

A wearable sign-to-speech translation system. a, Schematic illustration of a subject’s hand with the skin-attached YSSA and the wireless PCB. b, Schematic illustration to show the structural design of a yarn-based stretchable sensing unit and its working principle. Inset: enlarged view of the conductive yarn. c, Photographs of the sign language hand gestures according to American Sign Language and the corresponding generated voltage profiles as recognition patterns, which are capable of expressing letters, numbers, and short phrases. d, User-friendly interface for final speech expression, and the real-time electrical signals display as an intermediate state.

Our wearable sign-to-speech translation system offers good mechanical and chemical durability, high sensitivity, quick response times, and excellent stretchability. To illustrate the capabilities of the wearable sign-to-speech translation system, a total of 660 sign language hand gestures based on American Sign Language (ASL) were acquired and successfully analysed with the assistance of a machine learning algorithm. The system has a high recognition rate of 98.63% and a short recognition time of less than one second. We envision that the wearable sign-to-speech translation system could not only improve the effective communications between signers and non-signers, but also help the people who want to learn sign language.

This work was covered by over 200 media worldwide, including [abc] [CNN] [NBC] [UCLA] [NPR][ScienceDaily]. The current research of the Wearable Bioelectronics Group at UCLA focuses on nanotechnology and bioelectronics for energy, sensing, and therapy applications in the form of smart textiles, wearables, and body area networks. If you are desired to join force and work on these challenging scientific projects, please send an email to jun.chen@ucla.edu to express your interest.  

Jun Chen

Assistant Professor, University of California, Los Angeles

Dr. Jun Chen is currently an assistant professor in the Department of Bioengineering, University of California, Los Angeles. His current research focuses on nanotechnology and bioelectronics for energy, sensing, environment, and therapy applications in the form of smart textiles, wearables, and body area sensor networks. He has already published 2 books, 115 journal articles and 70 of them are as first/corresponding authors in Chemical Reviews, Nature Energy, Nature Electronics, Nature Sustainability, Nature Communications, Joule, Matter, and many others. His works were selected as research highlights by Nature and Science 6 times. He also filed 14 US patents and licensed 1. He is currently an Associate Editor of Biosensors and Bioelectronics, and an Editorial Board Member of Advanced Fiber Materials, Nano-Micro Letters, Frontiers in Pharmacology, Frontiers in Chemistry, and Smart Materials in Medicine. With a current h-index of 61, he was identified to be one of the world’s most influential researchers in the field of Materials Science by the Web of Science Group, and on the global list of The Highly Cited Researchers 2019.

No comments yet.