Lipreading app helps voiceless patients communicate

Dr Shondipon Laha, a consultant in critical care at Lancashire Teaching Hospitals NHS Trust, often treats patients with difficulty speaking. 

"I was doing my ward rounds during the World Cup 2018 when my patient started mouthing something at me," says Dr Laha, describing one such patient. “He had undergone a tracheostomy and was unable to speak. After a while, I realised that he wanted me to move out of the way so he could watch England taking penalties!"

This incident was light-hearted, but being unable to speak can have serious consequences. Intensive care units (ICUs) frequently look after patients who have lost their voices. They may, for instance, have undergone throat surgery or suffered a stroke. Losing the ability to communicate can increase their anxiety and impede recovery, which is frustrating for everyone involved. Patients cannot speak to nurses, doctors, or visitors, who often resort to imperfect attempts at lipreading.

Dr Laha regularly encounters this problem, and he was keen to find a solution. He knew that lipreading could help patients, but it can take years to become a proficient lipreader. He believed that artificial intelligence (AI) might hold an answer, so he searched for companies working to develop the relevant software. He discovered Liopa, an AI firm in Northern Ireland spun out from Queens University Belfast (QUB).  Together, they developed SRAVI – a smartphone app that records and interprets patients’ lips as they speak.

Liopa has secured contracts for SRAVI with Lancashire Teaching Hospitals NHS Trust and is in talks with other healthcare providers across the UK and North America. In addition, Innovate NI has recognised Liopa's success with a Gold Level Innovator Award. 

They worked with Dr Laha and followed the four-step Innovation Framework to develop their app – here's how it worked for them.

Ideation

Dr Laha understood the anxiety that non-vocal patients experienced when trying to communicate. ICU nurses and doctors had long resorted to gestures and writing to help them communicate with their patients, but these solutions had practical drawbacks. He realised that lipreading was the key to unlocking this problem and was determined to find a tech solution that could help.

Selection

Dr Laha searched online for lipreading software and discovered Liopa, a firm born out of 15 years of AI, speech, and image processing research at Queens University Belfast (QUB). Their research had a particular focus on robust speech recognition in real-world environments.

Liopa worked with Dr Laha and the ICU team at Lancashire Teaching Hospitals NHS Trust and Professor Danny McAuley of QUB to research the feasibility of using AI-based lipreading in an intensive care environment. 

Funding from Innovate UK's Biomedical Catalyst programme enabled the team to capture videos of voiceless patients mouthing phrases that help hospital staff understand their needs. They used this data to test and refine Liopa's visual speech recognition. This project's success confirmed that Liopa's technology was a good fit for lipreading in a hospital setting. 

Development & testing

The project received further Innovate UK funding to develop a prototype mobile app called Speech Recognition App for the Voice Impaired (SRAVI). ICU staff could use SRAVI to video patients mouthing important phrases and receive a transcription.   

Liopa collaborated with Dr Laha and Lancashire Teaching Hospitals NHS Trust between 2019 and 2021 to test and finetune the SRAVI prototype, supported by Invest NI's R&D and go-to-market programmes. The result is the version of SRAVI available today – the first commercial solution for automated lipreading in healthcare settings. It has at least 95% accuracy when lipreading a pre-defined list of essential phrases such as "I'm in pain". 

Commercialisation

SRAVI has proven its worth in the context of the ICU, but it has the potential to help other voiceless patients communicate. After a successful service evaluation at Royal Preston Hospital, the Lancashire Teaching Hospitals NHS Trust decided to purchase the app's first commercial license. It is now available for all patients and staff throughout the hospital. Liopa is now in discussions with other NHS Trusts and hospitals in other regions, including North America. 

"I see Liopa's technology as fundamental to the recovery and wellbeing of critically ill patients," says Dr Laha. "SRAVI allows these patients to participate in their recovery."

If Dr Laha and Liopa’s story has inspired you to put your ideas into action, then get in touch to find out how Innovate NI can help.

Assessment

The Innovation Accreditation Programme is open to all businesses in Northern Ireland. So, take the first step today – complete the innovation assessment and discover where you are on the Innovation Framework.