Our Biometrics and Profiling (B&P) Centre of Expertise (CoE) explores a new AI tech that uses micro-vibrations of facial muscles to uncover the emotions that people try to hide, to help augment mass event security.
The Profiling team working to test their new tech. (Photo: HTX)
It was a sweltering day when a team of engineers from the Biometrics and Profiling (B&P) team gathered outside a happening event. As attendees streamed into the venue, the team set up camp outside the entrance outside the hot sun. They could be seen busy hooking a computer up to a camera mounted atop a tripod. The question was, whatever for?
It turns out, the B&P team was trialling an AI technology that scans a person’s face to read their emotions. Basically – an emotion detector.
The full-size camera used for mass-screening operations. (Photo: HTX)
While lie detectors are common today, they use a fairly outmoded method of polygraph testing, where testers need to use instruments directly on subjects to detect typical stress responses such as increased heart rate, blood pressure, and respiration rate. This new AI technology, however, uses a different method to determine a person’s emotional state: micro-vibrations of one’s facial muscles. Not only can it do so from afar, but also in a matter of seconds.
By detecting a person’s emotions at an event, profilers on duty can quickly identify individuals who are in a state of agitation and anxiety through Tell-Tale Indicators (TTI). TTIs that aren’t so visible to the human eye but exhibit as micro-vibrations include fidgeting or shivering, suggesting that a person is nervous. These are emotions that suspicious people tend to exhibit before or after committing a crime, and can help profilers in their analysis when conducting security checks so that they can take decisive action.
That’s the reason why the B&P team were out in the wild – to use the event as an opportunity to do mass-screening for event security. While their indoor trials were already a success, they wanted to test the tech capability in an outdoor setting where there would be more disturbances captured in the background.
How the tech works:
- A high-resolution camera picks up moving targets in crowds and outdoor environments with a lot of image noise. Points on their neck and face are identified and mapped as pixels.
- The AI-trained algorithm detects minute pixel movements, or micro-vibrations, of their facial muscles when they exhibit behaviours such as fidgeting or shivering.
- The micro-vibrations are then translated into frequencies and classified into emotions based on a heat map, determining a person’s emotional state.
Points of a person’s face are picked up and converted into frequencies. States of emotion can then be detected up based on micro-vibrations as reflected on the heat map. (Photo: HTX)
Heading this project was Profiling Engineer Liang Kaiying, and it’s her first project as project lead at HTX. After her first posting at ICA as a Senior Officer, she joined HTX in July this year where her research in human physiology brought her to discover the technology of micro-vibration detection.
Inspired by the Marvel movies, particularly of Iron Man, Kaiying has always been interested in helping to discover and develop new science and tech capabilities. (Photo: Kaiying)
Kaiying explained that this system allows for better allocation of manpower resources for large-scale events: “Instead of having profilers on the ground to manually profile everyone, using the micro-vibrations detector allows profilers to prioritise which individuals to flag out. It’s fast, contactless, and accurate.”
With her understanding of the technical aspects of this existing technology, she worked closely with Deputy Director of Profiling, Kee Ein Cern, to acquire the tech from vendors and align it with the Home Team’s Concept-of-operations (Con-ops). Through research and conversations with various stakeholders, she determined the optimal location of the trial and what variables should be accounted for, such as the camera distance from moving targets, crowd movement, and ambient lighting conditions.
Assisting Kaiying in this project was co-project lead Profiling Engineer Shaun Wee, who handled the coordination of manpower on the ground and helped to ensure that the trial was carried out smoothly. Prior to this project, Shaun was already interested in the use of behavioral cues such as flushing of the face and pupil dilation for profiling but had not heard of micro-vibrations detection. This was also his first time being involved in a trial deployment at a live event, which made it exciting.
While initially interested in leveraging on machine-learning algorithms to make business decisions, Shaun realised over time that what he learnt could be more broadly applied across various domains like national security. That’s how Shaun ended up at HTX as an engineer, first joining the Associate Programme. (Photo: Shaun)
“I had never come across such a capability where AI can read someone’s hidden emotions. When Kaiying shared the tech with the team, it sounded like something straight out of science-fiction. I really wanted to see how it would work in real life,” he said.
Both Kaiying and Shaun were enthusiastic about testing the tech in an actual operation, but several challenges were posed to them leading up to the actual day.
“We encountered logistic limitations to set-up the system in an outdoor environment, such as the absence of power supply to operate the system,” Kaiying shared. “We also had to work around unknown factors, like where road closures and queue barriers would be, when trying to find a suitable operational area. This made it stressful and difficult to prepare for the trial.
“Thankfully, we managed to have open communication with the stakeholders to work around them,” she continued. “DD Ein Cern was also there to readily provide guidance. Eventually, the team got to recce the deployment site a day before the event and the trial was able to be carried out without disruptions.
Taking on the role as project lead was a valuable experience for Kaiying in managing a team.
“The team ‘suffered’ under the hot sun together. Despite the torturous heat, we supported each other so that we could finish the trial,” she reflected. “More personally, this project has taught me to be more adaptable to challenges and to keep an open mind.”
The Profiling team after the trial. (Photo: B&P)
Shaun supported Kaiying on ground to ensure that the trial ran smoothly. (Photo: HTX)
Based on the results of the trial, she was able to better understand the factors that would affect the sensitivity of the detector and how to troubleshoot issues that users might face when using it. “Right now, we’re hosting Home Team stakeholders at our Credibility Assessment lab to showcase the system in an indoor setting,” Kaiying shared. “Our goal is for them to trial the system in their indoor work premises and ultimately, implement this system permanently.”
Shaun also reflected on the trial, saying, “Ultimately, it was satisfying to see how the work we do helps improve operational efficiencies on the ground for frontline security officers. Knowing that your efforts make a tangible impact on national security makes the hard work feel worth it.
“Although, we will have to bring more sunblock the next time we go out on a field trial like this again,” he added with a laugh, “My skin was quite literally burning.”