The HoloDeX allows Home Team officers to record immersive and interactive 3-dimensional digital twins of crime scenes and other incidents.
Sivakumar Ponapalam (Photo: HTX)
You have been called in to investigate a burglary. Upon arriving, you find a ransacked storage room. What’s the best way to preserve a visual record of the scene before you proceed to collect evidence? Traditionally, you would take multiple photographs of the scene. Sivakumar Ponapalam, a Principal Engineer with HTX’s Disruptive Technologies Office (DTO), has developed a more powerful option: the HoloDeX – an immersive and interactive 3-dimensional virtual “twin” of the scene which you can “walk through” and study at any time and place, even after the scene has been cleaned up in real life! This video shows how you can use the HoloDeX to investigate the burglary:
The HoloDeX originated from Sivakumar’s earlier research and development (R&D) on using smart wearables to give users the sense of touch in the virtual world of the Metaverse. This means that, if you are equipped with a smart haptic wearable, not only will you be able to hear and see a soccer ball hitting you in the Metaverse, you will also feel the impact! As the Metaverse already allows users to experience sights and sounds in its virtual world, the addition of the sense of touch would bring it a step forward towards a complete multisensory experience.
Sivakumar showcased this precursor of the HoloDeX – which he called the Internet of Senses – at HTX’s
TechXplore 4 exhibition in November 2022. How did this become the HoloDeX? He had a long-term vision. Inspired by
Star Trek’s Holodeck and Marvel’s
Iron Man, he wanted to “develop a Holodeck using Emerging Technologies such as Digital Twin, Extended Reality* (XR), Internet of Senses (IoS) and Generative Artificial Intelligence (AI).”
Unfortunately, it was not easy to turn the HoloDeX from vision into reality. In fact, Sivakumar shared that “The HoloDeX was identified as impossible by some of the industry partners.” But this did not deter him, “I knew it could be done based on the R&D work I had done for TechXplore, so I pursued it further.” He added that HTX’s management provided valuable support and guidance during this crucial time.
Several months later, he had built a miniature proof-of-concept recreating one of HTX’s pantries in XR. This recreation was not static – you could pick up and manipulate objects in this XR pantry, just as you could in the real-life pantry. In October 2023, Sivakumar showcased this miniature proof-of-concept to fellow Xponents at a Hatch Spotlight Event and then to visiting Home Team officers at HTX’s inaugural
Learning & Innovation Festival. Validated by the success of his miniature proof-of-concept, Sivakumar went on to build the more ambitious proof-of-concept based on the burglary investigation.
How to build a HoloDeX
How do you build a HoloDeX? Sivakumar explained that you will need to create a Digital Twin of the object and load this Digital Twin into an XR environment. There are several ways you can create a Digital Twin, which is a photorealistic 3-dimensional digital counterpart of the real-life object.
With the HoloDeX, you can explore and interact with the scene using your voice and gestures. (Photo: HTX)
If you are good at math, you can use the mathematical techniques of photogrammetry to convert the 2-dimensional photographs of the object into a 3-dimensional virtual object. (Thankfully, there are software packages like NeRFStudio, RealityCapture and Unity 3D to do this for you!)
Once you have the 3-dimensional virtual object, you can load it into a XR headset to experience and interact with the Digital Twin. The HoloDeX proof-of-concept currently uses the Microsoft HoloLens 2 as its XR interface, but the system can also be used with other XR headsets.
Sivakumar using Generative AI to analyze the crime scene. (Photo: HTX)
As shown in the video, the HoloDeX was able to recognize Sivakumar’s spoken commands and create written notes based on his spoken words. He also had the option of allowing Generative AI to study the crime scene and record its findings. Not only does the HoloDeX allow investigators to revisit Digital Twins of crime scenes at any time and place, its powerful voice recognition and Generative AI capabilities will allow them to rapidly create notes based on their fresh analyses of the crime scenes.
While the HoloDeX is currently still a nascent technology, it already has several use cases for the Home Team. Aside from preserving crime scenes, accident scenes, or fire scenes, the HoloDeX can also send first responders who are equipped with smart wearables crucial real-time data and insights about the incidents they are responding to. The HoloDeX can also be used for training. With immersive simulations of post-incident fire investigations, crime scene reconstructions, and tactical operations, the HoloDex gives Home Team officers the option of realistic and risk-free XR training.
In April 2024, the HoloDeX was showcased to international attendees at the
Milipol Asia-Pacific – TechX Summit where Sivakumar received good feedback and even received enquiries from foreign homeland security agencies about the innovative technology! What’s next? He is currently exploring a deeper integration with AI which will give the Digital Twin advanced cognitive capabilities such as predictive analytics and adaptive decision-making.
* Extended reality (XR) is the term for technology that alters reality by adding digital elements to the physical or real-world environment. XR includes but is not limited to Augmented Reality (AR), Mixed Reality (MR) and Virtual Reality (VR).