DIgital DYnaMic and respOnsible twinS for XR

The digital transformation and the availability of more diverse and cost-effective methods for 3D capture has led to the creation of digitised representations of parts of our public spaces, machinery and processes, commonly referred to as digital twins. In order to realise advanced virtual reality (VR) and augmented reality (AR) applications for cityscapes and industrial environments, continuously updated digital twins are a crucial requirement.

The vision of DIDYMOS-XR is to enable advanced, more realistic and more dynamic extended reality (XR) applications, enabled by artificial intelligence (AI). The project thus focuses on advancing the state of the art of enabling technologies that will allow creating large-scale digital twins, synchronised in real-time with the real world. DIDYMOS-XR will develop cost-efficient ways for continuously updating digital information, using numerous but heterogeneous sensors already deployed in cities and buildings, utilizing automated methods based on AI for fusing sensor information, analyzing scene semantics and make the appropriate changes to the digital twin. The developed application will be aware of scene dynamics in order to efficiently obtain reliable information for localization. The capture of scenes at scale, as well as using cameras and other sensor data for synchronising the digital representation, bears the risk of capturing personal and sensitive data and using them for unethical applications.Hence, the technologies resulting from DIDYMOS-XR must be and will be ethical and privacy-aware by design.

In order to demonstrate and validate the enabling technologies researched and developed in DIDYMOS-XR, the project will address use cases of XR applications in two domains that differ in their scale and characteristics.

• Cityscape: Create digital twins of cities and their infrastructure, based on existing digital information, captured with dedicated equipment and low-quality sensors available at scale (e.g., cameras in cars), and keep them up to date relying on deployed static and moving cameras, sensor data and data from relevant systems. This enables XR applications for urban planning, citizen involvement, smart mobility and tourism.
• Industry: Create cost-effective digital twins of existing production facilities and keep them synchronised with data from autonomous mobile robots (AMR) and humans. Collaborative mapping with a human in the loop will ensure high-fidelity digital twins, ensure worker safety in applications relying on digital twin data and collect data for further training of automatic reconstruction and mapping approaches.