Creating a Digital Twin VR-Environment

DUKE is a collaboration project between Lapland University of Applied Sciences (Lapland UAS) and Rovaniemi Municipal Federation of Education (REDU). The idea of DUKE is to examine how VR solutions can be implemented in educational purposes while leveraging renewable energy. Our main pilot is a digital twin of an educational district heating plant located in Rovaniemi and our secondary pilot is a digital twin of a customer grade heat pump. Digital twins allow us to visualize the environment with great immersion, this enables us to implement accurate functionality to various devices and systems. High immersion helps users to subconsciously connect the virtual object to the real-life object, thus seeing a virtual lever would have the same relationship as seeing a real lever; it should be interactable.

DUKE’s customer grade heat pump was made in cooperation with a project partner. Our partner wanted to incorporate VR solutions alongside their user manual. This way, the user can visualize more accurately basic procedures and maintenance work for the heat pump. Our DUKE engineers tackled this project by working closely with the heat pump specialists. The specialists taught us how to do maintenance on the heat pump and we, in turn, translated that knowledge into a VR simulation. The development process fundamentally revolved around an iterative workflow, where we had assessment and review meetings back and forth. In doing so, we ensure that the 3D-recreation was accurate and at a satisfactory quality.

DUKE’s official logo

Creating the digital twin VR-environment

A digital twin is a 3D virtual recreation of an object or a real-world environment. The definition ‘digital twin’ can be perceived as an instruction regarding how closely or realistically the virtual recreation represents its real-world counterpart. Here in Frostbit Software Lab, we aim to create our digital twins to the extent where the 3D-recreation is highly accurate and unmistakable when comparing the 3D-version to its reference.

Side-by-side image of the rendered 3D-scene and the raw 3D-scene.

The workflow and subsequently the challenges we face depends largely on the use case scenario. The VR simulation hosts a physics simulation system in addition to the high-fidelity 3D-visual representation of the environment. For this reason, we chose to use Unity for its lightweight and adaptable nature to implement our real time VR simulation. Unity allows us to easily manipulate and optimize the VR simulation to work with customer grade computers and high-end computers for a higher fidelity experience.

The developer’s viewport inside Unity game-engine. The screenshot is taken from the work-in-progress district heating plant.

We kicked off the visual implementation process by taking reference images on-site. The images are captured from multiple angles in order to get an understanding of the object’s shape and contour. Moreover, it is exceedingly useful in figuring out the object’s material properties, such as how shiny it is, what type of metal it is and what kind of reflections it produces.

A reference image of the district heating plant.

The 3D-modeling process uses blueprints in tandem with the reference images to get accurate measurements for a high-detail recreation. A keen observer may still find some discrepancies when comparing the screenshot image to the reference image. This is due to readjustments, some parts of the district heating plant were deemed obsolete for the VR simulation and therefore had to be cut or modified.

The texturing process uses the multi-angle images to figure out what type of material would be the most accurate to its real-world version. We use HDRi-images to create realistic lighting to the 3D-scene. HDRi’s are essentially 360-degree panoramic images that contain a vast amount of lighting data. Ultimately, we are able to create believable environments by following real-world lighting conditions while incorporating physical based rendering techniques.

A screenshot of the disctrict heating plant digital twin.

VR-Implementation in a game-engine

The first step our developers need to make is to choose an ecosystem, in other words, which development pipeline to use. Locking into a development pipeline is essential in guaranteeing efficient feature implementation down the line. The ecosystem in this context consists of choosing which manufacturers VR-gear to use, what game-engine and development toolkit to use. Our DUKE engineers ended up on using Unity game-engine for its lightweight and adaptable nature.

A test-player trying out the playable demo of the DUKE VR-simulation

Our DUKE engineers chose to use SteamVR software development kit for its wide VR-gear support. This SDK provides accessible script blueprints, that we can tailor for our own usage. For example, the SDK provides a basic functionality for object interaction and player movement. This way, more time can be allocated to polish more advanced features such as the dynamic interaction between different objects while a mathematic simulation is running in the background.

The mathematic simulation is built with modularity in mind. Each part of the simulation is broken down to individual pieces. This grants us the customizability to assign the simulation scripts to the 3D-objects that would logically handle the same physics in the real world. For example, the scripts that handle pressure loss are assigned to the 3D-objects that look like valves or pipes. One of the challenges we faced when working with a VR solution was that real-life dimensions did not translate well in a VR environment. EU-standard sized doors seemed too narrow, standardized knobs, bolts and valves seemed too small and ultimately play-testers reported having a difficult time maneuvering in the VR environment overall. A silver lining needed to be struck in order to satisfy the digital twin -nature of the project and the ease-of-use side for the players.

The current DUKE heat pump build contains four scenarios: “Tutorial”, “Water filter”, “Pressure adjustment” and “Water leakage”. Each scenario provides information on how to operate the customer grade heat pump and how to perform basic maintenance.

Tutorial -scenario

Tutorial scenario explains thoroughly all the functions available in the project. This scenario also includes information covering the main parts of the heating system. This is usually the first scenario the player should launch, since it also teaches the player how to use the VR-gear.

Water filter -scenario

The user can learn how to clean the water filter using this scenario.

Pressure adjustment -scenario

The user can learn how to adjust the heat pump to a correct pressure level in the system and how to use the safety valve to release the pressure from the system.

Water leakage -scenario

Water leakage -scenario provides a good guideline on how to store the heat-pump system. We show the most common issues that can be encountered in such a space.

The total budget of the project is 761 732 euros, of which Regional Council of Lapland has granted European Regional Development Fund (ERDF) and state funding 609 385 euros. This project will be executed in the timeframe of 1.1.2020-31.12.2022.

This article has been evaluated by the FrostBit’s publishing committee, which includes Heikki Konttaniemi, Toni Westerlund, Jarkko Piippo, Tuomas Valtanen, Pertti Rauhala and Tuuli Nivala. The article publications can be found on the FrostBit publication blog.


20/10/2021

Written by
Onni Li, Bachelor of Engineering (Lapland UAS), Project Engineer, FrostBit, Lapland UAS
Severi Kangas, Bachelor of Engineering (Lapland UAS), Project Engineer, FrostBit, Lapland UAS