Detecting berries with Deep Learning

BerryMachine is a project funded by Interreg Nord, the partners being Lapland University of Applied Sciences, Natural Resources Institute Finland (LUKE) and NIBIO (Norway). The purpose of the project is to research the possibilities of artificial intelligence in detecting the amount and quality of the current berry harvest, based on a photo taken by a smartphone camera. The artificial intelligence system prototype implementation will be developed by FrostBit Software Lab.

Traditionally berry harvest sighting data is produced manually by calculating berries within one square meter forest area at a time. These sightings are later used by certain calculations to determine the overall berry harvest situation. This method is considered to take excessive amounts of time, especially during the berry counting phase. Therefore, using artificial intelligence as a tool to speed up berry counting is a potential resource saving opportunity, which would also enable others than experts to count berries easily, which would improve overall efficiency.

The phases to implement the artificial intelligence system during the project include planning and executing the acquisition of the berry photo dataset, preprocessing the berry photo dataset as well as studying, testing and implementing suitable image recognition and object detection technologies into a one complete system based on the project dataset. Creating an artificial intelligence system is a naturally iterative process, which means, earlier phases have to be re-visited if the project requires it, excluding the acquisition of the project’s photo dataset.

Preliminary technological study, Case BerryMachine

The modern artificial intelligence is an extremely wide concept, which can be divided into three different main categories, from software architecture’s point of view. These categories are traditional machine learning, deep learning and reinforced learning (Krishnan 2019). Since the BerryMachine –project focuses on image and object detection, the most natural approach is to use deep learning technologies in the project.

Figure 1. A neural network, illustration.

By using deep learning, the artificial intelligence learns important features that a photo consists of independently. These features can be, for example, the shape or color of the recognized object, or even the typical background where the recognized object is usually attached to (for example, a car tire is typically attached to a car). The core concept of image and object detection is a neural network, which is practically a complex and a multi-layer information processing method, which is greatly similar to the way how humans learn new things. Basically a single layer of a neural network analyzes one feature of a photo at a time. For example, one layer could focus on the object’s shape, while some other layer could analyze the object’s color (Bonner 2019; Patel 2020; Géron 2019, 448). There are multiple types of neural networks (CNN, ANN, RNN). However, in this article, we will always refer to the CNN type of neural networks, which is also known as the convolutional neural network (Pai 2020).

The theory of neural networks goes back as far as the year 1943 (McCulloch & Pitts 1943), but it hasn’t been until the last couple of decades for the modern processing power to begin being powerful enough for the needs of applying neural networks in practical applications in the field of information technology.  

When it comes to image recognition, two main alternatives exist, the first being image classification and the second being object detection (Browniee 2021; Sharma 2019). Multiple approaches exist for both alternatives, and none of them is naturally better for all purposes, since it always depends on the situation. The most suitable approach and alternative that fit best the given purpose and dataset is typically found by boldly experimenting on different alternatives. (Leo 2020; Dwivedi 2020.) This is also the reason why developing image recognition systems take a considerable amount of development time. Because of this, any artificial intelligence project should have a flexible project structure to ensure the best possible outcome with the given resources.

Image classification and object detection are often used for similar purposes, however, they have different outcomes and implementations. From the outcome point of view, image classification aims to recognize what the photo depicts as a whole, while object detection aims to find every instance of the desired objects and their locations within the photo.  In other words, image classification recognizes that a photo contains flowers, while object detection also points out where all the recognized flowers are located within the photo.

Figure 2. The difference between image classification and object detection.

Common image classification technologies, neural network models and algorithms include VGG16/19, ResNet, MobileNet, Inception and Exception, just to name a few. Other viable alternatives exist also. Common object detection technologies include EfficientDet, YOLO and R-CNN. Which machine learning technology should be used in a given situation depends often on practicality related to software architecture, the compromise between detection accuracy and efficiency as well as the suitability of the technology for the project’s needs.

The practicality related to software architecture focuses on the compatibilities between technologies that only function correctly on certain software platforms and versions, which can make the software development process extremely slow and unnecessary difficult, if the compatibility issues are too frequent. The compromise between detection accuracy and efficiency refers to the process of selecting a recognition technology for the project that has the best possible accuracy detection, but it’s also light enough to be realistically used in the systems used by the project. For example, the most accurate and heaviest technologies might not be viable to be used in mobile devices, if the calculations required by the artificial intelligence is done within the mobile device itself.  In the BerryMachine –project, all heavy computations are calculated on servers, which allows us to use the heavier technologies if needed. The suitability for the project’s needs refers to how well the chosen technology performs with the project’s dataset in reality. For example, some certain pre-trained image recognition model (VGG19, ResNet etc.) could perform better in recognizing vehicles in photos than some other model. (Github 2021; Hofesmann 2021; Lendave 2021; Özgenel & Sorguç 2018.)

Producing the berry photo dataset and preprocessing

The technological development of the BerryMachine –project started in spring 2021 by designing the acquisition phase of the berry photo dataset. When it comes to image recognition, the photo dataset has to be considerably large. One classic definition states that each recognizable category should have at least 1000 different photos within the dataset. On the other hand, some others imply that it can be less in some situations. The photo dataset should also be versatile, so that the artificial intelligence learns to detect the desired objects against different backgrounds as well. (Warden 2017; Huellmann 2021.) We decided to acquire a photo dataset as large and versatile as possible, so that we would surely have enough material for the project’s needs.

Figure 3. Some photos within the BerryMachine berry photo dataset.


More than 10000 berry photos were collected during summer and autumn 2021 in the BerryMachine –project. A single observation square in this case was an aluminium frame sized one square meter, from within the berries were calculated. To process the photo dataset into the format understood by the artificial intelligence, the dataset had to be annotated first. Annotation is the process of marking down recognizable objects into each photo within the photo dataset in such a way, that the artificial intelligence can train itself to recognize the desired objects from new photos. In this case, the desired objects are raw and ripe berries as well as their flowers. The actual berry types found within the berry photo dataset are bilberry and lingonberry.

A software solution called Label Studio was used to annotate the berry photo dataset. Label Studio supports multi-user annotation process on same dataset simultaneously and also provides various different annotation data formats to be used with artificial intelligence training. Label Studio also supports using a previous self-trained machine vision model to help annotating photos automatically. In this case, the artificial intelligence finds the potential berries in the photo, and the user has to verify and fix the results, if needed.

Figure 4. The annotation view of Label Studio and accepting the results of automated annotations.

If using traditional image classification, annotation is not needed, because the photo dataset can be processed by the artificial intelligence based on separate folders. In this case, each folder represents a single recognizable category in the image classification dataset. Since we focused on object detection in the first version of the BerryMachine artificial intelligence system, we concentrated on annotating the photo dataset in this phase.


When a single photo is fed to the neural network process, the photo has to be resized into a certain size and shape. The typical image size in image classification is 224 x 224, which ensures that the photo processing time does not increase too much because of the image size. Since object detection involves mainly annotation data, we can use a greater image size by default. For example, the first version of the BerryMachine artificial intelligence system uses the YOLOv5l –object detection model, which supports photos either in the size of 640 x 640 or 1280 x 1280. The photos in the BerryMachine –photo dataset are typically larger than this, for example 2024 x 4032, which causes certain challenges in the development, since the sizes do not match with supported image sizes. If using a larger image size, it will make the training phase of the artificial intelligence considerably heavier as well.

Finally the processed photo is converted to tensor format (multidimensional matrix), which means the photo is converted to a numeric format. Since neural networks do not understand photos themselves, the numeric format is necessary so that the neural network is able to process the data. The numeric data within a tensor is often also normalized, so that each value in the data is between -1 and 1, which is optimal for training the neural network.

First technological prototype

After the photo dataset was in usable format, the next phase was to feed it to the artificial intelligence. This phase is called training the artificial intelligence, which will result in an artificial intelligence model that is able to detect those objects from a photo it has been trained to detect. In the case of BerryMachine, the artificial intelligence is trained to recognize raw and ripe berries as well as their flowers from a photo.

In order to get even the smallest photo details on board for the training phase, cropping the photo dataset was necessary. There were no ready-made automatic tools to do this cropping phase, which forced us to create our own tools. During the cropping phase, some challenges emerged, for example, berries that were in the middle of the cropping lines or close-up shots of berries that existed in multiple cropped pictures even if they were overlapping.  While cropping the photo dataset automatically, the annotation data was also split simultaneously, in order to keep the dataset synchronized.

Figure 5. Some usable image dataset cropping techniques.


After the photo dataset was cropped and splitted and the training software of the artificial intelligence was developed, the training software and the dataset was transferred to the Mahti –supercomputer at CSC, which was launched to be used by higher education institutes in Finland in 2020 (see CSC 2020 and Docs CSC 2021).

Training an artificial intelligence often takes a considerable amount of time, and the more complex the neural network and the larger the dataset is, the longer the training time and the more processing power the training phase requires. This is the reason why graphical processors (GPU) are usually preferred over traditional processors (CPU) when training a neural network. The reason for this is the processor architecture of a GPU, which is more efficient in processing specialized calculations in parallel when compared to CPUs. This is especially useful when training neural networks (Sharabok 2020).

For the first version of the BerryMachine’s artifical intelligence system, YOLOv5l –object detection technology was chosen as the main technology. YOLOv5l was chosen mostly due the practicality and detection accuracy. YOLOv5l is programmed by using the popular PyTorch –module (Python), which makes the implementation phase far easier. YOLOv5l is also very competetive in its detection accuracy, even if compared to other common object detection technologies.

During the training phase of the artificial intelligence, the amount of available graphics card memory affects considerably the training efficiency. The Mahti –supercomputer at CSC has 40 Gb of graphics memory in total. While training the BerryMachine –artificial intelligence, even this amount of memory proved difficult, since if we tried to use the larger image size of 1280 x 1280, the memory of a single graphics card ran out. One way to solve this challenge is to revert back to the image size of 640 x 640. However, this requires a longer training phase since the photo dataset has to be cropped into smaller pieces in order to achieve the same accuracy.

In the end, it took approximately 30 hours to train the latest version of the artificial intelligence on the Mahti supercomputer. After the training phase, the next step was to evaluate the results of the training process and the detection results.

Evaluating the first results and planning for the future

The first version of the BerryMachine artificial intelligence system aimed to recognize the following categories from a given photo:

  • bilberry (flower, raw, ripe)
  • lingonberry (flower, raw, ripe)
  • lingonberry bunch (flower, raw, ripe)

We decided to create a separate category for the lingonberry bunch. The reason for this is, based on our own experiences, it seems the artificial intelligence is more adept in recognizing a bunch than all the lingonberries separately within a bunch. This happens most likely due to the way how lingonberries grow in nature when compared, for example, to bilberries.

Figure 6. Lingonberries and berry detections.

The detection results produced by the trained artificial intelligence system with different photos:

Figure 7. The trained artificial intelligence system recognizes berries from a photo. The decimal number after the recognized category is the confidence of detection in percentage format (e.g. 0.7 = 70% confidence of correct detection).

Evaluating an image classification and/or object detection model can be done by using a variety of tools. One of the most common tools is the so-called confusion matrix, which allows us to quickly see how well the trained artificial intelligence model is able to recognize objects correctly and incorrectly. The confusion matrix created by the first version of the BerryMachine artificial intelligence is the following:

Figure 8. The confusion matrix produce by the first version of BerryMachine artificial intelligence system. The numbers on the diagonal line are the correct detections within the test dataset, everything else are false detections. The number values are percentages in decimal format (e.g. 0.72 = 72%).

The challenges with the lingonberry detection can be seen in the figure above. The background of the photos also is problematic, especially when detecting lingonberries. Even if all of the accuracies could be greatly improved, it’s especially the lingonberry that provides most of the difficulties at the time.

The next questions therefore is: why does this happen, and what can we do to improve the artificial intelligence further? The first step is to examine the dataset itself:

Figure 9. The distribution of berry photos by berry type in the dataset.

Figure 10.The distribution of berry photos by berry type and ripeness level in the dataset.

Figure 11. The dataset divided by ripeness levels and growing phases.

Figure 12. The dataset divided by growing phases and berry types, including lingonberry bunches.

By examining the figures above, we can see the imbalances within the photo dataset. First it seems the amount of bilberry photos is too small, but if examining especially the amount of annotations, the greatest deficit is within the amount of lingonberry photos. One approach to improve the dataset would to obtain more material for this category, and then retrain the artificial intelligence again.

We can also notice the following:

  • Lingonberries have more annotations than bilberries as a whole
  • Amount of annotations based on ripeness level are balanced within all photos
  • Amount of annotations are balanced between growing phases within a single berry type
  • The amount of annotations is considerably lower in raw lingonberries and lingonberry bunches when comparing to other categories. This correlates with the other difficulties considering the detection of lingonberries (see the earlier confusion matrix), but doesn’t most likely explain all the difficulties within lingonberry detection

In addition to complementing the dataset, we can also consider other methods to improve the artificial intelligence system. These include:

  • Replacing the currently used object detection technology completely
  • Using a heavier object detection model, for example, YOLOv5x
  • Creating more additional artificial intelligence tools to further process lingonberry detections in cases, where the artificial intelligence is not confident enough of the correct detection
  • Using image classification technology to support the object detection technology

Generally it seems we are not going to achieve a satisfying detection accuracy solely relying on object detection technologies. Because of this, FrostBit is going to combine object detection with conventional image classification next, to further process problematic detection cases.

Even if we can’t provide exact numbers, we can provide an educated guess on typical detection accuracies based on different image detection technologies. Based on our own experiences, the typical detection accuracy in object detection is between 40-70%, while the typical detection accuracy in image classification is typically between 80-95% per single photo. The limitation in image classification is that they only analyze the photo as a whole, for example, whether the photo has a raw lingonberry or a ripe bilberry etc.

Since image classification can only detect what the photo depicts as a whole, it’s not effective in recognizing multiple berries from a single photo at once. Because of this, the next version of the BerryMachine –project’s artificial intelligence system will use object detection only to find berries on the general level within a photo, all of which are cropped into separate smaller photos, which will finally be processed by image classification technologies. Therefore, the general software architecture can be visualized in the following way:

Figure 13. The next version of the BerryMachine artificial intelligence system, software architecture design.

Creating any kind of artificial intelligence system can potentially take an infinite amount of development time, the only limitations being creativity and available processing power. Because of this, FrostBit aims to get as much as possible out of the artificial intelligence technologies within given resources during the project. After the BerryMachine –project, it will be interesting to see how far did we get with the developed berry detection system and what can we create next based on our previous findings. Every development iteration seems to take us closer to our original objective.

The BerryMachine –project is being funded by Interreg Nord 2014-2020. The total budget of the project is 144 431 euros, of which 122 766 euros is being funded by the EU. The project schedule is 1.1.2021 – 30.9.2022.

This article has been evaluated by the FrostBit’s publishing committee, which includes Heikki Konttaniemi, Toni Westerlund, Jarkko Piippo, Tuomas Valtanen, Pertti Rauhala and Tuuli Nivala. The article publications can be found on the FrostBit publication blog.

30.12.2021

SHARE IN

Tuomas Valtanen, Project Manager

Tuomas works as the web/mobile team leader, a software engineer and a part-time teacher in FrostBit at Lapland UAS. His tasks include project management, project planning as well as software engineering and expertise in web, mobile and AI applications.

Master of Engineering

Mikko Pajula, Specialist

Mikko is a coder and part time teacher. His programming tasks are mainly full stack development with extra knowlegde in GIS and deep learning.

Bachelor of Engineering

References

Bonner, A. 2019. The Complete Beginner’s Guide to Deep Learning: Convolutional Neural Networks and Image Classification. Towards Data Science 2.2.2019. Accessed on 17.12.2021 https://towardsdatascience.com/wtf-is-image-classification-8e78a8235acb

Browniee, J. 2021. A Gentle Introduction to Object Recognition With Deep Learning. Machine Learning Mastery 27.1.2019. Accessed on 17.12.2021 https://machinelearningmastery.com/object-recognition-with-deep-learning/

CSC 2021. Supercomputer Mahti is now available to researchers and students – Finland’s next generation computing and data management environment is complete. CSC 26.8.2020. Accessed on 17.12.2021 https://www.csc.fi/en/-/supercomputer-mahti-is-now-available-to-researchers-and-students

Docs CSC 2021. Technical details about Mahti. Docs CSC 14.4.2021. Accessed on 17.12.2021 https://docs.csc.fi/computing/systems-mahti/

Dwivedi, P. 2020. YOLOv5 compared to Faster RCNN. Who wins?. Towards Data Science 30.1.2020.  Accessed on 17.2.2021 https://towardsdatascience.com/yolov5-compared-to-faster-rcnn-who-wins-a771cd6c9fb4

Géron, A. 2019. Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow. 2nd edition. Sebastobol: O’Reilly Media

Github.com 2021. TensorFlow 2 Detection Model Zoo.  Github.com 7.5.2021. Accessed on 17.12.2021 https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md

Hofesmann, E. 2021. Guide to Conda for TensorFlow and PyTorch. Towards Data Science 11.1.2021. Accessed on 17.12.2021
https://towardsdatascience.com/guide-to-conda-for-tensorflow-and-pytorch-db69585e32b8

Huellmann, T. 2021. How to build a dataset for image classification. Levity 9.11.2021. Accessed on 17.12.2021 https://levity.ai/blog/create-image-classification-dataset

Krishnan, B. P. 2019. Machine learning Vs Deep learning Vs Reinforcement learning. Medium 18.9.2019. Accessed on 16.12.2021 https://medium.com/analytics-vidhya/machinelearning-deeplearning-reinforcementlearning-ed7b217861c5

Lendave, V. 2021. A Comparison of 4 Popular Transfer Learning Models. Analytics India Magazine 1.9.2021. Accessed on 17.12.2021 https://analyticsindiamag.com/a-comparison-of-4-popular-transfer-learning-models/

Leo, M. S. 2020. How to Choose the Best Keras Pre-Trained Model for Image Classification. Towards Data Science 15.11.2020. Accessed on 17.12.2021 https://towardsdatascience.com/how-to-choose-the-best-keras-pre-trained-model-for-image-classification-b850ca4428d4

McCulloch, W.S. & Pitts, W. 1943. A Logical Calculus of Ideas Immanent in Nervous Activity. Bulletin of Mathematical Biophysics 5, 1943, 115–133. https://doi.org/10.1007/BF02478259

Pai, A. 2020. CNN vs. RNN vs. ANN – Analyzing 3 Types of Neural Networks in Deep Learning. Analytics Vidhya 17.2.2020. Accessed on 17.12.2021 https://www.analyticsvidhya.com/blog/2020/02/cnn-vs-rnn-vs-mlp-analyzing-3-types-of-neural-networks-in-deep-learning/

Patel, K. 2020. Image Feature Extraction: Traditional and Deep Learning Techniques. Towards Data Science 9.9.2020. Accessed on 17.12.2021 https://towardsdatascience.com/image-feature-extraction-traditional-and-deep-learning-techniques-ccc059195d04

Sharabok, G. 2020. Why Deep Learning Uses GPUs? Towards Data Science 26.7.2020. Accessed on 17.12.2021 https://towardsdatascience.com/why-deep-learning-uses-gpus-c61b399e93a0

Sharma, P. 2019. Image Classification vs. Object Detection vs. Image Segmentation. Analytics Vidhya 21.8.2019. Accessed on 17.12.2021 https://medium.com/analytics-vidhya/image-classification-vs-object-detection-vs-image-segmentation-f36db85fe81

Warden, P. 2017. How many images do you need to train a neural network? Pete Warden’s Blog 14.12.2017. Accessed on 17.12.2021 https://petewarden.com/2017/12/14/how-many-images-do-you-need-to-train-a-neural-network/

Özgenel, Ç.F. & Sorguç, A. G. 2018. Performance Comparison of Pretrained Convolutional Neural Networks on Crack Detection in Buildings. Berlin. 35th International Symposium on Automation and Robotics in Construction (ISARC 2018). Accessed on 17.12.2021 https://www.iaarc.org/publications/fulltext/ISARC2018-Paper154.pdf

Creating a Digital Twin VR-Environment

DUKE is a collaboration project between Lapland University of Applied Sciences (Lapland UAS) and Rovaniemi Municipal Federation of Education (REDU). The idea of DUKE is to examine how VR solutions can be implemented in educational purposes while leveraging renewable energy. Our main pilot is a digital twin of an educational district heating plant located in Rovaniemi and our secondary pilot is a digital twin of a customer grade heat pump.

Digital twins allow us to visualize the environment with great immersion, this enables us to implement accurate functionality to various devices and systems. High immersion helps users to subconsciously connect the virtual object to the real-life object, thus seeing a virtual lever would have the same relationship as seeing a real lever; it should be interactable.

DUKE’s customer grade heat pump was made in cooperation with a project partner. Our partner wanted to incorporate VR solutions alongside their user manual. This way, the user can visualize more accurately basic procedures and maintenance work for the heat pump. Our DUKE engineers tackled this project by working closely with the heat pump specialists. The specialists taught us how to do maintenance on the heat pump and we, in turn, translated that knowledge into a VR simulation. The development process fundamentally revolved around an iterative workflow, where we had assessment and review meetings back and forth. In doing so, we ensure that the 3D-recreation was accurate and at a satisfactory quality.

Creating the digital twin VR-environment

A digital twin is a 3D virtual recreation of an object or a real-world environment. The definition ‘digital twin’ can be perceived as an instruction regarding how closely or realistically the virtual recreation represents its real-world counterpart. Here in Frostbit Software Lab, we aim to create our digital twins to the extent where the 3D-recreation is highly accurate and unmistakable when comparing the 3D-version to its reference.

Side-by-side image of the rendered 3D-scene and the raw 3D-scene.

The workflow and subsequently the challenges we face depends largely on the use case scenario. The VR simulation hosts a physics simulation system in addition to the high-fidelity 3D-visual representation of the environment. For this reason, we chose to use Unity for its lightweight and adaptable nature to implement our real time VR simulation. Unity allows us to easily manipulate and optimize the VR simulation to work with customer grade computers and high-end computers for a higher fidelity experience.

The developer’s viewport inside Unity game-engine. The screenshot is taken from the work-in-progress district heating plant.

We kicked off the visual implementation process by taking reference images on-site. The images are captured from multiple angles in order to get an understanding of the object’s shape and contour. Moreover, it is exceedingly useful in figuring out the object’s material properties, such as how shiny it is, what type of metal it is and what kind of reflections it produces.

A reference image of the district heating plant.

The 3D-modeling process uses blueprints in tandem with the reference images to get accurate measurements for a high-detail recreation. A keen observer may still find some discrepancies when comparing the screenshot image to the reference image. This is due to readjustments, some parts of the district heating plant were deemed obsolete for the VR simulation and therefore had to be cut or modified.

The texturing process uses the multi-angle images to figure out what type of material would be the most accurate to its real-world version. We use HDRi-images to create realistic lighting to the 3D-scene. HDRi’s are essentially 360-degree panoramic images that contain a vast amount of lighting data. Ultimately, we are able to create believable environments by following real-world lighting conditions while incorporating physical based rendering techniques.

A screenshot of the disctrict heating plant digital twin.

VR-Implementation in a game-engine

The first step our developers need to make is to choose an ecosystem, in other words, which development pipeline to use. Locking into a development pipeline is essential in guaranteeing efficient feature implementation down the line. The ecosystem in this context consists of choosing which manufacturers VR-gear to use, what game-engine and development toolkit to use. Our DUKE engineers ended up on using Unity game-engine for its lightweight and adaptable nature.

A test-player trying out the playable demo of the DUKE VR-simulation

Our DUKE engineers chose to use SteamVR software development kit for its wide VR-gear support. This SDK provides accessible script blueprints, that we can tailor for our own usage. For example, the SDK provides a basic functionality for object interaction and player movement. This way, more time can be allocated to polish more advanced features such as the dynamic interaction between different objects while a mathematic simulation is running in the background.

The mathematic simulation is built with modularity in mind. Each part of the simulation is broken down to individual pieces. This grants us the customizability to assign the simulation scripts to the 3D-objects that would logically handle the same physics in the real world. For example, the scripts that handle pressure loss are assigned to the 3D-objects that look like valves or pipes. One of the challenges we faced when working with a VR solution was that real-life dimensions did not translate well in a VR environment. EU-standard sized doors seemed too narrow, standardized knobs, bolts and valves seemed too small and ultimately play-testers reported having a difficult time maneuvering in the VR environment overall. A silver lining needed to be struck in order to satisfy the digital twin -nature of the project and the ease-of-use side for the players.

The current DUKE heat pump build contains four scenarios: “Tutorial”, “Water filter”, “Pressure adjustment” and “Water leakage”. Each scenario provides information on how to operate the customer grade heat pump and how to perform basic maintenance.

Tutorial -scenario

Tutorial scenario explains thoroughly all the functions available in the project. This scenario also includes information covering the main parts of the heating system. This is usually the first scenario the player should launch, since it also teaches the player how to use the VR-gear.

Water filter -scenario

The user can learn how to clean the water filter using this scenario.

Pressure adjustment -scenario

The user can learn how to adjust the heat pump to a correct pressure level in the system and how to use the safety valve to release the pressure from the system.

Water leakage -scenario

Water leakage -scenario provides a good guideline on how to store the heat-pump system. We show the most common issues that can be encountered in such a space.

The total budget of the project is 761 732 euros, of which Regional Council of Lapland has granted European Regional Development Fund (ERDF) and state funding 609 385 euros. This project will be executed in the timeframe of 1.1.2020-31.12.2022.

This article has been evaluated by the FrostBit’s publishing committee, which includes Heikki Konttaniemi, Toni Westerlund, Jarkko Piippo, Tuomas Valtanen, Pertti Rauhala and Tuuli Nivala. The article publications can be found on the FrostBit publication blog.

20.10.2021

SHARE IN

Onni Li, Specialist

Onni specializes in CG-Graphics In Frostbit Software Lab. The majority of his responsibilities revolve around creating 3D-models, virtual environments and procedural materials for texturing. Onni is proficient with various different game and render engines by understanding how physically based rendering techniques work.

Bachelor of Engineering

Severi Kangas, Specialist

Severi worked as a programmer in FrostBit at the DUKE-project.

Bachelor of Engineering

A new era of educational games – learning through digital solutions and gamification

Advances in technology have made it possible to create more advanced games for educational purposes. It is no longer just “point and click” style computer games for children in the late 90s, but developments have given us the keys to assist in more complex learning. The wildest dreams of digital twins that utilize virtual reality have already come true and being constantly developed in the FrostBit laboratory. However, for the consumer, the virtual world viewed through glasses can still be foreign.

At the same time as the spearhead of technological development is far ahead of the consumer, we have also reached a different milestone in the world of educational games: each of us has access to a digital terminal that opens the door to learning in increasingly diverse ways. In practice, therefore, everyone can use educational games in their daily learning via a smartphone or computer. The quality and diverse content of games have also improved in the same proportion, and in recent years various device manufacturers also seem to be slowly waking up to the opportunities brought by digitalization. Virtual reality will reach us all at one point, but at this point we must at least leave behind the old moldy cereal-pack games.

Virtual learning in healthcare

In co-operation with the social and healthcare section, we are currently implementing a simple project as part of the Nursing RDI activity, the aim of which is to create a tutorial that teaches how to use an infusion pump. This Braun Infusomat Space is an everyday tool for the nurse and learning how to use it is one small but important part of nursing studies. Traditionally, the infusion pump is learned to be used in the classroom without the patient and in theory it is possible for the student to go through the operation of the device as much as they wish. In practice, however, there are often many students, limited time, and personal learning difficulties that can get in the way of learning, and because of these aspects, traditional classroom instruction does not always guarantee that all students will be able to use the device enough. A nursing student focuses mainly in learning to take care of the patient, and the use of technical equipment is not necessarily their primary focus. Infusomat is a reasonably valuable and even a little “threatening” device, and when practicing it, the student often learns certain procedures and tries to avoid mistakes. Thus, learning to use a device does not actually involve deep learning, but mainly the ability to repeat what has been learned. Such learning procedures lead to the expected result, but as an unfortunate side effect, the student does not actually get to know the device but merely uses it in predetermined situations. In case of problem situations or in the event of a disturbance, the ability to react and act on the situation is not the best, and the student may not feel confident when using the device.

The functions of an infusion pump (alpha)

Gamified learning can enhance the students’ deep learning. Instead of learning how to use an infusion pump only in a controlled way in the classroom, the student can continue to “play with the device” in the virtual world. Once the operation of the entire device has been 3D-modelled accurately in the game and the interface is suitably simple, the player will be able to try out different functions and resolve problem situations without worrying about accidentally breaking the pump. By utilizing the game, the student achieves confidence in using the device and then the operation of the correct infusion pump feels much more familiar as well as easier to learn in practice.

How are the functions of the infusion pump transferred to the game?

The most important aspects of making this game are the following:

  • User interface
  • Device operation
  • Correspondence of the 3D model
The 3D-model of the infusion pump (alpha)

What is essential, of course, is that the developer not only thoroughly understands the operation of the device, but also models it with the accuracy required for its operation. A big factor in particular is user orientation: a game can be useless to its target audience if the target audience is unable to use it. The developer must assume that the user of the device has little or no experience in playing different games and therefore must understand to develop functionalities so that no user experience difficulties will alter the learning process. In practice, this means that features very familiar to many digital games, such as first-person character movement and camera control, must be omitted or made so simple that they don’t interfere with the gaming experience. It is unnecessary to force the player to learn the seemingly complicated interface that they even get the opportunity to learn how to use the device himself.

The development phase has two stages: device modelling and construction of the game mechanics. Blender-3D Software is used to model the infusion pump and game mechanics are created with the Unity game engine. In modelling the device, it is essential to model all those external features that matter either in terms of game mechanics (buttons, screens, power cords) or visual equivalence (round arc of the door, aspect ratio). In game mechanics, an accurate representation of the infusion pump operation is important. For example, when you press the start button on the device, the infusion pump emits a small beep, the lights turn on, and then a text appears on the display to indicate that it has started. All of those functions are essential to make it easy for a player to transfer what they have learned from the digital to their real world. However, it is not essential that the internal operating logic should be exactly the same as that of the device being modelled.

A real infusion pump

The end result can be considered as the crucial point: the game must be able to convince the player that it works in the same way as the real-world device it represents. The choices made in game development must therefore be viewed from the perspective of transferring what has been learned to the real world: if the function is important for using the right physical device, then it must also be implemented in the digital version. Thus, a game made of an infusion pump does not have to be a perfect digital twin, but mainly a model that simulates its operation as accurately as possible.

The project is still ongoing, but looks very promising. Combining game development with the development of learning outcomes is an interesting and important direction that will certainly provide interesting data as the game develops. The tutorial is scheduled to be ready in June 2021. The game will run on a regular computer, mobile device, and in addition, the possibility of virtual reality will also be explored.

‘TKI-toiminta digiaikaan’ -project is funded by Lapin Liitto with EAKR-funding.

29.04.2021

SHARE IN

Samuli Valkama, Specialist

Samuli Valkama works as a specialist in the FrostBit software laboratory at the University of Lapland. The focus of his work is on project and game design, but the increase in the number of video productions in particular has increased.

Bachelor of Arts (Media), University of Lapland

Historical accuracy versus playability – Designing the Struve Geodetic Arc mobile game

If a game, game-like learning environment or any other software using game technologies is based on real events, the relationship between realism and playability has to be taken into consideration during the development process. Imitating the real world precisely with all its limitations set by physics, biology or the passage of time doesn’t necessarily serve the original purpose or goal of the game or software. It doesn’t matter if the goal is to entertain or to educate, both can suffer from total realistic accuracy.

Making a game replicate its real-life counterpart as precisely as possible is sensible and sometimes absolutely necessary when a game is focused on a very limited subject, such as controlling a specific vehicle. In the case of vehicle simulators, the physical controllers are also often made to look and feel like the real thing. On the other hand, when the subject of the game is broader, decisions have to be made on which things to replicate and which things to change to enable e.g. playability, entertainment or better understanding of the bigger picture. The intended platform of the game also plays into these decisions. A keyboard-mouse combination, regular commercial game controller or a smartphone screen are all very generic controllers and as such can’t exactly replicate the feel and functions of any specific tools or items. If the chosen platform is mobile, physical size and performance of the devices as well as the inaccuracy of gesture controls set extra limitations.

This balancing between realism, hardware limitations and the goal of the game has been and will continue to be an integral part of the design process of Struve Geodetic Arc mobile game. The game is being produced as a part of The Northern parts of the World Heritage Struve Geodetic Arc project (struvenorth.net), partially financed by EU’s Interreg Nord programme (ERDF).

The background story: What is the Struve Geodetic Arc?

Friedrich Georg Wilhelm von Struve (1793-1864) was an astronomer who also had an interest in geodetic surveying.  As a part of his research he organized a triangulation survey with a triangle chain reaching from Black Sea to Hammerfest. The goal of this survey was to gain a better understanding of the shape of the Earth near its poles. The whole chain includes 258 main triangles and 265 main station points that stretch over 10 different countries. A total of 34 of the main station points are part of UNESCO World Heritage. (maanmittauslaitos.fi)

Struve Geodetic Arc was accepted into the UNESCO World Heritage list in 2005, but it isn’t particularly well known in the Nordics. The Northern parts of the World Heritage Struve Geodetic Arc project aims to improve the accessibility and knowledge of the Struve Geodetic Arc. The purpose of the mobile game is to attract new audience for the World Heritage site by means of entertainment.

The events and environment of the game are based on real events in what is currently Finland, Sweden and Norway, then Sweden-Norway and the Russian Empire, during the 19th century. However, the intent is not to make a “real” educational game or replicate historical events and circumstances as precisely as possible, even though it would be possible in theory. Struve Geodetic Arc mobile game has been intentionally designed to handle its real-life backstory with a lighthearted and entertaining touch. The reasons behind these decisions can roughly be divided into three parts: reaching casual gamers, creating a better flow for the story and limitations of the mobile platform. These reasons and historical accuracy aren’t necessarily mutually exclusive, but in this project the choice was made to favor entertainment and casual approach.

Handling time and order of events

When a several hundred pages long novel is adapted into a movie, the events of the story are shortened and perhaps rearranged to some degree, in order to make all the important things fit in a more compact format and to keep the viewer engaged for the entire duration of the movie. Time and order of events have been approached somewhat similarly in the development process of the Struve Geodetic Arc mobile game for the exact same reasons: limiting the total length of the game and making it easy for the players to follow the events and keep them interested in the game from start to finish.

In reality, measurements across the entire triangle chain spanned several decades. The focus of the game is mainly on the section going over Lapland where measurements also took years. Exactly replicating all events from a time period this long would inevitably make the game longer despite the way time is handled inside the game. A typical mobile game session isn’t very long: GameAnalytics, a company selling tools for analyzing game usage, states in their 2019 report that the average session length for a mobile adventure game was less than 15 minutes (GameAnalytics Mobile Gaming Benchmarks Report: H1 2019, p.13). If the total length of a game is measured in several dozen hours like in some current AAA-titles for PC or console, it would take a much larger amount of these average length sessions to finish the game compared to a situation where the total length is just a few hours. In the Struve Geodetic Arc mobile game one of the goals is to get the players to finish the game and go all the way to the northern endpoint of the triangle chain in Hammerfest. So, if the game in total is very long and following that the required amount of game sessions needed to finish the game is large, it is possible the players feel that the game is too long and won’t ever finish it. Keeping the players engaged until the end of the game is a good enough reason to give up accuracy.

It was never intended that the actual historical timeline would be completely removed from the game, however. At the current development phase (spring 2021) the game is being designed to have calendar years pass according to the game’s internal time and to give the player a goal to reach the endpoint the same year the actual measuring crew got there. Some known places of stay might also be matched with the correct years.

Much like handling time inside the game flexibly, it was decided that the order of events would be presented in a more linear fashion compared to reality. In the game the player’s party follows a route along the triangle chain in one direction, whereas in reality the measuring crew did not move linearly from point A to point B. During the measurements the crew stayed longer in some places than others and measurements in a certain area could be done as separate trips from one place of stay. In the early stages of design, it would have been possible to choose game mechanics that supported moving in a way more closely resembling the actual measurements, but the development team decided to make movement on the map more linear. The team felt it would be more motivating for the player to follow a clear route from one place to another compared to e.g. separate trips from one place and returning there every time. One factor in this decision was that in the earliest design stages the team didn’t have the information on the more exact routes, it became available at a later phase. Lacking this information, it was easier to design the game to have a linear route.

Handling time and events in a flexible manner sacrifices much of the historical accuracy. The reasoning behind this sacrifice is making the game more approachable. No background information or prior interest towards the subject of the game is required from the player. The game also doesn’t aim at educating players by force. The thought is first and foremost to offer a good experience. Then on the side tell the player about a subject that was possibly previously unknown to them and get them interested enough to find out more through other means.

Effects of mobile platform

Regardless of genre and content of the game, limitations set by the platform must be taken into consideration when developing a game for mobile devices. Even though the physical size and resolution of screens on mobile devices have been growing, the screens are still many times smaller compared to a typical laptop or desktop screen. This sets limitations to the number of menus or other elements that can be fit on the screen at one time while still maintaining legibility. In addition to this, mobile devices are typically controlled with finger input, which is considerably less accurate way of navigation compared to a game controller or a mouse. This in turn limits the smallest possible size e.g. buttons can be.

When better accuracy is required, mobile devices can be controlled with a stylus. A small number of stylus-controlled games have also been made. Controllers made specifically for mobile gaming are also being made and sold, these closely resemble console controllers. Both styluses and controllers are mostly sold separately, so when the aim is to reach as large audience as possible and make using the game easy, it is not reasonable to require the user to have any kind of special controller devices.

In terms of game design, screen size affects (among other things) the amount of information the player can be shown at one time. This amount can be regulated for example by showing only the information that is required in that precise menu or at that precise moment of the game. Another option is dividing the information into smaller sections and having separate menus or menu levels for each section. Neither of these options are without problems. If the total amount of the information the player needs is large and most of it is contextual and not shown to player at all times, the limits of short-term memory might prevent the player from keeping all required things in mind. Then again, multi-leveled menus can be hard to navigate. This means the only solution isn’t just dividing and categorizing information or changing the way it is presented. The game itself must be designed to be simple enough so that the amount of required information isn’t too great.

Poor accuracy of finger input means that the basic functions of the game can’t require precise or complex actions from the player. The controls of the game must be simplified to a few most important buttons and simple gestures so that the learning curve isn’t too steep and continued use is effortless.

The need for simplification means more adaptations and loss of historical accuracy. Actions that in reality are complex or demanding have to be made easier and more straightforward. In the case of Struve Geodetic Arc mobile game one example of this is the triangle measurement itself. The aim is to give the player factual information on the measuring process, but using the measurement devices required skill and precision and the actions won’t be precisely replicated in-game. The realistic measurement actions will be replaced with simplified versions or with an entirely another set of related activities. When the basics of the game are easy to learn and playing doesn’t require complex actions, more casual players or people who haven’t really played at all, can easily start playing Struve Geodetic Arc game.

What historical things are included in the game?

Despite putting game mechanics and the requirements set by the chosen platform first in the development process of Struve Geodetic Arc mobile game, the intention is not to make a generic adventure game and just have it wear a cosmetic skin with some historical influences. The project team has gathered information on e.g. the progression of the actual measurements, known places of stay and the day-to-day -life of people in the 19th century. The game development team has then incorporated this information into their designs. The game will have player to encounter problems similar to those the real measuring crew encountered. The remoteness and challenging weather conditions of Lapland affected e.g. the ways measuring equipment could be transported and when measurements could be performed in the first place. One example of these conditions is that during the 19th century the actual road network in Finnish Lapland, excluding winter roads, reached Kittilä, Kolari and Sodankylä, so when the player moves further north they can’t choose to use a road and have to advance by other means.

One goal of the game is also to have the player understand, how significant the measurements were to the society at the time. Using the information gathered from the measurements, new more correct maps could be drawn and the significance of good maps to trade and warfare for example was and still is great.

Visual elements are one of the ways used to convey information and feeling of the time period to the player. Even though the art style of the game is somewhat comic-like, the appearance of e.g. clothes, tools, buildings and measuring equipment is modelled after their real-life counterparts. The intent is to show events and life from nearly 200 years ago as lively and interesting despite deciding to give up true historical accuracy in many ways.

More on The Northern parts of the World Heritage Struve Geodetic Arc project: struvenorth.net

More on the Struve Geodetic Arc in general: https://www.maanmittauslaitos.fi/en/struvegeodeticarc

Game Analytics report of 2019: https://progamedev.net/wp-content/uploads/2019/07/Benchmarks2019.pdf

31.03.2021

SHARE IN

Sanni Mustonen, Project Manager

Sanni is a graphic designer with an interest in the technical side of things. Her work includes everything from UI design to visual identities and teaching in the Summer Game Studies.

Doctoral Student, University of Lapland’s Faculty of Art and Design

UX and Service Design in the FrostBit lab

Besides being an intern in the FrostBit Software Lab, I am also a PHD student major in service design in the University of Lapland. I am very honoured to be an intern student in FrostBit. I got a chance to apply to UX and service design in different projects, and I am happy to tell you about my experiences.

What is UX and service design?

Recently, UX design and service design have been very topical: you maybe have heard about many products and software applied with UX design and service design principles. However, what exactly is UX design and service design?

UX means User Experience. Normally, in terms of design software, UX designers focus more on the logic in software. Through UX design, users will not feel confused or get lost in the 2D (applications, website and etc.) or 3D (VR or AR technologies) worlds. In most situations UX design links closely together with UI design (User Interface design).

Until now, there is still no single official definition to point out what is service design. From my own understanding, the key point in service design is “empathy”, which means that service designer must think of the “service process” from all stakeholders’ perspectives. For example, if you are developing a B2C websites, customers and staff who used the websites, as well as even some third parties are included as stakeholders. Furthermore, service design involves the whole process given the process design solution. Let me demonstrate you an example with a restaurant: when you first time see the logo and face of the restaurant, the service design already begins. The dining environment, taste of food, personal service and even the feedback you give yourself – all of these can be part of service design processes.

From my perspective, service design is more like a strategic design: in addition to the product itself, service design provides a more efficient and valuable design process solving through the “pain points”. Service design has a broader scope, which does not only include UX design, but also graphic design, product design, information design and so on.

Below, you can see a basic design process that I always follow:

How did I end up in UX and service design in the FrostBit Software Lab?

During my intern period in FrostBit Software Lab, there are two main things I have done: one is re-designing the FrostBit websites and the other one is doing interior design to the lobby outside the lab office. You could think that the website design is connected to the UX design and service design – which is true. In my design process, I implemented animations for the website, which could catch customers’ glance immediately when they arrive at the website, and push them to pay attention on the important information on the site. Moreover, some icons are combined with texts: in this way, the info could be transferred to readers in a very short time. Although there are common tips designers always apply in the web design, what I want to talk about is the way of thinking: thinking for readers and information provider at the same time. When you do it this way, you can define the core of design solution, balance the requirements from both two sides, and understand what is the “empathy ” I mentioned before.

The interior design for the lobby has the same thinking process: the process is determining the target group as well as determining the aim and main functions as well as the design elements (for each group) in the lobby. From my experience, I used UX and service design to plan the “big picture” of the lobby, and then use the interior design as a method to apply and show my ideas and design solutions.

During my internship, I did not only increase my basic design skills, but also practiced the key points of service design in the FrostBit software lab. In my opinion, there are three certain key things that you need to keep in mind during design process:

  • How to increase the value
  • How to use co-design
  • What is the new idea or solution

What about the future of service and UX design in FrostBit?

I am definitely sure the service design and UX design have a bright future in the FrostBit Software Lab. No matter how the world is changing, every progress that Frostbit software made could change or instruct people’s behaviour. This is why we use service design to see the whole picture and give “correct ” directions for our projects. At FrostBit, we are not only applying the “real” design skills, but also the way of “empathy” thinking in our projects and we are continuously working towards ways to establish this. If you want to see how we use UX and service design in our project to create inspiring outcomes, stay tuned for updates on our websites!

10.03.2021

SHARE IN

Nan Li, Specialist

Nan Li works as a specialist in the FrostBit Lab in LapUAS. She is a doctoral candidate major in Service Design in University of Lapland. Service design and graphic design are her main work areas.

Doctoral Candidate of Service Design, University of Lapland

Game Dev team working in FrostBit

We already have about 40 employees working in the lab, as well as, of course, our trainees, exchange students and students working on their thesis. Much of this is happening in the Game Development team amongst various game projects. Our number of employees has grown tremendously in recent years and a lot of new different skills have come along. We are no longer just a team of engineers coding in a dark lab with the power of ‘Jolt Cola’. Professionals in 3D-modelling, graphic design, service design, audiovisual and game pedagogy, among others, have joined the team. This has brought along everyone’s personal skills to the team. Our team has really strong skills to establish holistic projects from the beginning to the end. We constantly work closely together with projects on the Mobile and Web team, since almost every one of our projects needs at least a website or back-end system. Building a complex background system is the core competency of the lab’s Mobile and Web team.

Getting to know the mining processes in the Mantovaara open mine

What do we actually do in the Game Dev team? Our main tools are different game engines: we mainly use Unity3D or Unreal game engine according to the project. While our team name is Game Dev Team, we do not only make games in the “traditional format”. We utilize game technology, for example, for visualizations, simulations, learning environments and marketing. This makes the work really versatile; we get to work with different substance industries and we have made various implementations utilizing game technology ranging from particle physics to medical care. However, we don’t only cooperate with different material industries – we also often get to get to know and study what it is like to work in the industries, such as mining. A good example is the KaiVi project, during which the programmers and modelers from the lab went on internship to the mine to get acquainted with the mining operations. Such action is paramount when it comes to building environments that reflect reality. It is also important for the development team to have some understanding of the substance industry and thus be able to communicate effectively with professionals.

Multidisciplinary skills and learning

The full use of gaming technologies requires the cooperation of several professionals. Each project begins with designing and defining, involving substance experts, designers, programmers as well as artists. At this stage, pedagogical solutions to learning environments are also considered and service design tools are included. The actual implementations involve different stages depending on the project, but typically the project starts with conception and technology tests or prototypes, from which the transition to the actual product development takes place. Depending on the project, suitable tools are selected for the project, as well as the necessary methods for carrying out the project. We use the Scrum Method in several projects: the method enables agile development, and works very well in the current situation, where work takes place largely remotely.

We use new technological equipment and solutions in several projects. For example, we get to implement different virtual reality environments and also look for new solutions to take advantage of virtual reality. However, virtual reality is just one of the new technologies we take advantage of. In addition to VR, we can utilize artificial intelligence, machine learning, sensors, motion platforms and controllers in our projects, amongst other things. We are always striving to find new ways to leverage technology in our operations to get the best results.

While we each have our job roles in the lab, that doesn’t mean the programmer just needs to code, or the modeler to model. Depending on your own skills and state of mind, you can participate in various tasks in a variety of ways. There are project planning, articles, presentations, development assignments, internship guidance, technical leadership, workshops, webinars, and many other tasks important to our operations. Cliché, but true; no workday is similar and no project is similar.

Some of our staff also work as part-time teachers in ICT training. It is rewarding to be able to share our know-how with future engineers, and additionally, through it be a laboratory that makes its own contribution to the development of the area. In the upcoming summer 2021, we are once again involved in arranging the Summer Game Studies in the Lapland UAS. The planning phase is ongoing and the greatest Summer Game studies are on the way! Be sure to follow the FrostBit publications, as we will definitely report the mood from the SGS during summer.

Go check out our projects in the lab Portfolio

26.02.2021

SHARE IN

Toni Westerlund, Project Manager

There is no problem that Toni will not solve. Toni is a raw professional in software engineering and programming: “propeller hat” is certainly the correct name for Toni. Gaming technology and virtual reality are especially close to Toni’s heart. He also enjoys teaching, which is more subtle than sharing his own knowledge and raising new propeller hats. Toni leads FrostBit’s Game & XR Team and also works as an alumni manager for the ICT-field in LapUAS.

Master of Engineering

Visibility for Mining Industry with Gamification

There are as many as 46 mines in Finland, with more than 5,000 employees in 2018. Especially in northern Finland, the mining industry is an important employer for many and is an source of livelihood in areas where diverse employment and welfare development can be otherwise challenging (Kaivosteollisuus.fi). There is also a wide range of education possibilities of mining industry throughout Finland, which is why the reform of degree structures and the development of modern learning methods are topical due to the growing demand of the field.

Since mining is such a practical field, how could its education be modernized? The Migael-project answers this question with gamification and modern technologies. The aim of the project is to develop mining education by creating a virtual learning environment with a variety of scenario-based exercises. By the beginning of 2021, three exercises have been completed in the project and the fourth exercise will be finalized during February. Each exercise focuses on different “scenarios” in the open-pit-mine as well as in the underground mine. For each exercise, different platforms and technologies have been utilized to achieve the learning goals:

Charging/Blasting
In the first exercise, the player will be able to practice the charging of explosives in 2D view, as well as view the simulation of charging and blasting in both 3D- and Oculus Guest virtual views. In the exercise, the player gets to simulate charges of different sizes and see what their blasting effect would look like.

Workcard
The second exercise was also created to be played with Oculus Guest’s VR-glasses. In this exercise, the player will be able to do an occupational safety card inspection at an underground mine. The player is able to move around the underground mine and observe the necessary safety measures and fill in the occupational safety card accordingly.

Driving departure
The third exercise was created to be played on both Oculus Rift VR-glasses and Android phones, and this exercise will go through a mining vehicle run-in inspection. The player must check the worker’s safety equipment and the condition and usability of the mining vehicle before it can departure.

PRE- and POST-blasting safety measures
In the fourth exercise, the player performs the pre-blast and post-blast safety measures. The exercise was made into a more traditional 3D-desktop “serious game”, since it is possible to make more use of textual teaching material on a wide range of topics. The player gets to observe objects in the open-pit-mine environment to secure the area before and after the blast.

Different technical implementations enable different ways of presenting or simulating the subject to be taught. VR-technology is able to take a player or learner to a realistic learning situation that responds as much as possible to a real life situation. In this way, for example, training situations can be usefully simulated when the implementations in real life would be demanding, costly and often dangerous. Implementation on a mobile platform establishes easier learning regardless of time and place, and this can help to reach a wider group of users. Therefore, in the third exercise, for example, the driving departure was implemented for both VR glasses and mobile.

However, virtual glasses create their own challenge if the material and subject area is extensive and requires more than observation and “tangible measures.” While instructions for “floating texts” can be established on virtual glasses, interactive texts combined to player movement and other activity can create an unnecessary challenge in the VR-world. For this reason, the fourth exercise of Migael was created as a 3D-desktop game, as the topic required a lot of text in the form of teaching material and other information. In this exercise, it was necessary to plan particularly carefully which operations should and could be “gamified”, as there are several safety measures related to blasting. Some of the most generic and difficult operations were implemented in the form of cinematic transitions and instructions given by the game’s characters:

Thus, different technological implementations can enable different ways of learning and achieve certain learning goals. Some technologies allow for a more realistic simulation of exercises, while others allow for a broader presentation of the content and the possibility to reach a larger target audience. The exercises produced in Migael form a gameful learning environment where you can experience the benefits of many new technologies and learn about the mining industry in an inspiring and safe way. The aim of the project is to make the mining industry more visible and interesting, especially for students, and to establish important and safe training regardless of time and place.

In addition of the fourth exercise being completed soon, the early part of the year 2021 is eventful for the project: a Teams-webinar titled as ‘Utilization of modern technologies in the mining sector’ will be held on 25.01.2021 as part of the project. In addition to the FrostBit lab, the webinar will feature speakers from VTT and Kajaani University of Applied Sciences, who will present, amongst other things, the possibilities of gamification, VR / AR technologies and new sensor technologies in the mining industry. Mining staff, project staff and students of Lapland University of Applied Sciences will be invited to join and listen to the webinar.

The project’s activities will be updated on its official website, which includes a blog and the downloadable exercises: www.migael.fi

In addition, see the FrostBit portfolio-page of Migael: https://www.frostbit.fi/en/portfolio/migael-en/

The northen Ostrobothnia of Centre for Economic Development, Transport and the Environment has granted 337 502 € from the European Social Fund (ESF) and state funding for the Migael-project. The total cost of the project is 450,002 €.

References

www.kaivosteollisuus.fi/fi/kaivosala-suomessa

15.01.2021

SHARE IN

Tuuli Nivala, Specialist

Tuuli works as a specialist in FrostBit at the LapUAS. In her current role she utilizes her game pedagogical knowledge in game design, as well as participates in project planning and some 2D graphic design. She also does marketing and AV-media material for the laboratory.

Master of Media Education, Lapland University

Cross-platform mobile development, the Flutter experience

FrostBit Software Lab has followed the development of the Flutter platform (created by Google) with great interest for the last few years. Our laboratory has also actively applied Flutter in various projects to enhance the development of our mobile applications. The need for an efficient cross-platform technology has been great, since producing two separate native mobile applications (Android + iOS) has proven to be too expensive and inefficient for our purposes, whether considering the development or the maintenance phase.

The strength of native mobile applications lies in the possibility of total customization and the native access to the features of the mobile phone (camera, sensors etc.). On the other hand, the weaknesses of the native mobile application approach are the need for a sizable human resource and the maintenance challenges provided by the rapidly evolving mobile phone ecosystems.

Previously our laboratory has created applications by using the PhoneGap –platform, but the limitations it provides to mobile application design proved to be too great for the technical needs of the FrostBit Software Lab.

Google released the first version of Flutter in the year 2017, and it immediately sparked some interest within the developers at FrostBit Software Lab. At that time, we had two sizable mobile projects upcoming in our project calendar, so we bravely decided to try Flutter out in both projects to avoid the challenges that come with the development of two separate native mobile applications.

https://en.m.wikipedia.org/wiki/File:Google-flutter-logo.png

Going towards the year 2021, we are tentatively yet extremely positively surprised and optimistic when it comes the possibilities of using Flutter in creating cross-platform applications. We are certainly going to use it further in upcoming projects as much as we can.  Then again, nothing in this world is perfect, and Flutter also shares its weaknesses, despite all its strengths. For this purpose we have gathered our own experiences on Flutter from the past few years, and here’s our conclusion of what we think of Flutter so far:

Flutter strengths / pros:

  • Development is easier when compared to PhoneGap or Xamarin, for example, since Flutter needs less platform-specific code
  • Flutter is cross-platform; it’s possible to develop applications for Android, iOS, Windows, Linux, MacOS, web etc. at the same time
  • It’s fast to develop applications on Flutter
  • Flutter allows the developer to create complex and fully customized UI components, since everything related to layout can be altered
  • Flutter has good documentation and a great number of examples are available. Flutter’s userbase also grows rapidly all the time
  • Flutter is powered by a very performance efficient 2D-UI engine (sky_engine), which works well on a 120Hz display as well
  • Flutter uses the Dart-programming language, which is an easy-to-learn OOP language
  • Google is a huge organization, which has the resources to develop Flutter efficiently. Google also uses Flutter in some of its own products
  • Flutter-code will eventually compile into native application code on different platforms and devices
  • Flutter already has good development tools, for example, for Android Studio and Visual Studio Code
  • Flutter is rapidly being updated

Flutter weaknesses / cons:

  • Complex user interfaces can be tedious to develop
  • Since Flutter is updated frequently, a great number of changes are integrated into the platform continuously, which can make large projects difficult
  • Both Dart-language and Flutter are relatively new technologies, because of which they are also rapidly being updated. This problem will fix itself in the future though.
  • Many of the best features of Flutter are still in development stage (null safety, web and desktop application support etc. )
  • If Flutter does not have a plugin for a certain platform specific feature (Android or iOS), you will need to make platform specific code
  • Because of Flutter’s popularity and its relative easiness for application development, there are numerous third party plugins which can have questionable quality
  • Since Flutter is a new technology, so called “best practices” and recommended architectural designs haven’t been really formed or standardized

In conclusion, we think Flutter has way more strengths than weaknesses, most of which are related to the fact that Flutter is still a new technology.

If we also take a look into a few internet articles related to this subject, we can conclude that many other developers in the world have similar thoughts with us (e.g. Rozwadowski 2020;  Costa 2019; Sannacode 2020;  Powalowski 2019). According to these articles, Flutter weaknesses also include the large size of the mobile application as well as certain compromises that are related to the mobile application’s layout and user interface design recommendations of different platforms (e.g. material design on Android or iOS). Here at FrostBit Software Lab, we do not consider these weaknesses to be too problematic in our projects for now, however.

We have used Flutter in two major projects, first being “Arktori” and the other being “DWELL”. In the Arktori project, we are developing a mobile application, with which the rectors of the northern part of Finland can network with each other, as well as enhance their professional knowledge and also mentor each other in their daily work. In the DWELL project, we are developing a mobile application, which promotes and endorses communal living in apartment buildings. The pilot apartment building in DWELL project is the DAS Kelo –student dormitory. We have been quite happy with Flutter in both projects when it comes to mobile application development.

Some screenshots from the Arktori and DWELL mobile applications:

Arktori mobile application (Flutter), development version, December 2020
Arktori mobile application (Flutter), development version, December 2020
DWELL mobile application (Flutter), development version, December 2020
DWELL mobile application (Flutter), development version, December 2020

Flutter has proven to be a powerful tool to create cross-platform mobile applications. However, it remains to be seen, how flexible and efficient Flutter will be in the future when it comes to web and desktop applications. If Flutter will be truly competitive outside mobile applications, it will be a realistic scenario (from the software developer’s point of view) to concentrate mostly on Flutter when developing applications, only to be supported by other technologies in cases, where Flutter is not an optimal approach.

For now, we are not completely certain here at FrostBit if Flutter is the all-powering efficient application development platform for all kinds of applications, but no one can deny that they surely are trying hard to do so!

In any case, we are going to continue watching the development of Flutter with great interest, and are always ready to try out Flutter in new projects and use cases in the future!

11.12.2020

SHARE IN

Tuomas Valtanen, Project Manager

Tuomas works as the web/mobile team leader, a software engineer and a part-time teacher in FrostBit at Lapland UAS. His tasks include project management, project planning as well as software engineering and expertise in web, mobile and AI applications.

Master of Engineering


How can immersive technologies (VR/AR) foster meaningful education and travel experiences during the pandemic?

1. VR in teaching at Lapland University of Applied Sciences (Photo credits: FrostBit Software Lab)

Often, challenging times bring resilience and inspiration for creative solutions. Now more than ever, digitalization and technology have kept us going and connected; nevertheless, we all have experienced the effects that a long day of working or teaching with digital tools has on our mental and physical health. For numerous reasons, working and connecting through a flat-screen is simply not enough.  As humans, we need more; we need to connect in a meaningful way. These challenges and difficult times have fostered the rise of creative digital solutions, especially immersive technologies (virtual and augmented reality, VR/AR) since they enable us to travel without the need to leave our home or perform tasks that we will not be able to complete otherwise. Is this the right time for VR/AR to be the next digital tool and tech disruption that will elevate the way we work, connect, and travel? All the odds are in favor, as well as the hype.

How can VR/AR add value to education?

Education is definitely one of the areas that VR/AR technologies can create a significant impact based on the fact that most of the teaching and training is conducted remotely at the moment. The current teaching and learning methods through online lectures and classes lack interactivity and versatility; hence, immersive technologies can enhance the way we teach and learn. Think about the possibility that a teacher would put a VR headset and ‘teleport’ worldwide during the geography classes using Google Earth. Similarly with other school subjects, students can have a virtual visit, which is immersive and engaging, for instance, to the Colosseum of Rome during the history classes. A recent Stanford University study conducted with middle school students examined the differences in cognitive learning by comparing the usage of desktop and VR videos to teach about crystal reefs and ocean acidity. The study showed that the students who watched the VR videos had higher scores in learning compared to the students who watched the desktop videos. In addition, the students who experienced VR videos about the coral reef showed higher scores on the perception of self-efficacy compared to the group who watched the desktop videos.

2. Virtual visit of Florence, Italy through Google Earth (Photo credits: Google Earth VR, Steam)

One major drawback is the costly prices of such technologies, although the devices are becoming more affordable and of higher quality, as time passes. Besides, teachers in the universities and especially high schools, have limited access to the technology and VR/AR devices because specific R&D or research groups mostly use them. An additional significant factor that has prevented the widespread of VR/AR technologies in teaching relies on the fact that teachers are often scared to implement new technologies in teaching, ensuring that they complement the curriculum goals. What is the starting point then? Being curious and embracing innovation in teaching! In the following steps, have a general understanding of the technology and how you could utilize it in teaching. Ideally, if your school has a dedicated lab or environment for VR/AR, that is the first place to go. Ask for a general introduction and possibly request to borrow one device that you can explore by yourself. If that is not possible, then there is a lot of information about the role of VR/AR in education online. Moreover, if your organization does not have professional devices, you can start by experimenting with more affordable options in the market, for example, Samsung Gear VR. Entering the world of immersive technologies is still a confusing pathway but fortunately, the accessibility to the VR/AR information and technology is increasing day by day.

3. Student painting in VR

Which are the subjects that VR/AR can play a critical role in at the moment?

Healthcare

Due to the complexity of the situation regarding the physical distancing, healthcare is one of the sectors that need solutions that would enable remote collaboration and simulations. Several studies have proven the effectiveness of VR/AR in healthcare training and simulations; however, the employment of immersive technologies in healthcare is in the very early stages. According to the Gartner Hype Cycle for Digital Care Delivery including Telemedicine and Virtual Care, VR/AR for care delivery are ranked as ‘on the rise’ technologies, indicating that the real potential will evolve simultaneously with the technologies and applications. Therefore, the XR (extended reality, VR/AR/MR) simulations in the healthcare sector are estimated to have a market value of around 850 million euros by 2025, followed by a significant improvement of the technical and market value chains globally.

Considering the importance of immersive technologies in enhancing learning and safety in the healthcare sector, Lapland University of Applied Sciences has applied for international funding for a project which will create cutting-edge VR/AR solutions for healthcare education and medical staff. Our goal is to make healthcare training and simulations accessible to the students and professionals, enable hands-on experience which is safe and enhances learning. Another recent study indicates that when medical students train in VR, they scored better in all categories compared to the traditionally trained group, especially regarding information retention. The total test score showed an overall improvement of 230%. The future of healthcare training and simulations is surely based on immersive technologies.

Industry Training and Simulations

Similarly, with the healthcare sector, enterprise training and simulations is another critical sector where immersive technologies can be a solution for the disruptions caused by the pandemic. Long-term remote work has called for solutions for remote training and simulations. XR training and simulations for different sectors of industry is one of the key areas of expertise at the FrostBit Software Lab. We have developed VR training and simulations for the mining industry, renewable energy production, reindeer herding, real estate, and forestry.

4. Creating the digital twin of the district heating power plant (Credits: DUKE project)

The on-going DUKE project will develop a digital twin (digital representation) of the district heating power plant in Jänkätie, Rovaniemi, which will give students and new operators hands-on experience on the operation of the plant without the need to physically be there. A cost-effective and safe solution, which will make learning more accessible and effective for the students.

Tourism

The tourism sector in Lapland has been massively impacted by the pandemic and the negative effects are predicted to be long-term. While hoping for a better future, very few companies have shifted their attention towards innovative and creative solutions to sell their experiences. That might be due to the lack of awareness about the possibilities on the market, technology readiness, or the costly technological solutions. This is a time when the travel industry should adapt and look for creative and innovative solutions to reach their customers. XR solutions will not solve all the problems, but they will give hope and new markets. How can we create an immersive experience of Santa Claus Village, Northern Lights, or Lapland Landscapes without the need to physically be here? Surely, it is a complex task but the technology is here. We have to step up and implement it. The Amazon Explore platform was launched to provide people with virtual traveling experiences. You can pay to have a tour with a private guide in different cities of the world, such as having a virtual walking tour through Mexico’s city urban art scene. The experiences are video and desktop-based; thus, not immersive. Traveling experiences are immersive and VR leads to immersion.

Fostering distance collaboration through immersive technologies – the Arctic perspective on XR at VR Days 2020

This year, we were invited to share our XR expertise and use-cases at the most comprehensive VR/AR event in Europe which this year was held remotely. VR Days blends different fields in which immersive technologies are applied such as business, art, training and simulation, education, hardware, funding etc. This was a major step for FrostBit Software Lab and Lapland University of Applied Sciences to feature among the most influential individuals and companies of the XR industry such as Oculus, HTC, Facebook, Google and much more. As the first Finnish VR laboratory, FrostBit has a long history of solving-real life challenges with immersive technologies. For instance, creating a virtual graveyard experience for Salla’ museum of War and Reconstruction which would allow the visitors to access the German soldier graveyard located in the Finland-Russia border area. The VR experience enables an authentic graveyard visit without the need to go through the border control between the two countries. Check out the speech for VR Days 2020 below:

Erson Halili speaking at VR Days 2020 New Horizons (Credits: FrostBit Software Lab)

What to consider when planning and creating immersive experiences?

The quality and the content of the VR/AR experience can vary on the desired outcome of the specific experience; however, there are certain key steps that are critical when planning a VR/AR project. Most importantly, the combination of engineering, psychology and education are essential in creating meaningful VR/AR experiences. My perspective when planning XR experiences: combining cognitive psychology, media education and user-experiences. What makes a VR experience meaningful? Consider these tips:

  1. The XR experience should solve real-life problems. Although that might not always be the case, technology will solve a problem that is not possible to be solved otherwise.
  2. Employing user-centric design. The XR experiences should emerge from the users and carefully designed with them.
  3. Strive for meaningful experiences. Meaningful experiences engage the user and there is a clear intended outcome at the end of the experience.
  4.  Make sure you combine multi-disciplinary teams and skills. As stated above, combining multi-disciplinary teams will possibly ensure that the three other points are considered. Multi-disciplinary teams and skills lead to holistic user-centric experiences. That is a core strength we have here at FrostBit, where we proudly co-work with a diverse team of engineers, educational specialists, designers, artists, etc.
6. Visitors experiencing VR at FrostBit Software Lab

We, here at FrostBit Software Lab (Lapland UAS) are on a mission to make technological solutions accessible to the community. Therefore, we are organizing info sessions with teachers, educational specialists, decision-makers, and companies in Rovaniemi and Lapland region on how to utilize immersive technologies during the pandemic. We want to give our support to overcome the barriers caused by the pandemic and we believe that immersive technologies are a powerful assisting tool. Are you ready to explore VR/AR technologies and understand how it can be of help in your subject or business? Send us a message in advance to have a personalized meeting and demo. For teachers at the Lapland UAS and the University of Lapland, you can pop-up anytime at the FrostBit lab facilities.


Written by Erson Halili

18.11.2020

SHARE IN

References

Hakkennes, Sh., Craft, L., Jones, M. (2020). Hype Cycle for Digital Care Delivery Including Telemedicine and Virtual Care. Retrieved on November 15 from: https://www.gartner.com/en/documents/3988593/hype-cycle-for-digital-care-delivery-including-telemedic

Muller Queiroz, Anna Carolina & Nascimento, Alexandre & Tori, Romero & da Silva Leme, Maria. (2018). Using HMD-Based Immersive Virtual Environments in Primary/K-12 Education. 10.1007/978-3-319-93596-6_11

Pottle J. 2019. Virtual reality and the transformation of medical education. Retrieved on November 12 from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6798020/

Digital Twins with Game Engines

A digital twin (Digital Twin) is a virtual model of a product, process, or service. It can be used to visualize the operation of the object and its related information. Digital Twins for Leveraging Renewable Energy (DUKE) project, funded by Lapin Liitto and executed by the Lapland University of Applied Sciences, has its first functional pilot done in the Lapland Education Centre REDU’s educational heating plant in Jänkätie, Rovaniemi. It will be implemented with the Unity 3D game engine.

The game engine is powerful

The strength of game engines is the efficient and fast implementation of the visual environments. They are particularly well suited for 3D modelling. In addition to the visual model, the digital twin contains functionalities that can also be found in the real-world counterpart. The most straightforward functionalities to be modeled are, for example, details related to the mechanical properties of the system. In the visual model of a heating plant, mechanical objects to be modeled include, for example, modeling of building doors, inspection hatches, manually adjustable valve handles and electrical switches. Modeling doors and inspection hatches involves a visual effect that the observer sees after some input information triggering the door opening has been provided. The input information can be a virtual operation of the user to open the door or some movement of the cursor by the user in the vicinity of a suitable visual key element.

The functional model can be implemented with a game engine

There are numerous different options for implementing a key element or measure. Behind the door, a view opens to another space and through the inspection hatch, a view inside, for example, a boiler. Depending on the operating mode of the system, the boiler can be passive or produce heating power. When the heating is active, the visual effect after opening the door can be, for example, a visible combustion reaction inside, i.e. flames of fire. In the real world, opening the inspection door may also be accompanied by an emergency alert or other similar process-related action. In a real-world situation, a cross-effect, for example, on a malfunction of a vacuum that takes care of the pressure difference in the furnace, can also cause smoke flaking out of the inspection hatch. As a combination of the functional scenarios of the digital model, a description of the process is created, which in the case of game engines has to be modeled and programmed with the accuracy and to the extent required in the model.

In addition to the visual effect, sufficient realism may also require the creation of sounds that corresponds to reality. The sound associated with the opening of the door itself and the sound world carried from another space enhances the immersion of the model, i.e. the sense of presence or realism felt by the user. The concept of immersion comes from the gaming industry and especially from experiencing virtual reality.

Producing a functional model requires modeling the process

Producing a functional model of a complex process requires an understanding of the basics of process modeling. Systems combining mechanical, thermodynamic, hydraulic, and electrical subsystems, for example, have typically been modeled by describing the subsystems individually or with tools suitable for process modeling, such as Matlab Simulink, Labview, or Ptolemy II. In these simulation programs, it is possible to implement a time model required by the processes.

The functional model of the process describes the progress of the process in real time

In the time model, the time of the process progresses in real time or even faster than the actual rate of occurrence. The subprocesses take place in order and by mimicking the chain of events of the actual process so that the dependencies, i.e. causality, are realistic. In this case, for example, the fuel entering the boiler burns and, as a result of the combustion, thermal energy is transferred to the liquid circulating in the system. In a district heating plant, the circulating liquid is water and, for example, in ground source heat pumps, a water-ethanol mixture. Modeling of the time model and process progression can be implemented in its simplest way by thinking of a stable operating point for the system, where all the functional elements to be modeled in the system have settled into a stable state.

For example, in the modelling of a heat plant, the steady state is one in which the temperature of the circulating liquid, the volume flow of the circulating water in the circuit, and the pressure differences in the piping and boiler systems are in a stable state. The heating plant model has two separate water circuits connected by a heat exchanger. Through the heat exchanger, the thermal energy of the primary water, i.e. the water heated in the boiler circuit, is transferred to the secondary circuit, i.e. the district heating network.

Progressing the process over time is worth doing simply

The functionality of the sample heating plant was modeled so that at the start of the simulation, all variables, i.e., temperatures, pressure differences, and flows, have zero initial values. The calculation of the stable operating mode starts from the pump of the primary or boiler circuit and first the pressure difference produced by this power is calculated on the basis of the power set on the pump. The calculation of the pressure difference is started from the numerical value zero and the pressure differences of all elements of the circuit are calculated in small steps and at each cycle the pressure is raised to a state where the pressure difference in the pump and the set pump power theoretically correspond to each other. The pressure information produced by the pump is transferred in each revolution to the next element in the primary circuit which is the pipe. In the simplest model, the pipes transfer the pressure difference directly forward to the next model as is, so there are no pressure losses in the pipes. The pressure drop for the pipes can of course be set as a parameter, but in the simplest model it is not necessary.

First, the static state of the process is determined

The calculation proceeds element by element and the pressure difference moves forward for each circuit element. The elements form a system in which, viewed in the direction of rotation, they have a pressure difference as input data and a pressure difference after the element as output data. If the element is a valve, the numerical value of the valve position as input data is checked and based on this the pressure change produced by the valve is calculated and passed to the next element as input data. When the entire circuit is computationally rotated element by element, the calculation proceeds to the second stage, where the flow and temperature losses of the water flowing in the circuit are calculated on the basis of the pressure difference, if they have been defined for that circuit element. The temperature of the liquid circulating in the circuit is determined on the basis of the boiler power data and the volume flow of the flowing water. The pressure differences, flows and temperatures of the elements of the secondary circuit are calculated in a similar way. In the secondary circuit, the calculation is started in the same way as in the primary circuit for the pump producing the pressure difference. There is no heating boiler in the secondary or district heating circuit, but it corresponds to the output connection of the heat exchanger, which produces the heating power of the primary circuit to the secondary circuit.

The static model is sufficient for some of the simplest modeling. A more realistic model also includes modeling the dynamic properties of the system. Dynamic properties are the change phenomena that occur in the system. Such dynamic properties include, for example, increases and decreases in boiler power and flow changes in the secondary or district heating circuit. Modeling the dynamic properties leads to solving the differential equations describing the system. The simplest model of the differential equation is the so-called first-order differential equation, the mathematical solution of which is an exponentially descending or ascending graph. For example, the boost circuit boost can be modeled with a first-order differential equation if, as a result of the boost, the circulating water temperature rises to some new temperature value without the temperature behaving unstably. For example, temperature oscillation is an example of instability. A first-order differential equation is often a sufficient model for the dynamic properties of a system.

The phenomena of change, i.e. dynamic properties, are modeled with differential equations

The functionality of a simple dynamic model can be determined, for example, by determining the time during which the temperature rise occurs, i.e. the time constant of the temperature rise rate, by means of a step response test. By determining the time constant, the temperature rise corresponding to the power increase and the delay in the system, i.e. the so-called dead time, a dynamic model sufficient for most practical cases can be implemented. The dynamic properties of the sample heating plant were modeled by performing the test runs of the heating plant on 25.5. – 27.5.2020. During the test times, e.g. stepwise boiler circuit power increases and district heating circuit flow changes. During the changes, changes in the temperatures and flows of the primary or boiler circuit and the secondary or district heating circuit, as well as pressure differences, were measured. Based on the test measurements, the dynamic properties of the heating center were determined using regression analysis methods. The models of dynamic properties calculated on the basis of test runs and measurements are transferred to a functional model describing the properties of the system by programming them in the game engine into the program code of the model. The programming language is the Unity 3D game engine C#.

The visual and functional model combines into a digital twin

The final digital twin combines a visual model and a functional model describing the functionality of the heating plant process into a single entity. The goal is to produce the desired functionalities feature by feature. Each increase in functionality develops the system in a more usable and real-world direction. The digital twin, in its more advanced form, can be in real-time communication with its physical counterpart, so that operating operations through the digital twin can even control its real-world counterpart or alternatively the real-world thermal power plant functional change is realistically reflected in the digital twin. For example, so that the popping of smoke in the real world is heard and seen in the virtual world. The smell of smoke in the virtual model is still a fictitious feature for the time being, but in the future, perhaps that too can be implemented.

Official name of the project: Digital twins for leveraging renewable energy
Project timetable: 01/01/2020 – 31/12/2022
Total budget: 761 732 €
Funding: EAKR 2014-2020
Contact person: Tauno Tepsa (+358 40 821 6865)

16.10.2020

SHARE IN

Tauno Tepsa, Senior Lecturer

Tauno’s work is divided into teaching and working as a Project Manager. He works in the DUKE project as a Project Manager. Tauno’s special areas are Electronics, Embedded Software, Automation and IoT.

DI, MSc(Eng.)