Category: Article

In the autumn of 2022, thousands of berry images were stored among the large image metadata, the machine learning model was finalized and the final project seminar was kept. Berry machine -project ended and it was a fascinating journey into practical machine learning.

The Berry machine project was financed by Interreg Nord with Natural Resources Institute Finland (LUKE) and the Norwegian Institute of Bioeconomy Research (NIBIO) participating, Frostbit Software Lab at Lapland University of Applied Sciences as a developer. The project goal was to study the possibility to use machine learning to make the current berry harvest estimation measurement process less dependent on manual labour, thus bringing up the possibility to do significantly more field measurements and open up new ways to estimate berry yields.

Current berry harvest estimation measurement is done by manually counting every single berry inside a frame created with various materials (Kilpeläinen 2016). The frame forms precisely one m² area. In the forest, five frames are installed in places with varying ground types (marjahavainnot.fi 2022). After all of this manual work, data is still not exhaustive enough to do more elaborated studies as the process is both expensive and laborious (Bohlin 2021).

The Berry machine project studied the way how to use deep learning to help count the berries and thus make the process significantly less laborious. The plan was to use deep-learning computer vision to detect berries and create a functioning prototype system with a mobile application and all required backend applications.  Only bilberry (Vaccinium Myrtillus) was required as the initial target species, but lingonberry (Vaccinium Vitis-idea) was also included to give a more comprehensive view of how the berry machine system would work across species. Both of the berries were counted in growth phases “flower”, “raw” and “ripe”.

The main steps to fulfill the Berry machine project task were to train a computer deep learning model to detect berries from images. Using this computer vision, berries were counted from images and the detected berries count was compared to the number of berries counted by field measurement.

Traditional Artificial Neural Network (ANN) is formed with input neurons, hidden layers and an output layer (Figure 1). In the case of image classification, input image pixel values are added as values in the input layer (Gogul 2017). Weight and bias values in hidden layer neurons and connections are derived in the training process. Weight and bias are then used to recalculate new values for input values when all original pixel values propagate from left to right arriving finally at the final output layer representing the result in numerical form.

Compared to the ANN, image classification results are significantly improved changing to breakthrough technology, Convolutional Neural Network (CNN) (Marmanis 2016). CNN architecture improves ANN architecture by adding convolution and pooling layers. New layers simplify and extract features of the image while keeping the feature position with the original image (Figure 2). Pooling layers sub-sample the original image and feature extraction layers detect more high-order features from the image. With CNN architecture, computer vision is capable of human-level or higher accuracy (Galanty 2021).

CNN architecture on its own only classifies one image as a whole. The project goal with computer vision was to detect multiple instances of berries in various growth phases from a single image. Therefore object detection architecture is required instead of image classification architecture as object detection also seeks to locate berry instances from their natural background (Liu 2018).

Object detection architecture YOLO (You Only Look Once) was used in the project. YOLO was designed to be fast, simple and accurate (Redmon 2015). The YOLO process splits the image into a grid and detects objects in each grid and predicts the bounding boxes. Bounding box results are joined with the classification probability of each grid to filter more accurate results (Redmon 2015).  The YOLO process is visualized in Figure 3. YOLO version 5 was used in the project as it was built using PyTorch framework (Solawetz 2020) which did make it easier to implement. As a bonus, YOLOv5 added data augmentation to the training process out of the box (Benjumea 2021).

A deep-learning training dataset was formed by taking thousands of berry images. Approximately 10 000 images were collected using mostly consumer-level mobile phones. One of the ideas in the original project plan was to get closer to crowdsourcing the berry harvest measurements with an ease-of-use mobile application. Therefore training was to be done mostly using phones available to the general public.

After and during the dataset collection, the annotation process was initiated. The annotation process was very labour-intensive and not all images were annotated due to limits in resources. The annotation process is simply manually drawing the correct berry class on top of the image. A significant difficulty was that berries are small and especially raw berries hide well in the background. After the main dataset of annotated images was complete, additional annotation processes were initiated to balance the dataset. The balance dataset is a training dataset with a roughly equal number of instances from each class. The final dataset was formed of six classes shown with examples in Figure 4.

The density formula is the number of instances divided by area. Computer vision was used to estimate the number of berries but the surface area was missing. The field measurement counting frames were also annotated for density estimation as in Figure 5. Frame polygon was used to crop deep learning detected berries and thus all berries left in the image were inside from a one m² and 0,25 m² area. Two different sizes of count frames were used to allow comparing detection correlation differences in images taken closer and further off the ground.

The Berry machine tasks were visualised in Figure 6. A significant amount of work was used to analyse and improve iteratively training results. Improving training results meant that more annotation was done as sometimes data was not balanced enough. The computer was not the only one with difficulties detecting berries as annotating was also difficult due to a larger number of obscure berries barely visible in images. It is likely that if personnel was directed to annotate the same images, results would vary.

The main dataset processing before the training was the process of splitting the image into smaller pieces. The YOLOv5 model used in the project had a native resolution of 640×640 pixels. Inputting a larger image meant downscaling into the native resolution. As images were taken with, for example, 4032×3024 resolution, significant loss of detail would occur which is detrimental when detecting small instances like berries. Therefore all annotated images annotated in their original resolution and all images from where berries were detected, would be split into smaller images before detection and training. A complex splitting algorithm with overlapping zoom levels was initially used, but the splitting process was reverted to a more simple split process as simpler process advantages were more numerous than using more complex methods. Once again, simplicity wins.

Training iterations were not exhaustively laborious by themselves, but with deep learning, improvements were done by modifying the dataset and the dataset pre-processing which was time-consuming. Project time was also used in post-process as training results required analysis and detection quality checks in some cases using a benchmark dataset.

The final training dataset of image pieces normalized to 640×640 pixels used for training was 41 101 images with 95 065 annotated berries. Dataset was supplemented with synthetic data created randomly adding clipped berry images with transparent backgrounds on top of various background images. Background images were mostly images taken around the campus and plain white background images. Figure 7 visualizes the final balance of the dataset. Balance was deemed sufficient based on the graph.

As presented in Figure 8, the project’s backend architecture was made into four separate services. The prototype backend was the main backend for the prototype mobile application. The prototype backend’s main tasks were to add login property, act as image storage for images uploaded and store location and result values. The dataset inference process service’s purpose was to batch-run the detection process to thousands of images and save results for data analysis. The resulting metadata of detected berries was used for the density estimation study. The pipeline backend’s main task was to split the image and request detection backend for each split image. After all the image slices were executed through the detection backend, the detected berries’ position in the image slice was converted to the original main image. Finally, all results were sent to the address defined in the original request. The final backend was the inference backend which ran the model detection with the image received and responded with the detection results.

Prototype application development was initiated after the first viable detection model was trained. The main architectural decision with the prototype application backend was to build the actual deep learning backend (inference backend) into a separate service. The decision was made to allow separate backend deployment on various Graphical Processing Unit (GPU) or non-GPU servers. In addition, the image split backend was developed into a separate service to minimally modify the standard prototype backend used by the prototype application. This was to make it easiest to deploy the prototype backend using an existing headless API and allow using the image split-enabled detection backend (pipeline backend) past the prototype backend presented in Figure 8. Modular design as architectural design separates concerns more reliably, but in practice, it was noted, that deployment of multiple services on each edit slowed the process significantly and thus increased complexity.

The prototype application was developed using a multiplatform Flutter framework that enables developing for both IOS, web and Android using the same codebase (GitHub.com 2022). On measurement location, the user created a new location from the UI. The location user interface is visualized in Figure 9. The location page showed the estimated berry density among other information. The target berry phase and species were defined by the most prominent berry detected from multiple images. Density was estimated as the average of all images and corrected with a linear regression formula derived from the correlation analysis. Surface area estimation for density calculation was tested with two different methods. First, estimate the area from the berry average size. The second method was to estimate the area using augmented reality (AR) libraries. AR was implemented only in Apple IOS devices.

In the project, multiple different results emerged: Deep learning detection model results and density estimation results. Detection results are covered first. The results are from the last training iteration.

In the YOLOv5 deep learning process dataset of annotated image pieces was split into three parts. Training test and validation. The training dataset was used to train the model, the test dataset was used to test the model’s progress during training and the validation dataset was to do final testing after the training was done. Validation dataset results indicated a detection rate from 83% to 93% depending on the class. Results are visualized with the confusion matrix in Figure 10. The confusion matrix depicts percentages of predicted detections matching the true values. Reading the confusion matrix in the case of class bilberry ripe detections, 1% was bilberry raw, 91% correct and 17% of the ripe bilberries detected were part of the background. According to the confusion matrix, ripe lingonberries were the most detected class while bilberry raw had significantly worst results. Overall, the results were deemed acceptable.

For the density estimation study, the trained model was executed to detect berries in nearly 5000 images with one m² and 0,25 m² frame annotations. Frame annotation was then used to crop detected berries and only inner berries were kept (Figure 11).

estimated density = (number of detected berries of the correct type) / (frame surface area)

Equation 1. Calculate the estimated berry density

For density calculation, the surface area was defined by frame area and the number of berries was the cropped value of deep learning detected berries of the same type as calculated in the image. Every image file path defined the target berry species, growth phase and location. This meta-information was used to filter only target classes into density estimation. Density estimation was calculated for each image with Equation 1.

The estimated density was not the final result used in the prototype. The assumption was that image will not show all berries present, but berry detection in the image correlates with the field measurements. Linear regression formula was derived using the correlation. For each class, the regression formula in the prototype was different.

Estimated density and field measurement value was plotted into scatter plot for each class separately (Figure 12). Pearson correlation for each class was shown in the scatter plot in the plot title. The Scatter plot was coloured by location to visualize how location affects the relation between detected berries and the field berry count. The Scatter plot shows several points:

• Ripe lingonberry has fewer data and all of the ripe lingonberry images are only from 2021 images. More homogenous ripe lingonberry data hints at the need for more varied data as homogenous datasets can lead to early conclusions.
• To practically use correlation with for example linear regression to estimate density from the image, more variables need to be taken into consideration. Location colouring hints at one additional way to improve correlation by considering growth location.
• Bilberry and lingonberry behaviour were different. With bilberry, possible causes of noise can be more lush undergrowth than lingonberry blocking camera view of the berries.
• Lingonberry was detected as bunches, but field measurement calculated lingonberries as individual berries. The difference between berry count and bunch count is likely to increase with increased yield. Lingonberry may grow more berries in a single bunch when the berry harvest improves.

One method of increasing correlation results was to take all the images taken from the same frame and use only the image with the most berries detected. In theory, only the image with the best angle and quality to capture most berries is used. This required taking multiple images from a single frame. Practical challenges in image quality were sun position with the camera causing difficulty to detect individual berries, camera shake and angle with more ground vegetation in the way. Using the image with the most berries was justifiable and the maximum value image did give a better correlation than the average value. With the maximum value image, correlation with the field measurement value was improved (Figure 13).

Detecting berries in different growth phases using consumer-level cameras and deep-learning object detection yielded successful results. Computer vision with object detection is a highly viable method of detecting difficult instances from the natural background. Invaluable information about practical machine learning processes especially concerning data pre- and post-processing was collected. In the project, large and valuable training data was created from the field measurements to the additional synthetic dataset.

Density estimation was as not simple as many things in nature are. A perfect example of the challenges was shown in Figure 14 with bilberry. High-yield bilberry harvest is accompanied by lush ground vegetation. This inversely lowers the number of detected berries in the image.

The project gave a more in-depth view of the ways to estimate density and how to approach the problem. The main challenges with the density estimation were berry detection, surface area estimation and density estimation with the information collected. Berry detection was deemed feasible, but the other two require more studies.

Surface area estimation was tested in the prototype with two different methods. Using detected berries as a base measurement to estimate image cover area and using the operating system’s augmented reality (AR) framework. AR was deemed the preferable method in the long term. The project prototype proved that AR can be used to measure ground area, but requires a significant amount of software development to be viable. Challenges include detecting flat plane on a practically uneven surface and handling noise in AR measurement caused by constantly moving cameras and implementing AR on both of the most common phone operating systems.

Concerning density estimation with surface area and berry detection results, more research is required. Correlation does exist but is too weak to be practical. Directing users to take multiple images and using images with the most berries detected, is one viable option to improve the correlation, but more data dimensions are needed. Berry’s growth location shows promise as one dimension to factor in.

With the deep-learning process, a considerable amount of experience was collected. Practical experiences in the project correlated with the argument expressed in the article “Hidden Technical Debt in Machine Learning Systems” that Machine Learning (ML) systems have a special capacity for causing technical dept (Sculley 2015).  The article also states that only a tiny fraction of the code in an ML system is for the actual training or prediction (Figure 15). Berry machine project code base supported this statement. A large amount of code was required to modify and create a dataset in the post-process. ML backend deployment with modular design, version verification and generic request and response formats also increased the codebase a significant amount. The final part often overlooked was the analysis process tools to draw conclusions and guide the iterative training process in the correct direction.

As stated by Sculley 2015, Using generic packages in a hope that it reduces the amount of coding required, often results into glue code to join generic packages together. The glue code issue led to the project developing custom packages for project purposes. Version control was one difficult aspect, as the ML system was improved by modifying the large training dataset not managed by a version control system. Thus it is difficult to duplicate results in a specific version without the correct training dataset.

Implementing a real-world ML system can be a hassle. A large number of surrounding infrastructure is required in the system. Infrastructure is not inherently negative, but a valuable lesson is to consider this requirement when creating an ML system. When implementing an ML system, obstacles differ from a normal software project. Before and during the process, developers need to be careful not to end up filling the system with anti-patterns like glue-code, pipeline jungles and abstraction dept (Sculley 2015).

In the end, the Berry machine project was a priceless experience filled with hope and excitement. The project generated numerous internships and launched two bachelor’s theses and one master’s thesis. Practical ML system experience was learned. Hopefully, we will see more ML usage in the future as the potential is tangible.

Benjumea, Aduen, Izzedin Teeti, Fabio Cuzzolin, and Andrew Bradley. YOLO-Z: Improving small object detection in YOLOv5 for autonomous vehicles. arXiv preprint arXiv:2112.11798, 2021.

Bohlin, Inka. Maltamo, Matti. Hedenås, Henrik. Lämås, Tomas. Dahlgren, Jonas. Mehtätalo, Lauri. ”Predicting bilberry and cowberry yields using airborne laser scanning and other auxiliary data combined with National Forest Inventory field plot data.” Forest Ecology and Management, Volume 502. ISSN 0378-1127, 2021.

Galanty, Agnieszka . Tomasz Danel, Michał We˛grzyn, Irma Podolak. ”Deep convolutional neural network for preliminary in-field classification of lichen species.” 2021.

GitHub.com. Flutter SDK. 2022. https://github.com/flutter/flutter (retrieved 7. 12 2022).

Gogul, I. and Kumar, Sathiesh. ”Flower species recognition system using convolution neural networks and transfer learning.” 2017.

Kilpeläinen, Harri. Miina, Jari. Store, Ron. Salo, Kauko Salo. Kurttila, Mikko. ”Evaluation of bilberry and cowberry yield models by comparing model predictions with field measurements from North Karelia, Finland.” Forest Ecology and Management, Volume 363. ISSN 0378-1127, 2016. 120-129.

Liu, Li & Ouyang, Wanli. Wang, Xiaogang. Fieguth, Paul. Chen, Jie. Liu, Xinwang. Pietikäinen, Matti. ”Deep Learning for Generic Object Detection: A Survey.” International Journal of Computer Vision. 2018.

marjahavainnot.fi. ”Luonnonmarjojen satohavainnot.” 2022. https://marjahavainnot.fi/assets/info/Havaintometsan_perustaminen_v2.pdf (retrieved 25. 4 2022).

Marmanis, Dimitris and Wegner, Jermaine and Galliani, Silvano and Schindler, Konrad and Datcu, Mihai and Stilla, Uwe. ”SEMANTIC SEGMENTATION OF AERIAL IMAGES WITH AN ENSEMBLE OF CNNS.” ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences. 2016. 473-480.

Redmon, Joseph. Divvala, Santosh Kumar. Girshick, Ross B. Farhadi, Ali. ”You Only Look Once: Unified, Real-Time Object Detection.” CoRR abs/1506.02640. 2015.

Sculley, D. and Holt, Gary and Golovin, Daniel and Davydov, Eugene and Phillips, Todd and Ebner, Dietmar and Chaudhary, Vinay and Young, Michael and Crespo, Jean-François and Dennison, Dan. ”Hidden Technical Debt in Machine Learning Systems.” Advances in Neural Information Processing Systems. 2015.

Solawetz, Jacob. YOLOv5 New Version – Improvements And Evaluation. 29. 6 2020. https://blog.roboflow.com/yolov5-improvements-and-evaluation/ (retrieved 25. 4 2022).

This article has been evaluated by the FrostBit’s publishing committee, which includes Heikki Konttaniemi, Toni Westerlund, Jarkko Piippo, Tuomas Valtanen, Pertti Rauhala and Tuuli Nivala. The article publications can be found on the FrostBit publication blog.

—— 17.12.2022 ——

SHARE IN

Mikko is a coder and part time teacher. His programming tasks are mainly full stack development with extra knowlegde in GIS and deep learning.

—— Bachelor of Engineering

Bonner, A. 2019. The Complete Beginner’s Guide to Deep Learning: Convolutional Neural Networks and Image Classification. Towards Data Science 2.2.2019. Accessed on 17.12.2021 https://towardsdatascience.com/wtf-is-image-classification-8e78a8235acb

Browniee, J. 2021. A Gentle Introduction to Object Recognition With Deep Learning. Machine Learning Mastery 27.1.2019. Accessed on 17.12.2021 https://machinelearningmastery.com/object-recognition-with-deep-learning/

CSC 2021. Supercomputer Mahti is now available to researchers and students – Finland’s next generation computing and data management environment is complete. CSC 26.8.2020. Accessed on 17.12.2021 https://www.csc.fi/en/-/supercomputer-mahti-is-now-available-to-researchers-and-students

Docs CSC 2021. Technical details about Mahti. Docs CSC 14.4.2021. Accessed on 17.12.2021 https://docs.csc.fi/computing/systems-mahti/

Dwivedi, P. 2020. YOLOv5 compared to Faster RCNN. Who wins?. Towards Data Science 30.1.2020.  Accessed on 17.2.2021 https://towardsdatascience.com/yolov5-compared-to-faster-rcnn-who-wins-a771cd6c9fb4

Géron, A. 2019. Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow. 2^nd edition. Sebastobol: O’Reilly Media

Github.com 2021. TensorFlow 2 Detection Model Zoo.  Github.com 7.5.2021. Accessed on 17.12.2021 https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md

Hofesmann, E. 2021. Guide to Conda for TensorFlow and PyTorch. Towards Data Science 11.1.2021. Accessed on 17.12.2021
https://towardsdatascience.com/guide-to-conda-for-tensorflow-and-pytorch-db69585e32b8

Huellmann, T. 2021. How to build a dataset for image classification. Levity 9.11.2021. Accessed on 17.12.2021 https://levity.ai/blog/create-image-classification-dataset

Krishnan, B. P. 2019. Machine learning Vs Deep learning Vs Reinforcement learning. Medium 18.9.2019. Accessed on 16.12.2021 https://medium.com/analytics-vidhya/machinelearning-deeplearning-reinforcementlearning-ed7b217861c5

Lendave, V. 2021. A Comparison of 4 Popular Transfer Learning Models. Analytics India Magazine 1.9.2021. Accessed on 17.12.2021 https://analyticsindiamag.com/a-comparison-of-4-popular-transfer-learning-models/

Leo, M. S. 2020. How to Choose the Best Keras Pre-Trained Model for Image Classification. Towards Data Science 15.11.2020. Accessed on 17.12.2021 https://towardsdatascience.com/how-to-choose-the-best-keras-pre-trained-model-for-image-classification-b850ca4428d4

McCulloch, W.S. & Pitts, W. 1943. A Logical Calculus of Ideas Immanent in Nervous Activity. Bulletin of Mathematical Biophysics 5, 1943, 115–133. https://doi.org/10.1007/BF02478259

Pai, A. 2020. CNN vs. RNN vs. ANN – Analyzing 3 Types of Neural Networks in Deep Learning. Analytics Vidhya 17.2.2020. Accessed on 17.12.2021 https://www.analyticsvidhya.com/blog/2020/02/cnn-vs-rnn-vs-mlp-analyzing-3-types-of-neural-networks-in-deep-learning/

Patel, K. 2020. Image Feature Extraction: Traditional and Deep Learning Techniques. Towards Data Science 9.9.2020. Accessed on 17.12.2021 https://towardsdatascience.com/image-feature-extraction-traditional-and-deep-learning-techniques-ccc059195d04

Sharabok, G. 2020. Why Deep Learning Uses GPUs? Towards Data Science 26.7.2020. Accessed on 17.12.2021 https://towardsdatascience.com/why-deep-learning-uses-gpus-c61b399e93a0

Sharma, P. 2019. Image Classification vs. Object Detection vs. Image Segmentation. Analytics Vidhya 21.8.2019. Accessed on 17.12.2021 https://medium.com/analytics-vidhya/image-classification-vs-object-detection-vs-image-segmentation-f36db85fe81

Warden, P. 2017. How many images do you need to train a neural network? Pete Warden’s Blog 14.12.2017. Accessed on 17.12.2021 https://petewarden.com/2017/12/14/how-many-images-do-you-need-to-train-a-neural-network/

Özgenel, Ç.F. & Sorguç, A. G. 2018. Performance Comparison of Pretrained Convolutional Neural Networks on Crack Detection in Buildings. Berlin. 35th International Symposium on Automation and Robotics in Construction (ISARC 2018). Accessed on 17.12.2021 https://www.iaarc.org/publications/fulltext/ISARC2018-Paper154.pdf

—— 30.12.2021 ——

SHARE IN

Tuomas Valtanen, Project Manager

Tuomas works as the web/mobile team leader, a software engineer and a part-time teacher in FrostBit at Lapland UAS. His tasks include project management, project planning as well as software engineering and expertise in web, mobile and AI applications.

—— Master of Engineering

Mikko is a coder and part time teacher. His programming tasks are mainly full stack development with extra knowlegde in GIS and deep learning.

—— Bachelor of Engineering

When a several hundred pages long novel is adapted into a movie, the events of the story are shortened and perhaps rearranged to some degree, in order to make all the important things fit in a more compact format and to keep the viewer engaged for the entire duration of the movie. Time and order of events have been approached somewhat similarly in the development process of the Struve Geodetic Arc mobile game for the exact same reasons: limiting the total length of the game and making it easy for the players to follow the events and keep them interested in the game from start to finish.

In reality, measurements across the entire triangle chain spanned several decades. The focus of the game is mainly on the section going over Lapland where measurements also took years. Exactly replicating all events from a time period this long would inevitably make the game longer despite the way time is handled inside the game. A typical mobile game session isn’t very long: GameAnalytics, a company selling tools for analyzing game usage, states in their 2019 report that the average session length for a mobile adventure game was less than 15 minutes (GameAnalytics Mobile Gaming Benchmarks Report: H1 2019, p.13). If the total length of a game is measured in several dozen hours like in some current AAA-titles for PC or console, it would take a much larger amount of these average length sessions to finish the game compared to a situation where the total length is just a few hours. In the Struve Geodetic Arc mobile game one of the goals is to get the players to finish the game and go all the way to the northern endpoint of the triangle chain in Hammerfest. So, if the game in total is very long and following that the required amount of game sessions needed to finish the game is large, it is possible the players feel that the game is too long and won’t ever finish it. Keeping the players engaged until the end of the game is a good enough reason to give up accuracy.

It was never intended that the actual historical timeline would be completely removed from the game, however. At the current development phase (spring 2021) the game is being designed to have calendar years pass according to the game’s internal time and to give the player a goal to reach the endpoint the same year the actual measuring crew got there. Some known places of stay might also be matched with the correct years.

Much like handling time inside the game flexibly, it was decided that the order of events would be presented in a more linear fashion compared to reality. In the game the player’s party follows a route along the triangle chain in one direction, whereas in reality the measuring crew did not move linearly from point A to point B. During the measurements the crew stayed longer in some places than others and measurements in a certain area could be done as separate trips from one place of stay. In the early stages of design, it would have been possible to choose game mechanics that supported moving in a way more closely resembling the actual measurements, but the development team decided to make movement on the map more linear. The team felt it would be more motivating for the player to follow a clear route from one place to another compared to e.g. separate trips from one place and returning there every time. One factor in this decision was that in the earliest design stages the team didn’t have the information on the more exact routes, it became available at a later phase. Lacking this information, it was easier to design the game to have a linear route.

Handling time and events in a flexible manner sacrifices much of the historical accuracy. The reasoning behind this sacrifice is making the game more approachable. No background information or prior interest towards the subject of the game is required from the player. The game also doesn’t aim at educating players by force. The thought is first and foremost to offer a good experience. Then on the side tell the player about a subject that was possibly previously unknown to them and get them interested enough to find out more through other means.

Regardless of genre and content of the game, limitations set by the platform must be taken into consideration when developing a game for mobile devices. Even though the physical size and resolution of screens on mobile devices have been growing, the screens are still many times smaller compared to a typical laptop or desktop screen. This sets limitations to the number of menus or other elements that can be fit on the screen at one time while still maintaining legibility. In addition to this, mobile devices are typically controlled with finger input, which is considerably less accurate way of navigation compared to a game controller or a mouse. This in turn limits the smallest possible size e.g. buttons can be.

When better accuracy is required, mobile devices can be controlled with a stylus. A small number of stylus-controlled games have also been made. Controllers made specifically for mobile gaming are also being made and sold, these closely resemble console controllers. Both styluses and controllers are mostly sold separately, so when the aim is to reach as large audience as possible and make using the game easy, it is not reasonable to require the user to have any kind of special controller devices.

In terms of game design, screen size affects (among other things) the amount of information the player can be shown at one time. This amount can be regulated for example by showing only the information that is required in that precise menu or at that precise moment of the game. Another option is dividing the information into smaller sections and having separate menus or menu levels for each section. Neither of these options are without problems. If the total amount of the information the player needs is large and most of it is contextual and not shown to player at all times, the limits of short-term memory might prevent the player from keeping all required things in mind. Then again, multi-leveled menus can be hard to navigate. This means the only solution isn’t just dividing and categorizing information or changing the way it is presented. The game itself must be designed to be simple enough so that the amount of required information isn’t too great.

Poor accuracy of finger input means that the basic functions of the game can’t require precise or complex actions from the player. The controls of the game must be simplified to a few most important buttons and simple gestures so that the learning curve isn’t too steep and continued use is effortless.

The need for simplification means more adaptations and loss of historical accuracy. Actions that in reality are complex or demanding have to be made easier and more straightforward. In the case of Struve Geodetic Arc mobile game one example of this is the triangle measurement itself. The aim is to give the player factual information on the measuring process, but using the measurement devices required skill and precision and the actions won’t be precisely replicated in-game. The realistic measurement actions will be replaced with simplified versions or with an entirely another set of related activities. When the basics of the game are easy to learn and playing doesn’t require complex actions, more casual players or people who haven’t really played at all, can easily start playing Struve Geodetic Arc game.

Despite putting game mechanics and the requirements set by the chosen platform first in the development process of Struve Geodetic Arc mobile game, the intention is not to make a generic adventure game and just have it wear a cosmetic skin with some historical influences. The project team has gathered information on e.g. the progression of the actual measurements, known places of stay and the day-to-day -life of people in the 19^th century. The game development team has then incorporated this information into their designs. The game will have player to encounter problems similar to those the real measuring crew encountered. The remoteness and challenging weather conditions of Lapland affected e.g. the ways measuring equipment could be transported and when measurements could be performed in the first place. One example of these conditions is that during the 19^th century the actual road network in Finnish Lapland, excluding winter roads, reached Kittilä, Kolari and Sodankylä, so when the player moves further north they can’t choose to use a road and have to advance by other means.

One goal of the game is also to have the player understand, how significant the measurements were to the society at the time. Using the information gathered from the measurements, new more correct maps could be drawn and the significance of good maps to trade and warfare for example was and still is great.

Visual elements are one of the ways used to convey information and feeling of the time period to the player. Even though the art style of the game is somewhat comic-like, the appearance of e.g. clothes, tools, buildings and measuring equipment is modelled after their real-life counterparts. The intent is to show events and life from nearly 200 years ago as lively and interesting despite deciding to give up true historical accuracy in many ways.

More on The Northern parts of the World Heritage Struve Geodetic Arc project: struvenorth.net

More on the Struve Geodetic Arc in general: https://www.maanmittauslaitos.fi/en/struvegeodeticarc

Game Analytics report of 2019: https://progamedev.net/wp-content/uploads/2019/07/Benchmarks2019.pdf

There are as many as 46 mines in Finland, with more than 5,000 employees in 2018. Especially in northern Finland, the mining industry is an important employer for many and is an source of livelihood in areas where diverse employment and welfare development can be otherwise challenging (Kaivosteollisuus.fi). There is also a wide range of education possibilities of mining industry throughout Finland, which is why the reform of degree structures and the development of modern learning methods are topical due to the growing demand of the field.

Since mining is such a practical field, how could its education be modernized? The Migael-project answers this question with gamification and modern technologies. The aim of the project is to develop mining education by creating a virtual learning environment with a variety of scenario-based exercises. By the beginning of 2021, three exercises have been completed in the project and the fourth exercise will be finalized during February. Each exercise focuses on different “scenarios” in the open-pit-mine as well as in the underground mine. For each exercise, different platforms and technologies have been utilized to achieve the learning goals:

Charging/Blasting
In the first exercise, the player will be able to practice the charging of explosives in 2D view, as well as view the simulation of charging and blasting in both 3D- and Oculus Guest virtual views. In the exercise, the player gets to simulate charges of different sizes and see what their blasting effect would look like.

Workcard
The second exercise was also created to be played with Oculus Guest’s VR-glasses. In this exercise, the player will be able to do an occupational safety card inspection at an underground mine. The player is able to move around the underground mine and observe the necessary safety measures and fill in the occupational safety card accordingly.

Driving departure
The third exercise was created to be played on both Oculus Rift VR-glasses and Android phones, and this exercise will go through a mining vehicle run-in inspection. The player must check the worker’s safety equipment and the condition and usability of the mining vehicle before it can departure.

PRE- and POST-blasting safety measures
In the fourth exercise, the player performs the pre-blast and post-blast safety measures. The exercise was made into a more traditional 3D-desktop “serious game”, since it is possible to make more use of textual teaching material on a wide range of topics. The player gets to observe objects in the open-pit-mine environment to secure the area before and after the blast.

Different technical implementations enable different ways of presenting or simulating the subject to be taught. VR-technology is able to take a player or learner to a realistic learning situation that responds as much as possible to a real life situation. In this way, for example, training situations can be usefully simulated when the implementations in real life would be demanding, costly and often dangerous. Implementation on a mobile platform establishes easier learning regardless of time and place, and this can help to reach a wider group of users. Therefore, in the third exercise, for example, the driving departure was implemented for both VR glasses and mobile.

However, virtual glasses create their own challenge if the material and subject area is extensive and requires more than observation and “tangible measures.” While instructions for “floating texts” can be established on virtual glasses, interactive texts combined to player movement and other activity can create an unnecessary challenge in the VR-world. For this reason, the fourth exercise of Migael was created as a 3D-desktop game, as the topic required a lot of text in the form of teaching material and other information. In this exercise, it was necessary to plan particularly carefully which operations should and could be “gamified”, as there are several safety measures related to blasting. Some of the most generic and difficult operations were implemented in the form of cinematic transitions and instructions given by the game’s characters:

Thus, different technological implementations can enable different ways of learning and achieve certain learning goals. Some technologies allow for a more realistic simulation of exercises, while others allow for a broader presentation of the content and the possibility to reach a larger target audience. The exercises produced in Migael form a gameful learning environment where you can experience the benefits of many new technologies and learn about the mining industry in an inspiring and safe way. The aim of the project is to make the mining industry more visible and interesting, especially for students, and to establish important and safe training regardless of time and place.

In addition of the fourth exercise being completed soon, the early part of the year 2021 is eventful for the project: a Teams-webinar titled as ‘Utilization of modern technologies in the mining sector’ will be held on 25.01.2021 as part of the project. In addition to the FrostBit lab, the webinar will feature speakers from VTT and Kajaani University of Applied Sciences, who will present, amongst other things, the possibilities of gamification, VR / AR technologies and new sensor technologies in the mining industry. Mining staff, project staff and students of Lapland University of Applied Sciences will be invited to join and listen to the webinar.

The project’s activities will be updated on its official website, which includes a blog and the downloadable exercises: www.migael.fi

In addition, see the FrostBit portfolio-page of Migael: https://www.frostbit.fi/en/portfolio/migael-en/

The northen Ostrobothnia of Centre for Economic Development, Transport and the Environment has granted 337 502 € from the European Social Fund (ESF) and state funding for the Migael-project. The total cost of the project is 450,002 €.

www.kaivosteollisuus.fi/fi/kaivosala-suomessa

—— 15.01.2021 ——

SHARE IN

Tuuli Nivala, Specialist

Tuuli works as a specialist in FrostBit at the LapUAS. In her current role she utilizes her game pedagogical knowledge in game design, as well as participates in project planning and some 2D graphic design. She also does marketing and AV-media material for the laboratory.

—— Master of Media Education, Lapland University

FrostBit Software Lab has followed the development of the Flutter platform (created by Google) with great interest for the last few years. Our laboratory has also actively applied Flutter in various projects to enhance the development of our mobile applications. The need for an efficient cross-platform technology has been great, since producing two separate native mobile applications (Android + iOS) has proven to be too expensive and inefficient for our purposes, whether considering the development or the maintenance phase.

The strength of native mobile applications lies in the possibility of total customization and the native access to the features of the mobile phone (camera, sensors etc.). On the other hand, the weaknesses of the native mobile application approach are the need for a sizable human resource and the maintenance challenges provided by the rapidly evolving mobile phone ecosystems.

Previously our laboratory has created applications by using the PhoneGap –platform, but the limitations it provides to mobile application design proved to be too great for the technical needs of the FrostBit Software Lab.

Google released the first version of Flutter in the year 2017, and it immediately sparked some interest within the developers at FrostBit Software Lab. At that time, we had two sizable mobile projects upcoming in our project calendar, so we bravely decided to try Flutter out in both projects to avoid the challenges that come with the development of two separate native mobile applications.

Going towards the year 2021, we are tentatively yet extremely positively surprised and optimistic when it comes the possibilities of using Flutter in creating cross-platform applications. We are certainly going to use it further in upcoming projects as much as we can.  Then again, nothing in this world is perfect, and Flutter also shares its weaknesses, despite all its strengths. For this purpose we have gathered our own experiences on Flutter from the past few years, and here’s our conclusion of what we think of Flutter so far:

• Development is easier when compared to PhoneGap or Xamarin, for example, since Flutter needs less platform-specific code
• Flutter is cross-platform; it’s possible to develop applications for Android, iOS, Windows, Linux, MacOS, web etc. at the same time
• It’s fast to develop applications on Flutter
• Flutter allows the developer to create complex and fully customized UI components, since everything related to layout can be altered
• Flutter has good documentation and a great number of examples are available. Flutter’s userbase also grows rapidly all the time
• Flutter is powered by a very performance efficient 2D-UI engine (sky_engine), which works well on a 120Hz display as well
• Flutter uses the Dart-programming language, which is an easy-to-learn OOP language
• Google is a huge organization, which has the resources to develop Flutter efficiently. Google also uses Flutter in some of its own products
• Flutter-code will eventually compile into native application code on different platforms and devices
• Flutter already has good development tools, for example, for Android Studio and Visual Studio Code
• Flutter is rapidly being updated

• Complex user interfaces can be tedious to develop
• Since Flutter is updated frequently, a great number of changes are integrated into the platform continuously, which can make large projects difficult
• Both Dart-language and Flutter are relatively new technologies, because of which they are also rapidly being updated. This problem will fix itself in the future though.
• Many of the best features of Flutter are still in development stage (null safety, web and desktop application support etc. )
• If Flutter does not have a plugin for a certain platform specific feature (Android or iOS), you will need to make platform specific code
• Because of Flutter’s popularity and its relative easiness for application development, there are numerous third party plugins which can have questionable quality
• Since Flutter is a new technology, so called “best practices” and recommended architectural designs haven’t been really formed or standardized

In conclusion, we think Flutter has way more strengths than weaknesses, most of which are related to the fact that Flutter is still a new technology.

If we also take a look into a few internet articles related to this subject, we can conclude that many other developers in the world have similar thoughts with us (e.g. Rozwadowski 2020;  Costa 2019; Sannacode 2020;  Powalowski 2019). According to these articles, Flutter weaknesses also include the large size of the mobile application as well as certain compromises that are related to the mobile application’s layout and user interface design recommendations of different platforms (e.g. material design on Android or iOS). Here at FrostBit Software Lab, we do not consider these weaknesses to be too problematic in our projects for now, however.

We have used Flutter in two major projects, first being “Arktori” and the other being “DWELL”. In the Arktori project, we are developing a mobile application, with which the rectors of the northern part of Finland can network with each other, as well as enhance their professional knowledge and also mentor each other in their daily work. In the DWELL project, we are developing a mobile application, which promotes and endorses communal living in apartment buildings. The pilot apartment building in DWELL project is the DAS Kelo –student dormitory. We have been quite happy with Flutter in both projects when it comes to mobile application development.

Some screenshots from the Arktori and DWELL mobile applications:

Flutter has proven to be a powerful tool to create cross-platform mobile applications. However, it remains to be seen, how flexible and efficient Flutter will be in the future when it comes to web and desktop applications. If Flutter will be truly competitive outside mobile applications, it will be a realistic scenario (from the software developer’s point of view) to concentrate mostly on Flutter when developing applications, only to be supported by other technologies in cases, where Flutter is not an optimal approach.

For now, we are not completely certain here at FrostBit if Flutter is the all-powering efficient application development platform for all kinds of applications, but no one can deny that they surely are trying hard to do so!

In any case, we are going to continue watching the development of Flutter with great interest, and are always ready to try out Flutter in new projects and use cases in the future!

—— 11.12.2020 ——

SHARE IN

Tuomas Valtanen, Project Manager

Tuomas works as the web/mobile team leader, a software engineer and a part-time teacher in FrostBit at Lapland UAS. His tasks include project management, project planning as well as software engineering and expertise in web, mobile and AI applications.

—— Master of Engineering