Intelligent Immersive Environments with Digital Twins and Machine Learning for Industrial Applications

The fourth industrial revolution requires humans to actively participate in the process of digital transformation of industry and manufacturing. As more services become digital, humans must remain at the center of the process. Meanwhile, the term “digital” can also refer to advancements in general artificial intelligence. As a result, the modes of interaction between the human and the “digital” become critical. This thesis topic is devoted to the concept of creating an intelligent immersive virtual environment in which the environment is built using digital twin and machine learning technologies and is perceived by the user in extended reality. The emphasis here is on the human-in-the-loop aspect: “how can coherent modes of interaction between the human and the digital environment be used?” which is particularly relevant in the context of Industry 4.0 transition towards Industry 5.0 human-centric concept and manufacturing applications.

Research field:	Information and communication technology
Supervisors:	Dr. Vladimir Kuts Dr. Aleksei Tepljakov
Availability:	This position is available.
Offered by:	School of Information Technologies Department of Computer Systems
Application deadline:	Applications are accepted between June 01, 2022 00:00 and June 30, 2022 23:59 (Europe/Zurich)

The fourth industrial revolution (Industry 4.0) has resulted in the acceleration of the digital transformation in the sphere of manufacturing [1]. Central to the digital transformation is the digital twin (DT) technology [2], [3]. Meanwhile, another transformation is occurring—from the sheer technological advancements enabled by Industry 4.0, the next generation, dubbed Industry 5.0, is, in its core values, human-centric [4]. Hence, one of the most important driving aspects for the current PhD track proposal is the study, design, and development of a human-centric system in the context of the manufacturing industry.

Machine learning (ML) and extended reality (XR) are critical components of the digital transformation. The former has already become a staple in many industries, whilst the latter is still in the early stages of adoption [5], yet has significant disruptive potential in many areas where humans interact with technology.

When these three disruptive technologies—DT, ML, and XR—are combined, the concept of an intelligent immersive environment is created (IIE) [6]. When such an environment is applied in Industry 5.0, the result is human-centric. Contrary to the beliefs that artificial intelligence will replace humans, an IIE will instead augment and enhance the experience of the human user in a relevant industrial context.

The present PhD track proposal therefore focuses on a research and development effort to design and implement IIEs in a manufacturing context. The expected deliverables are mostly of a technical nature.

As part of the first year’s activities on the PhD track, a survey in companies should be conducted about the requirements regarding IIEs. In parallel, the state-of-the-art literature review should be completed on the topic of IIEs and their relation to DT, ML, and XR in the manufacturing context. Finally, some development activities in the domain of XR application design should take place to create the initial framework used during the rest of the PhD track for conducting research by formulating hypotheses and testing them using the framework.

The following research questions are proposed:

• How to design and conduct experiments for collecting relevant user data (such as interactions with the industrial DTs[VK1] [AT2]  and other users) in the immersive environment that will be used to develop the IIEs? This question should also consider the Quality of Experience (QoE) concepts while designing the applications.
• What is the best possible modeling approach and model type(s) for user motion/action/decision/state from which it is possible to derive proactive intelligent user interface and user experience elements for improving the immersive experience in terms of data exploration, analysis, and learning with the digital twins in the IIE?
• How to apply innovative ML technologies, such as Transformers for NLP [7], to enhance the general artificial intelligence aspect of IIEs?
• What are the best applications and implementations of IIEs for specific digital twins in manufacturing in the context of Industry 5.0 and Service 5.0?

 

References

[1]        M. Ghobakhloo and M. Iranmanesh, “Digital transformation success under Industry 4.0: a strategic guideline for manufacturing SMEs,” Journal of Manufacturing Technology Management, vol. 32, no. 8, pp. 1533–1556, Oct. 2021, doi: 10.1108/JMTM-11-2020-0455.

[2]        F. Tao, H. Zhang, A. Liu, and A. Y. C. Nee, “Digital Twin in Industry: State-of-the-Art,” IEEE Transactions on Industrial Informatics, vol. 15, no. 4, pp. 2405–2415, Apr. 2019, doi: 10.1109/TII.2018.2873186.

[3]        A. el Saddik, “Digital Twins: The Convergence of Multimedia Technologies,” IEEE MultiMedia, vol. 25, no. 2, pp. 87–92, Apr. 2018, doi: 10.1109/MMUL.2018.023121167.

[4]        X. Xu, Y. Lu, B. Vogel-Heuser, and L. Wang, “Industry 4.0 and Industry 5.0—Inception, conception and perception,” Journal of Manufacturing Systems, vol. 61, pp. 530–535, Oct. 2021, doi: 10.1016/j.jmsy.2021.10.006.

[5]        L. Gong, A. Fast-Berglund, and B. Johansson, “A Framework for Extended Reality System Development in Manufacturing,” IEEE Access, vol. 9, pp. 24796–24813, 2021, doi: 10.1109/ACCESS.2021.3056752.

[6]        A. Pena-Rios, H. Hagras, G. Owusu, and M. Gardner, “Furthering Service 4.0: Harnessing Intelligent Immersive Environments and Systems,” IEEE Systems, Man, and Cybernetics Magazine, vol. 4, no. 1, pp. 20–31, Jan. 2018, doi: 10.1109/MSMC.2017.2769199.

[7]        T. Wolf et al., “Transformers: State-of-the-Art Natural Language Processing,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020, pp. 38–45. doi: 10.18653/v1/2020.emnlp-demos.6.

 

About the Supervisors

Main supervisor: Dr. Aleksei Tepljakov
Co-supervisor: Dr. Vladimir Kuts

The supervisors have a record of accomplishment in working with intelligent environments in the context of the academic labs aimed for the usage of XR tools for control and monitoring of industrial assets. The present PhD track aims to bring the previous results into the industrial domain, to validate the presented concept further in the real manufacturing shopfloor and bring the value and impact to the wider number of use-cases aside of academical environment.

Dr. Aleksei Tepljakov earned his PhD in August 2015 from Tallinn University of Technology after conducting extensive research on advanced modeling and control with fractional calculus. He has also been researching extended reality (XR) and digital twins (DT) technologies in the context of modeling and control with industrial applications since 2015. From November 2021, Dr. Tepljakov holds a Senior Research Scientist position with Centre for Intelligent Systems, Department of Computer Systems, Tallinn University of Technology. His primary research is on cyber-physical systems, specifically advanced modeling and control (including machine learning approaches for computer vision and dynamic system modeling with genetic programming and symbolic regression), digital twins, and industrial applications. Dr. Tepljakov's research record and citation metrics demonstrate that he is one of the finest possible advisors for the proposed multidisciplinary thesis topic.

Dr. Vladimir Kuts received his Ph.D. in Mechanical Engineering from Tallinn University of Technology (TalTech) in 2019. From the Year 2017, Dr. Kuts is Head of Industrial Virtual and Augmented Reality laboratory (www.ivar.taltech.ee) in the Department of Mechanical and Industrial Engineering Department of TalTech. His main research interests include Industrial Digital Twins synchronized with the real industrial equipment such as robots and usage of Virtual Reality technologies for human-robot interaction standards validation. Currently Dr. Kuts is a postdoctoral researcher in Confirm Smart Manufacturing Centre of Ireland and doing research project between two Universities - University of Limerick and Technological University of the Shannon, where he works on XR interfaces towards Industry 5.0 concept. Since 2018, he has also been collaborating intensively with Dr. Tepljakov's XR research group. He is the ideal co-supervisor for this proposed PhD research project.

 

Requirements

The prospective candidate for the proposed PhD position must have the following background and qualifications:

• The applicant must hold a Master of Science degree in a field closely related to the problems outlined in the proposal (preferably, in information and communication technology or mechanical engineering);
• Previously demonstrated excellent academic ability;
• Working knowledge of Unreal Engine 4/5 or Unity engines and C/C++, Python, or C# programming languages;
• Significant interest in developing innovative interaction mechanics with immersive technologies using machine learning;
• At least basic experience with Machine Learning algorithms and Extended Reality technologies;
• Previously published peer-reviewed research in a is a strong asset;
• Very good command of the English language and overall soft skills.