Cross-modal deep learning enhanced mixed reality accelerates construction skill transfer from experts to students
Li, X. et al. A critical review of virtual and augmented reality (VR/AR) applications in construction safety. Autom. Constr. 86, 150–162 (2018).
Google Scholar
Wang, P. et al. A critical review of the use of virtual reality in construction engineering education and training. Int. J. Environ. Res. Public Health 15(6), 1204 (2018).
Google Scholar
Nonaka, I. & von Krogh, G. Perspective—Tacit knowledge and knowledge conversion: Controversy and advancement in organizational knowledge creation theory. Organ. Sci. 20(3), 635–652 (2009).
Google Scholar
Sujan, S. F. et al. Digitally capturing and managing tacit knowledge for construction safety: A study protocol. Buildings 11(11), 522 (2021).
Wang, X. et al. Integrating augmented reality with building information modeling: Onsite construction process controlling for liquefied natural gas industry. Autom. Constr. 40, 96–105 (2014).
Google Scholar
Baltrusaitis, T., Ahuja, C. & Morency, L. P. Multimodal machine learning: A survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41(2), 423–443 (2019).
Google Scholar
Li, X. et al. Digital twin-enabled virtual reality interaction for construction design, planning, monitoring, and quality control. J. Manag. Eng. 38(1), 04021075 (2022).
Pan, Y. & Zhang, L. Roles of artificial intelligence in construction engineering and management: A critical review and future trends. Autom. Constr. 122, 103517 (2021).
Google Scholar
Huang Y, Bianchi-berthouze N, Coutrix C, et al. Leveraging both verbal and non-verbal communications in conversational interfaces[C]//Proc. of the 2022 CHI Conference on Human Factors in Computing Systems. New Orleans LA USA: ACM 1–18. (2022)
Ramachandram, D. & Taylor, G. W. Deep multimodal learning: A survey on recent advances and trends[J]. IEEE Signal Process. Mag. 34(6), 96–108 (2017).
Google Scholar
Huang H, Zhang L, Tan K C, et al. Cross-modal common representation learning by hybrid transfer network[C]//Proc. of the Twenty-Ninth International Joint Conference on Artificial Intelligence. Yokohama, Japan: International Joint Conferences on Artificial Intelligence Organization, 2345–2351. (2021)
Lahat D, Adali T, Jutten C. Multimodal data fusion: An overview of methods, challenges, and prospects. Proc. of the IEEE 103(9) 1449-1477 2015
Alam, M. S. & Kwon, G. R. Robust multi-modal feature fusion using deep networks and multi-source information for face recognition. Sensors 21(18), 6164 (2021).
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach, California, USA: Curran Associates Inc., 2017: 6000–6010.
Zhang, Y. et al. Prior-based self-supervised learning for fully convolutional network. IEEE Trans. Med. Imaging 40(8), 2226–2238 (2021).
Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//Proc. of the 38th International Conference on Machine Learning. PMLR 8748–8763. (2021)
Chen T, Kornblith S, Norouzi M, et al. A simple framework for contrastive learning of visual representations[C]//Proc. of the 37th International Conference on Machine Learning. PMLR 1597–1607 (2020).
Zhou, T., Wang, S. & Billinghurst, M. Analysis of gaze behavior and verbal instructions for enhanced AR-based task guidance. IEEE Trans. Visual Comput. Graphics 28(5), 2196–2206 (2022).
Martinez-Maldonado, R. et al. Physical learning analytics: A multimodal perspective. J. Learn. Anal. 7(3), 45–68 (2020).
Schneider, B. et al. Using mobile eye-trackers to unpack the perceptual benefits of a tangible user interface for collaborative learning. ACM Trans. Comput.-Hum. Interact. 25(6), 1–23 (2018).
Google Scholar
Pathirage, C. P., Amaratunga, D. G. & Haigh, R. P. Tacit knowledge and organisational performance: Construction industry perspective. J. Knowl. Manag. 11(1), 115–126 (2007).
Google Scholar
Abbasnejad, B. et al. Building Information Modelling (BIM) adoption and implementation enablers in AEC firms: A systematic literature review. Arch. Eng. Design Manag. 17(5–6), 411–433 (2021).
Chen, Y. C. et al. Attention-aware bidirectional gated recurrent unit for asymmetric multimodal fusion in mixed reality-aided facilities management. Adv. Eng. Inform. 47, 101252 (2021).
Wang, X. et al. Augmented reality in built environment: Classification and implications for future research. Autom. Constr. 32, 1–13 (2013).
Google Scholar
Diao, P. Y. & Shih, N. J. Trends and research issues of augmented reality studies in architectural and civil engineering education—A review of academic journal publications. Appl. Sci. 9(9), 1840 (2019).
Google Scholar
Chalhoub, J. & Ayer, S. K. Using mixed reality for electrical construction design communication. Autom. Constr. 103, 235–243 (2019).
Chen, K. et al. Automatic generation of BIM models and their implementation in an extended reality construction verification system. Autom. Constr. 147, 104727 (2023).
Park, C. S. et al. A framework for proactive construction defect management using BIM, augmented reality and ontology-based data collection template. Autom. Constr. 33, 61–71 (2013).
Google Scholar
Pham, H. C. et al. Development of construction hazard database for automated hazard identification process using natural language processing. J. Constr. Eng. Manag. 147(1), 04020147 (2021).
Du, J. et al. CoVR: Cloud-based multiuser virtual reality headset system for project communication of remote users. J. Constr. Eng. Manag. 144(2), 04017109 (2018).
Google Scholar
He, Z., Wu, L. & Li, X. When architecture meets computation: Interactive and collaborative design-based approaches for augmented reality. J. Comput. Design Eng. 8(1), 583–602 (2021).
Le, Q. T. et al. A framework for using mobile based virtual reality and augmented reality for experiential construction safety education. Int. J. Eng. Educ. 31(3), 713–725 (2015).
Kim, J. et al. Mixed reality training for construction safety: Visual haptic floor-drilling simulation[J]. IEEE Trans. Visual Comput. Graphics 29(4), 2013–2023 (2023).
Sacks, R., Perlman, A. & Barak, R. Construction safety training using immersive virtual reality. Constr. Manag. Econ. 31(9), 1005–1017 (2013).
Google Scholar
Cheng, T. & Teizer, J. Real-time resource location data collection and visualization technology for construction safety and activity monitoring applications. Autom. Constr. 34, 3–15 (2013).
Google Scholar
Baduge, S. K. et al. The past, present and future of the apprenticeship training in construction industry. Educ. Train. 64(2), 141–157 (2022).
Addis, M. Tacit and explicit knowledge in construction management. Constr. Manag. Econ. 34(7–8), 439–445 (2016).
Google Scholar
Lin, Y. C. et al. Developing construction defect management system using BIM and ontology-based semantic web technology. Autom. Constr. 134, 104090 (2022).
Wang, Y. et al. Engagement-aware behaviors analysis for construction workers using wearable sensors. Adv. Eng. Inform. 47, 101258 (2021).
Chen M, Feng A, Mccullough K, et al. Multisensory social intelligence embodiment for human-centered AI. Proc. of the IEEE, 110(7): 910-932. (2022)
El-Diraby, T., Krijnen, T. & Papagelis, M. BIM-based collaborative design and socio-technical analytics of green buildings. Autom. Constr. 82, 59–74 (2017).
Google Scholar
Kassem, M., Benomran, L. & Teizer, J. Digital twin workflows in construction: A systematic literature review. Comput. Constr. 7(2), 136–157 (2022).
Zhang, J. et al. The BIM-enabled geotechnical information management of a construction project. Comput. Civil Eng. 30(4), 04016003 (2016).
Faghihi, V. et al. Objective-driven and Pareto Front analysis: Optimizing time, cost, and job-site movements. Autom. Constr. 69, 79–88 (2016).
Google Scholar
Lu, Q. et al. Activity theory-based analysis of BIM implementation in building O&M and first response. Autom. Constr. 85, 317–332 (2018).
Google Scholar
Ghalap, P. et al. BIM-based mixed reality application for construction progress monitoring and quality inspection: A state-of-the-art review and future research agenda. J. Build. Eng. 52, 104437 (2022).
Golparvar-Fard, M. et al. Evaluation of image-based modeling and laser scanning accuracy for emerging automated performance monitoring techniques. Autom. Constr. 20(8), 1143–1155 (2011).
Google Scholar
Kalman, R. E. A new approach to linear filtering and prediction problems. J. Basic Eng. 82(1), 35–45 (1960).
Google Scholar
Zadeh A, Liang P P, Mazumder N, et al. Tensor fusion network for multimodal sentiment analysis[C]//Proc. of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark: Association for Computational Linguistics, 1103–1114. (2017)
de Maesschalck, R., Jouan-Rimbaud, D. & Massart, D. L. The Mahalanobis distance. Chemom. Intell. Lab. Syst. 50(1), 1–18 (2000).
Google Scholar
Diao, P. Y. & Shih, N. J. BIM-based AR maintenance system (BARMS) as an intelligent instruction platform for complex plumbing facilities. Appl. Sci. 9(8), 1592 (2019).
Google Scholar
Ellis, S. R. et al. Factors influencing operator interaction with virtual objects viewed via head-mounted see-through displays: Viewing conditions and rendering latency[C]//IEEE Virtual Reality 105–112 (IEEE, 2003).
Duan Y, Fu G, Zhou N, et al. Everything as a service: Towards constructing service-oriented cloud computing architecture[C]//First International Conference on Cloud Computing. Beijing, China: IEEE 475–480. (2009)
Ren, J. et al. Edge computing for 5G-enabled Internet of Things: Dynamic resource allocation, computation offloading and task collaboration[J]. IEEE Internet Things J. 7(6), 5172–5184 (2020).
Paas, F., Renkl, A. & Sweller, J. Cognitive load theory and instructional design: Recent developments. Edu. Psychol. 38(1), 1–4 (2003).
Google Scholar
Hegarty, M. Cognition, metacognition, and the design of maps. Curr. Dir. Psychol. Sci. 20(3), 167–172 (2011).
Zhang, J. et al. Mixed-reality technologies for safety management. Smart Sustain. Built Environ. 11(3), 761–784 (2022).
Cohen, J. A coefficient of agreement for nominal scales. Educ. Psychol. Measur. 20(1), 37–46 (1960).
Google Scholar
Marszalek M, Laptev I, Schmid C. Actions in context[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL, USA: IEEE, 2009: 2929–2936.
Qi, S. et al. Region-aware temporal network for video scene graph generation. IEEE Trans. Pattern Anal. Mach. Intell. 44(10), 6830–6845 (2022).
Bai S, Kolter J Z, Koltun V. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling[J]. arXiv preprint arXiv:1803.01271, 2018.
Caruana, R. Multitask learning. Mach. Learn. 28(1), 41–75 (1997).
Google Scholar
Lecun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521(7553), 436–444 (2015).
Google Scholar
Zhao, R. et al. Deep learning and its applications to machine health monitoring. Mech. Syst. Signal Process. 115, 213–237 (2019).
Google Scholar
Wang, P., Anumba, C. & Liapi, K. Development of a skills transfer system using immersive virtual reality for high-rise steel beam installation. J. Constr. Eng. Manag. 147(9), 04021116 (2021).
Cheng, T. et al. Automated task-level activity analysis through fusion of real time location sensors and worker’s thoracic posture data. Autom. Constr. 29, 24–39 (2013).
Google Scholar
Chen, Y. C. et al. Attention-aware bidirectional gated recurrent unit for asymmetric multimodal fusion in mixed reality-aided facilities management. Adv. Eng. Inform. 47, 101252 (2021).
Teizer, J., Cheng, T. & Fang, Y. Location tracking and data visualization technology to advance construction ironworkers’ education and training in safety and productivity. Autom. Constr. 35, 53–68 (2013).
Google Scholar
Butkiewicz T. Low-cost coastal mapping using Kinect v2 time-of-flight cameras[C]//2014 Oceans – St. John’s. St. John’s, NL, Canada: IEEE, 2014: 1–9.
Kirkpatrick, D. L. & Kirkpatrick, J. D. Evaluating training programs: The four levels[M] 3rd edn. (Berrett-Koehler Publishers, 2006).
Kraiger, K., Ford, J. K. & Salas, E. Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. J. Appl. Psychol. 78(2), 311–328 (1993).
Google Scholar
Bhattacharya, B. et al. A deep learning system for automated whole-body tracking of fish from low-cost cameras. Sci. Rep. 12(1), 19266 (2022).
Ritter, F. E. & Schooler, L. J. The learning curve. Int. Encycl. Soc. Behav. Sci. 13, 8602–8605 (2001).
Murre, J. M. J. & Dros, J. Replication and analysis of Ebbinghaus’ forgetting curve. PLoS ONE 10(7), e0120644 (2015).
Google Scholar
Chun, M. M., Badre, D. & Olson, I. R. Transfer of learning: New perspectives on an enduring issue. Trends Cogn. Sci. 26(7), 599–612 (2022).
Brooke, J. SUS: A quick and dirty usability scale. Usability Eval. Ind. 189(194), 4–7 (1996).
Lewis J R, Sauro J The factor structure of the System Usability Scale[C]//International Conference on Human Centered Design San Diego CA USA Springer 94 103. (2009)
Ryan, R. M. & Deci, E. L. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am. Psychol. 55(1), 68–78 (2000).
Google Scholar
Paas, F. et al. Cognitive load measurement as a means to advance cognitive load theory. Edu. Psychol. 38(1), 63–71 (2003).
Google Scholar
Moosman, F. et al. An open-source workflow for 3D object detection in close range imagery. ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci. 8, 75–82 (2021).
Kruger, J. & Dunning, D. Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J. Pers. Soc. Psychol. 77(6), 1121–1134 (1999).
Google Scholar
Mahmoud, A. H. & Elshafay, A. Virtual reality: A tool for environmental design education. J. Eng. Appl. Sci. 69(1), 1–18 (2022).
Chowdhury, S., Schnabel, M. A. & Zhang, Y. A conceptual framework for an IVR system to promote enhanced learning in architectural design education. Archit. Sci. Rev. 65(3), 307–318 (2022).
Tao, W. et al. Extended reality for enhanced sensory inputs to buildings and cities: A systematic review. Adv. Eng. Inform. 54, 101760 (2022).
Zheng, Z. et al. Geometric deep learning for construction BIM data: A review. Adv. Eng. Inform. 51, 101495 (2022).
Tang, S. et al. A review of building information modeling (BIM) and the internet of things (IoT) devices integration: Present status and future trends. Autom. Constr. 101, 127–139 (2019).
Google Scholar
link
