Significant advances in new emerging technologies such as the 5th generation mobile networks (5G), Expand the reality (XR), and Artificial Intelligence (AI) enable extensive three-dimensional (3D) experience and interaction. The vivid 3D virtual dynamic displays and immersive experiences will become new normal in near future. The XR-based virtuality-reality co-existing classroom goes beyond the limitations of Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). Such technology also enables integration of the digital and physical worlds and further creates a smart classroom featuring co-existed virtuality and reality. In this paper, we show an application of the XR enabling human-environment interaction. Through the theory explanation, practice platform and combination of virtual and reality, using XR we construct a new type of class. We develop the teaching methods using digital enabling, building information model library, and offline XR immersive teaching, and design a mixed teaching method which is virtual and real, and is available in online and offline formats. In our developed system the working scene as it is seen in practice is shown to the students. We consider the steel structure construction technology course in civil engineering and architecture as an example, to help the learners develop their theoretical understanding and practical skills. We also discuss the application and development of the class based on virtual and reality. This work shows the feasibility and significance of learning methods and the reconstruction and transformation of learning spaces empowered by new technologies in the digital era.
Cone of Learning (see
Nevertheless, in some discipline, exercising the taught materials are not often easily attainable. For instance, in subjects such as traditional medical education which is mainly focused on animal specimens, cadavers, and teaching AIDS, the main issues are insufficient resources and a high risk of patients’ safety [
To address the above issue, it is an immediate need to seek technological solutions enabling the simulation of the actual scenes and construct virtuality-reality-coexisting classrooms. Such a classroom significantly improves the learners’ experience and meets the surging demand for diversified learning scenarios. In 2019, China Digital Library and Filchuang Technology Co., Ltd. cooperated to develop the “XR Global Digital Content Center” in Shanghai to integrate the virtual and real worlds through extended reality (XR). Besides, there are more possibilities to apply such technologies in education and create learning experiences for the learners which are otherwise hard to realize.
In this paper we introduce a mixed teaching method which is virtual and real, and available both online and offline. This is based on improving the teaching method with digital enabling, teaching carrier of building information model library, building information model library and offline XR immersive teaching. The steel structure construction technology course of civil engineering and architecture is used as an example to show how using XR enables restoring the real working scenario and helps learners better understand the concepts and apply their knowledge in practice.
In 1956, Heileg M. developed a motorcycle simulator, Sensorama, with a three-dimensional display and stereo effect and the ability to generate vibrating sensation. His 1962 patent “Sensorama Simulator” includes some basic ideas of VR technology [
According to Virtual Reality Technology [
In 1997, Ronald Azuma, a professor at the University of North Carolina, defined Augmented Reality (AG) in a paper titled “A Survey of Augmented Reality” [
The “Extended reality” or “cross reality”, also known as “XR” or “ER”, refers to a real-virtual environment for human-machine interaction generated by computers and wearable devices. The XR can be regarded as an umbrella term, covering VR, AR, MR, and other new immersive technologies that possibly occurred due to technological progress, and it could be V(R), A(R), or M(R), and even any point on the “virtuality-reality continuum” model. Therefore, XR covers all hardware, software, method, and immersive technology experience that can integrate the physical and virtual world [
In the XR, the physical objects are integrated with the simulated situations and generate the experience in both reality and virtuality. According to [
In 1991, Wyckoff and Mann developed a head-mounted display device that was able to present XR visual pictures. In their devise high-dynamic-range image processing techniques, VR, and AR technologies were combined to improve the sensory abilities [
The human-environment interacting XR technology constructs a virtuality-reality coexisting new classroom by combining the theoretical explanation and practice platforms. This approach enables safety, vivid reproduction of scenes, lower cost compared to the actual scenes, and repeatable operation and training. Through the integration of teaching, learning, and practice, enables the implementation of Piaget’s concept. Such technology also enables “moving the laboratory to the classroom” and constructivism learning theory which is based on the following “learning is an experience of a real-world situation” [
Development of the virtuality-reality co-existing classroom (
This component covers the in-depth integration of virtuality and reality, to create unlimited interactions. Combining teaching resources, communication technologies, and mobile terminals enables multi-scene ubiquitous learning possible [
As it is seen in (
i. Design of virtuality-reality co-existing course: The teaching design stage mainly comprised of the following two tasks: teaching strategy design and construction of the teaching resources. Teaching strategy design is based on talent training program and learning situation analysis to determine teaching objectives and difficult points in teaching. Teaching strategy design then optimizes the teaching process and application of the available technologies and methods. Teaching resource construction includes the allocation of physical resources e.g., animated micro class and teaching aid model, digital models, and XR resource construction.
ii. Implementation of Virtuality-reality co-existing classroom: In teaching implementation, XR is integrated into the entire teaching process before, during, and after the class. The virtual-real and 3D XR classroom teaching field highlights the emotional experience and social interaction values in real-time interaction. This incentivizes both the teachers’ and students’ subjective initiative to the maximum degree. Furthermore, the teachers and students are immersed in vivid classroom activities to significantly improve the learning experience [
iii. Development of the virtuality-reality co-existing evaluation system: An individualized electronic learning portfolio is generated for each learner based on the entire record of classroom activities in the teaching evaluation stage. The teaching is then dynamically adjusted according to the overall and individual portrait analysis to enable students’ learning according to their aptitude.
We consider a course in civil engineering, ‘the steel structure construction technology’ to illustrate the practical application of VR technology in higher vocational education. Due to the complexity of the process, it is almost impossible to have steel structure installations in a lab or a classroom for the students to acquire experience with the actual structures. Therefore, it is challenging for the student to gain mastery of the topic and construction sites’ real situations. Due to the safety risks in construction and the severe shortage of on-site training staff, it is often difficult to arrange on-site practice for the students. Therefore, developing a virtual training system based on VR technology is an ideal solution and an affordable alternative to address the above issues.
This course is composed of three modules: Module 1: Steel Structure Recognition, Module 2: Steel Structure BIM Modeling, and Module 3 Steel Structure Fabrication and Installation. In the XR classroom, the hardware is already available and therefore we focus on developing the software.
From the students’ perspective, it is essential to work with real engineering cases and perform careful analysis. The challenges include designing a customized part of the teaching aid model that can be synchronized in the classroom which cannot be done on-site. The XR resources are then developed and solve the difficult points of the course with the help of XR technology (see,
(1) Development of cloud teaching materials using AR technology: This is to assist reading, simulate the real scenes, and addressing the difficulties in theory and practice, two-dimensional drawing, and three-dimensional real scenes do not correspond well.
(2) Addressing the teaching difficulties in module 1, steel structure drawing recognition: The main objectives here are facilitating the training of spatial thinking, and development of the “Steel Structure 3D Simulation Recognition Training Platform”. We use the detailed drawing of common steel structure joints as the teaching material for two-dimensional design. The three-dimensional model is then generated and displayed. The system is equipped with professional voice-over node explanations and simulated expert on-site teaching. We also provide 3D models that are easily splittable and can be assembled step-by-step to support one-click quick assembly and restoration of the overall components.
Dynamic display of the section process of the joint sections, intelligent matching of joint details corresponds to the section position are also supported. The same interface shows a two-dimensional diagram of the current section, which can be zoomed in and out and also moved. The joint member dimensions can be displayed in three-dimensional space, with a full sense of stereo feeling, and can accurately correspond to the matching position so clearly. The joint members can be also displayed in all directions and all joints can be zoomed and moved by 360 degrees, which is convenient for learning. After students complete the exercises in the training platform, it will automatically come out the score and give a portrait analysis.
(3) Aiming at the error checking of the model, and addressing the teaching difficulty in steel structure BIM modeling in module 2, "one-click VR conversion" software is developed. The 3D model created by the students can be converted by the software and can be imported into the VR system to check and then be modified. Through all these steps, virtual construction is performed.
(4) Aiming at the lack of intuitive sense of the construction process and installation process, and addressing the difficulty in steel structure production and installation in module 3, the “steel structure VR training platform” is developed. This platform consists of three modes including the teaching mode, training mode, and training management mode [
i. Three operational modes:
Teaching mode: In this mode, the steel structure hoisting project construction can be played, and paused. This enables special observation and learning which is suitable for teaching demonstrations by the teachers.
Practical training mode: This is for interactive learning and training in the VR environment. This model combines with the knowledge points to be mastered to set practical training questions. After answering the questions the student can progress to the next process. The number and the content of the questions can be edited.
The training teaching management mode: This mode is accessible through a unified management interface. The main functions include managing the VR equipment terminals, students’ accounts, training course data resource package, and student VR training course task completion status and assessment scores. This mode also analyzes and feedback knowledge blind spots.
ii. Three experiences:
Roaming experience: The user can perform roaming and walking functions in the scene. There are two ways, one is to walk by themselves, and the other is to walk through the handle ray.
Perspective experience: The user can choose a high perspective and perform an interactive experience below the corresponding height.
Construction environment experience: This is designed to enable adjustments on lighting brightness and experience the construction environment at different times.
iii. The two resources:
Machine tool information view: The user can check the XR model of the machine tool used in the construction process.
Relevant resources view: The user can view drawings and drawings, safety and technical instructions, plan review, project summary, and other relevant materials.
Using B/S technology architecture (browser/server mode), the core part of the system function implementation is on the server. Only one browser is installed on the client, e.g., Netscape Navigator, or Internet Explorer, and the server is installed with SQL Server, Oracle, MYSQL, and other database tools. The browser interacts with the database through the Web Server (
The steel structure frame is the considered classroom teaching task, and the steel structure roof truss is the after-school expansion task. The teaching modules include picture recognition, three-dimensional modeling, steel structure production, and installation. Both cases are based on the actual projects. The advantage is that the teaching can be organized in combination with the actual work process and this motivates the students to learn. The main disadvantage is that there are fewer nodes covered which is not conducive to extensive learning of the knowledge points. In this case, different types of nodes are added to the project. To make the selected knowledge points more similar to the actual project, the large samples of the nodes are taken from the national standard atlas.
Module 1 is the learning stage of the steel structure drawing. To read two-dimensional drawings faster, develop web-based steel structure drawing training software, and set up simulation training on the MOOC platform. Which realizes a two-dimensional plane. For the change of thinking in three-dimensional space (
Module 2 is the steel structure BIM modeling. In this project, the student performs BIM modeling according to the drawings. The built model is then transformed into a VR model through the designed software. The model is then checked in the virtual world and, if required, returned to the student for modification. The modified model is then uploaded to the blockchain. and through the blockchain, it is scored by the teacher (
Module 3 is on the production and installation of steel structures. In this project, the steel structure VR training platform, BIM data, videos, and materials can be viewed by the students in real-time in the VR environment. The students can watch realistic construction scenes with digital model through simulation and advanced light effect rendering and rich functional interaction, as well as virtual reproduction construction process (see,
After completing the steel structure construction technology course, the students are asked to fill in an online questionnaire. The seventy-two filled-in questionnaires indicate that the students liked the practical training the most (
1. In this case, the knowledge of steel structure drawing, modeling, production, and installation has been taught through three modules. However, due to the limited time of the class, the mechanical analysis of the connection joint is not described. Therefore, simulation experiment of steel structure joints should be developed, the graphics of experiment outcomes should be visualized, and the analysis of steel structure joint should be conducted visually and in reality to explain the obscure mechanical principles through vivid graphics.
2. VR needs to be equipped with special glasses and handles. Therefore, virtual simulation training courses on the online MOOC platform are still not suitable. It is expected that technological developments soon enable the versatile development of such a system.
3. This case is based on teaching mode of XR, which is a combination of virtual and real environments which enables exposing the students to both theory and practice and enable them understand diagrams, models, manufacturing processes and installation methods. This can be extended to the study of reinforced concrete structure, masonry structure, wood structure, and decoration engineering in civil engineering. This can be also extended to other engineering topics.
In this paper, we outlined the system framework of the virtual and real classroom-based on XR technology. We discussed the steps including developing the XR-based environment, curriculum design, classroom implementation, and evaluation system. The digital multi-scene and multi-resource characteristics of the virtual and real curriculum satisfy the demand for digital transformation and diversified learning and provides unlimited possibilities for future classrooms. The main conclusions are as the following. Technology promotes changes in the classroom teaching from the perspectives of application scenarios, and learner’s demands. In the XR-based virtuality-reality coexisting classroom teaching is the driving force for innovation and reform of classroom-based education. The complicated virtual environment created in a virtuality-reality coexisting classroom facilitates implementing Labs with fixed seats and partition walls cannot adapt to the demands of XR-based virtuality-reality coexisting classrooms. Therefore, smart classrooms with XR equipment are required where seats can are mobilized, and space is divided for physical assembling and virtual construction. The operating mechanism, framework, functions, teaching contents, and teaching design of XR-based virtuality-reality coexisting classroom need further research and development. Such research activities can provide operable and exercisable research results for the teaching reform in the digital age which affects the methods of study, study process, and evaluation.
Development of 5G and front-projected holographic display [
The authors would like to express their gratitude to EditSprings (