METAL Plus: Context-aware multi-user MR collaborative teaching system

The Collaborative Virtual Environments (CVEs) created by Mixed Reality (MR) technologies have been classified as symmetric and asymmetric CVEs, where the latter aim to provide different authorities for different collaborator roles utilizing heterogeneous techniques that cover the full gamut of Milgram’s Mixed Reality continuum. The Light Field Display (LFD), as a new type of MR display that generates an autostereoscopic viewing experience without head-mounted devices, has been incorporated with AR and VR headsets to create remote and co-located asymmetric collaborative environments. In previous asymmetric CVE research, LFDs were adapted to simultaneously render multi-contents for multiple students to lower down average device costs for the MR vet training. However, multiple students sharing one LFD to interact with the teacher may weaken the teacher’s understanding of individual student’s current learning progress, which may make teaching decisions even harder.

Therefore, this project provides an enhanced solution that supports teaching decisions targeted at each student without increasing the device costs. The context-aware LFD student clients, which render a dynamic viewing zone for each student by face encoding tracking, are implemented and applied for anti-cheat quiz support. By synchronizing each student’s tracking data with a Local Area Network (LAN) middleware, the AR teacher client can distinguish different students to in-situ superimpose the quiz progress and targeted-explainable teaching decision support over each corresponding student’s head.

This system is developed on the basis of prior METAL co-located content sharing system, which achieves more teaching decision support features and the novel anti-cheat quiz assistance feature for co-located teaching application scenarios. This system still maintains the high-level METAL setup for one AR teacher client and multiple LFD student clients, while instead of communicating both clients using Wide Area Network (WAN), a LAN synchronization middleware is developed for speedy and secure network communication between the co-located teacher and student clients. In addition, to allow for teaching decisions targeted at each individual student, and LFD provides each student with a context-aware dynamic privacy viewing zone that follows their real-time movement. By doing so, the teacher client is allowed to distinguish different students and adjust the teaching materials targeted at each student. Exploiting such a dynamic privacy viewing zone, an anti-cheat quiz support is integrated to both clients, which further allows for the in-situ visualization of the explainable teaching decision targeted at each individual student.

In this project, I designed and developed the LFD clients and network middleware. This paper is currently under review by IEEE Access.