When AI and virtual reality converge

IEC E-tech

article here

Artificial intelligence (AI) is enabling extended reality environments (XR) to push boundaries even further. While standards for XR exist, more will be required to foster interoperability between both worlds.

 

Over the last ten years, technologies such as artificial intelligence, machine learning (ML), virtual reality (VR) and augmented reality (AR) have all advanced at a remarkable speed. At the same time, innovations in optics, display engineering, microprocessors and rendering algorithms have helped to create increasingly sophisticated immersive environments. Today, people can explore a wide range of virtual spaces, whether they use a headset or not, from videogames and virtual tours to social platforms and educational tools.

While VR and AR technologies (usually bracketed under XR for extended reality) have developed independently of AI/ML, they are increasingly converging.

“AI can enhance XR in numerous ways,” says Dr Takeshi Kurata, Director, Research Institute on Human and Societal Augmentation, AIST in Japan, and the Acting Chair of the joint technical committee between ISO and IEC,  ISO/IEC JTC 1/SC 24, responsible for work on computer graphics, image processing and environmental data representation. “AI helps create, maintain and update virtual content at a far lower cost than manual methods,” he adds.

For example, when re-creating real‑world environments, human appearances, or objects, generative AI can deliver hyper realistic results at a fraction of the cost of conventional methods. The caveat is that unless checked by human expertise, this can sometimes lead to hallucinations.

How AI can elevate XR

A growing field is augmented intelligence - using AI to enhance human capability rather than replace it. When paired with VR or AR, augmented intelligence could transform creative and industrial workflows.

Nearly every XR application requires onboarding or instructions. Instead of delivering training outside the immersive environment, XR can embed it directly into the experience, much like videogames that teach players through interactive play rather than manuals.

“People learn better when practice is interactive, contextual, safe, and supported by intelligent guidance,” says Donggil Song, Associate Professor of Engineering Technology and Industrial Distribution at Texas A&M University. “One of the clearest examples is the creation of virtual assistants capable of understanding natural speech and offering personalized guidance.” This approach can be extended far beyond application tutorials. When combined with AI techniques such as natural language processing, sentiment analysis and pathfinding, VR becomes a powerful platform for education and skills development.

“XR already gives us immersive learning or training experiences, but AI gives us intelligence inside that experience,” says Song, whose work at EinBrain Lab focuses on combining AI with XR and data‑driven feedback to improve learning and human performance. EinBrain’s flagship project is AlgeVerse, a fully VR, gamified college algebra platform that uses AI-based peer mentors to give the learner guidance, feedback, and suggestions. The project, which is in year one of a three‑year cycle, supports natural communication and is expanding into a multi‑user environment. “Users might not be able to differentiate which avatars are real students and which are AI peer mentors,” Song explains.

The lab has also created a playful VR environment called Prehistoric Protocols to make learning about computer networking more engaging. “Students travel to a [virtual] prehistoric world with cavemen and dinosaurs. Students must teach them using foundational IT concepts,” Song describes.

AI-powered characters in video games

The video-game sector is one of the earliest fields to adopt practical AI in real products. Modern games use AI for world generation, pathfinding, data analysis and player‑experience modelling. One of the most intriguing recent developments is the population of virtual environments with non‑player characters (NPCs) and AI agents. “As it becomes more difficult to distinguish whether entities are controlled by humans or AI, it may eventually become unclear whether a human society actually exists inside the environment at all,” notes Kurata. “From this perspective, I believe AI-driven NPC control is one of the most impactful developments in this field.”

In rural areas of Japan facing depopulation, there are places where large numbers of scarecrows are placed outdoors to ease the loneliness of declining communities. “In a sense, NPCs can be viewed as a virtual counterpart of such scarecrows,” he suggests. “However, NPCs are controlled by AI in the backend, making them far more realistic and interactive, even though their existence is virtual compared with physical scarecrows.”

Challenges include computational resources

Increased adoption of 5G connectivity will accelerate AI-driven XR applications. With high bandwidth, low latency and the ability to connect vast numbers of devices, 5G enables new forms of edge and cloud computing. This could offload heavy processing from headsets, allowing them to become lighter, cooler and more power‑efficient while still delivering richer experiences.

However, integrating AI into VR and AR is not without constraints. AI workloads often demand substantial computational resources, which increases power draw and heat, potentially making headset devices heavier or less comfortable. Research is underway to create more efficient AI‑specific chips and to run AI on low‑power processors, but as AI applications grow more complex, the tension between capability and efficiency is likely to persist.

Kurata says, “Often we see evaluation indicators like accuracy or precision. However, there are often no metrics for energy effectiveness. If we successfully standardize such metrics for industry, competing companies in this space may try to reduce energy consumption as much as possible.”

Other challenges are social rather than technical. Bias is one of the most serious. Any AI system can reflect the biases present in its training data. Speech recognition, for instance, often performs best for accents similar to those of the engineers who built the system—typically male, English‑speaking developers with American, Indian or Chinese accents. Consequently, it is essential to evaluate AI systems to avoid unintended harm or discrimination.

The need for interoperability standards

In addition to metrics and standards relating to the energy efficiency of these converging systems, the field urgently needs national and international standards for interoperability purposes, according to many practitioners in the field, including Song. “Developing for multiple headsets is easier than before, but still requires significant work,” he says. “Standards reduce friction and improve interoperability, safety, reliability and quality assurance.  If we have strong standards between AI and XR, compatibility and interoperability can be solved. Then more research and more products become possible.”

A foundational step is to agree on XR-related terminology without which, Kurata argues, there will be no standards and no future.

Terminology is where it all starts

Kurata’s argument, outlined in an IEEE paper, is that when concepts span multiple domains, such as AI, XR and the Metaverse, ambiguity in terminology often leads to misunderstanding and hinders effective collaboration.  “Much like technical interoperability in software and systems, the clarification of terminology facilitates semantic interoperability, which serves as a foundation for cooperation among diverse stakeholders,” he states.

The term XR is currently widely used as an expression encompassing VR, AR, and MR. However, there is no clear consensus regarding its origin or meaning.  “XR is sometimes explained as an abbreviation for Extended Reality, but multiple interpretations exist regarding its etymology and formation process,” Kurata writes in the paper XR is XR: Rethinking MR and XR as Neutral Umbrella Terms published earlier this year.

He suggests that XR functions as a “neutral symbolic label” by encompassing multiple “reality”-related terms. Stable usage of such terminology requires governance through collaboration among academia, industry, and standardisation organisations, he insists.

What standards are in the pipeline?

Kurata points to two XR-related standards currently under development that have a strong relevance to AI. The first, ISO/IEC AWI 26073, covers spatio-temporal mixed and augmented reality experience description (MAR-ED) for interactive playback

“MAR-ED is highly compatible with AI because it represents experiences semantically through meaningful events, interactions, and narrative structures rather than only raw spatial or visual data. This enables AI agents or AI avatars to understand, adapt, and dynamically interact with immersive experiences, including adaptive playback and real-time branching. In this sense, MAR-ED can also serve as a semantic experiential framework for AI-driven XR and metaverse applications.”

Related work categorized under document ISO/IEC CD 25767 specifically focuses on avatar face representation for XR communication. It aims to standardize the human-to-3D avatar face modelling process, tailored for users of XR glasses. Kurata says this work is highly relevant to AI because realistic avatar face generation and real-time facial animation rely heavily on AI-based face tracking, expression recognition, and generative modelling technologies.

As both technologies increasingly merge, more standards will be required. IEC and ISO are on the case.