Immersive Research Environment for Cultural Heritage

Project Overview

In collaboration with SURF (the Dutch collaborative organization for educational IT), this project created an immersive research environment for cultural heritage. The application allows researchers and students to interact with high-fidelity 3D-scanned cultural objects in a Mixed Reality space, providing contextual information and natural interaction powered by multiple AI models — all running on edge infrastructure hosted by SURF.

The goal was to demonstrate how immersive technologies combined with edge-hosted AI can transform access to cultural heritage collections, making them explorable in ways that physical exhibits cannot match.

Technical Implementation

• Mixed Reality Framework — Built with AR Foundation and Meta SDK for spatial mapping, allowing virtual artifacts to sit naturally on real-world surfaces
• 4 LLM Models at the Edge — Speech-to-text, text generation (RAG on cultural heritage datasets), image generation (DALL-E style), and text-to-speech — all hosted on SURF's edge infrastructure
• Unified Application Architecture — A single application managing all 4 LLM pipelines simultaneously, with intelligent routing between models based on user intent
• 3D Object Interaction — Grabbing, rotating, scaling, and inspecting photogrammetry-scanned artifacts with realistic physics and haptic feedback
• Spatial Audio Cues — Audio feedback tied to artifact positions, guiding users through the virtual exhibition space
• Edge Computing Integration — Low-latency AI inference running on SURF's edge nodes, keeping data processing close to the user for responsive interactions

Design Approach

Cultural heritage researchers are not typically XR users, so the interface was designed for zero learning curve. Users enter the space and see artifacts arranged around them. Speaking naturally activates the AI — "Tell me about this vase" triggers the text generation LLM, which retrieves information from a curated knowledge base of museum metadata. The system combines visual, auditory, and textual information to create a rich, multimodal research experience.

The spatial layout mirrors a physical gallery, with artifacts grouped by historical period and geography. Users can teleport between groupings, pull up detailed information panels, and even generate visual reconstructions of how artifacts might have looked when new.

Key Results & Impact

• 4 simultaneous LLM models managed in a single XR application
• Edge-hosted AI with sub-second response times for text generation
• Demonstrated at SURF events to showcase the potential of XR + AI in education
• Proved the concept of combining MR with edge computing for cultural heritage research

Tech Stack