Haptic Harmonics

Project Overview

Haptic Harmonics is a mixed reality music creation experience developed during the Immersive Tech Week 2023 Hackathon. Using the Varjo XR-3 headset for high-fidelity mixed reality and Manus haptic gloves for tactile feedback, users can compose custom music tracks by physically interacting with virtual instruments that feel real to the touch.

The project won Best Use of Immersive Technologies at the event, recognized for its innovative combination of haptics, spatial computing, and musical expression.

Technical Implementation

• Varjo XR-3 Integration — Leveraging the XR-3's human-eye-resolution passthrough and wide field of view for seamless blending of virtual instruments with the real environment
• Manus Haptic Gloves — Full finger tracking with vibrotactile feedback at each fingertip, creating the sensation of touching, plucking, and striking virtual instruments
• Spatial Music Engine — Custom Unity-based audio engine that places sound sources in 3D space, with positional audio that responds to head and hand movement
• Instrument Physics — Each virtual instrument has unique haptic profiles: strings vibrate with different frequencies, drums provide percussive thumps, and wind instruments simulate air pressure
• Multi-Track Composition — Users can layer instruments, record loops, and arrange compositions by physically moving and placing sound objects in the space around them
• Performance Optimization — Real-time haptic feedback and spatial audio processing while maintaining high framerate on the Varjo XR-3's demanding display resolution

Design Approach

Music creation in traditional DAWs (Digital Audio Workstations) is a visual, mouse-driven process. Haptic Harmonics asked: what if making music was physical? The design replaces sliders and menus with tangible instruments that exist in your space. A virtual synthesizer sits on your left, a drum pad floats to your right, and strings stretch across the room in front of you.

The haptic feedback was the key innovation — without it, virtual instruments feel like air-guitar. With the Manus gloves, users reported feeling genuine resistance when plucking strings and satisfying thuds when hitting drums. This tactile dimension transformed the experience from a gimmick into a genuinely new way of composing.

Key Results & Impact

• Winner — Best Use of Immersive Technologies at Immersive Tech Week 2023
• First-of-its-kind — Combined enterprise-grade XR (Varjo) with professional haptics (Manus) for creative expression
• Live demonstration — Presented and demoed at the Immersive Tech Week conference in Rotterdam
• Industry recognition — Featured on LinkedIn with significant engagement from the XR professional community

Tech Stack