All Case Studies

Tutorio

September, 2023

Tutorio is a Mixed Reality HoloLens application designed for creating and sharing interactive spatial workspace guides. Built with Azure Spatial Anchors, cloud synchronization, and a custom MVVM framework, it supports seamless collaboration and precise tutorial placement

Introduction

• Overview: Tutorio is a Mixed Reality application for HoloLens that enables users to create, edit, and follow virtual workspace instructions. The application evolved from a prototype to a polished product over two major releases on the Microsoft Store during its three-year development, leveraging cutting-edge technologies like Azure Spatial Anchors and Azure Blob Storage for robust and scalable functionality.

• Collaboration: Tutorio’s success was a highly collaborative effort. As the only Unity HoloLens developer on the project, I worked closely with our project leader, two designers, and a web developer who built the Blazor-based web app. Initially, I developed the web app’s first version before handing it off to the web developer. Customer feedback played a crucial role in shaping the development and ensuring the app met real-world needs. This synergy ensured that all components – HoloLens, web app, and user experience – worked seamlessly together.

• Motivation: The challenge was to transform a promising concept into a full-featured application. Starting with a prototype, I developed scalable systems and intuitive interactions, ensuring Tutorio’s operation across real-world environments.

Key Features

• Persistent Spatial Anchors: Azure Spatial Anchors ensure tutorial steps stay accurately positioned in the physical world across sessions and devices. Anchors update in real-time, providing continuity for multiple users.
• Interactive Tutorials: Users create multi-step guides with spatially anchored text, images, and video content, with each step fully adjustable in 3D space.
• Media Capture and Playback: Record images or videos directly in tutorial steps. Files are uploaded to Azure Blob Storage for persistent cloud access.
• Hand Menu: A custom, hands-free menu appears by looking at your palm. It follows the user’s hand dynamically and closes automatically when no longer needed.
• Role-Based Access Control: User roles dictate editing and viewing permissions. The UI adapts dynamically based on role, simplifying the experience for end users.
• Multi-Column Layout and Pagination: A custom layout component adjusts dynamically based on the number of tutorials, paginating content for larger lists.
• Navigator Arrow: Guides users to tutorial steps when out of sight, improving navigation and usability.
• Upload Queue System: Handles media uploads asynchronously, queuing files for reliable transfers even with Wi-Fi interruptions or app restarts.
• Authentication and Integration: Uses Microsoft Authentication Library (MSAL) for secure sign-in, with data synchronized through the Microsoft Graph API.
• Custom Data Binding System: A lightweight MVVM-style binding system supports automatic updates between the UI and ViewModels when the data changes.
• Voice Operation: The entire app is voice-operated using MRTK features. Users can activate controls by speaking out their labels, and tutorial step text can be dictated through voice commands.
• Localization: The app supports full localization, including UI text and voice dictation in multiple languages.

Key Challenges & Solutions

1. Persistent Virtual Content with Azure Spatial Anchors

• Challenge: Maintaining spatially anchored tutorial steps across sessions and devices was essential for a consistent user experience.
• Solution: Used Azure Spatial Anchors SDK to anchor tutorial steps in the physical environment, enabling synchronization across devices. Anchors and updates are stored and retrieved from the cloud in real time, ensuring persistence and accuracy.

2. Scalable Cloud Data Management

• Challenge: Storing and accessing tutorial data, including media files, required a reliable and scalable solution.
• Solution: Integrated Azure Blob Storage to handle data persistence, with tutorials automatically synchronized to ensure consistent access across devices.

3. Annotation System

• Challenge: Enable users to create visual annotations in physical or virtual spaces for instructional clarity.
• Solution: Implemented a 3D annotation system using fingertip tracking with pinching gestures to capture spatial (Vector3) data. Annotations were rendered with Unity’s LineRenderer, supporting customizable colors and positioning on spatial meshes (physical real life environment) or mid-air. Saved annotations were stored in the cloud alongside other tutorial data.

4. Spatial UI vs. 2D UI

• Challenge: Developing intuitive layouts, interactions, and navigation in 3D required a different approach from traditional 2D UI paradigms.
• Solution: Utilized MRTK3 for spatial UI interactions and developed custom components for menus, pagination, and dynamic placement. Iterative prototyping and testing refined interactions to ensure usability and consistency in 3D space.

5. Architectural Scalability

• Challenge: The initial prototype’s lack of structure prevented maintainability as the project expanded.
• Solution: Adopted MVVM architecture with Extenject for dependency injection, ensuring modularity and a clear separation of UI, logic, and data layers for scalable development.

6. Data Binding in Unity

• Challenge: Unity’s UGUI lacks native data binding support, essential for MVVM.
• Solution: Developed a custom data binding system using reflection, expressions, and events. Integrated Loxodon Fody to automate INotifyPropertyChanged boilerplate, simplifying implementation.

7. Media Synchronization and Reliability

• Challenge: Slow Wi-Fi and app interruptions caused delays and data loss during media uploads.
• Solution: Implemented an asynchronous upload queue system with resumable uploads. Metadata for each file was stored locally, allowing the system to check for incomplete uploads and retry after interruptions, such as a device running out of battery.

8. Hands-Free Interaction

• Challenge: Default MRTK menu interactions required two hands and would close when the user palm was not visible, conflicting with our hands-free interaction goals.
• Solution: Developed a custom hand menu activated by looking at the user’s palm. The menu persists in space even after the palm is no longer visible, allowing the same hand to interact with it. The menu automatically closes when the hand moves beyond a set distance from the menu, providing a flexible and intuitive experience.

9. Custom UI Components

• Challenge: Unity’s default UI layout tools lacked flexibility for specific design needs like multi-column layouts.
• Solution: Developed a MultiColumnLayoutGroup component that calculates and assigns child positions in a variable column formation with adjustable spacing.

Impact

I developed Tutorio from a basic prototype into a fully realized application released on the Microsoft Store. It was adopted by paying customers who used it to solve real-world challenges in their organizations. This project demonstrated the value of Mixed Reality in enhancing workflows and training processes, providing a robust, scalable solution tailored to customer needs.

Key Takeaways

Developing Tutorio highlighted the complexities of Mixed Reality development, from spatial UI design to overcoming HoloLens-specific challenges. Over three years, I tackled persistent spatial content, cloud integration, and performance optimization while navigating the intricacies of publishing on the Microsoft Store. This project deepened my expertise in Unity, Azure, and scalable architectures, and proves the potential of Mixed Reality in workplace transformation.