All Case Studies

Tutorio

September, 2023

A Mixed Reality HoloLens app enabling users to create and share virtual workspace instructions. Developed it over three years as the sole Unity developer, in collaboration with a design and feedback team. Features include Azure Spatial Anchors, Azure cloud integration, MVVM architecture, and cloud-synced tutorials.

Introduction

• Overview: Tutorio is a Mixed Reality application for HoloLens that enables users to create, edit, and follow virtual workspace instructions. The application evolved from a prototype to a polished product over two major releases on the Microsoft Store during its three-year development, leveraging cutting-edge technologies like Azure Spatial Anchors and Azure Blob Storage for robust and scalable functionality.

• Collaboration: Tutorio’s success was a highly collaborative effort. As the only Unity HoloLens developer on the project, I worked closely with our project leader, two designers, and a web developer who built the web app. Initially, I developed the Blazor-based web app’s first version before handing it off to a dedicated web developer. Customer feedback played a crucial role in shaping the development and ensuring the app met real-world needs. This synergy ensured that all components—HoloLens, web app, and user experience—worked seamlessly together.

• Motivation: The challenge was to transform a promising concept into a full-featured application. Starting with a prototype, I designed and implemented scalable systems and intuitive interactions, ensuring Tutorio’s operation across real-world environments.

Key Features

• Persistent Spatial Anchors: Azure Spatial Anchors ensure tutorial steps stay accurately positioned in the physical world across sessions and devices. Anchors update in real-time, providing continuity for multiple users.
• Interactive Tutorials: Users create multi-step guides with spatially anchored text, images, and video content, with each step fully adjustable in 3D space.
• Media Capture and Playback: Record images or videos directly in tutorial steps. Files are uploaded to Azure Blob Storage for persistent cloud access.
• Hand Menu: A custom, hands-free menu appears by looking at your palm. It follows the user’s hand dynamically and closes automatically when no longer needed.
• Role-Based Access Control: User roles dictate editing and viewing permissions. The UI adapts dynamically based on role, simplifying the experience for end users.
• Multi-Column Layout and Pagination: A custom layout component adjusts dynamically based on the number of tutorials, paginating content for larger lists.
• Navigator Arrow: Guides users to tutorial steps when out of sight, improving navigation and usability.
• Upload Queue System: Handles media uploads asynchronously, queuing files for reliable transfers even with slow Wi-Fi or app restarts.
• Authentication and Integration: Uses Microsoft Authentication Library (MSAL) for secure sign-in, with data synchronized through the Microsoft Graph API.
• Custom Data Binding System: A lightweight MVVM-style binding system supports automatic updates between the UI and ViewModels when the data changes.
• Voice Operation: The entire app is voice-operated using MRTK features. Users can activate controls by speaking their names, and tutorial step text can be dictated through voice commands.
• Localization: The app supports full localization, including UI text and voice dictation in multiple languages.

Key Challenges & Solutions

1. Persistent Virtual Content with Azure Spatial Anchors

• Challenge: Maintaining spatially anchored tutorial steps across sessions and devices was essential for a consistent user experience.
• Solution: Used Azure Spatial Anchors SDK to anchor tutorial steps in the physical environment, enabling synchronization across devices. Anchors and updates are stored and retrieved from the cloud in real time, ensuring persistence and accuracy.

2. Scalable Cloud Data Management

• Challenge: Storing and accessing tutorial data, including media files, required a reliable and scalable solution.
• Solution: Integrated Azure Blob Storage to handle data persistence, with tutorials automatically synchronized to ensure consistent access across devices.

3. Annotation System

• Challenge: Enable users to create visual annotations in physical or virtual spaces for instructional clarity.
• Solution: Implemented a 3D annotation system using fingertip tracking with pinching gestures to capture spatial (Vector3) data. Annotations were rendered with Unity’s LineRenderer, supporting customizable colors and positioning on spatial meshes or mid-air. Saved annotations were stored in the cloud alongside other tutorial data.

4. Spatial UI vs. 2D UI

• Challenge: Designing intuitive layouts, interactions, and navigation in 3D required a different approach from traditional 2D UI paradigms.
• Solution: Utilized MRTK3 for spatial UI interactions and developed custom components for menus, pagination, and dynamic placement. Iterative prototyping and testing refined interactions to ensure usability and consistency in 3D space.

5. Architectural Scalability

• Challenge: The initial prototype’s lack of structure hindered maintainability as the project expanded.
• Solution: Adopted MVVM architecture with Extenject for dependency injection, ensuring modularity and a clear separation of UI, logic, and data layers for scalable development.

6. Data Binding in Unity

• Challenge: Unity’s UGUI lacks native data binding support, essential for MVVM.
• Solution: Developed a custom data binding system using reflection, expressions, and UnityEvents. Integrated Fody to automate INotifyPropertyChanged boilerplate, simplifying implementation.

7. Media Synchronization and Reliability

• Challenge: Slow Wi-Fi and app interruptions caused delays and data loss during media uploads.
• Solution: Implemented an asynchronous upload queue system with resumable uploads. Metadata for each file was stored locally, allowing the system to check for incomplete uploads and retry after interruptions, such as a device running out of battery.

8. Hands-Free Interaction

• Challenge: Default MRTK menus required two hands and closed when the palm was not visible, conflicting with our hands-free interaction goals.
• Solution: Developed a custom hand menu activated by looking at the user’s palm. The menu persists in space even after the palm is no longer visible, allowing the same hand to interact with it. The menu automatically closes when the hand moves beyond a set distance, providing a flexible and intuitive experience.

9. Custom UI Components

• Challenge: Unity’s default layout tools lacked flexibility for specific design needs like multi-column layouts.
• Solution: Developed a MultiColumnLayoutGroup component that calculates and assigns child positions in a variable column formation with adjustable spacing.

Impact

I developed Tutorio from a basic prototype into a fully realized application released on the Microsoft Store. It was adopted by paying customers who used it to solve real-world challenges in their organizations. This project demonstrated the value of Mixed Reality in enhancing workflows and training processes, providing a robust, scalable solution tailored to customer needs.

Key Takeaways

Developing Tutorio highlighted the complexities of Mixed Reality development, from spatial UI design to overcoming HoloLens-specific challenges. Over three years, I tackled persistent spatial content, cloud integration, and performance optimization while navigating the intricacies of publishing on the Microsoft Store. This project deepened my expertise in Unity, Azure, and scalable architectures, and proves the potential of Mixed Reality in workplace transformation.