TOM-Client-Unity

A Unity implementation of the client to support smart glasses and phones that receive data from the server

• This Unity3D client serves as the primary front-end interface for devices such as smart glasses and phones.
• Users directly interact with this interface.
• It's constructed using MRTK 2.8 and employs web socket communication to connect with the TOM-Server-Python.

Requirements (HoloLens)

To develop for HoloLens 2, you need a Windows 10 or later PC. Follow these steps to set up your development environment:

1. Install Required Tools

Before you begin, make sure you have the necessary prerequisites for HoloLens development. You can find the full list here: HoloLens Development Prerequisites.

• Install the required development tools, including:
  • Unity3D with Universal Windows Platform Build Support
    • Use the version specified at ProjectSettings/ProjectVersion.txt
  • Visual Studio with Universal Windows Platform development or Windows application development

2. Connect HoloLens 2 to Your PC

To connect your HoloLens 2 device to your PC using a browser:

• Use WiFi for connection: Guide to Connect via WiFi.
• If you're connecting for the first time and don't have a username and password:
  • Request a PIN: How to Create a Username and Password.

3. Enable Voice Dictation

For voice dictation on HoloLens, make sure you have:

• An active internet connection.
• Online speech recognition enabled:
  1. Go to Settings → Privacy → Speech.
  2. Turn on "Online speech recognition" and "Speech recognition".

Requirements (XReal, Quest)

To develop for XReal and Quest, follow these steps to set up your development environment:

1. Install Required Development Tools

Before starting, ensure you have the necessary software installed:

• Unity Hub
  • Install Unity3D with the version specified in the ProjectSettings/ProjectVersion.txt file.
  • During installation, make sure to include the following modules:
    • Android Build Support (Required for building applications for XReal and Quest which runs on Android)
    • Visual Studio (Needed for scripting and debugging)

Installation Guide

Follow the steps below to set up and run the project for different platforms.

1. Open the Project in Unity

• Launch Unity Hub and open the project.
• Follow the platform-specific instructions below.

2. HoloLens2

2.1 Configure Build Settings

• Configure only the Build Settings. Follow the guide: Hololens with MRTK.

2.2 Create Configuration File

• Create a tom_config.json file inside the Videos/TOM directory on the HoloLens 2.
• Transfer this file to the device using one of the methods below:
  • WiFi: Use the Device Portal File Explorer.
  • USB Connection: Connect the device as a media device and replace the file manually.

2.3 Add Server Address

• Edit tom_config.json and specify the server address:

  {"host":"<IP_ADDRESS>","port":"8090"}
    ```
  - Note: Both the **HoloLens 2** and the **server** must be on a **private network** (e.g., phone hotspot).
  

2.4 Build & Deploy

• For real device testing: Build the project and deploy it to HoloLens 2.

  • Ensure the correct scene is the first scene in Build Settings (default: Scenes/Home).
  • 

• For simulation using Unity Editor: Use Holographic Remoting Player.

  • Install the Holographic Remoting Player on HoloLens 2.
  • In Unity, go to Mixed Reality -> Remoting -> Holographic Remoting for Play Mode.
  • Enter the HoloLens 2 IP Address and enable remoting (Enable Holographic Remoting for Play Mode).
  • Press Play in Unity to test in real-time.

2.5 Voice Dictation in Simulator

• Ensure your Windows PC has online speech recognition enabled:
• Settings → Privacy & Security → Speech → Turn ON Online Speech Recognition.
  • For voice input in simulator uses the Windows PC's microphone.

2.6 Check Project Settings

• Go to Project Settings → Player → Resolution and Presentation.
• Ensure "Default Orientation" is set to "Auto-rotation".

3. XREAL Setup Guide

Note: The XRSDK package is not included in this repository. You need to download and import it manually.

3.1 Install NRSDK

• Download NRSDK version 1.10.2 XRSDK from: NRSDK 1.10.2 Release Notes & Download

  • Note: Higher versions might work, but they are not tested yet.

• Open Unity, then go to:

  • Assets → Import Package → Custom Package...
  • Select the NRSDK package file you downloaded.

3.2 Configure Unity for XREAL

• Switch platform to Android.
  • Go to File → Build Settings.
  • Select Android as the target platform.
  • Click Switch Platform.
• Add BUILD_XREAL to Scripting Define Symbols:
  • Go to Project Settings → Android Tab → Other Settings → Scripting Define Symbols.
  • Add BUILD_XREAL to it.
  • This enables XREALController code segments.
    • Note: Unity automatically excludes UnityEngine.Windows.WebCam classes when UNITY_ANDROID is used. -- More details on Platform-Dependent Compilation.

3.3 Check Orientation Settings

• Go to Project Settings → Player → Resolution and Presentation → Orientation.
• Ensure "Default Orientation" is set to "Portrait"
  • This setting is auto-applied by the XREAL Project Validator.

4. Simulator (Unity Editor)

4.1 Server Setup

• Ensure both Unity Editor (Simulator) and server are running on the same computer.
• The server address should be 127.0.0.1 (localhost).

4.2 Create Configuration File

• Create a tom_config.json file inside the Videos/TOM directory on your PC.
• Set the server address as:

    {"host":"127.0.0.1","port":"8090"}

Application Execution

To run the application, follow these steps:

1. Start the TOM Server

   • Launch the TOM-Server-Python to establish a socket connection.
   • Ensure the server is running and accessible before starting the client application.

2. Run the Unity Client Application

   • Depending on your platform, start the Unity application on:
     • HoloLens 2
     • XREAL (Nreal)
     • Simulator (Unity Editor)

Development

• For more details on development guidelines and best practices, refer to: 📄 DeveloperGuide.md

References

• Unity Learn
• MixedRealityToolkit-Unity
• mixed-reality-toolkit-project-unity
• HoloLens 2 fundamentals: develop mixed reality applications

Eye tracking

• Eye tracking on HoloLens 2
• Enable eye-tracking
• Getting started with eye tracking in MRTK2
• Eyes and hands — MRTK2