A Unity implementation of the client to support smart glasses and phones that receive data from the server
- This Unity3D client serves as the primary front-end interface for devices such as smart glasses and phones.
- Users directly interact with this interface.
- It's constructed using MRTK 2.8 and employs web socket communication to connect with the TOM-Server-Python.
To develop for HoloLens 2, you need a Windows 10 or later PC. Follow these steps to set up your development environment:
Before you begin, make sure you have the necessary prerequisites for HoloLens development. You can find the full list here: HoloLens Development Prerequisites.
- Install the required development tools, including:
- Unity3D with Universal Windows Platform Build Support
- Use the version specified at
ProjectSettings/ProjectVersion.txt
- Use the version specified at
- Visual Studio with Universal Windows Platform development or Windows application development
- Unity3D with Universal Windows Platform Build Support
To connect your HoloLens 2 device to your PC using a browser:
- Use WiFi for connection: Guide to Connect via WiFi.
- If you're connecting for the first time and don't have a username and password:
- Request a PIN: How to Create a Username and Password.
For voice dictation on HoloLens, make sure you have:
- An active internet connection.
- Online speech recognition enabled:
- Go to Settings → Privacy → Speech.
- Turn on "Online speech recognition" and "Speech recognition".
To develop for XReal and Quest, follow these steps to set up your development environment:
Before starting, ensure you have the necessary software installed:
- Unity Hub
- Install Unity3D with the version specified in the
ProjectSettings/ProjectVersion.txt
file. - During installation, make sure to include the following modules:
- Android Build Support (Required for building applications for XReal and Quest which runs on Android)
- Visual Studio (Needed for scripting and debugging)
- Install Unity3D with the version specified in the
Follow the steps below to set up and run the project for different platforms.
- Launch Unity Hub and open the project.
- Follow the platform-specific instructions below.
- Configure only the Build Settings. Follow the guide: Hololens with MRTK.
- Create a
tom_config.json
file inside theVideos/TOM
directory on the HoloLens 2. - Transfer this file to the device using one of the methods below:
- WiFi: Use the Device Portal File Explorer.
- USB Connection: Connect the device as a media device and replace the file manually.
- Edit
tom_config.json
and specify the server address:{"host":"<IP_ADDRESS>","port":"8090"} ``` - Note: Both the **HoloLens 2** and the **server** must be on a **private network** (e.g., phone hotspot).
-
For real device testing: Build the project and deploy it to HoloLens 2.
-
For simulation using Unity Editor: Use Holographic Remoting Player.
- Install the Holographic Remoting Player on HoloLens 2.
- In Unity, go to
Mixed Reality -> Remoting -> Holographic Remoting for Play Mode
. - Enter the HoloLens 2 IP Address and enable remoting (
Enable Holographic Remoting for Play Mode
). - Press
Play
in Unity to test in real-time.
- Ensure your Windows PC has online speech recognition enabled:
Settings → Privacy & Security → Speech
→ Turn ONOnline Speech Recognition
.- For voice input in simulator uses the Windows PC's microphone.
- Go to
Project Settings → Player → Resolution and Presentation
. - Ensure "Default Orientation" is set to "Auto-rotation".
Note: The XRSDK package is not included in this repository. You need to download and import it manually.
-
Download NRSDK version 1.10.2 XRSDK from: NRSDK 1.10.2 Release Notes & Download
- Note: Higher versions might work, but they are not tested yet.
-
Open Unity, then go to:
Assets → Import Package → Custom Package...
- Select the NRSDK package file you downloaded.
- Switch platform to Android.
- Go to
File → Build Settings
. - Select Android as the target platform.
- Click Switch Platform.
- Go to
- Add
BUILD_XREAL
to Scripting Define Symbols:- Go to
Project Settings → Android Tab → Other Settings → Scripting Define Symbols
. - Add
BUILD_XREAL
to it. - This enables XREALController code segments.
- Note: Unity automatically excludes
UnityEngine.Windows.WebCam
classes whenUNITY_ANDROID
is used. -- More details on Platform-Dependent Compilation.
- Note: Unity automatically excludes
- Go to
- Go to
Project Settings → Player → Resolution and Presentation → Orientation
. - Ensure "Default Orientation" is set to "Portrait"
- This setting is auto-applied by the XREAL Project Validator.
- Ensure both Unity Editor (Simulator) and server are running on the same computer.
- The server address should be 127.0.0.1 (localhost).
- Create a
tom_config.json
file inside theVideos/TOM
directory on your PC. - Set the server address as:
{"host":"127.0.0.1","port":"8090"}
To run the application, follow these steps:
-
Start the TOM Server
- Launch the TOM-Server-Python to establish a socket connection.
- Ensure the server is running and accessible before starting the client application.
-
Run the Unity Client Application
- Depending on your platform, start the Unity application on:
- HoloLens 2
- XREAL (Nreal)
- Simulator (Unity Editor)
- Depending on your platform, start the Unity application on:
- For more details on development guidelines and best practices, refer to: 📄 DeveloperGuide.md