Skip to content

A Unity implementation of the client to support smart glasses and phones that receive data from the server

License

Notifications You must be signed in to change notification settings

TOM-Platform/TOM-Client-Unity

Repository files navigation

TOM-Client-Unity

A Unity implementation of the client to support smart glasses and phones that receive data from the server

  • This Unity3D client serves as the primary front-end interface for devices such as smart glasses and phones.
  • Users directly interact with this interface.
  • It's constructed using MRTK 2.8 and employs web socket communication to connect with the TOM-Server-Python.

Requirements (HoloLens)

To develop for HoloLens 2, you need a Windows 10 or later PC. Follow these steps to set up your development environment:

1. Install Required Tools

Before you begin, make sure you have the necessary prerequisites for HoloLens development. You can find the full list here: HoloLens Development Prerequisites.

2. Connect HoloLens 2 to Your PC

To connect your HoloLens 2 device to your PC using a browser:

3. Enable Voice Dictation

For voice dictation on HoloLens, make sure you have:

  • An active internet connection.
  • Online speech recognition enabled:
    1. Go to SettingsPrivacySpeech.
    2. Turn on "Online speech recognition" and "Speech recognition".

Requirements (XReal, Quest)

To develop for XReal and Quest, follow these steps to set up your development environment:

1. Install Required Development Tools

Before starting, ensure you have the necessary software installed:

  • Unity Hub
    • Install Unity3D with the version specified in the ProjectSettings/ProjectVersion.txt file.
    • During installation, make sure to include the following modules:
      • Android Build Support (Required for building applications for XReal and Quest which runs on Android)
      • Visual Studio (Needed for scripting and debugging)

Installation Guide

Follow the steps below to set up and run the project for different platforms.

1. Open the Project in Unity

  • Launch Unity Hub and open the project.
  • Follow the platform-specific instructions below.

2. HoloLens2

2.1 Configure Build Settings

2.2 Create Configuration File

  • Create a tom_config.json file inside the Videos/TOM directory on the HoloLens 2.
  • Transfer this file to the device using one of the methods below:

2.3 Add Server Address

  • Edit tom_config.json and specify the server address:
    {"host":"<IP_ADDRESS>","port":"8090"}
      ```
    - Note: Both the **HoloLens 2** and the **server** must be on a **private network** (e.g., phone hotspot).
    

2.4 Build & Deploy

  • For real device testing: Build the project and deploy it to HoloLens 2.

    • Ensure the correct scene is the first scene in Build Settings (default: Scenes/Home).
    • Build Settings Screenshot
  • For simulation using Unity Editor: Use Holographic Remoting Player.

    • Install the Holographic Remoting Player on HoloLens 2.
    • In Unity, go to Mixed Reality -> Remoting -> Holographic Remoting for Play Mode.
    • Enter the HoloLens 2 IP Address and enable remoting (Enable Holographic Remoting for Play Mode).
    • Press Play in Unity to test in real-time.

2.5 Voice Dictation in Simulator

  • Ensure your Windows PC has online speech recognition enabled:
  • Settings → Privacy & Security → Speech → Turn ON Online Speech Recognition.
    • For voice input in simulator uses the Windows PC's microphone.

2.6 Check Project Settings

  • Go to Project Settings → Player → Resolution and Presentation.
  • Ensure "Default Orientation" is set to "Auto-rotation".

3. XREAL Setup Guide

Note: The XRSDK package is not included in this repository. You need to download and import it manually.

3.1 Install NRSDK

  • Download NRSDK version 1.10.2 XRSDK from: NRSDK 1.10.2 Release Notes & Download

    • Note: Higher versions might work, but they are not tested yet.
  • Open Unity, then go to:

    • Assets → Import Package → Custom Package...
    • Select the NRSDK package file you downloaded.

3.2 Configure Unity for XREAL

  • Switch platform to Android.
    • Go to File → Build Settings.
    • Select Android as the target platform.
    • Click Switch Platform.
  • Add BUILD_XREAL to Scripting Define Symbols:
    • Go to Project Settings → Android Tab → Other Settings → Scripting Define Symbols.
    • Add BUILD_XREAL to it.
    • This enables XREALController code segments.

3.3 Check Orientation Settings

  • Go to Project Settings → Player → Resolution and Presentation → Orientation.
  • Ensure "Default Orientation" is set to "Portrait"

4. Simulator (Unity Editor)

4.1 Server Setup

  • Ensure both Unity Editor (Simulator) and server are running on the same computer.
  • The server address should be 127.0.0.1 (localhost).

4.2 Create Configuration File

  • Create a tom_config.json file inside the Videos/TOM directory on your PC.
  • Set the server address as:
      {"host":"127.0.0.1","port":"8090"}

Application Execution

To run the application, follow these steps:

  1. Start the TOM Server

    • Launch the TOM-Server-Python to establish a socket connection.
    • Ensure the server is running and accessible before starting the client application.
  2. Run the Unity Client Application

    • Depending on your platform, start the Unity application on:
      • HoloLens 2
      • XREAL (Nreal)
      • Simulator (Unity Editor)

Development

  • For more details on development guidelines and best practices, refer to: 📄 DeveloperGuide.md

References

Eye tracking

About

A Unity implementation of the client to support smart glasses and phones that receive data from the server

Resources

License

Stars

Watchers

Forks

Packages

No packages published