PrISM: Procedure Interaction from Sensing Module

This is a repository for the research code in which we aim to develop a real-time intelligent assistant that navigates users through dialogues during procedural tasks (e.g., cooking, latte-making, medical self-care).

Publications

The code is structured in a modular manner, from underlying sensing mechanisms to user interactions. We value your citation of the relevant publication.

• PrISM-Tracker: A Framework for Multimodal Procedure Tracking Using Wearable Sensors and State Transition Information with User-Driven Handling of Errors and Uncertainty. Riku Arakawa, Hiromu Yakura, Vimal Mollyn, Suzanne Nie, Emma Russell, Dustin P. Dimeo, Haarika A. Reddy, Alexander K. Maytin, Bryan T. Carroll, Jill Fain Lehman, Mayank Goel. Proceedings of the ACM on Interactive Mobile Wearable Ubiquitous Technology, Volume 6, Issue 4. (Ubicomp'23)
• PrISM-Observer: Intervention Agent to Help Users Perform Everyday Procedures Sensed using a Smartwatch. Riku Arakawa, Hiromu Yakura, Mayank Goel. Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST’24), Pittsburgh USA, Oct. 2024.
• PrISM-Q&A: Step-Aware Question Answering with Large Language Models Enabled by Multimodal Procedure Tracking using a Smartwatch. Riku Arakawa, Jill Fain Lefman, Mayank Goel. Proceedings of the ACM on Interactive Mobile Wearable Ubiquitous Technology, Volume 8, Issue 4. (Ubicomp'25)

What's in this repository now

For now, we have

• data_collection: smartwatch app + preprocess script
• src: modularized pipeline
  • HAR (frame-level human activity recognition)
  • Tracker (postprocess with an extended Viterbi algorithm)
  • Observer (proactive intervention based on the tracking result)
  • Q&A (coming soon, question-answering interaction with LLMs augmented by sensing output)

We will also release the real-time server app shortly.

Setup

Install the prism module into your environment

$ conda create -n "prism" python=3.10
$ conda activate prism
$ conda install --file requirements.txt

Create a datadrive folder at your convenience. Make sure to update src/prism_tracker/config.py.

datadrive = Path('Path / To / Your / Datadrive')

After that, please run

$ python -m pip install -e src

In the datadrive, the structure will be

datadrive
│
└───pretrained_models
│   └───audio_model.h5
│   └───motion_model.h5
│   └───motion_norm_params.pkl
│  
└───tasks
    └───latte_making
          └───dataset
          └───qa (will be generated)
          └───models (will be generated)

Download the required files from the following links:

• pretrained_models: https://www.dropbox.com/sh/w3lo0f1k6w90b5w/AADuoDVSKuY9kQSPx2RRGJ_Ma?dl=0
• sample datasets: https://www.dropbox.com/sh/93jd6elugxgvm6k/AACL3XGiP8-UXPKIWK-h9Ud1a?dl=0
  • For now, there are cooking and latte-making tasks. We are expanding the dataset with different tasks and additional interesting sensor sources. Stay tuned!

License

This repository is published under MIT license. Please contact Riku Arakawa and Mayank Goel if you would like another license for your use.

Contact

Feel free to contact Riku Arakawa for any help, questions or general feedback!

Acknowledgements

• Hiromu Yakura helped with the implementation of the tracker and observer.
• Vimal Mollyn helped with the implementation of the HAR module.
• Suzanne Nie and Vicky Liu helped with the data collection pipeline.