[DeepDive] Oculus Quest (+VR/AR/MR/XR deep dive) #6
Description
(Playing with the concept outlined in #5)
Background
Sideloading Apps
- https://github.com/the-expanse/SideQuest
- https://github.com/the-expanse/SideQuest/issues/31 : Ability to type/copy/paste into headset
- https://github.com/the-expanse/SideQuest/issues/34 : How to sideload shared libraries? (+ apktool, uses-library, Chrome DevTools, accessing the headset's browser, deep linking, etc)
Sideloadable
- Lists
- Facebook, Messenger, Instagram, etc
- Gaming
- AltspaceVR
- https://github.com/polygraphene/ALVR
- https://riftcat.com/vridge
- https://wccftech.com/ps4-remote-play-app-android-device/
- https://github.com/majido/clipper : Simple android app to interact with system clipboard service via adb shell
- Unsure?
- https://play.google.com/store/apps/details?id=com.playstation.remoteplay
- https://play.google.com/store/apps/details?id=com.valvesoftware.steamlink
- https://play.google.com/store/apps/details?id=com.plexapp.android.vr
- https://play.google.com/store/apps/details?id=com.google.android.vr.home
- https://play.google.com/store/apps/details?id=com.google.samples.apps.cardboarddemo
- https://blog.mozilla.org/firefox/firefox-reality-oculus-go-vr/ (Oculus Go Store)
Controllers
- https://medium.com/badvr/oculus-quest-rift-s-touch-controller-teardown-f551933cbb0c
- https://uploadvr.com/oculus-go-ps4-switch-xbox/
Debugging, etc
adb- https://developer.android.com/studio/command-line/adb
- https://developer.oculus.com/documentation/quest/latest/concepts/mobile-adb/
brew cask install android-platform-tools
- https://github.com/Genymobile/scrcpy : Display and control your Android device
brew install scrcpy
Etc
My related StackOverflow posts
- https://stackoverflow.com/questions/56386237/is-it-possible-to-copy-paste-type-eg-a-password-from-an-app-into-the-oculus-q
- https://stackoverflow.com/questions/56403318/can-i-remotely-access-web-browser-tabs-in-my-oculus-quest-oculus-go-from-my-comp/56403319#56403319
Unpacking/Reverse Engineering APK's/Android apps
- https://ibotpeaches.github.io/Apktool/
- https://github.com/iBotPeaches/Apktool : A tool for reverse engineering Android apk files
brew install apktool
- https://github.com/skylot/jadx : Dex to Java decompiler
brew install jadx
- https://github.com/pxb1988/dex2jar : Tools to work with android .dex and java .class files
brew install dex2jar
- https://github.com/java-decompiler/jd-gui : A standalone Java Decompiler GUI
brew cask install jd-gui
- https://adoptopenjdk.net/ : Prebuilt OpenJDK Binaries for Free
brew cask install adoptopenjdk8
- https://medium.com/mindorks/how-i-decreased-my-app-size-to-70-using-apk-analyser-4a6f79512072
Other potentially cool VR/AR/3D related things
- https://studio.vroid.com/
- https://hub.vroid.com/ : VRoid Hub is a platform for 3D characters
- https://vrm.dev/
- https://vrm.dev/vrm_applications/
- https://github.com/vrm-c/UniVRM : Unity package that can import and export VRM format
- https://www.adobe.com/products/projectaero.html
- https://sketchfab.com/
- https://developer.vuforia.com/
- https://developer.vuforia.com/vui/pricing
- Vuforia Engine is the most widely used platform for AR development, with support for leading phones, tablets, and eyewear. Developers can easily add advanced computer vision functionality to Android, iOS, and UWP apps, to create AR experiences that realistically interact with objects and the environment.
- https://sine.space/
- https://medium.com/@narpoghos5/hacked-an-oculus-quest-level-immersion-with-a-20-headset-and-arkit-arcore-df6df5189738
- https://aframe.io/
- A web framework for building virtual reality experiences
- https://poly.google.com/
- https://vr.google.com/jump/ (360 video capture)
- https://renderman.pixar.com/
- Haptic gloves, hand tracking, etc
VR Deep Dives
- https://hackernoon.com/behold-the-next-generation-vr-technology-part-1-facial-tracking-cd839eaa6697
- https://hackernoon.com/behold-the-next-generation-vr-technology-part-2-full-body-tracking-1210ab27ebe4
- https://hackernoon.com/behold-the-next-generation-vr-technology-part-3-avatars-a761239f6b2d
- https://hackernoon.com/behold-the-next-generation-vr-technology-part-4-physical-world-mapping-a16a6f3ad99d
- realtime object reconstruction (YouTube)
- environment scaling (YouTube)
- https://www.media.mit.edu/projects/oasis/overview/ : Oasis is a novel system for automatically generating immersive and interactive virtual reality environments using the real world as a template.
- https://medium.com/inborn-experience/behold-the-next-generation-vr-technology-part-5-locomotion-c92d66d4a33e
- https://medium.com/inborn-experience/behold-the-next-generation-vr-technology-part-6-brain-interface-89b1d31a0a96
- https://medium.com/@spammaleros/behold-the-next-generation-vr-technology-part-7-controllers-and-haptics-3243e8399d29
Oculus
- https://www.oculus.com/blog/
- Oculus Venues
- Oculus Rooms
- https://developer.oculus.com/blog/
- https://developer.oculus.com/design/latest/concepts/book-bp/
- https://developer.oculus.com/develop/
- https://developer.oculus.com/documentation/quest/latest/
- https://developer.oculus.com/documentation/quest/latest/concepts/unity-sample-framework/
- https://developer.oculus.com/documentation/quest/latest/concepts/unity-tutorial/
- https://developer.oculus.com/documentation/quest/latest/concepts/unity-resources/
- https://developer.oculus.com/documentation/quest/latest/concepts/unity-utilities-overview/
- OVRCameraRig: a custom VR camera that may be used to replace the regular Unity Camera in a scene
- OVRPlayerController: the easiest way to start navigating a virtual environment. It is basically an OVRCameraRig prefab attached to a simple character controller.
- OVRCubemapCaptureProbe: This prefab allows you to capture a static 360 screenshot of your application while it is running.
- OVRGrabber, OVRGrabbable
- OVRInput, Haptics
- OVRTouchpad
- OVRBoundary (Guardian System)
- https://developer.oculus.com/documentation/quest/latest/concepts/unity-mobile-performance-intro/
- https://developer.oculus.com/documentation/quest/latest/concepts/unity-perf/
- https://developer.oculus.com/documentation/quest/latest/concepts/unity-best-practices-intro/
- https://developer.oculus.com/documentation/quest/latest/concepts/mobile-vrapi-input-api/
- https://developer.oculus.com/documentation/quest/latest/concepts/mobile-guardian/
- Mixed Reality Capture
- https://developer.oculus.com/documentation/platform/latest/
- https://developer.oculus.com/documentation/avatarsdk/latest/
- https://developer.oculus.com/documentation/oculus-browser/latest/concepts/browser-intro/
- https://developer.oculus.com/documentation/unity/latest/concepts/unity-mrc/
- https://developer.oculus.com/downloads/unity/
- https://developer.oculus.com/downloads/package/oculus-remote-monitor-for-os-x/
- https://developer.oculus.com/downloads/web-vr/
- https://developer.oculus.com/documentation/quest/latest/
- https://creator.oculus.com/blog/
- Oculus Rooms
- Oculus Venues
- https://www.oculus.com/medium/
- OC5:
- Reconstruction: https://youtu.be/o7OpS7pZ5ok?t=5938
- Apartment Reconstruction: https://youtu.be/o7OpS7pZ5ok?t=6015
- Face Tracking: https://youtu.be/o7OpS7pZ5ok?t=6185
Unity (ARFoundation)
- https://unity.com/
- https://store.unity.com/
- https://learn.unity.com/
- https://docs.unity3d.com/Manual/index.html
- https://blogs.unity3d.com/
- https://blogs.unity3d.com/2019/06/06/ar-foundation-support-for-arkit-3/
- https://blogs.unity3d.com/2019/04/16/introducing-unity-2019-1/
- https://blogs.unity3d.com/2018/12/18/unitys-handheld-ar-ecosystem-ar-foundation-arcore-and-arkit/
- https://blogs.unity3d.com/2018/06/15/multi-platform-handheld-ar-in-2018-part-1/
- ARFoundation
- AR Foundation allows you to work with augmented reality platforms in a multi-platform way within Unity.
- https://docs.unity3d.com/Packages/[email protected]/manual/index.html
- https://github.com/Unity-Technologies/arfoundation-samples
- https://bitbucket.org/Unity-Technologies/unity-arkit-plugin/
- https://unity.github.com/
Apple (ARKit)
- https://www.apple.com/au/ios/augmented-reality/
- https://developer.apple.com/arkit/
- https://developer.apple.com/documentation/arkit
- https://developer.apple.com/documentation/arkit/building_your_first_ar_experience
- https://developer.apple.com/documentation/arkit/understanding_world_tracking_in_arkit
- https://developer.apple.com/documentation/arkit/creating_a_multiuser_ar_experience
- https://developer.apple.com/documentation/arkit/creating_a_persistent_ar_experience
- https://developer.apple.com/documentation/arkit/scanning_and_detecting_3d_objects
- https://www.imore.com/oculus-quest-arena-demo-impresses-arkit-component#arkit
Google (ARCore)
- ARCore
- https://developers.google.com/ar/
- https://developers.google.com/ar/reference/
- https://developers.google.com/ar/develop/
- https://developers.google.com/ar/discover/supported-devices
- iOS: ARCore requires an ARKit compatible device running iOS 11.0 or later.
- https://play.google.com/store/apps/details?id=com.google.ar.core
- https://github.com/google-ar/arcore-ios-sdk
- https://github.com/google-ar/arcore-android-sdk
- https://github.com/google-ar/arcore-unity-sdk/
- https://github.com/lvonasek/tango/wiki/3D-Scanner-for-ARcore
- Project Tango
- https://www.theverge.com/2017/12/15/16782556/project-tango-google-shutting-down-arcore-augmented-reality
- https://sparkar.facebook.com/ar-studio/
- https://facebook.github.io/react-360/
- https://github.com/facebook/react-360 : Create amazing 360 and VR content using React
- https://facebook.github.io/react-360/blog/
- https://www.facebook.com/spaces (Oculus Store)
- VR Avatars
- https://tech.fb.com/codec-avatars-facebook-reality-labs/
- https://edition.cnn.com/2019/05/01/tech/facebook-vr-avatar/index.html
- F8 2019: Research Towards VR Full Body Tracking & Avatars
- Start with an anatomical model...: https://youtu.be/FhiAFo9U_sM?t=207
- Example live interaction: https://youtu.be/FhiAFo9U_sM?t=360
- https://research.fb.com/category/augmented-reality-virtual-reality/
- https://research.fb.com/downloads/detectron/
- https://research.fb.com/downloads/densepose/ : Facebook’s real-time approach for mapping all human pixels of 2D RGB images to a 3D surface-based model of the body.
React 360
Facebook 360
- https://facebook360.fb.com/
- https://github.com/facebook/360-Capture-SDK : A developer focused sample SDK that allows game and virtual Reality devs to be able to easily and quickly integrate 360 photo/video capture capability into their game apps.
- https://github.com/facebook/transform360 : Transform360 is an equirectangular to cubemap transform for 360 video.
- https://github.com/facebook/Surround360 : Open source hardware and software for capturing stereoscopic 3D 360 video for VR
- https://github.com/facebookincubator/facebook-360-spatial-workstation : The Facebook 360 Spatial Workstation is a software suite for designing spatial audio for 360 video and cinematic VR.
Valve / Steam (SteamVR, OpenVR)
- https://steamcommunity.com/steamvr
- https://store.steampowered.com/steamvr
- SteamVR for MacOS: Beta Available Now
- Introducing SteamVR Home Beta
- https://store.steampowered.com/app/992490/iVRy_Driver_for_SteamVR/
- https://play.google.com/store/apps/details?id=com.mediator_software.ivry&hl=en
- https://github.com/ValveSoftware/openvr
Leap Motion
- https://www.leapmotion.com/
- https://developer.leapmotion.com/
- https://developer.leapmotion.com/sdk/v2
- https://developer.leapmotion.com/documentation
- https://developer.leapmotion.com/unity
- https://developer.leapmotion.com/leapjs
- https://developer.leapmotion.com/android
- http://blog.leapmotion.com/
- https://www.leapmotion.com/news/the-future-of-vr-is-here-qualcomm-and-leap-motion-work-together-to-demonstrate-natural-interaction-for-mobile-vr/
- "The combination of the Qualcomm Snapdragon 835 mobile platform and Leap Motion’s cutting-edge hand tracking technology, which renders the precise movement of hands and fingers with very low latency, is designed to allow developers to tap into an incredible software and hardware ecosystem."
- Oculus Quest runs Snapdragon 835, leap motion has an android platform coming out, I expect they will likely work together and run on the base quest hardware
- https://www.roadtovr.com/qualcomm-vrdk-vr835-leap-motion-hand-tracking/
- "Leap Motion says that the module runs at 10x the speed of the original device at significantly lower power. It’s also very tiny."
- https://github.com/leapmotion
- https://github.com/leapmotion/LeapMotionCoreAssets : Unity Assets for Leap Motion v2 Skeletal Tracking
- https://github.com/leapmotion/ProjectNorthStar : Leap Motion's Project North Star AR Headset.
WebXR / WebVR / WebAR
- https://webxr.io/
- https://github.com/immersive-web/webxr
- https://www.w3.org/TR/webxr/
- https://hacks.mozilla.org/2018/09/webxr/
- https://blog.mozvr.com/tag/webxr/
- https://webvr.info/
- https://createwebvr.com/webAR.html
Concepts / Topics
Social VR Worlds / Open MMO / etc
- https://orbusvr.com/
- https://rec.net/ (Recroom)
- https://rec.net/room/browse
- https://rec.net/event/browse
- https://rec.net/mobile
- https://www.playstation.com/en-us/games/rec-room-ps4/
- https://www.againstgrav.com/rec-room
- https://www.againstgrav.com/community/
- https://recroom.happyfox.com/kb/article/43-custom-rooms-tools-and-tutorials/
- YouTube: How To Rec Room: Customizing your Dorm Room!
- YouTube: How To Rec Room - Costumes - ^CostumesTemplate Room
- YouTube: Maker Pen 101 - How to build a book in Rec Room
- YouTube: How To Rec Room - Streamer Cam
- https://rec-room.fandom.com/wiki/Rec_Room_Wiki
- https://altvr.com/
- https://www.vrchat.net/
- https://www.sinespace.com/
- https://www.highfidelity.com/
- https://www.sansar.com/
- https://improbable.io/
- https://twitter.com/metaworldvr
Automating Builds (CI/CD)
- https://unity3d.com/unity/features/cloud-build
- https://forum.unity.com/threads/share-your-build-server-specs-and-setup.507060/
- https://medium.com/@jounimannonen/unity-build-automation-96c8a22860c3
- https://docs.unity3d.com/Manual/CommandLineArguments.html
- https://stablekernel.com/continuous-integration-for-unity-5-using-travisci/
- https://github.com/DragonBox/u3d : U3d is a cross-platform set of tools to interact with Unity3D from command line
- https://github.com/kevinta893/unity3d-travis-ci
- https://wiki.jenkins.io/display/JENKINS/Unity3dBuilder+Plugin
- https://developer.apple.com/testflight/ : Beta Testing Made Simple
- https://fastlane.tools/
- https://github.com/fastlane/fastlane : The easiest way to automate building and releasing your iOS and Android apps
- https://www.macstadium.com/
360 Image/Video/Audio Capture/Playback
- https://developer.oculus.com/blog/announcing-360-capture-sdk/
- https://blogs.unity3d.com/2018/01/26/stereo-360-image-and-video-capture/
- https://docs.unity3d.com/ScriptReference/Camera.RenderToCubemap.html
- https://docs.unity3d.com/ScriptReference/Rendering.EditorCameraUtils.RenderToCubemap.html
- https://docs.unity3d.com/ScriptReference/RenderTexture.ConvertToEquirect.html
- https://assetstore.unity.com/packages/essentials/unity-recorder-94079
- https://veer.tv/blog/how-to-use-unity-2018-1-to-capture-stereoscopic-360-images-and-videos/
- https://blogs.unity3d.com/2017/07/27/how-to-integrate-360-video-with-unity (playback)
- https://facebookincubator.github.io/facebook-360-spatial-workstation/Documentation/SpatialWorkstation/SpatialWorkstation.html
- Unity Asset Store
- Live streaming 360
- https://facebook360.fb.com/editing-360-photos-injecting-metadata/
User Interface Design: VR Menu, Inventory, etc
- Concepts
- Smartphone (eg. Dance Central)
- Smart watch + detachable menu (eg. RecRoom, Fallout's 'pip boy', etc)
- 'Swipe menu' (eg. Sword Art Online)
- Reach over shoulder/into backpack (eg. RecRoom 'favourite tool')
- Hovering buttons near hand/wrist when certain gestures (eg. Leap Motion demo's)
- Could be hand, forearm, etc
- Point + button, menu apears on object, can react (eg. Facebook likes, reacts, etc)
- Gestures
- thumbs up (like/yes) (eg. Dance Central)
- hold up palm (stop/no) (eg. RecRoom)
- etc
- VR UI
Smartwatch / wrist menu / etc
- https://www.ims.tuwien.ac.at/projects/smartwatch (pdf)
- https://github.com/lolzoloz9/VR-Capstone/wiki
- [YouTube: Unrealty Simon
- Unreal Engine - [VR] Interactive Watch Menu](https://www.youtube.com/watch?v=KQexLS2xjOs)
- Examples
- Models
Apple Watch
Sword Art Online inspired menu system, etc
- http://age-of-aincrad.com/
- https://github.com/Scub3d/Project-SAOUI : SAO Menu System Orion Version
- SAO Utils : The SAO Utils program is a Sword Art Online Look'n'Feel launcher with varieties of powerful utilities.
- YouTube: HOCGaming - Sword Art Online GUI - VR Dev Log - 02
- YouTube: HOCGaming - Sword Art Online GUI - V2! [VR Dev Log - 05]
- https://www.reddit.com/r/Unity3D/comments/739k7a/i_made_a_sword_art_online_themed_vr_menu_has_some/?st=jx6mp6n5&sh=ba72652a
- https://github.com/MeesMD/VR-UI : The concept is based on an animation series called 'Sword art online'. In the series an interface is used that gives the players of the world access to the options, inventory, friends, messages and maps.
External Peripherals
- https://www.researchgate.net/publication/309710568_TickTockRay_smartwatch-based_3D_pointing_for_smartphone-based_virtual_reality
- "TickTockRay is a smartwatch-based raycasting technique designed for smartphone-based head mounted displays. It demonstrates that smartwatch-based raycasting can be reliably implemented on an off-the-shelf smartphone and may provide a feasible alternative for specialized input devices."
- https://assetstore.unity.com/packages/templates/systems/apple-watch-kit-88245
- eg. smartwatch that communicates sensor data into VR (heart rate, gyroscope, haptics, etc?)
- Apple Watch
- Android
- Oculus Quest
- Uses android, runs on Qualcomm Snapdragon 835, which supports bluetooth 5