several orbitcontrols: one per canvas in several views?

[MainzerKaiser]

I want to render several models and each shall have its own canvas and orbitcontrols. I made that work by the following code, but I make two observation:

1. clicking first time on a model doesnt initiate the rotation and looking into your code you somehow try to detect the first touch and only regard everything afterwards as rotation control. is there a way to customize that?
2. when keeping the finger on the display of my android (i run via expo) and leaving the current canvas the target camera position resets somehow to a weird position that i can make visible within the handleCameraChange function via console.logs. This does not happen when testing rotation outside of the canvas in a browser (firefox or chrome). Is that behavior on android intentional?

import React, { Suspense, useRef, useMemo, useContext, useCallback, useEffect, useState, lazy } from 'react';
import Chair from './src/components/chair';
import { SafeAreaView, StyleSheet, View } from 'react-native';
import { Canvas } from '@react-three/fiber/native';
import useControls from 'r3f-native-orbitcontrols';
import * as THREE from "three";
import Trigger from './src/components/Trigger';
import Loader from './src/components/Loader';
import { useVisibleStore } from './src/context/MyZustand';
import { Dimensions } from 'react-native';


const Game: React.FC = () => {
  const { width, height } = Dimensions.get('window');
  const isPortrait = height > width;
  const nCubes= useVisibleStore((state) => state.nCubes)
  const [loading, setLoading] = useState<boolean>(false);

  const controlsAndEvents = Array.from({ length: nCubes }, (_, index) => {
    const [OrbitControls, events] = useControls();
    return { OrbitControls, events, index };
  });

return (
      <SafeAreaView style={styles.container}>
        <View style={styles.modelsContainer}>
        {loading && <Loader />}
          <View style={isPortrait ? styles.columnContainer : styles.rowContainer}>
            {controlsAndEvents.map(({ OrbitControls, events, index }) => (
              <View key={index} style={isPortrait ? styles.modelColumn : styles.modelRow}{...events}>
                  <Canvas key={index}>
                    <OrbitControls
                      key={index} 
                      enabled={true}
                      enableZoom={false}
                      enablePan={false}
                      dampingFactor={0.5}
                      enableRotate={true}
                      rotateSpeed={0.5} 
                      // onChange={(event) => handleCameraChange(event, index, letters)} 
                      {...OrbitControls}
                    />
                    <directionalLight position={[1, -1, -1]} intensity={5} args={["yellow", 10]}  />
                    <directionalLight position={[-1, -1, -1]} intensity={7} args={["white", 10]}  />
                    <Suspense fallback={<Trigger setLoading={setLoading} />}>
                      <Chair />
                    </Suspense>
                  </Canvas>
              </View> 
             ))}
          </View>
         </View>
      </SafeAreaView>
  );
};

const styles = StyleSheet.create({
  container: {
    flex: 1,
    backgroundColor: 'grey', // Add this line to set the background color to grey
  },
  modelsContainer: {
    flex: 0.9,
  },
   columnContainer: {
    // flexDirection: 'column',
    flex: 1,
  },
  rowContainer: {
    // flexDirection: 'row',
    flex: 1,
  },
  modelColumn: {
    flex: 1,
    justifyContent: 'center',
  },
  modelRow: {
    flex: 1,
    justifyContent: 'center',
  },
});

export default Game;

[TiagoCavalcante]

I think (1) should be solved in 1.0.10 — can you check it? —. I still have to look at (2).

[MainzerKaiser]

Hi Tiago,
thanks a lot for responding. your fix in 1.0.10 works very well on web and android!

Regarding the second topic, I uploaded this animation, which is a recording from my android phone. (android 12 and expo on dev server)
What you don't see is the touch location of my finger, but I am moving the middle chair and hover around the canvas border to the upper chair. In the web version, I can keep touching even over other canvases/chairs, and I still move the chair where I started touching the screen. On android there is always a reset of the camera location as soon as you leave the canvas where you started the rotation.

chair_leaves_canvas.mp4

[TiagoCavalcante]

@MainzerKaiser I couldn't reproduce this (but still didn't test in an Android though).

Maybe you can share the whole code, or try to reduce the code where the bug happens?

Also, does this bug happen on web too?

[MainzerKaiser]

@TiagoCavalcante, the bug does not happen on web, it only appears on anroid. I could not test ios, though.
I created this repo with the chair, although if loaded via expo it should show 6 chairs, aligned vertically.
It should reflect the code in the first comment. And it is independent of the model loaded.

https://github.com/MainzerKaiser/testing_r3f_orbit_in_reactnative.git

[TiagoCavalcante]

The bug isn't happening on iOS, I'll test it on Android soon.

[MainzerKaiser]

How would you go about resetting the camera settings on button click?

[TiagoCavalcante]

If you are using an "external" camera like in the second example from README.md, you can set its target with camera.lookAt(new Vector3(x, y, z)) and its position (which is only useful if pan is enabled) with camera.position.set(x, y, z).

If you are using the default Canvas camera you can obtain the camera object with useThree (similarly to what is made below) and you can update it like above.

If you are using frameloop="demand" on the Canvas you may need to call Fiber's invalidate:

function ComponentWhichMustBeInsideTheCanvas() {
  const invalidate = useThree((state) => state.invalidate)

  useEffect(() => {
    // Click stuff
    invalidate()
  }, [dependencies])

  return null as unknown as JSX.Element // This must be a JSX element
}

[MainzerKaiser]

Unfortunately I could not make it work. I cannot use the external camera as it screws up the rest of my code (placement of all objects). How would i go about resetting each of my canvas cameras, if they are introduced by the mapping of controlsAndEvents.
I try to use a useEffect triggered by the button click and reloading boolean starting the resetCameraTarget() function, but my code does not reset the individual cameras in the nCubes=6 canvases.
I get the error: OrbitControls instance not available.

  const controlsAndEvents = Array.from({ length: nCubes }, (_, index) => {
    const [OrbitControls, events] = useControls();
    const orbitControlsRef = useRef<typeof OrbitControls>();
    const canvasRef = useRef<any>(null);

    return { OrbitControls, events, orbitControlsRef, canvasRef, index };
  });

  interface ModelCallProps {
    index: number;
    canvasRef: React.RefObject<any>; // Adjust the type as needed
    reloading: boolean;
  }


  function ModelCall({ index, canvasRef, reloading }: ModelCallProps) {
    const { camera } = useThree();
  
    const resetCameraTarget = () => {
      if (camera && camera.userData && camera.userData.controls) {
        const orbitControls = camera.userData.controls;
        orbitControls.target.set(0, 0, 0); // Set the target position, adjust as needed
        orbitControls.update(); // Make sure to call update after modifying properties
      } else {
        console.error("OrbitControls instance not available.");
      }
    };
  
  
    // Reset camera target when reloading is true
    useEffect(() => {
      if (reloading) {
        resetCameraTarget();
        setQALoading(false); // Assuming setQALoading is a function to update qaloading state
      }
    }, [reloading]);
    return (
      <Model
        letters={letters[index]}
        cubeindex={index}
      />
    );
  }
  
  return (

      <SafeAreaView style={styles.container}>
        <View style={styles.modelsContainer}>
        {loading && <Loader />}
          <View style={isPortrait ? styles.columnContainer : styles.rowContainer}>
            {controlsAndEvents.map(({ OrbitControls, events, orbitControlsRef, canvasRef, index }) => {
                return (
                  <View
                  key={index}
                  style={isPortrait ? styles.modelColumn : styles.modelRow}
                  {...events}
                >
                    <Canvas 
                      key={index} 
                      ref={canvasRef}
                      
                    >
                      <OrbitControls
                        key={index} 
                        enableZoom={false}
                        enablePan={false}
                        dampingFactor={0.75}
                        enableRotate={true}
                        rotateSpeed={0.45}
                        onChange={(event) => handleCameraChange(event, index, letters)} 
                        {...OrbitControls}
                      />
                      <directionalLight position={[-1, 1, 1]} intensity={4} args={["white", 10]}  />
                      <directionalLight position={[1, -1, 1]} intensity={5} args={["blue", 10]}  />
                      <directionalLight position={[-1, -1, 1]} intensity={3} args={["orange", 10]}  />
                      <directionalLight position={[1, 1, -1]} intensity={5} args={["white", 10]}  />
                      <directionalLight position={[1, -1, -1]} intensity={5} args={["yellow", 10]}  />
                      <directionalLight position={[-1, -1, -1]} intensity={7} args={["white", 10]}  />
                      <Suspense fallback={<Trigger setLoading={setLoading} />}>
                          <ModelCall index={index} canvasRef={canvasRef} reloading={qaloading}/>
                        {/* <Chair /> */}
                      </Suspense>
                    </Canvas>
                    {/* </View> 
                    </Pressable>*/}
                </View> 
                )
              })}
          </View>
         </View>
</SafeAreaView>
    )
  // );
};

[TiagoCavalcante]

@MainzerKaiser can you check if the error is still happening with 1.0.12?

[MainzerKaiser]

Hi Tiago,
i get this error when hitting the web version in expo:

ERROR in ./node_modules/r3f-native-orbitcontrols/lib/index.esm.js:3
Module not found: Can't resolve '@react-three/fiber/native'
Did you mean 'react-three-fiber-native.esm.js'?
BREAKING CHANGE: The request '@react-three/fiber/native' failed to resolve only because it was resolved as fully specified
(probably because the origin is strict EcmaScript Module, e. g. a module with javascript mimetype, a '*.mjs' file, or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
  1 | import React, { useMemo, useEffect } from 'react';
  2 | import { Vector3, Vector2, Spherical, Quaternion } from 'three';
> 3 | import { invalidate, useThree, useFrame } from '@react-three/fiber/native';

This is my package.json with comparable packages as in your package.json. Do you see what the problem is?

{
  "name": "cubewordle",
  "version": "1.0.0",
  "main": "node_modules/expo/AppEntry.js",
  "scripts": {
    "start": "expo start",
    "android": "expo start --android",
    "ios": "expo start --ios",
    "web": "expo start --web"
  },
  "dependencies": {
    "@expo/webpack-config": "~19.0.1",
    "@mediapipe/tasks-vision": "^0.10.8",
    "@react-three/drei": "^9.89.0",
    "@react-three/fiber": "^8.15.11",
    "base64-arraybuffer": "^1.0.2",
    "expo": "^50.0.14",
    "expo-crypto": "^12.8.1",
    "expo-gl": "~13.6.0",
    "expo-status-bar": "~1.11.1",
    "jotai": "^2.6.1",
    "nanoid": "^5.0.4",
    "r3f-native-orbitcontrols": "^1.0.12",
    "react": "18.2.0",
    "react-native": "0.73.6",
    "react-native-3d-model-view": "^1.2.0",
    "react-native-gesture-handler": "~2.14.0",
    "react-native-safe-area-context": "4.8.2",
    "react-native-screens": "^3.29.0",
    "react-native-web": "~0.19.6",
    "react-promise-suspense": "^0.3.4",
    "react-use-refs": "^1.0.1",
    "rn-fetch-blob": "^0.12.0",
    "three": "^0.159.0",
    "three-bmfont-text": "^2.3.0",
    "three-stdlib": "^2.29.4",
    "xlsx": "^0.18.5",
    "zustand": "^4.4.7"
  },
  "devDependencies": {
    "@babel/core": "^7.20.0",
    "@types/expo": "^33.0.1",
    "@types/expo__vector-icons": "^10.0.0",
    "@types/react-dom": "~18.0.10",
    "@types/uuid": "^9.0.7",
    "typescript": "^5.3.3",
    "vite": "^5.0.10",
    "@types/react": "^18.2.56",
    "@types/react-native": "^0.72.8",
    "@types/three": "^0.161.2"
  },
  "lint-staged": {
    "*.{ts,tsx,md}": "prettier --write --no-semi"
  },
  "private": true
}

and the tsconfig.json:

{
  "compilerOptions": {
    "target": "es2020",
    "module": "commonjs",
    "jsx": "react-native",
    "strict": true,
    "esModuleInterop": true
  },
  "include": [
    "src/**/*.ts",
    "src/**/*.tsx",
    "App.tsx"
  ],
  "exclude": [
    "node_modules"
  ],
  "extends": "expo/tsconfig.base"
}

[MainzerKaiser]

Hi Tiago,
when starting the web version, i still get the error above, that ./node_modules/r3f-native-orbitcontrols/lib/index.esm.js:3 cannot resolve '@react-three/fiber/native'. Don't know why that is. Could it be, that the types are not working in the new version?

But i could test the android version with your new 1.0.12.. And my original second issue:

when keeping the finger on the display of my android (i run via expo) and leaving the current canvas the target camera position resets somehow to a weird position that i can make visible within the handleCameraChange function via console.logs. This does not happen when testing rotation outside of the canvas in a browser (firefox or chrome). Is that behavior on android intentional?

still persists, unfortunately.

Best regards,
MK

[markhuang19994]

@MainzerKaiser

when keeping the finger on the display of my android (i run via expo) and leaving the current canvas the target camera position resets somehow to a weird position that i can make visible within the handleCameraChange function via console.logs. This does not happen when testing rotation outside of the canvas in a browser (firefox or chrome). Is that behavior on android intentional?

I encountered the same issue as you and have resolved it. Hopefully, the same approach works for you.

I calculate the Euclidean distance between two positions using a formula. If the distance is greater than 2, I assume the position update is incorrect.

const cameraRef = useRef<Camera>();
const lastCameraPositionRef = useRef<Vector3>();

const handleOrbitPositionUpdate = (position?: Vector3) => {
  if (!position) return;

  const lastCameraPosition = lastCameraPositionRef.current;
  if (lastCameraPosition) {
    const { x, y, z } = position;
    const [x0, y0, z0] = lastCameraPosition;

    const camera = cameraRef.current;
    if (camera) {
      // if the distance exceeds 2, assume the new position is incorrect.
      const distance = Math.sqrt((x0 - x) ** 2 + (y0 - y) ** 2 + (z0 - z) ** 2);
      if (distance >= 2) {
        camera.position.set(x0, y0, z0);
        invalidate();
        return;
      }
    }

    if(lastCameraPositionRef.current) {
      lastCameraPositionRef.current = position;    
    }
  }
};


<OrbitControls
  onChange={event =>
    handleOrbitPositionUpdate(event.target.camera?.position)
  }
/>

[MainzerKaiser]

Thanks for that idea. I adopted it to my problem and in principle it works fine. What does your invalidate() function do?
The important part of my code is now:

  const handleCameraChange = useCallback((event: any, index: number, letters:string[][], lastCameraPositionRef:any) => {
    const { x, y, z } = event.target.camera.position;
    const { x: x0, y: y0, z: z0 } = lastCameraPositionRef.current;

    const distance = Math.sqrt((x0 - x) ** 2 + (y0 - y) ** 2 + (z0 - z) ** 2);
    // console.log(distance, y)
    if (distance >= 2) {
      event.target.camera.position.set(x0,y0,z0);
      return;
    } else {
      lastCameraPositionRef.current = new THREE.Vector3(x, y, z);
  [...]
  
  
  const controlsAndEvents = Array.from({ length: nCubes }, (_, index) => {
    const [OrbitControls, events] = useControls();
    const orbitControlsRef = useRef<typeof OrbitControls>();
    const canvasRef = useRef<any>(null);
    const lastCameraPositionRef = useRef<THREE.Vector3>(new THREE.Vector3(0,0,5));

    return { OrbitControls, events, orbitControlsRef, canvasRef, index, lastCameraPositionRef};
  });
  
  [...]
{controlsAndEvents.map(({ OrbitControls, events, orbitControlsRef, canvasRef, index, lastCameraPositionRef }) => {
             return (
                  <View
                  key={index*3}
                  style={isPortrait ? styles.modelColumn : styles.modelRow}
                  {...events}
                >
                    <Canvas 
                      key={index*5} 
                      ref={canvasRef}
                      
                    >
                      <OrbitControls
                        key={index} 
                        enableZoom={false}
                        enablePan={false}
                        dampingFactor={0.75}
                        enableRotate={true}
                        rotateSpeed={0.45}
                        onChange={(event) => handleCameraChange(event, index, letters, lastCameraPositionRef)} 
                        {...OrbitControls}
                      />
                      <directionalLight position={[-1, 1, 1]} intensity={4} args={["white", 10]}  />
                      <directionalLight position={[1, -1, 1]} intensity={5} args={["blue", 10]}  />
                      <directionalLight position={[-1, -1, 1]} intensity={3} args={["orange", 10]}  />
                      <directionalLight position={[1, 1, -1]} intensity={5} args={["white", 10]}  />
                      <directionalLight position={[1, -1, -1]} intensity={5} args={["yellow", 10]}  />
                      <directionalLight position={[-1, -1, -1]} intensity={7} args={["white", 10]}  />
                      <Suspense fallback={<Trigger setLoading={setLoading} />}>
                          <ModelCall index={index} canvasRef={canvasRef} reloading={qaloading}/>
                      </Suspense>
                    </Canvas>

                </View> 
                )
              })}

  
  

Setting the event.target.camera.position.set(x0,y0,z0); to the lastCameraPosition in case the distance>2 is visibile as a small stutter.
But apart from that it works fine..

I'd rather have an option to invalidate every touch event outside of the canvas.

WhatsApp.Video.2024-07-14.um.12.07.27_96d1e932.mp4

[markhuang19994]

Thanks for that idea. I adopted it to my problem and in principle it works fine. What does your invalidate() function do?

I use the invalidate function to quickly update the corrected position on the canvas. This is especially useful when your canvas frameLoop is set to demand (On-demand rendering).