How to force the drone flying with the specific velocity and Eular angle without fast-physics ？

[misakabribri]

OS: Windows 10
IDE: VS 2019;
Unreal: UE 4.24.3

Hello. I already have the fligt data (including positions and Euler angles) gathered from other simulation platforms. I want to use AirSim to 'playback' the flight data (just like a 3D-film) and capture images of the drone.

I guess there are at least two things to do:

1. disable the simple fight model;
2. write a new API funtion to command the drone move with the specific velocity and Eular angle;
   but I don't know how to do these things.

[ahmed-elsaharti]

Hi @misakabribri , this will mainly depend on the flight data you gathered. For example, if its a PX4 mission, you can simply run AirSim in PX4 SITL or HITL mode and upload your mission to be executed.

To address the points you mentioned, if I understand what you're trying to do correctly,

1. I don't think there is a way to have the drone spawn in AirSim without an autopilot mode selected. ie: you can get rid of simpleflight if you want, But you will need to select another vehicleSetting if you want to spawn a UAV within Airsim (assuming you're in Multirotor mode) which means that there will have to be some sort of autopilot on there (SimpleFlight, PX4Multirotor, ArduCopter or ArduRover).
2. There currently is multiple API functions that would let you set the moving velocity of a drone in any direction with reference to the world frame (moveByVelocityAsync) or the drone's frame (moveByVelocityBodyFrameAsync). This is done using the physics simulation and the autopilot though rather than just spawning the drone based on flight data.

[misakabribri]

@ahmed-elsaharti
Thanks! The flight data is gathered from a fixed-wing aricraft, therefore the mission can't be executed via multirotor mode on AirSim. I kown it sounds weird, but I am also trying to modify the 3D model (use the fixed-wing model instead of the default multirotor).

I found some API functions in RpcLibClientBase.hpp:
// sensor omniscient APIs
Pose simGetVehiclePose(const std::string& vehicle_name = "") const;
void simSetVehiclePose(const Pose& pose, bool ignore_collision, const std::string& vehicle_name = "");
void simSetTraceLine(const std::vector& color_rgba, float thickness=3.0f, const std::string& vehicle_name = "");

1. Can I treat the vehicle as a "sensor" and change its position mannully using the fucniton simSetVehiclePose()?
2. The ComputerVision Mode won't use any phisics and vehicles. Can use the CV mode to get rid of the phisc engine? (maybe it is not a good idea)

[ahmed-elsaharti]

@misakabribri you could try using simSetVehiclePose to teleport the vehicle along a path but it would still be using multirotor physics.

With CV mode you pretty much just have a camera moving around in the scene.

Have a look at #2955, my understanding is that the author is working on a fixed wing model there

[jonyMarino]

@ahmed-elsaharti, do you think that using CV mode and using the blueprint drone (that can be added through the ue editor) can be a way of accomplishing this? Then we can move the drone through simSetObjectPose(). I think this is an excellent use case for easy integration with other physics simulators.

[ahmed-elsaharti]

@jonyMarino This is a BRILLIANT idea! This could also be used as a hacky solution for #3208
I just tested it and it works perfectly. Using the UE4 Editor i placed a random object (in this case a drone called BP_FlyingPawn_5, any fixed-wing model could be used too)


@misakabribri note that the drone mesh I placed in here could be replaced with any mesh/drone
on executing:

import airsim
client = airsim.VehicleClient()
client.confirmConnection()
pose1 = client.simGetObjectPose("BP_FlyingPawn_5");
pose1.position = pose1.position + airsim.Vector3r(-2, -2, -2)
success = client.simSetObjectPose("BP_FlyingPawn_5", pose1, True);

We get:

[saihv]

The preferred way is to use CV non-CV mode and simSetVehiclePose(), and this is also the way we can integrate external physics engines etc. simSetObjectPose (while it may sometimes work), is generally not recommended for the vehicles because it targets not the physics body but more of an abstract Unreal object type. I am not sure about the following: but there might also be issue with what's considered as origin when applying simSetObjectPose to the vehicle itself: because the UE objects are generally referenced with the vehicle as the origin, and moving the vehicle's body to another spot may not transform the origin accurately.

That said, there is currently an open issue with simSetVehiclePose, where calling it even just once disables physics entirely (#2324 should fix it).

The interesting next step is to go one level beyond setting the vehicle pose, and instead overriding the entire state. Currently, simSetVehiclePose internally calls kinematics_->setPose(pose). There is also a kinematics_->setState that can affect the entirety of the state: pose, twist and accelerations too.

AirSim/AirLib/include/physics/Kinematics.hpp

Line 13 in f423d9f

class Kinematics : public UpdatableObject {

[ahmed-elsaharti]

@saihv I haven't experimented with CV mode so far so I might (will probably) be completely wrong here, but doesn't CV mode not spawn any vehicles? My understanding at this point is that the only vehicle that we can set the pose of in CV mode is the actual camera. right?

[saihv]

Sorry! I made a typo. I meant [non-CV mode (Multirotor) and simSetVehiclePose] is preferred for simple visualization purposes (position and orientation) instead of simSetObjectPose. Another possible issue (just speculating, not sure) with CV mode + simSetObjectPose might be if we want to move the camera while moving and visualizing the drone: change in camera positions might change the origin of the UE frame within which we reference the objects, because the camera is the main reference in CV mode.

Teleporting the drone through simSetVehiclePose seems to disable physics forever, so this might actually be a good thing if we want to continue visualizing. To actually command velocities and higher order terms, setState might be interesting (whereas with setObjectPose we might be just limited to positions)

[ahmed-elsaharti]

@saihv Oh, no worries it makes sense now and I completely agree 👍. I was actually going to test moving the camera around in Non-CV mode to see if it messes up with anything in an attempt to see if I could create a 'fixed position' camera that tracks the drone as is needed for #3208

[ahmed-elsaharti]

@saihv I've tested the CV mode + simSetObjectPose side of things and it does seem like the UE origin does not change once created during initialization of the simulation (insert my regular 'I'm probably wrong' here 😂)

What I did to test this was to set up a scene with a drone_pawn as an object in there:

This is the initial camera pose and object location:

I then grabbed the object's location and moved it 4m backwards using

pose1 = client.simGetObjectPose("BP_FlyingPawn_5");
pose1.position = pose1.position + airsim.Vector3r(-4, 0, 0)
 success = client.simSetObjectPose("BP_FlyingPawn_5", pose1, True);

I then moved the camera backwards an equal amount

camera_pose = airsim.Pose(airsim.Vector3r(-4, 0, 0), airsim.to_quaternion(0, 0, 0))
client.simSetCameraPose("0", camera_pose)

This resulted in only the camera moving within the original frame.

Just to make sure that everything is still referenced to that coordinate system i moved the drone backwards again to pose2:

and compared both pose1 and pose2:

which showed that they're both 4m apart with pose1 grabbed/changed before camera movement while pose2 was grabbed/changed after it. which leads me to conclude that the camera movement did not affect the coordinate system the drone/object is referenced to or the rest of the scene objects.

Is it safe to assume that this is a viable solution or is my testing method flawed?

Edit: forgot to mention that moving the camera manually after and before these works fine too

[saihv]

@ahmed-elsaharti Thanks for the quick test! simSetCameraPose should also just move the camera relative to an invisible vehicle (imagine an infinitely long camera arm, essentially, whereas the invisible origin stays constant). This is certainly a way to get around the origin problem. With regards to your last comment, did you mean moving the camera with the M key?

EDIT: Actually nvm, it does make sense. Because NedTransform::NedTransform() is instantiated at the start with a global transform, it will keep a constant origin reference and it shouldn't matter what happens to the pose of the camera or the computer vision pawn itself during the simulation: because simGet/SetObjectPose operates on the global NED transform as well.

[ahmed-elsaharti]

@saihv I meant the manual WASD camera controls available through the view mode enabled through the M F key.

Also, would you recommend using this method to programmatically set the orientation of a fixed-location camera trying to follow track a drone (or any object really) being moved around the scene to simulate a 'fixed location ground observer' scenario?

[saihv]

Also, would you recommend using this method to programmatically set the orientation of a fixed-location camera trying to follow track a drone (or any object really) being moved around the scene to simulate a 'fixed location ground observer' scenario?

Yeah I think it should work. I forgot about the simSetCameraPose function entirely (although I originally wrote that function :D) and was thinking only about simSetVehiclePose, but regardless, set camera pose should safely let you operate local to the computer vision pawn origin anyway. I am assuming in this case your drone is also just behaving as an object (and not an actual FlyingPawn vehicle)

[ahmed-elsaharti]

I forgot about the simSetCameraPose function entirely (although I originally wrote that function :D)

@saihv I honestly do not blame you. AirSim's got ALOT of functions that are very similarly named. This would be a good segue into the 'API documentation' topic but thats out of the issue's scope anyway 😂.

I am assuming in this case your drone is also just behaving as an object (and not an actual FlyingPawn vehicle)

Yup for the case of #3208 my understanding is that AirSim would be used as a tool to 'visualize/draw' the data. So the drone would be an arbitrary object.

[saihv]

Side note, I am noticing some weird behavior with M in CV mode. WASD + simSetVehiclePose() seems to work okay w.r.t. the transforms, but pressing M in CV mode does seem to destroy some camera transform somewhere because the subwindow images are going wonky. Likely irrelevant to this discussion because we don't really need M in CV mode..

[ahmed-elsaharti]

Woops, i just realized that i meant F-mode rather than M-mode. Sorry about that

[saihv]

Yeah I think as long as F / WASD works it's fine, M is probably decoupling the camera director from the pawn. WASD movement actually displaces the ComputerVisionPawn in the UE world, so that was what I was mainly curious about

[nikitabeebe]

If this is used for a fixed wing UE4 model, will the setPose account for the difference between the movement of a copter (like the default FastPhysicsEngine) and that of a plane?

[jonyMarino]

Hi @nikitabeebe! Now the preferable way to accomplish this is using the "ExternalPhysicsEngine" along with setVehiclePose. In that way, you would get the drone's sensors.

[nikitabeebe]

Does the external physics engine support JSBSim or Gazebo? I saw the example you've made in the pull request, but is there any more work towards an actual flight engine?

[jonyMarino]

The only thing that ExternalPhysicsEngine does for now is disabling the update of the drone position except for setVehiclePose. So any FDM that you connect with that method will "drive" the drone. It can be JSBSim, Gazebo, or any other.