Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial implementation of mujoco texture support #423

Merged
merged 1 commit into from
Jan 16, 2025
Merged

Initial implementation of mujoco texture support #423

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions book/robot/BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,15 @@ rt_ipynb_test(
],
)

rt_ipynb_test(
name = "menagerie",
srcs = ["menagerie.ipynb"],
deps = [
"//manipulation",
"//manipulation:make_drake_compatible_model",
],
)

rt_ipynb_test(
name = "simulation",
srcs = ["simulation.ipynb"],
Expand Down
107 changes: 11 additions & 96 deletions book/robot/inspector.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 9,
"id": "99268d0c",
"metadata": {
"colab": {},
Expand All @@ -23,19 +23,9 @@
},
"outputs": [],
"source": [
"from pydrake.all import (\n",
" AddDefaultVisualization,\n",
" ModelVisualizer,\n",
" PackageMap,\n",
" Parser,\n",
" RobotDiagramBuilder,\n",
" Simulator,\n",
" StartMeshcat,\n",
")\n",
"from pydrake.all import ModelVisualizer, PackageMap, StartMeshcat\n",
"\n",
"from manipulation import running_as_notebook\n",
"from manipulation.make_drake_compatible_model import MakeDrakeCompatibleModel\n",
"from manipulation.utils import AddMujocoMenagerie"
"from manipulation import running_as_notebook"
]
},
{
Expand Down Expand Up @@ -152,84 +142,7 @@
"source": [
"# Robots description files from other repositories\n",
"\n",
"We've tried to make it very easy to pull in resources from other repositories. Here's an example of pulling in files directly from github. Drake's MuJoCo parser is the least used/complete of our parsers (I only added a minimal version of it quite recently), so many of these don't parse beautifully yet -- I'm working on it! But it's a good example of how to pull in resources from other repositories, and if there is a particular model you need that doesn't parse well for you, you can let me know."
]
},
{
"cell_type": "markdown",
"id": "074ad941",
"metadata": {},
"source": [
"## MuJoCo Menagerie\n",
"\n",
"Note that I'm tracking the remaining issues for parsing models from this repo [Drake issue #20444](https://github.com/RobotLocomotion/drake/issues/20444)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ec925772",
"metadata": {
"colab": {},
"colab_type": "code",
"id": "5SjOClhTltPk"
},
"outputs": [],
"source": [
"if running_as_notebook: # I don't want to download the menagerie in CI.\n",
" meshcat.Delete()\n",
" visualizer = ModelVisualizer(meshcat=meshcat)\n",
" plant = visualizer.parser().plant()\n",
" package_map = visualizer.parser().package_map()\n",
" AddMujocoMenagerie(package_map)\n",
" original_file = package_map.ResolveUrl(\n",
" # \"package://mujoco_menagerie/aloha/scene.xml\" # drake#22444\n",
" \"package://mujoco_menagerie/kuka_iiwa_14/iiwa14.xml\"\n",
" )\n",
" drake_compatible_file = original_file.replace(\".xml\", \".drake.xml\")\n",
" MakeDrakeCompatibleModel(original_file, drake_compatible_file)\n",
" visualizer.AddModels(filename=drake_compatible_file)\n",
" visualizer.Finalize()\n",
" visualizer.Run(loop_once=not running_as_notebook)\n",
" meshcat.DeleteAddedControls()"
]
},
{
"cell_type": "markdown",
"id": "15b804b0",
"metadata": {},
"source": [
"Sometimes it's useful to run a short simulation. Here's an example of how to do that."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a86287d0",
"metadata": {},
"outputs": [],
"source": [
"if running_as_notebook:\n",
" meshcat.Delete()\n",
" builder = RobotDiagramBuilder()\n",
" plant = builder.plant()\n",
" parser = Parser(plant)\n",
" package_map = parser.package_map()\n",
" AddMujocoMenagerie(package_map)\n",
" original_file = package_map.ResolveUrl(\n",
" \"package://mujoco_menagerie/agility_cassie/scene.xml\"\n",
" )\n",
" drake_compatible_file = original_file.replace(\".xml\", \".drake.xml\")\n",
" MakeDrakeCompatibleModel(original_file, drake_compatible_file)\n",
" parser.AddModels(drake_compatible_file)\n",
" plant.Finalize()\n",
" AddDefaultVisualization(builder.builder(), meshcat=meshcat)\n",
" diagram = builder.Build()\n",
" meshcat.StartRecording()\n",
" simulator = Simulator(diagram)\n",
" simulator.AdvanceTo(1.0)\n",
" meshcat.StopRecording()\n",
" meshcat.PublishRecording()"
"We've tried to make it very easy to pull in resources from other repositories. Here's an example of pulling in files directly from github."
]
},
{
Expand Down Expand Up @@ -267,12 +180,14 @@
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cfaf4640",
"cell_type": "markdown",
"id": "074ad941",
"metadata": {},
"outputs": [],
"source": []
"source": [
"## MuJoCo Menagerie\n",
"\n",
"Another nice repository of robot models is the [MuJoCo Menagerie](https://github.com/google-deepmind/mujoco_menagerie). See the [menagerie.ipynb](menagerie.ipynb) notebook to browse these models in Drake+Meshcat.\n"
]
}
],
"metadata": {
Expand Down
180 changes: 180 additions & 0 deletions book/robot/menagerie.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "05124c35",
"metadata": {
"colab_type": "text",
"id": "EgiF12Hf1Dhs"
},
"source": [
"This notebook provides examples to go along with the [textbook](http://manipulation.csail.mit.edu/robot.html). I recommend having both windows open, side-by-side!"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "99268d0c",
"metadata": {
"colab": {},
"colab_type": "code",
"id": "eeMrMI0-1Dhu",
"lines_to_end_of_cell_marker": 2
},
"outputs": [],
"source": [
"import logging\n",
"\n",
"import numpy as np\n",
"from IPython.display import Image, display\n",
"from pydrake.all import Parser, RobotDiagramBuilder, Simulator, StartMeshcat\n",
"\n",
"from manipulation import running_as_notebook\n",
"from manipulation.make_drake_compatible_model import MakeDrakeCompatibleModel\n",
"from manipulation.utils import AddMujocoMenagerie, ApplyDefaultVisualization"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3db6138d",
"metadata": {},
"outputs": [],
"source": [
"# Start the visualizer.\n",
"meshcat = StartMeshcat()"
]
},
{
"cell_type": "markdown",
"id": "074ad941",
"metadata": {},
"source": [
"## MuJoCo Menagerie\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0163ee04",
"metadata": {},
"outputs": [],
"source": [
"logging.getLogger(\"drake\").setLevel(logging.ERROR)\n",
"\n",
"\n",
"def show_model(model: str):\n",
" \"\"\"For `model` as one of the keys in the `model_data` defined below, this will load the model in Drake and display it in Meshcat, and it will display the corresponding render from the Menagerie documentation.\"\"\"\n",
" model_url = f\"package://mujoco_menagerie/{model}/{model_data[model][0]}\"\n",
" image_url = f\"package://mujoco_menagerie/{model}/{model_data[model][1]}\"\n",
"\n",
" meshcat.Delete()\n",
" builder = RobotDiagramBuilder()\n",
" plant = builder.plant()\n",
" parser = Parser(plant)\n",
" package_map = parser.package_map()\n",
" AddMujocoMenagerie(package_map)\n",
" model_path = package_map.ResolveUrl(model_url)\n",
" image_path = package_map.ResolveUrl(image_url)\n",
" drake_model_path = model_path.replace(\".xml\", \".drake.xml\")\n",
" MakeDrakeCompatibleModel(model_path, drake_model_path, overwrite=True)\n",
" model_instances = parser.AddModels(drake_model_path)\n",
" plant.Finalize()\n",
" ApplyDefaultVisualization(builder.builder(), meshcat=meshcat)\n",
" diagram = builder.Build()\n",
" simulator = Simulator(diagram)\n",
"\n",
" display(Image(filename=image_path))\n",
"\n",
" # Workaround for drake#22444: set the desired state to zero.\n",
" for model_instance in model_instances:\n",
" desired_state_port = plant.get_desired_state_input_port(model_instance)\n",
" if desired_state_port.size() > 0:\n",
" desired_state_port.FixValue(\n",
" plant.GetMyContextFromRoot(simulator.get_context()),\n",
" np.zeros(desired_state_port.size()),\n",
" )\n",
"\n",
" diagram.ForcedPublish(simulator.get_context())\n",
"\n",
"\n",
"model_data = {\n",
" \"agility_cassie\": (\"scene.xml\", \"cassie.png\"), # missing some textures\n",
" \"aloha\": (\"scene.xml\", \"aloha.png\"), # textures make it disappear\n",
" \"anybotics_anymal_b\": (\"scene.xml\", \"anymal_b.png\"), # a few missing bits\n",
" \"anybotics_anymal_c\": (\"scene.xml\", \"anymal_c.png\"), # mostly invisible\n",
" \"berkeley_humanoid\": (\"scene.xml\", \"berkeley_humanoid.png\"), # invisible\n",
" \"bitcraze_crazyflie_2\": (\"scene.xml\", \"cf2.png\"),\n",
" \"boston_dynamics_spot\": (\"scene_arm.xml\", \"spot.png\"),\n",
" # \"flybody\": (\"scene.xml\", \"flybody.png\"), # Drake#22372\n",
" \"franka_emika_panda\": (\"scene.xml\", \"panda.png\"),\n",
" \"franka_fr3\": (\"scene.xml\", \"fr3.png\"),\n",
" \"google_barkour_v0\": (\"scene.xml\", \"barkour_v0.png\"), # invisible\n",
" \"google_barkour_vb\": (\"scene.xml\", \"barkour_vb.png\"),\n",
" \"google_robot\": (\"scene.xml\", \"robot.png\"), # missing textures\n",
" \"hello_robot_stretch\": (\"scene.xml\", \"stretch.png\"), # gripper is invisible\n",
" \"hello_robot_stretch_3\": (\"scene.xml\", \"stretch.png\"), # missing texture on hand\n",
" \"kinova_gen3\": (\"scene.xml\", \"gen3.png\"),\n",
" \"kuka_iiwa_14\": (\"scene.xml\", \"iiwa14.png\"),\n",
" \"leap_hand\": (\"scene_right.xml\", \"right_hand.png\"),\n",
" \"pal_talos\": (\"scene_motor.xml\", \"talos.png\"), # missing body\n",
" \"pal_tiago\": (\"scene_motor.xml\", \"tiago.png\"), # mostly invisible\n",
" # \"pal_tiago_dual\": (\"scene_motor.xml\", \"tiago_dual.png\"), # Drake error\n",
" # \"realsense_d435i\": (\"d435i.xml\", \"d435i.png\"), # Drake#22372\n",
" \"rethink_robotics_sawyer\": (\"scene.xml\", \"sawyer.png\"),\n",
" \"robotiq_2f85\": (\"scene.xml\", \"2f85.png\"), # invisible\n",
" \"robotiq_2f85_v4\": (\"scene.xml\", \"2f85.png\"), # invisible\n",
" \"robotis_op3\": (\"scene.xml\", \"op3.png\"), # invisible\n",
" \"shadow_dexee\": (\"scene.xml\", \"shadow_dexee.png\"), # somethings off\n",
" \"skydio_x2\": (\"scene.xml\", \"x2.png\"), # invisible\n",
" \"trs_so_arm100\": (\"scene.xml\", \"so_arm100.png\"),\n",
" \"trossen_vx300s\": (\"scene.xml\", \"vx300s.png\"), # invisible\n",
" \"trossen_wx250s\": (\"scene.xml\", \"wx250s.png\"), # invisible\n",
" \"ufactory_lite6\": (\"scene.xml\", \"lite6.png\"), # invisible\n",
" \"ufactory_xarm7\": (\"scene.xml\", \"xarm7.png\"), # invisible\n",
" \"unitree_a1\": (\"scene.xml\", \"a1.png\"), # missing body\n",
" \"unitree_g1\": (\"scene_with_hands.xml\", \"g1.png\"), # invisible\n",
" \"unitree_go1\": (\"scene.xml\", \"go1.png\"), # invisible\n",
" \"unitree_go2\": (\"scene.xml\", \"go2.png\"),\n",
" \"unitree_h1\": (\"scene.xml\", \"h1.png\"), # invisible\n",
" \"unitree_z1\": (\"scene.xml\", \"z1.png\"), # invisible\n",
" \"universal_robots_ur10e\": (\"scene.xml\", \"ur10e.png\"),\n",
" \"universal_robots_ur5e\": (\"scene.xml\", \"ur5e.png\"),\n",
" \"wonik_allegro\": (\"scene_left.xml\", \"allegro_hand.png\"), # invisible\n",
"}\n",
"\n",
"if running_as_notebook:\n",
" show_model(\"rethink_robotics_sawyer\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1fc30a41",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Loading
Loading