Figure out the correct way of supporting task instances (and other elements) cross-dataset #1846
Description
We currently have the main scenes etc. in the behavior-1k-assets dataset and the task instances in the 2025-challenge-task-instances dataset. This means that if the scene has its dataset name set to the former, the JSON will not be searched for inside the latter. To patch this, the team made the scene code specifically look in the latter dataset. This is not clean, specific to this dataset, and has other issues (you can't pick scene instances in the orig dataset).
We need to find a good solution to this. How can the code know where to look for?
One alternative is overlaying the datasets on top of another (e.g. objects would no longer be namespaced by dataset, everything forms one overlay dataset), but this induces problems in terms of what happens if object or scene names overlap between datasets. To avoid that, one solution might be adding another layer of namespacing inside the objects/ and scenes/ directories and THEN doing the overlaying.
All of this should be done in the context of upcoming support for procthor/ai2thor/hssd/habitat etc. datasets.