add human trajectories

Author: shuyanzhou

README.md

@@ -21,10 +21,9 @@
 
 ![Overview](media/overview.png)
 
-## Roadmap
-- [ ] Support more agents with different prompting mechanisms such as [ASH](https://arxiv.org/pdf/2305.14257.pdf).
 
 ## News
+* [12/21/2023] We release the recording of trajectories performed by human annotators on ~170 tasks. Check out the [resource page](./resources/README.md#12212023-human-trajectories) for more details.
 * [11/3/2023] Multiple features!
   * Uploaded newest [execution trajectories](./resources/README.md#1132023-execution-traces-from-our-experiments-v2)
   * Added [Amazon Machine Image](./environment_docker/README.md#pre-installed-amazon-machine-image) that pre-installed all websites so that you don't have to!

resources/README.md

@@ -1,4 +1,12 @@
 # WebArena Resources
+## [12/21/2023] Human Trajectories
+We collected human trajectories on 179 tasks and the recording files are [here](https://drive.google.com/drive/folders/1NrN_sawtYK2V_uHnmmS8ugmGIKUAsPgt?usp=sharing).
+
+We sample one task from each template or templates that share similar task semantic. Each file is named as `<task_id>.zip`, and the corresponding template id can be found in the [task config file](../config_files/test.raw.json). The trajectories are presented as playwright trace files. You can view the concrete HTML, network traffic etc by `playwright show-trace <example_idx>.zip`.
+
+Human task success rate: 78.24%
+
+
 ## [11/3/2023] Execution Traces from Our Experiments (v2)
 ![v2 results](../media/v2_result.png)
 The results on the release v2 can be found in this [folder](https://drive.google.com/drive/folders/1H4wkzDkY2ufiC63DISMXllri0j-ipWcs?usp=sharing). It contains

scripts/collect_obs.py

@@ -22,7 +22,7 @@
 
 def gen_tmp_storage_state() -> None:
     with open(f"scripts/tmp_storage_state.json", "w") as f:
-        json.dump({"storage_state": ".auth/gitlab_state.json"}, f)
+        json.dump({"storage_state": ".auth/shopping_admin_state.json"}, f)
 
 
 def get_observation(
@@ -32,10 +32,12 @@ def get_observation(
         observation_type=observation_type,
         current_viewport_only=current_viewport_only,
         headless=HEADLESS,
+        sleep_after_execution=2.0,
     )
     env.reset(options={"config_file": f"scripts/tmp_storage_state.json"})
-    s = f"""page.goto("{GITLAB}/byteblaze/a11y-syntax-highlighting")
-    page.scroll(down)
+    s = f"""page.goto("http://ec2-3-131-244-37.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/")
+    page.get_by_label("", exact=True).fill("reviews")
+    page.get_by_label("", exact=True).press("Enter")
     page.scroll(down)"""
     action_seq = s.split("\n")