Skip to content

Commit

Permalink
Deployed dc9db58 with MkDocs version: 1.6.1
Browse files Browse the repository at this point in the history
  • Loading branch information
@marvin233
marvin233 committed Jan 21, 2025
1 parent bccf906 commit c98d9ec
Show file tree
Hide file tree
Showing 4 changed files with 106 additions and 62 deletions.
154 changes: 93 additions & 61 deletions pages/leaderboard/index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -356,67 +356,99 @@ <h1 style="color: #4A90E2;">Leaderboard</h1>

<div class=table-container style="text-align:center">
<table id="sortableTable">
<tr>
<th >Agent Name</th>
<th onclick="sortTable(1)" >Avg &#x21C5;</th>
<th onclick="sortTable(2)" >Detection &#x21C5;</th>
<th onclick="sortTable(3)" >Localization &#x21C5;</th>
<th onclick="sortTable(4)" >Diagnosis &#x21C5;</th>
<th onclick="sortTable(5)" >Mitigation &#x21C5;</th>
<th onclick="sortTable(6)" >Time &#x21C5;</th>
<th >Org.</th>
<th >Model</th>
<th >Link</th>
</tr>
<tr>
<td>🥇FLASH</td>
<td><b>59.27</b></td>
<td><b>100</b></td>
<td>46.15</td>
<td>36.36</td>
<td><b>54.55</b></td>
<td>102.57</td>
<td>AIOpsLab</td>
<td>GPT 4</td>
<td><a href="">🔗</a></td>
</tr>
<tr>
<td>🥈REACT</td>
<td>53.15</td>
<td>76.92</td>
<td>53.85</td>
<td><b>45.45</b></td>
<td>36.36</td>
<td>44.25</td>
<td>AIOpsLab</td>
<td>GPT 4</td>
<td><a href="">🔗</a></td>
</tr>
<tr>
<td>🥉GPT 4 w Shell</td>
<td>49.74</td>
<td>69.23</td>
<td><b>61.54<b></td>
<td>40.9</td>
<td>27.27</td>
<td>30.57</td>
<td>AIOpsLab</td>
<td>GPT 4</td>
<td><a href="">🔗</a></td>
</tr>
<tr>
<td>GPT 3.5 w Shell</td>
<td>15.73</td>
<td>23.07</td>
<td>30.77</td>
<td>9.09</td>
<td>0</td>
<td>12.79</td>
<td>AIOpsLab</td>
<td>GPT 3.5</td>
<td><a href="">🔗</a></td>
</tr>
</table>
<thead>
<tr>
<th>Agent Name</th>
<th>Avg</th>
<th>Detection</th>
<th>Localization</th>
<th>Diagnosis</th>
<th>Mitigation</th>
<th>Time</th>
<th>Organization</th>
<th>Link</th>
</tr>
</thead>
<tbody>
<tr>
<td>🥇FLASH (GPT-4)</td>
<td><b>59.27</b></td>
<td><b>100</b></td>
<td>46.15</td>
<td>36.36</td>
<td><b>54.55</b></td>
<td>102.57</td>
<td>AIOpsLab</td>
<td>🔗</td>
</tr>
<tr>
<td>🥈REACT (GPT-4)</td>
<td>53.15</td>
<td>76.92</td>
<td>53.85</td>
<td><b>45.45</b></td>
<td>36.36</td>
<td>44.25</td>
<td>AIOpsLab</td>
<td>🔗</td>
</tr>
<tr>
<td>🥉GPT-4 w Shell</td>
<td>49.74</td>
<td>69.23</td>
<td>61.54</td>
<td>40.9</td>
<td>27.27</td>
<td>30.57</td>
<td>AIOpsLab</td>
<td>🔗</td>
</tr>
<tr>
<td>FLASH (Llama3-8b)</td>
<td>33.34</td>
<td>80</td>
<td>20</td>
<td>0</td>
<td>33.34</td>
<td>63.16</td>
<td>AIOpsLab</td>
<td>🔗</td>
</tr>
<tr>
<td>GPT-3.5 w Shell</td>
<td>15.73</td>
<td>23.07</td>
<td>30.77</td>
<td>9.09</td>
<td>0</td>
<td>12.79</td>
<td>AIOpsLab</td>
<td>🔗</td>
</tr>
<tr>
<td>REACT (Llama3-8b)</td>
<td>15</td>
<td>60</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>230.74</td>
<td>AIOpsLab</td>
<td>🔗</td>
</tr>
<tr>
<td>LocaleXpert (Llama3-8b)</td>
<td>-</td>
<td>-</td>
<td><b>80</b></td>
<td>-</td>
<td>-</td>
<td>102.08</td>
<td>AIOpsLab</td>
<td>🔗</td>
</tr>
</tbody>
</table>
</div>


Expand Down
12 changes: 12 additions & 0 deletions pages/table.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
| Agent Name | Avg | Detection | Localization | Diagnosis | Mitigation | Time | Org. | Model | Link |
|-------------------|-------|-----------|--------------|-----------|------------|--------|-----------|--------|------|
| 🥇FLASH | **59.27** | **100** | 46.15 | 36.36 | **54.55** | 102.57 | AIOpsLab | GPT 4 | 🔗 |
| 🥈REACT | 53.15 | 76.92 | 53.85 | **45.45** | 36.36 | 44.25 | AIOpsLab | GPT 4 | 🔗 |
| 🥉GPT 4 w Shell | 49.74 | 69.23 | **61.54** | 40.9 | 27.27 | 30.57 | AIOpsLab | GPT 4 | 🔗 |
| GPT 3.5 w Shell | 15.73 | 23.07 | 30.77 | 9.09 | 0 | 12.79 | AIOpsLab | GPT 3.5| 🔗 |






2 changes: 1 addition & 1 deletion search/search_index.json
View file
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":"AIOpsLab A Holistic Framework to Design, Develop, and Evaluate AI Agents for Enabling Autonomous Clouds M365 Research - AIOps Team \u00a0Leaderboard \u00a0Paper \u00a0Code News <p>\ud83c\udd95 [12/2024] Microsoft Research features AIOpsLab in their latest blog post! \ud83c\udf10 [Link] </p> <p>\ud83c\udd95 [12/2024] Our code is now live on GitHub! \ud83d\ude80 [Link] </p> <p>\ud83c\udd95 [11/2024] Checkout our arxiv paper \"AIOpsLab: A Holistic Framework for Evaluating AI Agents for Enabling Autonomous Cloud\" \ud83d\udc40 [Link] </p> <p>\ud83c\udd95 [10/2024] Our vision paper \"Building AI Agents for Autonomous Clouds: Challenges and Design Principles\" was accepted by SoCC'24 \ud83d\udc40 [Link] </p> About <p>AIOpsLab is a holistic framework to enable the design, development, and evaluation of autonomous AIOps agents that, additionally, serves the purpose of building reproducible, standardized, interoperable and scalable benchmarks. AIOpsLab can deploy microservice cloud environments, inject faults, generate workloads, and export telemetry data, while orchestrating these components and providing interfaces for interacting with and evaluating agents. Moreover, AIOpsLab provides a built-in benchmark suite with a set of problems to evaluate AIOps agents in an interactive environment. This suite can be easily extended to meet user-specific needs. </p> <p>The Orchestrator coordinates interactions between various system components and serves as the Agent-Cloud-Interface (ACI). Agents engage with the Orchestrator to solve tasks, receiving a problem description, instructions, and relevant APIs. The Orchestrator generates diverse problems using the Workload and Fault Generators, injecting these into applications it can deploy. The deployed service has observability, providing telemetry such as metrics, traces, and logs. Agents act via the Orchestrator, which executes them and updates the service's state. The Orchestrator evaluates the final solution using predefined metrics for the task.</p> BibTeX <pre><code>\n @inproceedings{shetty2024building,\n title = {Building AI Agents for Autonomous Clouds: Challenges and Design Principles},\n author = {Shetty, Manish and Chen, Yinfang and Somashekar, Gagan and Ma, Minghua and Simmhan, Yogesh and Zhang, Xuchao and Mace, Jonathan and Vandevoorde, Dax and Las-Casas, Pedro and Gupta, Shachee Mishra and Nath, Suman and Bansal, Chetan and Rajmohan, Saravan},\n year = {2024},\n booktitle = {Proceedings of 15th ACM Symposium on Cloud Computing (SoCC'24)},\n }\n @misc{chen2024aiopslab,\n title = {AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds},\n author = {Chen, Yinfang and Shetty, Manish and Somashekar, Gagan and Ma, Minghua and Simmhan, Yogesh and Mace, Jonathan and Bansal, Chetan and Wang, Rujia and Rajmohan, Saravan},\n year = {2024},\n booktitle = {Arxiv}\n }\n </code>\n </pre>"},{"location":"pages/leaderboard/","title":"Leaderboard","text":"AIOpsLab A Holistic Framework to Design, Develop, and Evaluate AI Agents for Enabling Autonomous Clouds M365 Research - AIOps Team \u00a0Home \u00a0Paper \u00a0Code Leaderboard <p>We showcase the key results on the leaderboard. If you'd like your results to appear, please email us at [email protected]. In the table, AVG represents the average accuracy across all tasks. Time indicates the average runtime for the agents. Agent Name Avg \u21c5 Detection \u21c5 Localization \u21c5 Diagnosis \u21c5 Mitigation \u21c5 Time \u21c5 Org. Model Link \ud83e\udd47FLASH 59.27 100 46.15 36.36 54.55 102.57 AIOpsLab GPT 4 \ud83d\udd17 \ud83e\udd48REACT 53.15 76.92 53.85 45.45 36.36 44.25 AIOpsLab GPT 4 \ud83d\udd17 \ud83e\udd49GPT 4 w Shell 49.74 69.23 61.54 40.9 27.27 30.57 AIOpsLab GPT 4 \ud83d\udd17 GPT 3.5 w Shell 15.73 23.07 30.77 9.09 0 12.79 AIOpsLab GPT 3.5 \ud83d\udd17"}]}
{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":"AIOpsLab A Holistic Framework to Design, Develop, and Evaluate AI Agents for Enabling Autonomous Clouds M365 Research - AIOps Team \u00a0Leaderboard \u00a0Paper \u00a0Code News <p>\ud83c\udd95 [12/2024] Microsoft Research features AIOpsLab in their latest blog post! \ud83c\udf10 [Link] </p> <p>\ud83c\udd95 [12/2024] Our code is now live on GitHub! \ud83d\ude80 [Link] </p> <p>\ud83c\udd95 [11/2024] Checkout our arxiv paper \"AIOpsLab: A Holistic Framework for Evaluating AI Agents for Enabling Autonomous Cloud\" \ud83d\udc40 [Link] </p> <p>\ud83c\udd95 [10/2024] Our vision paper \"Building AI Agents for Autonomous Clouds: Challenges and Design Principles\" was accepted by SoCC'24 \ud83d\udc40 [Link] </p> About <p>AIOpsLab is a holistic framework to enable the design, development, and evaluation of autonomous AIOps agents that, additionally, serves the purpose of building reproducible, standardized, interoperable and scalable benchmarks. AIOpsLab can deploy microservice cloud environments, inject faults, generate workloads, and export telemetry data, while orchestrating these components and providing interfaces for interacting with and evaluating agents. Moreover, AIOpsLab provides a built-in benchmark suite with a set of problems to evaluate AIOps agents in an interactive environment. This suite can be easily extended to meet user-specific needs. </p> <p>The Orchestrator coordinates interactions between various system components and serves as the Agent-Cloud-Interface (ACI). Agents engage with the Orchestrator to solve tasks, receiving a problem description, instructions, and relevant APIs. The Orchestrator generates diverse problems using the Workload and Fault Generators, injecting these into applications it can deploy. The deployed service has observability, providing telemetry such as metrics, traces, and logs. Agents act via the Orchestrator, which executes them and updates the service's state. The Orchestrator evaluates the final solution using predefined metrics for the task.</p> BibTeX <pre><code>\n @inproceedings{shetty2024building,\n title = {Building AI Agents for Autonomous Clouds: Challenges and Design Principles},\n author = {Shetty, Manish and Chen, Yinfang and Somashekar, Gagan and Ma, Minghua and Simmhan, Yogesh and Zhang, Xuchao and Mace, Jonathan and Vandevoorde, Dax and Las-Casas, Pedro and Gupta, Shachee Mishra and Nath, Suman and Bansal, Chetan and Rajmohan, Saravan},\n year = {2024},\n booktitle = {Proceedings of 15th ACM Symposium on Cloud Computing (SoCC'24)},\n }\n @misc{chen2024aiopslab,\n title = {AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds},\n author = {Chen, Yinfang and Shetty, Manish and Somashekar, Gagan and Ma, Minghua and Simmhan, Yogesh and Mace, Jonathan and Bansal, Chetan and Wang, Rujia and Rajmohan, Saravan},\n year = {2024},\n booktitle = {Arxiv}\n }\n </code>\n </pre>"},{"location":"pages/leaderboard/","title":"Leaderboard","text":"AIOpsLab A Holistic Framework to Design, Develop, and Evaluate AI Agents for Enabling Autonomous Clouds M365 Research - AIOps Team \u00a0Home \u00a0Paper \u00a0Code Leaderboard <p>We showcase the key results on the leaderboard. If you'd like your results to appear, please email us at [email protected]. In the table, AVG represents the average accuracy across all tasks. Time indicates the average runtime for the agents. Agent Name Avg Detection Localization Diagnosis Mitigation Time Organization Link \ud83e\udd47FLASH (GPT-4) 59.27 100 46.15 36.36 54.55 102.57 AIOpsLab \ud83d\udd17 \ud83e\udd48REACT (GPT-4) 53.15 76.92 53.85 45.45 36.36 44.25 AIOpsLab \ud83d\udd17 \ud83e\udd49GPT-4 w Shell 49.74 69.23 61.54 40.9 27.27 30.57 AIOpsLab \ud83d\udd17 FLASH (Llama3-8b) 33.34 80 20 0 33.34 63.16 AIOpsLab \ud83d\udd17 GPT-3.5 w Shell 15.73 23.07 30.77 9.09 0 12.79 AIOpsLab \ud83d\udd17 REACT (Llama3-8b) 15 60 0 0 0 230.74 AIOpsLab \ud83d\udd17 LocaleXpert (Llama3-8b) - - 80 - - 102.08 AIOpsLab \ud83d\udd17"}]}
Binary file modified sitemap.xml.gz
View file
Binary file not shown.

0 comments on commit c98d9ec

Please sign in to comment.