|
1 | 1 | <!DOCTYPE html> |
2 | 2 | <html lang="en"> |
3 | | - <head> |
4 | | - <meta charset="utf-8"> |
5 | | - <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
6 | | - <meta name="viewport" content="width=device-width, initial-scale=1"> |
7 | | - <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> |
8 | | - <meta name="description" content=""> |
9 | | - <meta name="author" content=""> |
10 | | - <link rel="icon" href="../../favicon.ico"> |
| 3 | +<head> |
| 4 | +<meta charset="utf-8"> |
| 5 | +<meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| 6 | +<meta name="viewport" content="width=device-width, initial-scale=1"> |
| 7 | +<!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> |
| 8 | +<meta name="description" content=""> |
| 9 | +<meta name="author" content=""> |
| 10 | +<link rel="icon" href="../../favicon.ico"> |
11 | 11 |
|
12 | | - <title>Guide to the OWASP Benchmark v${version}</title> |
| 12 | +<title>Guide to the OWASP Benchmark v${version}</title> |
13 | 13 |
|
14 | | - <!-- Bootstrap core CSS --> |
15 | | - <link href="content/css/bootstrap.min.css" rel="stylesheet"> |
| 14 | +<!-- Bootstrap core CSS --> |
| 15 | +<link href="content/css/bootstrap.min.css" rel="stylesheet"> |
16 | 16 |
|
17 | | - <!-- Custom styles for this template --> |
18 | | - <link href="content/dashboard.css" rel="stylesheet"> |
| 17 | +<!-- Custom styles for this template --> |
| 18 | +<link href="content/dashboard.css" rel="stylesheet"> |
19 | 19 |
|
20 | | - <!-- Just for debugging purposes. Don't actually copy these 2 lines! --> |
21 | | - <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]--> |
22 | | - <script src="content/js/ie-emulation-modes-warning.js"></script> |
| 20 | +<script src="content/js/ie-emulation-modes-warning.js" type="text/javascript"></script> |
23 | 21 |
|
24 | | - <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> |
25 | | - <!--[if lt IE 9]> |
26 | | - <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script> |
27 | | - <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script> |
28 | | - <![endif]--> |
29 | | - </head> |
| 22 | +</head> |
30 | 23 |
|
31 | | - <body> |
| 24 | +<body> |
32 | 25 |
|
33 | 26 | <nav class="navbar navbar-inverse navbar-fixed-top"> |
34 | | - <div class="container-fluid"> |
35 | | - <div class="navbar-header"> |
36 | | - <a class="navbar-brand" href="OWASP_Benchmark_Home.html">OWASP Benchmark v${version}</a> |
| 27 | + <div class="container-fluid"> |
| 28 | + <div class="navbar-header"> |
| 29 | + <a class="navbar-brand" href="OWASP_Benchmark_Home.html">OWASP Benchmark v${version}</a> |
| 30 | + </div> |
| 31 | + <div id="navbar" class="navbar-collapse collapse"> |
| 32 | + <ul class="nav navbar-nav navbar-right"> |
| 33 | + <li><a href="OWASP_Benchmark_Home.html">Home</a></li> |
| 34 | + <li class="dropdown"><a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Tools<span class="caret"></span></a> |
| 35 | + <ul class="dropdown-menu">${toolmenu} |
| 36 | + </ul></li> |
| 37 | + <li class="dropdown"><a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Vulnerabilities<span class="caret"></span></a> |
| 38 | + <ul class="dropdown-menu">${vulnmenu} |
| 39 | + </ul></li> |
| 40 | + <li><a href="OWASP_Benchmark_Guide.html">Guide</a></li> |
| 41 | + </ul> |
| 42 | + </div> |
37 | 43 | </div> |
38 | | - <div id="navbar" class="navbar-collapse collapse"> |
39 | | - <ul class="nav navbar-nav navbar-right"> |
40 | | - <li><a href="OWASP_Benchmark_Home.html">Home</a></li> |
41 | | - <li class="dropdown"> |
42 | | - <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Tools<span class="caret"></span></a> |
43 | | - <ul class="dropdown-menu"> |
44 | | -${toolmenu} |
45 | | - </ul> |
46 | | - </li> |
47 | | - <li class="dropdown"> |
48 | | - <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Vulnerabilities<span class="caret"></span></a> |
49 | | - <ul class="dropdown-menu"> |
50 | | -${vulnmenu} |
51 | | - </ul> |
52 | | - </li> |
53 | | - <li><a href="OWASP_Benchmark_Guide.html">Guide</a></li> |
54 | | - </ul> |
55 | | - </div> |
56 | | - </div> |
57 | 44 | </nav> |
58 | 45 |
|
59 | 46 |
|
60 | 47 | <div class="container"> |
61 | 48 |
|
62 | | - <div class="starter-template"> |
| 49 | + <div class="starter-template"> |
| 50 | + |
| 51 | + <div>empty</div> |
| 52 | + <div>empty</div> |
63 | 53 |
|
64 | | -<div>empty</div> |
65 | | -<div>empty</div> |
| 54 | + <h2>Introduction</h2> |
| 55 | + <p>The OWASP Benchmark is a test suite designed to evaluate the speed, coverage, and accuracy of automated vulnerability detection tools. Without the ability to measure these tools, it is difficult to understand their strengths and weaknesses, and compare them to each other. The Benchmark contains thousands of test cases that are fully runnable and exploitable.</p> |
66 | 56 |
|
67 | | -<h2>Introduction</h2> |
68 | | -<p>The OWASP Benchmark is a test suite designed to evaluate the speed, coverage, and accuracy of automated vulnerability detection tools. Without the ability to measure these tools, |
69 | | -it is difficult to understand their strengths and weaknesses, and compare them to each other. The Benchmark contains thousands of test cases that are fully runnable and exploitable.</p> |
| 57 | + <p>You can analyze the Benchmark with any type of Application Security Testing (AST) tool, including Static (SAST), Dynamic (DAST) (like OWASP ZAP), and Interactive (IAST) tools. The current version is implemented in Java. Future versions may expand to include other languages.</p> |
70 | 58 |
|
71 | | -<p>You can analyze the Benchmark with any type of Application Security Testing (AST) tool, including Static (SAST), Dynamic (DAST) |
72 | | -(like OWASP ZAP), and Interactive (IAST) tools. The current version is implemented in Java. Future versions may expand to include other languages.</p> |
| 59 | + <p> |
| 60 | + For more information, please visit the <a href="https://owasp.org/www-project-benchmark/">OWASP Benchmark Project Site</a>. |
73 | 61 |
|
74 | | -<p>For more information, please visit the <a href="https://owasp.org/www-project-benchmark/">OWASP Benchmark Project Site</a>. |
| 62 | + </p> |
| 63 | + <h2>Interpretation Guide</h2> |
| 64 | + <p>Security tools (SAST, DAST, and IAST) are amazing when they find a complex vulnerability in your code. But they can drive everyone crazy with complexity, false alarms, and missed vulnerabilities. Using these tools without understanding their strengths and weaknesses can lead to a dangerous false sense of security.</p> |
75 | 65 |
|
76 | | -<h2>Interpretation Guide</h2> |
77 | | -<p>Security tools (SAST, DAST, and IAST) are amazing when they find a complex vulnerability in your code. But they can drive everyone crazy with complexity, false alarms, and missed |
78 | | -vulnerabilities. Using these tools without understanding their strengths and weaknesses can lead to a dangerous false sense of security.</p> |
| 66 | + <p>We are on a quest to measure just how good these tools are at discovering and properly diagnosing security problems in applications. We rely on the long history of military and medical evaluation of detection technology as a foundation for our research. Therefore, the test suite tests both real and fake vulnerabilities.</p> |
79 | 67 |
|
80 | | -<p>We are on a quest to measure just how good these tools are at discovering and properly diagnosing security problems in applications. We rely on the long history of military and |
81 | | -medical evaluation of detection technology as a foundation for our research. Therefore, the test suite tests both real and fake vulnerabilities.</p> |
| 68 | + <p>There are four possible test outcomes in the Benchmark:</p> |
| 69 | + <ul> |
| 70 | + <li>Tool correctly identifies a real vulnerability (True Positive - TP)</li> |
| 71 | + <li>Tool fails to identify a real vulnerability (False Negative - FN)</li> |
| 72 | + <li>Tool correctly ignores a false alarm (True Negative - TN)</li> |
| 73 | + <li>Tool fails to ignore a false alarm (False Positive - FP)</li> |
| 74 | + </ul> |
82 | 75 |
|
83 | | -<p>There are four possible test outcomes in the Benchmark:</p> |
84 | | -<ul> |
85 | | -<li>Tool correctly identifies a real vulnerability (True Positive - TP)</li> |
86 | | -<li>Tool fails to identify a real vulnerability (False Negative - FN)</li> |
87 | | -<li>Tool correctly ignores a false alarm (True Negative - TN)</li> |
88 | | -<li>Tool fails to ignore a false alarm (False Positive - FP)</li> |
89 | | -</ul> |
| 76 | + <p>We can learn a lot about a tool from these four metrics. A tool that simply flags every line of code as vulnerable will perfectly identify all vulnerabilities in an application, but will also have 100% false positives. Similarly, a tool that reports nothing will have zero false positives, but will also identify zero real vulnerabilities. Imagine a tool that flips a coin to decide whether to report each vulnerability for every test case. The result would be 50% true positives and 50% false positives. We need a way to distinguish valuable security tools from these trivial ones.</p> |
90 | 77 |
|
91 | | -<p>We can learn a lot about a tool from these four metrics. A tool that simply flags every line of code as vulnerable will perfectly identify all vulnerabilities in |
92 | | -an application, but will also have 100% false positives. Similarly, a tool that reports nothing will have zero false positives, but will also identify zero real vulnerabilities. |
93 | | -Imagine a tool that flips a coin to decide whether to report each vulnerability for every test case. The result would be 50% true positives and 50% false positives. We need a |
94 | | -way to distinguish valuable security tools from these trivial ones.</p> |
| 78 | + <p>If you imagine the line that connects all these points, from 0,0 to 100,100 establishes a line that roughly translates to "random guessing." The ultimate measure of a security tool is how much better it can do than this line. The diagram below shows how we will evaluate security tools against the Benchmark.</p> |
95 | 79 |
|
96 | | -<p>If you imagine the line that connects all these points, from 0,0 to 100,100 establishes a line that roughly translates to "random guessing." The ultimate measure of a |
97 | | -security tool is how much better it can do than this line. The diagram below shows how we will evaluate security tools against the Benchmark.</p> |
| 80 | + <img src="content/benchmark_guide.png" alt="" /> |
98 | 81 |
|
99 | | -<img src="content/benchmark_guide.png"/> |
| 82 | + <h3>Key:</h3> |
100 | 83 |
|
101 | | -<h3>Key:</h3> |
| 84 | + <table class="table"> |
| 85 | + <tr> |
| 86 | + <th>True Positive (TP)</th> |
| 87 | + <td>Tests with real vulnerabilities that were correctly reported as vulnerable by the tool.</td> |
| 88 | + </tr> |
102 | 89 |
|
103 | | -<table class="table"> |
104 | | -<tr> |
105 | | -<th>True Positive (TP)</th> |
106 | | -<td>Tests with real vulnerabilities that were correctly reported as vulnerable by the tool.</td> |
107 | | -</tr> |
| 90 | + <tr> |
| 91 | + <th>False Negative (FN)</th> |
| 92 | + <td>Tests with real vulnerabilities that were not correctly reported as vulnerable by the tool.</td> |
| 93 | + </tr> |
108 | 94 |
|
109 | | -<tr> |
110 | | -<th>False Negative (FN)</th> |
111 | | -<td>Tests with real vulnerabilities that were not correctly reported as vulnerable by the tool.</td> |
112 | | -</tr> |
| 95 | + <tr> |
| 96 | + <th>True Negative (TN)</th> |
| 97 | + <td>Tests with fake vulnerabilities that were correctly not reported as vulnerable by the tool.</td> |
| 98 | + </tr> |
113 | 99 |
|
114 | | -<tr> |
115 | | -<th>True Negative (TN)</th> |
116 | | -<td>Tests with fake vulnerabilities that were correctly not reported as vulnerable by the tool.</td> |
117 | | -</tr> |
| 100 | + <tr> |
| 101 | + <th>False Positive (FP)</th> |
| 102 | + <td>Tests with fake vulnerabilities that were incorrectly reported as vulnerable by the tool.</td> |
| 103 | + </tr> |
118 | 104 |
|
119 | | -<tr> |
120 | | -<th>False Positive (FP)</th> |
121 | | -<td>Tests with fake vulnerabilities that were incorrectly reported as vulnerable by the tool.</td> |
122 | | -</tr> |
| 105 | + <tr> |
| 106 | + <th>True Positive Rate (TPR) = TP / ( TP + FN )</th> |
| 107 | + <td>The rate at which the tool correctly reports real vulnerabilities. Also referred to as Recall, as defined at <a href="https://en.wikipedia.org/wiki/Precision_and_recall">Wikipedia</a>. |
| 108 | + </td> |
| 109 | + </tr> |
123 | 110 |
|
124 | | -<tr> |
125 | | -<th>True Positive Rate (TPR) = TP / ( TP + FN )</th> |
126 | | -<td>The rate at which the tool correctly reports real vulnerabilities. Also referred to as Recall, as defined at |
127 | | -<a href="https://en.wikipedia.org/wiki/Precision_and_recall">Wikipedia</a>.</td> |
128 | | -</tr> |
| 111 | + <tr> |
| 112 | + <th>False Positive Rate (FPR) = FP / ( FP + TN )</th> |
| 113 | + <td>The rate at which the tool incorrectly reports fake vulnerabilities as real.</td> |
| 114 | + </tr> |
129 | 115 |
|
130 | | -<tr> |
131 | | -<th>False Positive Rate (FPR) = FP / ( FP + TN )</th> |
132 | | -<td>The rate at which the tool incorrectly reports fake vulnerabilities as real.</td> |
133 | | -</tr> |
| 116 | + <tr> |
| 117 | + <th>Score = TPR - FPR</th> |
| 118 | + <td>Normalized distance from the random guess line.</td> |
| 119 | + </tr> |
| 120 | + </table> |
134 | 121 |
|
135 | | -<tr> |
136 | | -<th>Score = TPR - FPR</th> |
137 | | -<td>Normalized distance from the random guess line.</td> |
138 | | -</tr> |
139 | | -</table> |
| 122 | + </div> |
140 | 123 |
|
141 | | - </div> |
142 | | - |
143 | 124 | </div> |
144 | 125 |
|
145 | | - <!-- Bootstrap core JavaScript |
| 126 | + <!-- Bootstrap core JavaScript |
146 | 127 | ================================================== --> |
147 | 128 | <!-- jQuery (necessary for Bootstrap's JavaScript plugins) --> |
148 | | - <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script> |
| 129 | + <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js" type="text/javascript"></script> |
149 | 130 | <!-- Include all compiled plugins (below), or include individual files as needed --> |
150 | | - <script src="content/js/bootstrap.min.js"></script> |
151 | | - </body> |
| 131 | + <script src="content/js/bootstrap.min.js" type="text/javascript"></script> |
| 132 | +</body> |
152 | 133 | </html> |
| 134 | + |
0 commit comments