1
1
Reference Coreference Scorer
2
2
============================
3
3
4
- Description
4
+ DESCRIPTION
5
5
-----------
6
6
7
- This is the official implementation of the revised coreference scorer used for CoNLL-2011/2012 shared tasks on coreference resolution.
8
- It addresses issues that prevented the consistent scoring of predicted mentions in the past.
7
+ This is the official implementation of the revised coreference scorer
8
+ used for CoNLL-2011/2012 shared tasks on coreference resolution. It
9
+ addresses issues that prevented the consistent scoring of predicted
10
+ mentions in the past.
9
11
10
12
11
13
VERSION
12
14
-------
13
15
14
- The current stable version is v8.01
15
-
16
- Citation
17
- --------
18
-
19
- We would appreciate if you cite the paper when you use this scorer as some of us are academics or wanting to be academics, and citations matter.
20
-
21
- ::
22
-
23
- @InProceedings{pradhan-EtAl:2014:P14-2,
24
- author = {Pradhan, Sameer and Luo, Xiaoqiang and Recasens, Marta and Hovy, Eduard and Ng, Vincent and Strube, Michael},
25
- title = {Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation},
26
- booktitle = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
27
- month = {June},
28
- year = {2014},
29
- address = {Baltimore, Maryland},
30
- publisher = {Association for Computational Linguistics},
31
- pages = {30--35},
32
- url = {http://www.aclweb.org/anthology/P14-2006}
33
- }
34
-
35
-
36
- Usage
37
- -----
38
-
39
-
40
- ::
41
-
42
- perl scorer.pl <metric> <key> <response> [<document-id>]
43
-
44
-
45
- <metric>: the metric desired to score the results. one of the following values:
46
-
47
- muc: MUCScorer (Vilain et al, 1995)
48
- bcub: B-Cubed (Bagga and Baldwin, 1998)
49
- ceafm: CEAF (Luo et al., 2005) using mention-based similarity
50
- ceafe: CEAF (Luo et al., 2005) using entity-based similarity
51
- blanc: BLANC (Luo et al., 2014) BLANC metric for gold and predicted mentions
52
- all: uses all the metrics to score
53
-
54
- <key>: file with expected coreference chains in CoNLL-2011/2012 format
55
-
56
- <response>: file with output of coreference system (CoNLL-2011/2012 format)
57
-
58
- <document-id>: optional. The name of the document to score. If name is not
59
- given, all the documents in the dataset will be scored. If given
60
- name is "none" then all the documents are scored but only total
61
- results are shown.
62
-
63
-
64
- OUTPUT
65
- ------
66
-
67
- The score subroutine returns an array with four values in this order:
68
-
69
- Coreference Score
70
- ~~~~~~~~~~~~~~~~~
71
-
72
- Recall = recall_numerator / recall_denominator
73
- Precision = precision_numerator / precision_denominator
74
- F1 = 2 * Recall * Precision / (Recall + Precision)
75
-
76
- Also recall, precision and F1 are printed in the standard output when variable ``$VERBOSE `` is not null.
77
-
78
-
79
- Identification of mentions
80
- ~~~~~~~~~~~~~~~~~~~~~~~~~~
81
-
82
- A score for identification of mentions (recall, precision and F1) is
83
- also included. Mentions from system response are compared with key
84
- mentions. This version performs strict mention matching as was used in
85
- the CoNLL-2011 and 2012 shared tasks.
86
-
87
- AUTHORS
88
- -------
89
-
90
- Emili Sapena, Universitat Politècnica de Catalunya, http://www.lsi.upc.edu/~esapena, esapena <at> lsi.upc.edu
91
- Sameer Pradhan, http://cemantix.org, pradhan <at> cemantix.org
92
- Sebastian Martschat, sebastian.martschat <at> h-its.org
93
- Xiaoqiang Luo, xql <at> google.com
94
-
95
-
96
- LICENSE
97
- -------
98
- This program is free software; you can redistribute it and/or modify it
99
- under the terms of the GNU General Public License as published by the
100
- Free Software Foundation; either version 2 of the License, or (at your
101
- option) any later version. This program is distributed in the hope that
102
- it will be useful, but WITHOUT ANY WARRANTY; without even the implied
103
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
104
- GNU General Public License for more details.
105
-
106
- You should have received a copy of the GNU General Public License along
107
- with this program; if not, write to the Free Software Foundation, Inc.,
108
- 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
109
-
110
- COPYRIGHT
111
- ---------
112
-
113
- 2009-2011, Emili Sapena esapena <at> lsi.upc.edu
114
- 2011- Sameer Pradhan pradhan <at> cemantix.org
115
-
116
- Reference Coreference Scorer
117
- ============================
118
-
119
- Description
120
- -----------
121
-
122
- This is the official implementation of the revised coreference scorer as described in the following paper:
16
+ The current stable (official) version for scoring predicted mentions is **v8.01 **
123
17
124
- Citation
18
+ CITATION
125
19
--------
126
20
127
- We would appreciate if you cite the paper when you use this scorer as some of us are academics or wanting to be academics, and citations matter.
21
+ We would appreciate if you cite the paper when you use this scorer as
22
+ some of us are academics or wanting to be academics, and citations
23
+ matter.
128
24
129
25
::
130
26
@@ -141,12 +37,7 @@ We would appreciate if you cite the paper when you use this scorer as some of us
141
37
}
142
38
143
39
144
- VERSION
145
- -------
146
-
147
- The current stable (official) version for scoring predicted mentions is **v8.01 **
148
-
149
- Usage
40
+ USAGE
150
41
-----
151
42
152
43
::
@@ -190,7 +81,7 @@ Coreference Score
190
81
These values are to standard output when variable ``$VERBOSE `` is not null.
191
82
192
83
193
- Identification of mentions
84
+ Identification of Mentions
194
85
~~~~~~~~~~~~~~~~~~~~~~~~~~
195
86
196
87
A score for identification of mentions (recall, precision and F1) is
0 commit comments