-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathvfst-transition.html
190 lines (166 loc) · 11 KB
/
vfst-transition.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>Voikko – Transitioning the Finnish dictionary from Malaga to VFST</title>
<link rel="stylesheet" type="text/css" href="stylesheets/styles.css" />
<style type="text/css">
#perf caption { font-style: italic; }
#perf { border-collapse: collapse; }
#perf td, #perf th { border: 1px solid #555555; }
</style>
</head>
<body>
<p class="linkback"><a href=".">← Back to Index</a></p>
<h1>Voikko<br /><small>Transitioning the Finnish dictionary from Malaga to VFST</small></h1>
<p>
Libvoikko 4.0 and voikko-fi 2.0 were released on 2015-12-14. In these versions the default dictionary
format for Finnish language was changed and other configuration changes have also been made. This
document provides detailed information about the change and its effects on developers,
software distributors and end users.
</p>
<p style="font-size: 110%">
This document may occasionally be updated with corrections or new information.
If you are interested in the latest changes please look at
<a href="https://github.com/voikko/voikko.github.io/commits/master/vfst-transition.html">GitHub change history</a>.
</p>
<h2>History, rationale and goals</h2>
<p>
The first version of Voikko consisting of libvoikko 1.0 and Suomi-malaga 0.7.1 was released
in August 2006 to support spell checking and hyphenation for Finnish language. Grammar checking
was introduced during 2008 (libvoikko 2.0). Slight changes were made in the dictionary format
to add support for morphological analysis during 2009 (libvoikko 2.2).
</p>
<p>
Since those days all of
these features have seen gradual improvements but the limits of Malaga grammar formalism
have already been reached. There are no reasonable ways of making the Malaga interpreter
any faster than it already is and generating Finnish word forms is not possible either
which has made many interesting applications such as inflecting thesaurus impossible.
</p>
<p>
Re-writing the morphology in VFST (Varissuo Finite State Transducer) format was started in 2012.
The work has been done in parallel with maintenance work on Malaga dictionary. Indeed many
useful fixes that have been committed during the past three years to our Malaga dictionary have
resulted from this work. The primary goals for the VFST dictionary format were the following:
</p>
<ul>
<li>The format should be usable for algorithms that generate word forms.</li>
<li>Performance of existing functionality (spell checking, hyphenation and morphological analysis)
should increase.</li>
<li>The tools needed for compiling the dictionary should be easily available and actively maintained
(the Malaga software has not been developed since about 2008).</li>
<li>The dictionary format should be platform independent (the Malaga format depends on platform
endianess).
<li>Regressions in features or memory use are not allowed.</li>
</ul>
<h2>The benefits of VFST dictionary format</h2>
<p>
The initial version of Voikko with VFST dictionaries fulfills the goals mentioned above. Without loss
of functionality it offers the following improvements over Malaga format:</p>
<ul>
<li>The tools needed for building the dictionary are actively maintained. <kbd>voikkovfstc</kbd> is
developed as a part of libvoikko and <kbd>foma</kbd> has had its latest release in 2015. Additionally
it is relatively easy to replace Foma with another Xerox lexc compatible compiler if necessary.</li>
<li>The VFST dictionary format is platform independent. The dictionary data can be directly
mapped into process memory on little endian architectures. For the (rare) cases where this optimisation
is needed on a big endian system <kbd>voikkovfstc</kbd> can also produce big endian optimised dictionaries.</li>
<li>Processing speed using the VFST format is about twice as fast as it was with Malaga format. Additionally
there are some unexplored optimisation opportunities that may lead to additional improvements in future releases.</li>
<li>The size of the dictionary file on disc and memory is roughly half of an equivalent Malaga dictionary.</li>
<li>It is possible to transform the dictionary programmatically using regular expression operations.</li>
<li>The format is suitable for word form generation and some specialised uses in grammar checking.</li>
</ul>
<table id="perf">
<caption>Performance of different dictionary formats with equal coverage just before the release of libvoikko 4.0 and voikko-fi 2.0.
Speed measurements are from spell checking a list of 5 700 000 strings from Finnish Wikipedia using an
old laptop and single CPU core. The default configurations for Malaga and VFST are shown in <strong>bold</strong>.
Smaller numbers are better.</caption>
<tr><th>Dictionary</th><th>Time (min:sec)</th><th>Size (MB)</td></tr>
<tr><td>Malaga (full functionality)</td><td>9:06</td><td>12</td></tr>
<tr><td><strong>Malaga (spelling, grammar and hyphenation)</strong></td><td><strong>7:46</strong></td><td><strong>7.6</strong></td></tr>
<tr><td>VFST (full functionality)</td><td>4:44</td><td>5.8</td></tr>
<tr><td><strong>VFST (spelling, grammar, hyphenation and baseforms)</strong></td><td><strong>3:57</strong></td><td><strong>3.8</strong></td></tr>
<tr><td>VFST (spelling, grammar and hyphenation)</td><td>3:45</td><td>1.5</td></tr>
</table>
<p>Libvoikko 4.0.1 and 4.0.2 contain additional improvements that further improve the speed of VFST backend by more than 20 %.</p>
<h2>Programs using libvoikko do not require any changes</h2>
<p>
Programs that use libvoikko through C, Java, Python or .Net API do not need to be modified to use
the new dictionary format. Spell checking, hyphenation and grammar checker APIs will work as before.
Morphological analysis with VFST will produce all of the attributes that were produced with Malaga.
However the following changes may affect some programs:
</p>
<ul>
<li>The baseline of attributes that are expected to be returned in all dictionary configurations
will include the following attributes that were not part of the baseline Malaga dictionary:
<ul>
<li>TENSE</li>
<li>KYSYMYSLIITE</li>
<li>POSSESSIVE</li>
</ul>
Programs that behave badly when additional unknown attributes are introduced may need to be fixed not
to make such assumptions.
</li>
<li>BASEFORM is now produced in the standard dictionary configuration but may still be omitted in
dictionaries optimised for low memory use.</li>
<li>WORDBASES (and WORDIDS) will return more detailed chain of derivation for some words. For example
WORDBASES for word <em>kanavoitumisen</em> is now <strong>+kanavoi(kanavoida)+tua(+tua)</strong> instead of
<strong>+kanavoitua(kanavoida)</strong>. The syntax of these attributes is still the same as before.</li>
</ul>
<h2>Default build configuration of libvoikko has changed</h2>
<p>
The default build configuration of libvoikko 4.0 has been changed. Those who build libvoikko from source
code may need to adjust their configurations:
</p>
<ul>
<li>Previously only Malaga backend (dictionary format 2) was enabled by default. Now we enable HFST (dictionary format 3)
and Finnish VFST (dictionary format 5) instead.
<ul>
<li>HFST (dictionary format 3) requires the hfstospell library. It can be disabled with <kbd>--disable-hfst</kbd></li>
<li>Finnish VFST (dictionary format 5) can be disabled with <kbd>--disable-vfst</kbd></li>
<li>VFST compiler tool <kbd>voikkovfstc</kbd> is now built by default. It can be disabled with <kbd>--disable-buildtools</kbd></li>
<li>Some VFST related features are still experimental. These are still disabled by default but the switch to enable them has been renamed.
It is now called <kbd>--enable-expvfst</kbd></li>
<li>If you still need support for Malaga (dictionary format 2) please use libvoikko 4.1.1 and compile with <kbd>--enable-malaga</kbd></li>
</ul>
<li>The default dictionary path used as a last resort location to look for dictionary files is no longer set by default.
This choice has been made as such default path was really ever useful on Unix systems. It is still recommended to set a default dictionary
path on systems where it is useful: <kbd>--with-dictionary-path=/usr/lib/voikko</kbd></li>
<li>In the default build configuration it is now possible to use and distribute the libvoikko library under MPL 1.1 / GPL 2+ / LGPL 2.1+ tri-license.
This license offers some additional flexibility over plain "GPL 2 or later" license that was required by the Malaga backend. The new license is
the same as the one used by Hunspell spell checker library. Please note that the Finnish VFST dictionary is still only available under the GPL license.</li>
</ul>
<h2>Suomi-malaga is now voikko-fi</h2>
<p>
The Finnish dictionary source package suomi-malaga has been renamed to voikko-fi. Voikko-fi can be used to
build both Malaga and VFST dictionaries for use with libvoikko. It is now recommended to use VFST format
if possible.
</p>
<ul>
<li>Building the VFST dictionaries requires <a href="https://code.google.com/p/foma/">Foma</a>. For those who prefer building with
Autotools based build system the <a href="https://github.com/hatapitk/foma/commits/autotools2">autotools2 branch of Foma</a> is also available.
You only need the <kbd>foma</kbd> tool, shared libraries and development headers are not needed.</li>
<li>Building VFST dictionaries requires <kbd>voikkovfstc</kbd> tool which is built as a part of libvoikko. Thus you now need to build and
install libvoikko before building voikko-fi (previously the order was not sigificant).</li>
<li>Python 3 has replaced Python 2 as the build scripting tool. Python 3 is needed even if you build only Malaga dictionaries.</li>
<li>Commands to build and install the VFST dictionaries are <kbd>make vvfst</kbd> and <kbd>make vvfst-install</kbd>.</li>
<li>The VFST dictionaries will produce BASEFORM attributes by default for morphological analysis. We recommend that dictionaries
built for general use do not disable this feature. It is possible to make the dictionaries more compact by compiling with option
<kbd>VVFST_BASEFORMS=no</kbd>. You can do this when you know that the baseforms will not be used (embedded dictionaries produced just for
spell checking). Such dictionaries will still provide baseforms but they will be incorrect.</li>
<li>Some rarely used build options have been deprecated by not providing support for them with VFST format. The support may be restored in
future releases if demand for these options still exists. Please see README in the source distribution for detailed information.</li>
</ul>
<h2>Removal of Malaga based dictionary format</h2>
<p>To support the transition to VFST dictionary format libvoikko and voikko-fi supported both dictionary formats for a period of 14 months.
Support for Malaga based dictionary format was removed from the master branch of our Git repository in March 2017. The latest versions
supporting both formats were libvoikko 4.1.1 and voikko-fi 2.1.</p>
<h2>Questions, bug reports and support requests</h2>
<p>
If you have comments or need support related to this transition please send email to <a href="mailto:[email protected]">Harri Pitkänen ([email protected])</a>
or <a href="mail.html">the Voikko mailing lists</a>.
</p>
<div style="clear:both;" />
</body>
</html>