architecture.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>Voikko – General architecture</title>
<link rel="stylesheet" type="text/css" href="stylesheets/styles.css" />
</head>
<body>

<p class="linkback"><a href=".">← Back to Index</a></p>

<h1>Voikko<br /><small>General architecture</small></h1>

<p>Voikko consists of a set of separately released components that form a
stack of layers as illustrated in the picture below.</p>

<p><img src="images/architecture.png" style="border: 1px solid black" alt="General architecture of Voikko" /></p>

<p>In this project we develop the components shown with blue background, and
some of the components with yellow background:</p>
<dl>
  <dt>libvoikko</dt>
  <dd>Libvoikko is the high level library that contains among other things
   algorithms that generate spelling suggestions and perform rule based
   hyphenation. It is also capable of caching the results of common spell
   checking operations to improve performance. All of the grammar checking
   is also done within libvoikko. Libvoikko supports few different dictionary
   formats some of which require dependencies on other libraries such as hfst-ospell.
   The different dictionary formats have distinct version numbers so that it is
   possible to have multiple versions of libvoikko installed on the same computer
   without problems related to dictionary compatibility.</dd>
  <dt>voikko-fi (previously Suomi-malaga)</dt>
  <dd>Voikko-fi is a description of Finnish morphology based on finite state transducer
   technology (Foma and VFST). The package contains morpholgies
   that are suitable for both spell checking and text indexing.</dd>
  <dt>libreoffice-voikko</dt>
  <dd>Libreoffice-voikko is an LibreOffice extension that uses Voikko to provide
   Finnish spell checking, hyphenation and grammar checking.</dd>
  <dt>Enchant Voikko plugin</dt>
  <dd>Voikko provider plugin for multi-backend Enchant spell checker library
   is included in Enchant version 1.4 and later.</dd>
  <dt>tmispell-voikko</dt>
  <dd>Tmispell-voikko is an ispell compatible spell checker that uses Voikko to provide
   Finnish spell checking and falls back to real ispell for other languages. Tmispell-voikko
   was originally written by Pauli Virtanen for the freely distributable but closed source
   spell checker Soikko. Tmispell-voikko also contains an Enchant provider plugin for
   Enchant version 1.3. Tmispell-voikko is deprecated and not actively developed
   anymore. Developers using ispell to add spell checking capability in their
   applications should consider switching to Enchant instead.</dd>
  <dt>Joukahainen</dt>
  <dd>Joukahainen is our web application used to maintain the vocabulary. Joukahainen is
   designed to store and provide vocabulary data in an application independent format which
   should make it easier to use and experiment with the data outside the Voikko project.</dd>
</dl>

<h2>The reasons for using finite state technology instead of Hunspell</h2>
<p>This project started in late 2005 under the name Hunspell-fi, with an aim to create
Finnish vocabulary and affix files for Hunspell. The Hunspell based implementation was developed
roughly six months, and there were no serious problems but it was also evident that the
work progressed rather slowly. In early 2006 Hannu Väisänen published Suomi-malaga, which
contained a vocabulary that was (depending on how one defines "word") roughly ten times
larger than the Hunspell-fi vocabulary at that time. Additionally the Hunspell-fi implementation
did not support compound words and only a few derived word forms, which were both supported
by Suomi-malaga.</p>
<p>Between 2012 and 2015 the morphology was re-written on top of finite state technology
that has become the most widely used tool for computational linguists to model the morphology
of natural languages. The implementation used in Voikko is not a purely finite state solution
but very close to that. From the end user point of view it is very similar to the original
Malaga morphology.</p>
<p>There are major limitations (or at least there have been, some may have been fixed in recent versions) in Hunspell that
make it practically impossible to use as a replacement of for Voikko when processing
Finnish language:</p>
<ul>
<li>Hunspell only supports two consecutive suffixes for any word. This is of course a
major improvement compared to ispell or myspell that supported only one. But in Finnish
it is common to have three or more suffixes in a word. This requires merging of
suffixes and in practice some sort of tool has to be created to generate these merged
forms (apparently this method has been used in Hungarian Hunspell dictionaries, so it
is possible). This makes it is impossible to write Hunspell affix files manually and
adds another layer of complexity to the system. Of course having such tool and storing
morphological rules in an application independent format would help converting to other
formats later, which would be a good thing.</li>
<li>Hunspell has methods for allowing or forbidding certain inflected or derived forms
in compounds. But binary allow/forbid is not enough, we need a method for specifying
compound rules by word class and then specifying that certain derivational suffixes
cause the word to belong to a different class. And this must work recursively
(derived words can be derived too) for at least three levels. Additionally words can
be inflected (in a restricted way) inside compounds too, so this must be supported
as well.</li>
<li>Hunspell seems to need a lot more memory than libvoikko, especially when multiple
instances are used on the same machine. The total running set size for libvoikko and its
dictionary is about 5 megabytes shared + 0.5 megabytes data per running instance.
Hunspell (Hungarian) consumes 1 megabytes shared + 15 megabytes data per instance.
This would still be acceptable for single user systems, but in a classroom LTSP
server with 20 pupils using LibreOffice the total memory use would be
about 15 megabytes with Voikko and 300 megabytes with Hunspell. If they used
Firefox with spell checking extension at the same time, the numbers would become
25 megabytes for Voikko and 600 megabytes for Hunspell. Using precompiled binary
dictionaries and mmap, or a spelling server would fix this problem in Hunspell, but the
server solution would cause additional security and maintenance issues for the
administrator.</li>
<li>Voikko is not just a spell checking library. It currently provides morphological analyzer,
grammar checker, hyphenator and string tokeniser as well. Additionally Voikko has many extra options
related to spell checking not found in Hunspell, such as special suggestion mode
for fixing errors introduced during optical character recognition. Hunspell can be used
as a morphological analyzer and some other functionality from libvoikko could be ported there if needed. But other features
(such as hyphenator and grammar checker) would need to be designed differently to make them
language independent for Hunspell. Totally separating hyphenator and spell checker
(as is currently done in most free tools except Voikko) does
not make sense, since both rely on morphological analysis for their operation.</li>
</ul>
<p>All of the problems above could definitely be solved within Hunspell, but
migh require a lot of work.
Compromising quality just to become compatible with Hunspell is not an option,
because Finnish people have come to expect really good results from their
spell checkers (we have had advanced compound word checking in commercial text
editors for well over ten years).</p>
<p>The latest version of libvoikko is compatible with Hunspell licensing so it would be
possible to merging code between these projects. We are open to consider this but it
should be noted that dictionary formats of libvoikko are still evolving.</p>

<div style="clear:both;" />

</body>
</html>