v0.4.0

Main changes

• Handle zstd-compressed dictionaries in all CLIs #112

Precompiled dictionary files

We provide precompiled dictionaries for Vibrato, allowing you to get started with tokenization easily. You can download them from Assets in this release.

The following variants are distributed:

• ipadic-mecab-2_7_0/system.dic.zst from IPADIC v2.7.0
• ipadic-mecab-2_7_0-small/system.dic.zst from IPADIC v2.7.0
  • A smaller version that contains only the features 品詞-品詞細分類1 and 発音.
• jumandic-mecab-7_0/system.dic.zst from mecab-jumandic-utf8 v7.0
• naist-jdic-mecab-0_6_3b/system.dic.zst from NAIST Japanese Dictionary v0.6.3b
• unidic-mecab-2_1_2/system.dic.zst from UniDic v2.1.2
• unidic-cwj-3_1_1/system.dic.zst from UniDic v3.1.1

These system dictionaries were compiled and modified in the manners described in compile.md and map.md. We trained the mappings of connection ids using license-expired data obtained from Aozora Bunko, following the guideline.

The licenses are contained in each file.