From 4839a21a98b1ee7c85446b70aac1b1f43d80ddc7 Mon Sep 17 00:00:00 2001 From: Richard Zowalla <13417392+rzo1@users.noreply.github.com> Date: Sun, 26 Jan 2025 09:09:10 +0100 Subject: [PATCH] OPENNLP-1521: Add documentation to describe how to re-generate snowball stemmer code (#744) --- README.md | 4 ++ dev/Snowball-Stemmer.md | 82 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+) create mode 100644 dev/Snowball-Stemmer.md diff --git a/README.md b/README.md index e5cc51166..ddd98141a 100644 --- a/README.md +++ b/README.md @@ -114,6 +114,10 @@ After cloning the repository go into the destination directory and run: mvn install ``` +### Additional Developement Information + +- [Building and Integrating Snowball Stemmer for OpenNLP](dev/Snowball-Stemmer.md) + ## Contributing The Apache OpenNLP project is developed by volunteers and is always looking for new contributors to work on all parts of the project. Every contribution is welcome and needed to make it better. A contribution can be anything from a small documentation typo fix to a new component. diff --git a/dev/Snowball-Stemmer.md b/dev/Snowball-Stemmer.md new file mode 100644 index 000000000..701170234 --- /dev/null +++ b/dev/Snowball-Stemmer.md @@ -0,0 +1,82 @@ + + +# Building and Integrating Snowball Stemmer for OpenNLP + +This guide outlines the steps to build the Snowball compiler, generate stemmer classes, and integrate them into OpenNLP. + +--- + +## Prerequisites + +- A Unix-like environment with `make` installed. +- Access to the [Snowball repository](https://github.com/snowballstem/snowball). +- The OpenNLP repository checked out locally. + +--- + +## Procedure + +### 1. Clone and Build the Snowball Compiler + +Clone the Snowball repository and build the compiler using `make`: + +```bash +git clone https://github.com/snowballstem/snowball.git +cd snowball +make +``` + +This will generate the snowball compiler in the root directory of the repository. + +# Run the Snowball Compiler + +Run the Snowball compiler to generate the stemmer code. + +```bash +#!/bin/bash + +# Define an array of languages +languages=("arabic" "catalan" "danish" "dutch" "english" "finnish" "french" "german" "greek" "hungarian" "indonesian" "irish" "italian" "norwegian" "porter" "portuguese" "romanian" "russian" "spanish" "swedish" "turkish") + +# Base paths +snowball_exec_path="../snowball" +output_base="../../../../IdeaProjects/opennlp/opennlp-tools/src/main/java/opennlp/tools/stemmer/snowball" + +# Loop through the languages and execute the command +for lang in "${languages[@]}"; do + "${snowball_exec_path}" "${lang}.sbl" -java -o "${output_base}/${lang}Stemmer" +done +``` + +Usage: +1. Save this script as `generate_stemmers.sh` at the appropriate location. +2. Make it executable with `chmod +x generate_stemmers.sh`. +3. Run it using `./generate_stemmers.sh`. + +# Manually Reformat Code to Match OpenNLP Style + +- Open the generated Java files in your preferred IDE or text editor. +- Reformat the code to match the OpenNLP code style. This may include: +- Adjusting indentation. +- Renaming variables or methods as needed. +- Ensuring proper spacing and alignment. + +# Add License Information + +- Ensure each generated file includes the appropriate license information for both Snowball and OpenNLP. +