Merge pull request #8 from raszidzie/master

Added new similarity algorithm using Gensim Library

Author: luozhouyang

README.md

@@ -304,6 +304,19 @@ print(fourgram.distance(s1, s2))
 
 ```
 
+## Gensim
+Gensim is billed as a Natural Language Processing package that does ‘Topic Modeling for Humans’. But its practically much more than that.
+
+If you are unfamiliar with topic modeling, it is a technique to extract the underlying topics from large volumes of text. Gensim provides algorithms like LDA and LSI (which we will see later in this post) and the necessary sophistication to build high-quality topic models.
+
+You may argue that topic models and word embedding are available in other packages like scikit, R etc. But the width and scope of facilities to build and evaluate topic models are unparalleled in gensim, plus many more convenient facilities for text processing.
+
+It is a great package for processing texts, working with word vector models (such as Word2Vec, FastText etc) and for building topic models.
+
+Also, another significant advantage with gensim is: it lets you handle large text files without having to load the entire file in memory
+
+Gensim Tutorial – A Complete Beginners Guide: https://www.machinelearningplus.com/nlp/gensim-tutorial/
+
 ## Shingle (n-gram) based algorithms
 A few algorithms work by converting strings into sets of n-grams (sequences of n characters, also sometimes called k-shingles). The similarity or distance between the strings is then the similarity or distance between the sets.
 
@@ -366,8 +379,6 @@ SIFT4 is a general purpose string distance algorithm inspired by JaroWinkler and
 
 **Not implemented yet**
 
-
-
 ## Users
 * [StringSimilarity.NET](https://github.com/feature23/StringSimilarity.NET) a .NET port of java-string-similarity
 

requirements.txt

@@ -0,0 +1,2 @@
+gensim
+nltk

similarity/gensim_similarity.py

@@ -0,0 +1,42 @@
+import gensim
+from nltk.tokenize import word_tokenize
+
+class GensimSimilarity:
+    def __init__(self):
+        self.raw_documents = ["I'm taking the show on the road.",
+            "My socks are a force multiplier.",
+            "I am the barber who cuts everyone's hair who doesn't cut their own.",
+            "Legend has it that the mind is a mad monkey.",
+            "I make my own fun."]
+
+    def getSimilarity(gen):
+        gen_docs = [[w.lower() for w in word_tokenize(text)] 
+                for text in gen.raw_documents]
+        print(gen_docs)
+        dictionary = gensim.corpora.Dictionary(gen_docs)
+        print("Number of words in dictionary:",len(dictionary))
+
+        for i in range(len(dictionary)):
+            print(i, dictionary[i])
+
+        corpus = [dictionary.doc2bow(gen_doc) for gen_doc in gen_docs]
+        print(corpus)
+
+        tf_idf = gensim.models.TfidfModel(corpus)
+        print(tf_idf)
+        s = 0
+        for i in corpus:
+            s += len(i)
+        print(s)
+
+        sims = gensim.similarities.Similarity('workdir/',tf_idf[corpus],num_features=len(dictionary))
+
+        query_doc = [w.lower() for w in word_tokenize("Socks are a force for good.")]
+        print(query_doc)
+        query_doc_bow = dictionary.doc2bow(query_doc)
+        print(query_doc_bow)
+        query_doc_tf_idf = tf_idf[query_doc_bow]
+        print(f'Result: {sims[query_doc_tf_idf]}')
+
+similarity = GensimSimilarity()
+similarity.getSimilarity()