added more readme

Author: huseinzol05

README.md

@@ -1,6 +1,6 @@
 <p align="center">
     <a href="#readme">
-        <img alt="logo" width="50%" src="wordcloud.png">
+        <img alt="logo" width="50%" src="malay-dataset.png">
     </a>
 </p>
 <p align="center">
@@ -802,7 +802,7 @@ Total size: 1.4 MB
 1. Positive
 2. Negative
 
-Reference: https://www.nltk.org/api/nltk.corpus.reader.html#module-nltk.corpus.reader.categorized_sents
+Reference: http://www.cs.cornell.edu/people/pabo/movie-review-data/
 
 <img src="https://img.shields.io/badge/third--party-red.svg">
 
@@ -902,16 +902,6 @@ Total size: 482.4 MB
 
 <img src="https://img.shields.io/badge/third--party-red.svg">
 
-#### [Klook](crawl/klook)
-
-**The copyright data remains with the original owners of the data, do not use this data for commercial purpose, https://www.klook.com/policy/**
-
-Crawled up to 200 interesting locations from MY and SG klook.
-
-Total size: 10.3 MB
-
-<img src="https://img.shields.io/badge/third--party-red.svg">
-
 #### [IIUM-Confession](crawl/iium-confession)
 
 **The copyright data remains with the original owners of the data, do not use this data for commercial purpose.**
@@ -922,23 +912,23 @@ Total size: 75.1 MB
 
 <img src="https://img.shields.io/badge/third--party-red.svg">
 
-#### [Wattpad](crawl/wattpad)
+#### [Iproperty](crawl/iproperty)
 
-**The copyright data remains with the original owners of the data, do not use this data for commercial purpose, https://support.wattpad.com/hc/en-us/articles/216192503-Copyright-FAQ**
+**The copyright data remains with the original owners of the data, do not use this data for commercial purpose, https://www.iproperty.com.my/terms-of-use/**
 
-Crawled using keywords,
+crawled up to 16 states on sales residential, sales commercial, rent residential, rent commercial.
 
-1. melayu
-2. malaysia
-3. seram
-4. hantu
-5. puisi
-6. sajak
-7. cerita
+Total size: 1329 MB
 
-Crawled up to 7k fiction stories.
+<img src="https://img.shields.io/badge/third--party-red.svg">
 
-Total size: 97 MB
+#### [Klook](crawl/klook)
+
+**The copyright data remains with the original owners of the data, do not use this data for commercial purpose, https://www.klook.com/policy/**
+
+Crawled up to 200 interesting locations from MY and SG klook.
+
+Total size: 10.3 MB
 
 <img src="https://img.shields.io/badge/third--party-red.svg">
 
@@ -954,61 +944,27 @@ Total size: 7.9 GB
 
 **The copyright data remains with the original owners of the data, do not use this data for commercial purpose, https://www.ticket2u.com.my/copyright**
 
-Contains 4282 events in Malaysia from 2017,
-
-```python
-{'row': {'rownum': '4282',
-  'rowtotal': '4282',
-  'rowpp': '18',
-  'link': 'https://www.ticket2u.com.my/event/10223/emi-business-networking-3.0',
-  'time': '4:00PM',
-  'avatar': 'https://www.ticket2u.com.my/upload/event/listing/0-10223-8ce30523-200c-4bfa-98a9-daadd142989b-GYQ6_X.jpg',
-  'datefrom106': '26 Oct 2017',
-  'dateto106': '26 Oct 2017',
-  'day': 'Thursday',
-  'date': '26',
-  'month': 'Oct',
-  'year': '2017',
-  'datefrom': '2017-10-26T16:00:00',
-  'dateto': '2017-10-26T19:00:00',
-  'active': '1',
-  'id': '10223',
-  'name': 'EMI Business Networking 3.0',
-  'titlename': 'EMI Business Networking 3.0',
-  'excerpt': '',
-  'pid': '0',
-  'basecurrency': 'RM',
-  'online': '0',
-  'countryid': '1',
-  'stateid': '1',
-  'areaid': '0',
-  'locname': 'Denai Alam Recreational and Riding Club',
-  'statename': 'WP Kuala Lumpur',
-  'latitude': '3.150970999999999',
-  'type': '619',
-  'regboo': '0',
-  'pricefrom': '75.00',
-  'longitude': '101.51955099999998',
-  'eventcat': 'Business Sharing and Networking Event',
-  'eventcatcode': 'business',
-  'eventsubcat': 'Networking',
-  'eventsubcatcode': 'networking',
-  'showdate': '1',
-  'exclusive': '0',
-  'notexclusive': '0',
-  'issaleend': '1',
-  'status': 'expired'}}
-```
+Contains 4282 events in Malaysia from 2017.
 
 <img src="https://img.shields.io/badge/third--party-red.svg">
 
-#### [Iproperty](crawl/iproperty)
+#### [Wattpad](crawl/wattpad)
 
-**The copyright data remains with the original owners of the data, do not use this data for commercial purpose, https://www.iproperty.com.my/terms-of-use/**
+**The copyright data remains with the original owners of the data, do not use this data for commercial purpose, https://support.wattpad.com/hc/en-us/articles/216192503-Copyright-FAQ**
 
-crawled up to 16 states on sales residential, sales commercial, rent residential, rent commercial.
+Crawled using keywords,
 
-Total size: 1329 MB
+1. melayu
+2. malaysia
+3. seram
+4. hantu
+5. puisi
+6. sajak
+7. cerita
+
+Crawled up to 7k fiction stories.
+
+Total size: 97 MB
 
 <img src="https://img.shields.io/badge/third--party-red.svg">
 
@@ -2811,7 +2767,7 @@ Total size: 1.7 GB
 
 1. Please citate the repository if use these corpus.
 
-```
+```bibtex
 @misc{Malay-Dataset, We gather Bahasa Malaysia corpus!,
   author = {Husein, Zolkepli},
   title = {Malay-Dataset},

alignment/README.md

@@ -1,3 +1,14 @@
+# Alignment
+
+Purpose of this dataset to do word alignment, example,
+
+```text
+a black cat
+kucing hitam
+
+-> 1-1 2-0
+```
+
 ## how-to
 
 Or you can skip the steps and download the priors.

chatbot/blended-skill-talk/README.md

@@ -1,3 +1,30 @@
-## how-to
+# Blended Skill Talk
 
-1. https://f000.backblazeb2.com/file/malay-dataset/chatbot/blended-skill-talk/blended_skill_talk.json.translate
+Original paper, https://arxiv.org/abs/2004.08449
+
+## download
+
+1. blended_skill_talk.json.translate, https://f000.backblazeb2.com/file/malay-dataset/chatbot/blended-skill-talk/blended_skill_talk.json.translate
+
+## Citation
+
+```bibtex
+@article{DBLP:journals/corr/abs-2004-08449,
+  author    = {Eric Michael Smith and
+               Mary Williamson and
+               Kurt Shuster and
+               Jason Weston and
+               Y{-}Lan Boureau},
+  title     = {Can You Put it All Together: Evaluating Conversational Agents' Ability
+               to Blend Skills},
+  journal   = {CoRR},
+  volume    = {abs/2004.08449},
+  year      = {2020},
+  url       = {https://arxiv.org/abs/2004.08449},
+  archivePrefix = {arXiv},
+  eprint    = {2004.08449},
+  timestamp = {Sat, 23 Jan 2021 01:20:50 +0100},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-2004-08449.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```

chatbot/convai2/README.md

@@ -1,4 +1,46 @@
-## how-to
+# ConvAI2
 
-1. https://f000.backblazeb2.com/file/malay-dataset/chatbot/convai2/convai2-0.json.translate
-2. https://f000.backblazeb2.com/file/malay-dataset/chatbot/convai2/convai2-100000.json.translate
+Original website, https://parl.ai/projects/convai2/
+
+Original paper, https://arxiv.org/abs/1902.00098?
+
+## download
+
+1. convai2-0.json.translate, https://f000.backblazeb2.com/file/malay-dataset/chatbot/convai2/convai2-0.json.translate
+2. convai2-100000.json.translate, https://f000.backblazeb2.com/file/malay-dataset/chatbot/convai2/convai2-100000.json.translate
+
+## Citation
+
+```bibtex
+@article{DBLP:journals/corr/abs-1902-00098,
+  author    = {Emily Dinan and
+               Varvara Logacheva and
+               Valentin Malykh and
+               Alexander H. Miller and
+               Kurt Shuster and
+               Jack Urbanek and
+               Douwe Kiela and
+               Arthur Szlam and
+               Iulian Serban and
+               Ryan Lowe and
+               Shrimai Prabhumoye and
+               Alan W. Black and
+               Alexander I. Rudnicky and
+               Jason Williams and
+               Joelle Pineau and
+               Mikhail S. Burtsev and
+               Jason Weston},
+  title     = {The Second Conversational Intelligence Challenge (ConvAI2)},
+  journal   = {CoRR},
+  volume    = {abs/1902.00098},
+  year      = {2019},
+  url       = {http://arxiv.org/abs/1902.00098},
+  archivePrefix = {arXiv},
+  eprint    = {1902.00098},
+  timestamp = {Sat, 23 Jan 2021 01:11:58 +0100},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-1902-00098.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+a service of Schloss Dagstuhl - Leibniz Center for Informatics	homebrowsesearchabout
+
+```

chatbot/wiki-wizard/README.md

@@ -1,17 +1,38 @@
-# dialog
+# Wizard of Wikipedia
 
-prefix, https://f000.backblazeb2.com/file/malay-dataset/
+Original paper, https://arxiv.org/abs/1811.01241
 
-## how-to
+## download
 
-1. wiki-wizard/dialogs.translate
+### dialog
 
-# information
+1. dialogs.translate, https://f000.backblazeb2.com/file/malay-dataset/wiki-wizard/dialogs.translate
 
-prefix, https://f000.backblazeb2.com/file/malay-dataset/
+### information
 
-## how-to
+1. informations-0.json.translate, https://f000.backblazeb2.com/file/malay-dataset/wiki-wizard/informations-0.json.translate
+2. informations-100000.json.translate, https://f000.backblazeb2.com/file/malay-dataset/wiki-wizard/informations-100000.json.translate
+3. informations-200000.json.translate, https://f000.backblazeb2.com/file/malay-dataset/wiki-wizard/informations-200000.json.translate
 
-1. wiki-wizard/informations-0.json.translate
-1. wiki-wizard/informations-100000.json.translate
-1. wiki-wizard/informations-200000.json.translate
+## Citation
+
+```bibtex
+@article{DBLP:journals/corr/abs-1811-01241,
+  author    = {Emily Dinan and
+               Stephen Roller and
+               Kurt Shuster and
+               Angela Fan and
+               Michael Auli and
+               Jason Weston},
+  title     = {Wizard of Wikipedia: Knowledge-Powered Conversational agents},
+  journal   = {CoRR},
+  volume    = {abs/1811.01241},
+  year      = {2018},
+  url       = {http://arxiv.org/abs/1811.01241},
+  archivePrefix = {arXiv},
+  eprint    = {1811.01241},
+  timestamp = {Sat, 23 Jan 2021 01:19:39 +0100},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-1811-01241.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```

corpus/audience/README.md

@@ -0,0 +1,11 @@
+# Audience
+
+Original website, https://www.kaggle.com/crowdflower/political-social-media-posts
+
+## Citation
+
+Auto generated using https://www.bibme.org/bibtex/website-citation,
+
+```bibtex
+@misc{eight_2016, title={Political Social Media Posts}, url={https://www.kaggle.com/crowdflower/political-social-media-posts}, journal={Kaggle}, author={Eight, Figure}, year={2016}, month={Nov}}
+```

corpus/emotion/README.md

@@ -0,0 +1,29 @@
+# Emotion
+
+Gathered emotion dataset using lexicon, all steps in [notebook].
+
+## download
+
+1. download dataset from here, 27.4 MB, https://malaya-dataset.s3-ap-southeast-1.amazonaws.com/emotion/emotion-twitter-lexicon.json
+
+```
+anger 108813
+fear 20316
+happy 30962
+love 20783
+sadness 26468
+surprise 13107
+```
+
+## Citation
+
+```bibtex
+@misc{Malay-Dataset, We gather Bahasa Malaysia corpus!, Semi-Supervised Emotion dataset,
+  author = {Husein, Zolkepli},
+  title = {Malay-Dataset},
+  year = {2018},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/huseinzol05/malay-dataset/tree/master/corpus/emotion}}
+}
+```

corpus/emotion/lexicon/README.md

@@ -0,0 +1,11 @@
+# Gender
+
+Original website, https://www.kaggle.com/crowdflower/twitter-user-gender-classification
+
+## Citation
+
+Auto generated using https://www.bibme.org/bibtex/website-citation,
+
+```bibtex
+@misc{eight_2016, title={Twitter User Gender Classification}, url={https://www.kaggle.com/crowdflower/twitter-user-gender-classification}, journal={Kaggle}, author={Eight, Figure}, year={2016}, month={Nov}}
+```

corpus/emotion/lexicon/1.populate-emotion-lexicon.ipynb renamed to corpus/emotion/notebook/1.populate-emotion-lexicon.ipynb

@@ -0,0 +1,11 @@
+# Insincere Question
+
+Original website, https://www.kaggle.com/c/quora-insincere-questions-classification
+
+## Citation
+
+Auto generated using https://www.bibme.org/bibtex/website-citation,
+
+```bibtex
+@misc{kaggle, title={Quora Insincere Questions Classification}, url={https://www.kaggle.com/c/quora-insincere-questions-classification}, journal={Kaggle}}
+```