'Pre-processing'

Author: André Almada

README.md

@@ -0,0 +1 @@
+"# data-mining-py" 

bag-of-words.py

@@ -0,0 +1,42 @@
+import glob
+import os
+import re
+import pandas as pd
+import nltk
+nltk.download('punkt')
+nltk.download('stopwords')
+from nltk.tokenize import word_tokenize
+from nltk.corpus import stopwords
+from nltk.stem.porter import PorterStemmer
+
+path = './txt/'
+stemmer = PorterStemmer()
+
+def get_class (filename):
+    return filename.split('-')[0].split('\\')[1]
+
+def get_data (filename):
+    with open(filename, 'r', encoding='latin1') as f:
+        data = f.read()
+        data = re.sub('[^A-Za-z]', ' ', data)
+        data = data.lower()
+        data = word_tokenize(data)
+        
+        for token in data:
+            if token in stopwords.words('english'):
+                data.remove(token)
+
+        for i in range(len(data)):
+            data[i] = stemmer.stem(data[i])
+
+        return data
+
+names = [f for f in glob.glob(os.path.join(path, '*.txt'))]
+
+dataset = pd.DataFrame({'journal' : [get_class(f) for f in names], 'data' : [get_data(f) for f in names]})
+
+# Tira caracteres não alfabéticos e deixa o texto inteiro na minúscula  
+#dataset.data = dataset.data.map(lambda x: re.sub('[^A-Za-z]', ' ', x).lower())
+
+#dataset.data = pre_processing (dataset.data)
+

txt/CBR-1010Agr109-120.txt

@@ -0,0 +1,73 @@
+KBS Maintenance as Learning Two-Tiered
+Domain Representation *
+
+Gennady Agre
+
+Institute of Information Technologies - Bulgarian Academy of Sciences
+Acad. G. Bonchev St. Block 29A, 1113 Sofia, Bulgaria
+Email: [email protected]
+
+
+Abstract. The paper deals with the problem of improving problem-solving 
+behavior of traditional KBS in the course of its real operation
+which is a part of the maintenance task. The solution of the problem is
+searched in integration of the KBS with a specially designed case-based
+reasoning module used for correcting solutions produced by the KBS.
+Special attention is paid to the methods of case matching and reconciling 
+conflicts between CBR and RBR. The proposed solution for both
+problems is based on treating the maintenance task as a problem for
+learning two-tiered domain representation. From this view point rules
+form the first domain tier reflecting existing strong patterns in the representation 
+of domain concepts, while the second tier is formed by the
+newly solved cases along with a special domain-dependent procedure for
+case matching. The main ideas of the approach are illustrated by the
+results of some experiments with the experimental system CoRCase.
+References
+
+1.	Aamodt, A.: Knowledge-intensive, integrated approach to problem solving and sustained 
+learning. Ph.D. Dissertation, University of Trondheim (1991).
+2.	Agre, G.: Improvement of KBS Behavior by Using Problem-Solving Experience.
+Ph. Jorrand and V. Sgurev (Eds.), Proc. of the VIth Int. Conference AIMSA �94,
+World Scientific Pubi., Singapore (1994) 257�266.
+3.	Agre, G.: An Approach to Integration of Rule-Based and Case-Based Reasoning.
+Problems of Engineering Cybernetics and Robotics, 42, Bulgarian Academy of Sciences, 
+Sofia (1995) 40�49.
+4.	Barsalou, L., Hale, G.: Components of conceptual representation: from feature lists
+to recursive frames. I. Van Mechelen, J. Hampton, R.S. Michalsi and P. Theuns
+(Eds.) Categories and Concepts - Theoretical Views and Inductive Data Analysis,
+Academic Press (1993) 97�144.
+5.	Bergadano, F., Matwin, S., Michaiski R.S., Zhang, J.: Learning two-tiered description 
+of flexible concepts: the POSEIDON system. Machine Learning 8 (1992) 5�43.
+6.	Biberman, Y.: A Contex Similarity Measure. F. Bergadano and L. De Raedt (Eds.)
+Machine Learning: ECML-94. LNAJ 784, Springer-Verlag (1994) 48�63
+7.	Coenen, F., Bench-Capon, T.: Maintenance of Knowledge-Based Systems: Theory,
+Techniques and Tools, Academic Press (1993).
+8.	Golding A.R., Rosenbloom, P.S.: Improving rule-based systems through case-based
+reasoning. Proceedings of the National Conference on Artificial Intelligence, Anaheim, 
+MIT Press (1991) 22�27.
+9.	Hammond, K.: Case-Based Planning: Viewing Planning as a Memory Task. Academic Press (1989).
+10.	Iordanova l., Giacometti, A., Vila, A., Amy, B., Reymond, F., Abaoub, L., Dahou
+M., Rialle, V.: Shade - A Hybrid System for Diagnosis in Electromyography. Proc.
+of IXth Int. Congress on Electromyography, Jerusalem (1992).
+11.	Kolodner, J.L.: Extending problem solver capabilities through case-based inference.
+Proc. of 4th Workshop on Machine Learning, UC-Irvine, June 22-25 (1987) 167�
+178.
+12.	Markov, Z.: Private communication (1995).
+13.	Matwin, S., Plante, B.: A Deductive-Inductive Method For Theory Revision. R.S.
+Michalski and Gh. Tecuci (Eds.) Proc. of the First Int. Workshop on Multistrategy
+Learning (1991) Harpers Ferry, 160�174.
+14.	Michalski R.S.: Learning flexible concepts: fundamental ideas and a method based
+on two-tiered representation. Y. Kodratoff and R.S. Michalski (Eds.) Machine
+Learning: an Artificial Intelligence Approach 3, San Mateo, CA: Morgan Kaufmann (1990) 63�111.
+15.	Mitchell, T.M., R. Keller, Kedar-Cabelli, S.: Explanation-Based Generalization: A
+unifying view. Machine Learning 1 (1986) 47�80.
+16.	Cost, S., Salzberg, S.: A weighted nearest neighbor algorithm for learning with
+symbolic features. Machine Learning 10(1) (1991) 56�78.
+17.	Stanfill, C., Waltz, D.: Toward memory-based reasoning. Communication of ACM
+29(12) (1986) 1213�1229.
+18.	Zhang, J.: Integrating Symbolic and Subsymbolic Approaches in Learning Flexible
+Concepts. R.S. Michalski and Gh. Tecuci (Eds.) Proc. of the First Int. Workshop
+on Multistrategy Learning, Harpers Ferry (1991) 289�304.
+19.	Zhang, J.: Selecting Typical Instances in Instance-Based Learning. D. Sleeman and
+P. Edwards (Eds.) Machine Learning - Proc. of the Ninth Int. Workshop (ML02),
+San Mateo, CA: Morgan Kaufmann (1992) 470�479

txt/CBR-1010All1-10.txt

@@ -0,0 +1,34 @@
+Integration of Case Based Retrieval with a Relational
+Database System in Aircraft Technical Support
+
+Jonathan RC Allen1, David WR Patterson, Maurice D Mulvenna and John G Hughes
+
+Northern Ireland Knowledge Engineering Laboratory
+Faculty of Informatics, University of Ulster at Jordanstown Newtownabbey
+Northern Ireland, UK, BT37 0QB
+
+
+
+Abstract. Case-Based Reasoning (CBR) is suited to problem solving in
+domains where there are recurring problems. This paper describes the
+development of a CBR system for use in such a domain, the Technical
+Support department of an aircraft manufacturing company. The system uses
+three types of indexing: knowledge-guided induction, inductive indexing
+and nearest neighbour matching. The resultant system integrates case based
+retrieval with a relational database system to provide a rich environment to
+help manage the life cycle of a technical support query. In early tests with
+the system, staff can discern if a new query is a recurring problem and has
+been solved before or if it is a completely new unsolved technical query.
+References
+
+1.	Barletta R. An Introduction to Case-Based Reasoning, Al Expert 8, 1991.
+2.	Harmon, P. Case-Based Reasoning 2, Intelligent Software Strategies, 7(11), 1991.
+3.	Kriegsman, M., & Barletta, R. Building a Case-Based Help Application, IEEE
+Expert, 8(6), 1993.
+4.	Harmon, P. Case-Based Reasoning 3, Intelligent Software Strategies, 8(1), 1992.
+5.	Forsyth, R. Expert Systems: Principles and Case Studies, ed. Forsyth R, Chapman
+and Hall Ltd, (1989).
+6.	Michalski, R., Carbonnel, J., & Mitchell, T. Machine Learning: An Artificial
+Intelligence Approach, Tioga Publishing Corp, Palo alto, CA, 1983.
+7.	Magaldi, R.V., Maitaining Aeroplanes in Time-Constrained Operational Situations
+Using Case-Based Reasoning, EWCBR 1994 Chantilly, France.

txt/CBR-1010Alu121-132.txt

@@ -0,0 +1,87 @@
+A Case-Based Approach for Developing Writing Tools
+Aimed at Non-native English Users
+
+Sandra M. Alu�sio1 and Osvaldo N. Oliveira Jr.2
+
+
+1 Universidade de S�o Paulo
+Departamento de Ci�ncias de Computa��o e Estat�stica
+CP 668, 13560-970, S�o Carlos, SP, BRAZIL
+e-mail:[email protected]
+2 Universidade de S�o Paulo
+Instituto de F�sica de S�o Carlos
+CP 369, 13560-970 S�o Carlos, SP, BRAZIL
+e-mail:[email protected]
+
+
+
+Abstract
+
+	A writing tool has been developed for helping non-native English users to produce
+a first draft of Introductory Sections of scientific papers. A corpus analysis was carried
+out in 54 papers of Experimental Physics which allowed one to identify the schematic
+structure of Introductions and 30 rhetorical strategies generally employed. Each one of
+the Introductions analysed constituted a case. The user chooses from menus features
+related to the rhetorical strategies for each component and gives the intended order for
+his/her Introduction, thus forming the requisition. Using three types of metric, the tool
+recovers the best-match cases that can be later modified in a revision process.
+Preliminary experiments showed that high precision and recall will only be obtained if
+the number of cases in the case base is considerably increased. In the revision process,
+four operations are suggested which consist in modifying/adding/deleting the different
+rhetorical messages that constitute the strategies of the chosen case.
+References
+
+[Born-92] Born, G. A Hipertext -Based Support Aid for Writing Software
+Documentation. In Computers and Writing - State of the Art, P. O�Brian-Holt &
+N.Williams (eds), Kiuwer Academic Publishers, Dordrecht, pp. 266-277, 1992.
+[Buchanan-92] Buchanan, R.A. Textbase Technology: Writing with Reusable Text. In
+Computers and Writing - State of the Art, P. O�Brian-Holt & N.Williams (eds),
+Kluwer Academic Publishers, Dordrecht, pp. 254-265, 1992.
+[Caldeira-92] (Caldeira), Alufsio S.M.; De Oliveira, M.C.F.; Fontana, N.; Nacamatsu,
+C.O. and Oliveira Jr., O.N. Writing tools for non-native users of English.
+Proceedings of the XVIII Latinamerican Informatics Conference, Spain, p. 224-
+231, 1992.
+[Crookes-86] Crookes, G. Towards a Validate Analysis of Scientific Text Structure.
+Applied Linguistics, Vol. 7 No. 1, 1986, pp.57-70.
+[Fontana-93] Fontana, N.; (Caldeira), Alu�sio S.M.; De Oliveira, M.C.F. and Oliveira
+Jr., O.N. Computer Assisted Writing--Aplications to English as a Foreign
+Language. CALL, Volume 6 (2), p. 145-161, 1993.
+[Hovy-90] Hovy, E. Pragmatics and Natural Language Generation. Artificial
+Intelligence 43, p. 153-197, 1990.
+[Huckin-91] Huckin, T.N. and Olsen, L.A. Technical Writing and Professional
+Communication for Nonnative Speakers of English. McGraw-Hill, In. 1991.
+[Jacobs-85] Jacobs, P.S. PHRED: A Generator for Natural Language Interfaces.
+Computational Linguistics 11 (4):2 19-242.
+[Kettler-94] Kettler, B.P.; Hendler, J.A. Andersen, W.A; and Evett M.P. Massively
+Parallel Support for Case-Based Planning. IEEE Expert, pp. 8-14, February 1994.
+[Kitano-90] Kitano, H. Parallel Incremental Sentence Production for a Model of
+Simultaneous Interpretation. In Current Research in Natural Language Generation,
+Dale, R., Mellish, C. and Zock, M. (eds), Academic Press, Boston, 1990, pp. 321-
+351.
+[Kukich-83] Kukich, K. Knowledge-Based Report Generation: A Knowledge
+Engineering Approach to Natural Language Report Generation. PhD Thesis,
+University of Pittsburg, 1983.
+[Maybury-91] Maybury, M.T. Planning Multisentential English Text Using
+Communicative Acts. (PhD Thesis) Tech. R. 239, University of Cambridge, 1991.
+[Oliveira-91] Oliveira, Jr. O.N.; (Caldeira), Alufsio S.M. and Fontana, N. Chusaurus:
+A Writing Tool Resource for Non-Native Users of English, In Proceedings of the
+XI International Conference of The Chilean Computer Science Society, pp. 59-70.
+Also In Computer Science: Research and Applications, R. Baeza-Yates and U.
+Manber (eds), Plenum Press, N.Y. pp. 63-72, 1992.
+[Paice-90] Paice, C.D. Constructing Literature Abstracts by Computer: Techniques
+and Prospects. Information Processing & Management, Vol. 26, No. 1, pp. 171-
+186, 1990.
+[Pautler-94] Pautler, D. Planning and learning in domains providing little feedback.
+AAAI Fall Symposium on Planning & Learning Notes�94.
+[Smadja-91] Smadja, F. Retrieving Collocational Knowledge from Textual Corpora.
+An application: Language Generation. PhD Thesis, Computer Science
+Department, Columbia University, 1991.
+[Swales-90] Swales, J. Genre Analisys - English in academic and research settings.
+Cambridge University Press, 1990.
+[Taylor-91] Taylor, G. and Tingguang, C. Linguistic, Cultural and Subcultural Issues
+in Contrastive Discourse Analysis: Anglo-amarican and Chinese Scientific Texts.
+Applied Linguistics, Vol. 12, No. 3, 1991, pp. 3 19-336.
+[Trimble-85] Trimble, L. English for science and technology: a discourse approach.
+Cambridge University Press, 1985.
+[Weissberg-90] Weissberg R. and Buker, S. Writing up Research - Experimental
+Research Report Writing for Students of English. Prentice Hall Regents, 1990.

txt/CBR-1010Ash133-144.txt

@@ -0,0 +1,58 @@
+Reasoning with Reasons in
+Case-Based Comparisons
+
+Kevin D. Ashley and Bruce M. McLaren
+University of Pittsburgh, Pittsburgh, Pennsylvania 15260
+
+Abstract. In this work, we are interested in how rational decision makers reason
+with and about reasons in a domain, practical ethics, where they appear to reason
+about reasons symbolically in terms of both abstract moral principles and case
+comparisons. The challenge for reasoners, human and artificial, is to use abstract
+knowledge of reasons and principles to inform decisions about the salience of
+similarities and differences among cases while still accounting for a case's or
+problems specific contextual circumstances. TRUTH-TELLER is a program we
+have developed and tested that compares pairs of cases presenting ethical
+dilemmas about whether to tell the truth. The programs methods for reasoning
+about reasons help it to make context sensitive assessments of the salience of
+similarities and differences.
+
+References
+Aleven, V. and Ashley, K.D. (1994). An Instructional Environment for Practicing
+Argumentation Skills; In the Proceedings of AAAI-94, pages 485-492.
+Ashley, K.D. (1990). Modeling Legal Argument: Reasoning with Cases and
+Hypotheticals. MIT Press, Cambridge. Based on PhD. Dissertation.
+Ashley, K.D. and McLaren, B.M. (1994). A CBR Knowledge Representation for
+Practical Ethics; In Proceedings, 2d EWCBR.
+Bareiss, E. R. (1989). Exemplar-Based Knowledge Acquisition - A Unified Approach to
+Concept Representation, Classification, and Learning. Academic Press, San Diego,
+CA, 1989. Based on PhD dissertation, 1988.
+Bok, 5. (1989). Lying: Moral Choice in Public and Private Life. Random House, Inc.
+Vintage Books, New York.
+Branting, K. L. (1991). Building Explanations From Rules and Structured Cases. In the
+Journal of Man-Machine Studies. 34, 797-837.
+Edelson, D. C. (1992). When Should A Cheetah Remind You of a Bat? Reminding in
+Case-Based Teaching. In Proceedings of AAAI-92, 667-672. San Jose, CA.
+Gilligan, C. (1982). In a Different Voice. Harvard University Press.
+Hammond, K. (1989) Case-Based Planning -- Viewing Planning as a Memory Task. San
+Diego, CA: Academic Press.
+Jonsen A. R. and Toulmin 5. (1988). The Abuse of Casuistry: A History of Moral
+Reasoning. University of CA Press, Berkeley.
+Kass, A. M., Leake, D., and Owens, C. C. (1986). Swale: A Program that Explains. In
+Schank, R. C. (ed.), Explanation Patterns: Understanding Mechanically and
+Creatively. Lawrence Erlbaum Associates, Hillsdale, NJ.
+Kolodner, J. (1993) Case-Based Reasoning Morgan Kaufmann Publishers, Inc., San
+Mateo, CA.
+Koton, P. (1988) Using Experience in Learning and Problem Solving. PhD thesis, MIT.
+MacGregor R. and Burstein, M. H. (1991). Using a Description Classifier to Enhance
+Knowledge Representation. In IEEE Expert 6(3), pages 41-46.
+Rissland, E. L. and Skalak, D. B. (1991). CABARET: Rule Interpretation in a Hybrid
+Architecture. In the Journal of Man-Machine Studies. 34, pages 8 39-887.
+Rissland, E. L., Skalak, D. B., and Friedman, M. T. (1993). BankXX: A Program to
+Generate Argument through Case-Based Search. In Fourth International Conference
+on AI and Law, Vrie Universiteit, Amsterdam.
+Strong, C. (1988). Justification in Ethics. In Baruch A. Brody, ed., Moral Theory and
+Moral Judgments in Medical Ethics, pp. 193-211. Kiuwer, Dordrecht.
+Sycara, E. P. (1987). Resolving Adversarial Conflicts: An Approach Integrating Case-Based 
+and Analytic Methods Georgia Inst.Tech., Tech. Rep.GIT-ICS-87/26. Atlanta.
+Veloso, M. V. (1992). Learning by Analogical Reasoning in General Problem Solving.
+PhD thesis, Carnegie Mellon University. Technical Report No. CMU-CS-92-174.