Extracting Entities and Relationships from Wikipedia Articles using NER and OpenIE

Automatic knowledge extraction from Portuguese Wikipedia articles. Given any article, the system identifies who and what is mentioned, what relationships exist between them, and builds a queryable knowledge graph from the result.

• Language: Portuguese
• Input: any Wikipedia article (or arbitrary Portuguese text)
• Output: typed knowledge graph - entities labeled as PER, LOC, ORG or MISC, connected by extracted relations

Two complementary techniques are combined: NER identifies and classifies named entities, OpenIE extracts relational triples (subject, relation, object). A merge step links triple arguments to their entity types, producing typed triples - e.g. (PER) Salazar -fundou→ (ORG) Estado Novo. Results are validated against DBpedia and explored through an interactive Streamlit app.

Built with spaCy, Linguakit and Gemini.

---

Requirements

• Python 3.11+
• spaCy + pt_core_news_lg model
• Linguakit (Perl, for NER + OpenIE comparison)
• Gemini API key (free tier, for LLM-based OpenIE)
• SPARQLWrapper (for DBpedia validation)
• networkx + pyVis (graph construction + rendering)
• Streamlit (web interface)

Install:

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python -m spacy download pt_core_news_lg

Linguakit (Perl toolkit, for NER + OpenIE comparison):

git clone https://github.com/citiususc/Linguakit.git tools/linguakit

Gemini API key (free at aistudio.google.com):

# create .env in project root (or copy .env.example and edit)
echo "GEMINI_API_KEY=your-key-here" > .env

---

Files

• src/config.py - central settings, file paths, IO helpers, .env loader
• src/fetch.py - Wikipedia PT fetch
• src/preprocess.py - sentence segmentation
• src/ner.py - NER with spaCy and Linguakit
• src/openie_linguakit.py - rule-based OpenIE via Linguakit rel
• src/openie_llm.py - LLM-based OpenIE via Gemini
• src/merge.py - align triple arguments to NER entities -> typed triples
• src/validate.py - DBpedia SPARQL validation
• src/graph.py - networkx + pyVis graph construction
• app/streamlit_app.py - interactive web interface
• run.py - CLI orchestrator (all stages or any subset)

---

How to Run

CLI

# full pipeline (default page: Estado Novo (Portugal))
python run.py all

# different Wikipedia article
python run.py all --page "José Saramago"

# limit sentences (fast testing / save Gemini quota)
python run.py all --page "José Saramago" --limit 20

# custom text instead of Wikipedia
python run.py all --text "Saramago nasceu em Portugal em 1922." --page "Custom"

# choose NER + OpenIE engines
python run.py all --ner spacy                      # spaCy only
python run.py all --ner spacy --openie llm         # spaCy + Gemini (fastest)
python run.py all --ner spacy linguakit            # both NER engines
python run.py all --openie linguakit llm           # both OpenIE engines

# run specific stages only
python run.py fetch preprocess ner
python run.py merge graph
python run.py validate

Parameters:

• --page: Wikipedia article title (default: Estado Novo (Portugal))
• --text: Raw Portuguese text - skips Wikipedia fetch
• --limit: Cap sentences for preprocessing (affects all downstream stages)
• --ner: NER engines - spacy, linguakit, or both (default: both)
• --openie: OpenIE engines - linguakit, llm, or both (default: both)

Web Interface

streamlit run app/streamlit_app.py

Visit http://localhost:8501

Features:

• Fetch a Wikipedia article or paste custom text directly
• Select NER and OpenIE engines
• Set sentence limit
• Filter graph by source, entity type, typed-only edges
• Search nodes and relations
• View triples and entities as tables (with source sentence toggle)
• DBpedia knowledge graph tab (Wikipedia articles only)

---

Results

Example output (José Saramago, 20 sentences, spaCy + Linguakit):

Typed triples: 97
  linguakit   both-typed: 13 (33%)
  llm         both-typed: 24 (41%)

Sample fully-typed triples:
  (PER) Saramago            -assumiu->          (PER) cargo de Ministro das Finanças
  (ORG) o Estado Novo       -encerrou->         (LOC) Portugal
  (MISC) Ensaio sobre Ceg.  -foi dirigido por-> (PER) Fernando Meirelles
  (PER) Marcello Caetano    -pretendeu->        (LOC) rebatizar o regime

DBpedia entity grounding: 34/239 entities matched

---

Pipeline

                                         ┌─────────┐
  ┌────────────┐                    ┌───▶│   NER  │───┐
  │ Text Input │──┐                 │    └─────────┘   │   ┌──────────────────┐    ┌──────────────────────┐
  └────────────┘  ├──▶┌──────────┐──┤      Merge       ├─▶│  Knowledge Graph │──▶ │ Comparison w/DBpedia│
  ┌────────────┐  │   │   Data   │  │    (NER+OpenIE)  │   └──────────────────┘    └──────────────────────┘
  │ Wikipedia  │──┘   │Processing│  │    ┌─────────┐   │
  │ (Fetch)    │      └──────────┘  └───▶│ Open IE│───┘
  └────────────┘                         └─────────┘

Stage	Module	Output	Tools
Fetch	src/fetch.py	data/raw/page.json	Wikipedia-API (pt)
Preprocess	src/preprocess.py	data/processed/sentences.json	spaCy sentencizer
NER	src/ner.py	data/out/entities.json	spaCy + Linguakit tagger -nec
OpenIE	src/openie_linguakit.py	data/out/triples_linguakit.json	Linguakit rel
OpenIE	src/openie_llm.py	data/out/triples_llm.json	Gemini gemini-2.5-flash
Merge	src/merge.py	data/out/typed_triples.json	string matching
Validate	src/validate.py	data/out/validation.json	DBpedia SPARQL
Graph	src/graph.py	data/out/graph.html	networkx + pyVis

---

Project Layout

src/            pipeline modules + config.py
app/            streamlit_app.py
run.py          CLI orchestrator
tools/          linguakit/ (cloned Perl toolkit, git-ignored)
data/           raw/ processed/ out/ (regenerable, git-ignored)
.env            API keys (git-ignored)
requirements.txt

---

Presentation

Detailed explanation and walkthrough: Canva Presentation