transformers_named_entity_recognition.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transformer network: Named entity recognition\n",
    "\n",
    "We explore an application of the transformer architecture by performing the following tasks:\n",
    "\n",
    "* Use tokenizers and pre-trained models from the HuggingFace Library.\n",
    "* Fine-tune a pre-trained transformer model for named entity recognition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='0'></a>\n",
    "## Packages\n",
    "\n",
    "Run the following cell to load the packages you'll need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "import json\n",
    "import random\n",
    "import logging\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When faced with a large amount of unstructured text data, named entity recognition (NER) can help you detect and classify important information in your dataset. For instance, in the running example \"Jane vists Africa in September\", NER would help you detect \"Jane\", \"Africa\", and \"September\" as named entities and classify them as person, location, and time, respectively. \n",
    "\n",
    "* You will use the Transformer model to process a large dataset of resumes.\n",
    "* You will find and classify relevant information such as the companies the applicant worked at, skills, type of degree, etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 - Dataset cleaning\n",
    "\n",
    "You will optimize a Transformer model on a dataset of resumes. Take a look at how the data you will be working with are structured."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_data = pd.read_json(\"ner.json\", lines=True)\n",
    "df_data = df_data.drop(['extras'], axis=1)\n",
    "df_data['content'] = df_data['content'].str.replace(\"\\n\", \" \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>content</th>\n",
       "      <th>annotation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Abhishek Jha Application Development Associate...</td>\n",
       "      <td>[{'label': ['Skills'], 'points': [{'start': 12...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Afreen Jamadar Active member of IIIT Committee...</td>\n",
       "      <td>[{'label': ['Email Address'], 'points': [{'sta...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Akhil Yadav Polemaina Hyderabad, Telangana - E...</td>\n",
       "      <td>[{'label': ['Skills'], 'points': [{'start': 37...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Alok Khandai Operational Analyst (SQL DBA) Eng...</td>\n",
       "      <td>[{'label': ['Skills'], 'points': [{'start': 80...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Ananya Chavan lecturer - oracle tutorials  Mum...</td>\n",
       "      <td>[{'label': ['Degree'], 'points': [{'start': 20...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             content  \\\n",
       "0  Abhishek Jha Application Development Associate...   \n",
       "1  Afreen Jamadar Active member of IIIT Committee...   \n",
       "2  Akhil Yadav Polemaina Hyderabad, Telangana - E...   \n",
       "3  Alok Khandai Operational Analyst (SQL DBA) Eng...   \n",
       "4  Ananya Chavan lecturer - oracle tutorials  Mum...   \n",
       "\n",
       "                                          annotation  \n",
       "0  [{'label': ['Skills'], 'points': [{'start': 12...  \n",
       "1  [{'label': ['Email Address'], 'points': [{'sta...  \n",
       "2  [{'label': ['Skills'], 'points': [{'start': 37...  \n",
       "3  [{'label': ['Skills'], 'points': [{'start': 80...  \n",
       "4  [{'label': ['Degree'], 'points': [{'start': 20...  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'label': ['Skills'],\n",
       "  'points': [{'start': 1295,\n",
       "    'end': 1621,\n",
       "    'text': '\\n• Programming language: C, C++, Java\\n• Oracle PeopleSoft\\n• Internet Of Things\\n• Machine Learning\\n• Database Management System\\n• Computer Networks\\n• Operating System worked on: Linux, Windows, Mac\\n\\nNon - Technical Skills\\n\\n• Honest and Hard-Working\\n• Tolerant and Flexible to Different Situations\\n• Polite and Calm\\n• Team-Player'}]},\n",
       " {'label': ['Skills'],\n",
       "  'points': [{'start': 993,\n",
       "    'end': 1153,\n",
       "    'text': 'C (Less than 1 year), Database (Less than 1 year), Database Management (Less than 1 year),\\nDatabase Management System (Less than 1 year), Java (Less than 1 year)'}]},\n",
       " {'label': ['College Name'],\n",
       "  'points': [{'start': 939, 'end': 956, 'text': 'Kendriya Vidyalaya'}]},\n",
       " {'label': ['College Name'],\n",
       "  'points': [{'start': 883, 'end': 904, 'text': 'Woodbine modern school'}]},\n",
       " {'label': ['Graduation Year'],\n",
       "  'points': [{'start': 856, 'end': 860, 'text': '2017\\n'}]},\n",
       " {'label': ['College Name'],\n",
       "  'points': [{'start': 771,\n",
       "    'end': 813,\n",
       "    'text': 'B.v.b college of engineering and technology'}]},\n",
       " {'label': ['Designation'],\n",
       "  'points': [{'start': 727,\n",
       "    'end': 769,\n",
       "    'text': 'B.E in Information science and engineering\\n'}]},\n",
       " {'label': ['Companies worked at'],\n",
       "  'points': [{'start': 407, 'end': 415, 'text': 'Accenture'}]},\n",
       " {'label': ['Designation'],\n",
       "  'points': [{'start': 372,\n",
       "    'end': 404,\n",
       "    'text': 'Application Development Associate'}]},\n",
       " {'label': ['Email Address'],\n",
       "  'points': [{'start': 95,\n",
       "    'end': 145,\n",
       "    'text': 'Indeed: indeed.com/r/Abhishek-Jha/10e7a8cb732bc43a\\n'}]},\n",
       " {'label': ['Location'],\n",
       "  'points': [{'start': 60, 'end': 68, 'text': 'Bengaluru'}]},\n",
       " {'label': ['Companies worked at'],\n",
       "  'points': [{'start': 49, 'end': 57, 'text': 'Accenture'}]},\n",
       " {'label': ['Designation'],\n",
       "  'points': [{'start': 13,\n",
       "    'end': 45,\n",
       "    'text': 'Application Development Associate'}]},\n",
       " {'label': ['Name'],\n",
       "  'points': [{'start': 0, 'end': 11, 'text': 'Abhishek Jha'}]}]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_data.iloc[0]['annotation']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mergeIntervals(intervals):\n",
    "    sorted_by_lower_bound = sorted(intervals, key=lambda tup: tup[0])\n",
    "    merged = []\n",
    "\n",
    "    for higher in sorted_by_lower_bound:\n",
    "        if not merged:\n",
    "            merged.append(higher)\n",
    "        else:\n",
    "            lower = merged[-1]\n",
    "            if higher[0] <= lower[1]:\n",
    "                if lower[2] is higher[2]:\n",
    "                    upper_bound = max(lower[1], higher[1])\n",
    "                    merged[-1] = (lower[0], upper_bound, lower[2])\n",
    "                else:\n",
    "                    if lower[1] > higher[1]:\n",
    "                        merged[-1] = lower\n",
    "                    else:\n",
    "                        merged[-1] = (lower[0], higher[1], higher[2])\n",
    "            else:\n",
    "                merged.append(higher)\n",
    "    return merged"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_entities(df):\n",
    "    \n",
    "    entities = []\n",
    "    \n",
    "    for i in range(len(df)):\n",
    "        entity = []\n",
    "    \n",
    "        for annot in df['annotation'][i]:\n",
    "            try:\n",
    "                ent = annot['label'][0]\n",
    "                start = annot['points'][0]['start']\n",
    "                end = annot['points'][0]['end'] + 1\n",
    "                entity.append((start, end, ent))\n",
    "            except:\n",
    "                pass\n",
    "    \n",
    "        entity = mergeIntervals(entity)\n",
    "        entities.append(entity)\n",
    "    \n",
    "    return entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>content</th>\n",
       "      <th>annotation</th>\n",
       "      <th>entities</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Abhishek Jha Application Development Associate...</td>\n",
       "      <td>[{'label': ['Skills'], 'points': [{'start': 12...</td>\n",
       "      <td>[(0, 12, Name), (13, 46, Designation), (49, 58...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Afreen Jamadar Active member of IIIT Committee...</td>\n",
       "      <td>[{'label': ['Email Address'], 'points': [{'sta...</td>\n",
       "      <td>[(0, 14, Name), (62, 68, Location), (104, 148,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Akhil Yadav Polemaina Hyderabad, Telangana - E...</td>\n",
       "      <td>[{'label': ['Skills'], 'points': [{'start': 37...</td>\n",
       "      <td>[(0, 21, Name), (22, 31, Location), (65, 117, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Alok Khandai Operational Analyst (SQL DBA) Eng...</td>\n",
       "      <td>[{'label': ['Skills'], 'points': [{'start': 80...</td>\n",
       "      <td>[(0, 12, Name), (13, 51, Designation), (54, 60...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Ananya Chavan lecturer - oracle tutorials  Mum...</td>\n",
       "      <td>[{'label': ['Degree'], 'points': [{'start': 20...</td>\n",
       "      <td>[(0, 13, Name), (14, 22, Designation), (24, 41...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             content  \\\n",
       "0  Abhishek Jha Application Development Associate...   \n",
       "1  Afreen Jamadar Active member of IIIT Committee...   \n",
       "2  Akhil Yadav Polemaina Hyderabad, Telangana - E...   \n",
       "3  Alok Khandai Operational Analyst (SQL DBA) Eng...   \n",
       "4  Ananya Chavan lecturer - oracle tutorials  Mum...   \n",
       "\n",
       "                                          annotation  \\\n",
       "0  [{'label': ['Skills'], 'points': [{'start': 12...   \n",
       "1  [{'label': ['Email Address'], 'points': [{'sta...   \n",
       "2  [{'label': ['Skills'], 'points': [{'start': 37...   \n",
       "3  [{'label': ['Skills'], 'points': [{'start': 80...   \n",
       "4  [{'label': ['Degree'], 'points': [{'start': 20...   \n",
       "\n",
       "                                            entities  \n",
       "0  [(0, 12, Name), (13, 46, Designation), (49, 58...  \n",
       "1  [(0, 14, Name), (62, 68, Location), (104, 148,...  \n",
       "2  [(0, 21, Name), (22, 31, Location), (65, 117, ...  \n",
       "3  [(0, 12, Name), (13, 51, Designation), (54, 60...  \n",
       "4  [(0, 13, Name), (14, 22, Designation), (24, 41...  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_data['entities'] = get_entities(df_data)\n",
    "df_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_dataturks_to_spacy(dataturks_JSON_FilePath):\n",
    "    try:\n",
    "        training_data = []\n",
    "        lines=[]\n",
    "        with open(dataturks_JSON_FilePath, 'r') as f:\n",
    "            lines = f.readlines()\n",
    "\n",
    "        for line in lines:\n",
    "            data = json.loads(line)\n",
    "            text = data['content'].replace(\"\\n\", \" \")\n",
    "            entities = []\n",
    "            data_annotations = data['annotation']\n",
    "            if data_annotations is not None:\n",
    "                for annotation in data_annotations:\n",
    "                    #only a single point in text annotation.\n",
    "                    point = annotation['points'][0]\n",
    "                    labels = annotation['label']\n",
    "                    # handle both list of labels or a single label.\n",
    "                    if not isinstance(labels, list):\n",
    "                        labels = [labels]\n",
    "\n",
    "                    for label in labels:\n",
    "                        point_start = point['start']\n",
    "                        point_end = point['end']\n",
    "                        point_text = point['text']\n",
    "                        \n",
    "                        lstrip_diff = len(point_text) - len(point_text.lstrip())\n",
    "                        rstrip_diff = len(point_text) - len(point_text.rstrip())\n",
    "                        if lstrip_diff != 0:\n",
    "                            point_start = point_start + lstrip_diff\n",
    "                        if rstrip_diff != 0:\n",
    "                            point_end = point_end - rstrip_diff\n",
    "                        entities.append((point_start, point_end + 1 , label))\n",
    "            training_data.append((text, {\"entities\" : entities}))\n",
    "        return training_data\n",
    "    except Exception as e:\n",
    "        logging.exception(\"Unable to process \" + dataturks_JSON_FilePath + \"\\n\" + \"error = \" + str(e))\n",
    "        return None\n",
    "\n",
    "def trim_entity_spans(data: list) -> list:\n",
    "    \"\"\"Removes leading and trailing white spaces from entity spans.\n",
    "\n",
    "    Args:\n",
    "        data (list): The data to be cleaned in spaCy JSON format.\n",
    "\n",
    "    Returns:\n",
    "        list: The cleaned data.\n",
    "    \"\"\"\n",
    "    invalid_span_tokens = re.compile(r'\\s')\n",
    "\n",
    "    cleaned_data = []\n",
    "    for text, annotations in data:\n",
    "        entities = annotations['entities']\n",
    "        valid_entities = []\n",
    "        for start, end, label in entities:\n",
    "            valid_start = start\n",
    "            valid_end = end\n",
    "            while valid_start < len(text) and invalid_span_tokens.match(\n",
    "                    text[valid_start]):\n",
    "                valid_start += 1\n",
    "            while valid_end > 1 and invalid_span_tokens.match(\n",
    "                    text[valid_end - 1]):\n",
    "                valid_end -= 1\n",
    "            valid_entities.append([valid_start, valid_end, label])\n",
    "        cleaned_data.append([text, {'entities': valid_entities}])\n",
    "    return cleaned_data  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = trim_entity_spans(convert_dataturks_to_spacy(\"ner.json\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm.notebook import tqdm\n",
    "def clean_dataset(data):\n",
    "    cleanedDF = pd.DataFrame(columns=[\"setences_cleaned\"])\n",
    "    sum1 = 0\n",
    "    for i in tqdm(range(len(data))):\n",
    "        start = 0\n",
    "        emptyList = [\"Empty\"] * len(data[i][0].split())\n",
    "        numberOfWords = 0\n",
    "        lenOfString = len(data[i][0])\n",
    "        strData = data[i][0]\n",
    "        strDictData = data[i][1]\n",
    "        lastIndexOfSpace = strData.rfind(' ')\n",
    "        for i in range(lenOfString):\n",
    "            if (strData[i]==\" \" and strData[i+1]!=\" \"):\n",
    "                for k,v in strDictData.items():\n",
    "                    for j in range(len(v)):\n",
    "                        entList = v[len(v)-j-1]\n",
    "                        if (start>=int(entList[0]) and i<=int(entList[1])):\n",
    "                            emptyList[numberOfWords] = entList[2]\n",
    "                            break\n",
    "                        else:\n",
    "                            continue\n",
    "                start = i + 1  \n",
    "                numberOfWords += 1\n",
    "            if (i == lastIndexOfSpace):\n",
    "                for j in range(len(v)):\n",
    "                        entList = v[len(v)-j-1]\n",
    "                        if (lastIndexOfSpace>=int(entList[0]) and lenOfString<=int(entList[1])):\n",
    "                            emptyList[numberOfWords] = entList[2]\n",
    "                            numberOfWords += 1\n",
    "        cleanedDF = cleanedDF.append(pd.Series([emptyList],  index=cleanedDF.columns ), ignore_index=True )\n",
    "        sum1 = sum1 + numberOfWords\n",
    "    return cleanedDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "19db57824d5e4e5ab7f804c4317ef06d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=220.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "cleanedDF = clean_dataset(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at your cleaned dataset and the categories the named entities are matched to, or 'tags'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>setences_cleaned</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[Name, Name, Designation, Designation, Designa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[Name, Name, Empty, Empty, Empty, Empty, Empty...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[Name, Name, Name, Empty, Empty, Empty, Empty,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[Name, Name, Designation, Designation, Designa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[Name, Name, Designation, Empty, Companies wor...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    setences_cleaned\n",
       "0  [Name, Name, Designation, Designation, Designa...\n",
       "1  [Name, Name, Empty, Empty, Empty, Empty, Empty...\n",
       "2  [Name, Name, Name, Empty, Empty, Empty, Empty,...\n",
       "3  [Name, Name, Designation, Designation, Designa...\n",
       "4  [Name, Name, Designation, Empty, Companies wor..."
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleanedDF.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 - Padding and generating tags\n",
    "\n",
    "Now, it is time to generate a list of unique tags you will match the named entities to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "unique_tags = set(cleanedDF['setences_cleaned'].explode().unique())#pd.unique(cleanedDF['setences_cleaned'])#set(tag for doc in cleanedDF['setences_cleaned'].values.tolist() for tag in doc)\n",
    "tag2id = {tag: id for id, tag in enumerate(unique_tags)}\n",
    "id2tag = {id: tag for tag, id in tag2id.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'College Name',\n",
       " 'Companies worked at',\n",
       " 'Degree',\n",
       " 'Designation',\n",
       " 'Email Address',\n",
       " 'Empty',\n",
       " 'Graduation Year',\n",
       " 'Location',\n",
       " 'Name',\n",
       " 'Skills',\n",
       " 'UNKNOWN',\n",
       " 'Years of Experience'}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unique_tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, you will create an array of tags from your cleaned dataset. Often your input sequence will exceed the maximum length of a sequence your network can process. In this case, your sequence will be cut off, and you need to append zeroes onto the end of the shortened sequences using this [Keras padding API](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/sequence/pad_sequences)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing.sequence import pad_sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LEN = 512\n",
    "labels = cleanedDF['setences_cleaned'].values.tolist()\n",
    "\n",
    "tags = pad_sequences([[tag2id.get(l) for l in lab] for lab in labels],\n",
    "                     maxlen=MAX_LEN, value=tag2id[\"Empty\"], padding=\"post\",\n",
    "                     dtype=\"long\", truncating=\"post\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[9, 9, 5, ..., 8, 8, 8],\n",
       "       [9, 9, 8, ..., 8, 8, 8],\n",
       "       [9, 9, 9, ..., 8, 1, 8],\n",
       "       ...,\n",
       "       [9, 9, 5, ..., 8, 8, 8],\n",
       "       [9, 9, 5, ..., 8, 8, 8],\n",
       "       [9, 9, 5, ..., 8, 8, 8]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 - Tokenizing and aligning labels with 🤗 library\n",
    "\n",
    "Before feeding the texts to a Transformer model, you will need to tokenize your input using a [🤗 Transformer tokenizer](https://huggingface.co/transformers/main_classes/tokenizer.html). It is crucial that the tokenizer you use match the Transformer model type you are using. We will use the 🤗 [DistilBERT fast tokenizer](https://huggingface.co/transformers/model_doc/distilbert.html), which standardizes the length of your sequence to 512 and pads with zeros. Notice this matches the maximum length you used when creating tags. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import DistilBertTokenizerFast #, TFDistilBertModel\n",
    "tokenizer = DistilBertTokenizerFast.from_pretrained('tokenizer/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transformer models are often trained by tokenizers that split words into subwords. For instance, the word 'Africa' might get split into multiple subtokens. This can create some misalignment between the list of tags for the dataset and the list of labels generated by the tokenizer, since the tokenizer can split one word into several, or add special tokens. Before processing, it is important that you align the lists of tags and the list of labels generated by the selected tokenizer with a `tokenize_and_align_labels()` function.\n",
    "\n",
    "Implement `tokenize_and_align_labels()`. The function should perform the following:\n",
    "* The tokenizer cuts sequences that exceed the maximum size allowed by your model with the parameter `truncation=True`\n",
    "* Aligns the list of tags and labels with the tokenizer `word_ids` method returns a list that maps the subtokens to the original word in the sentence and special tokens to `None`. \n",
    "* Set the labels of all the special tokens (`None`) to -100 to prevent them from affecting the loss function. \n",
    "* Label of the first subtoken of a word and set the label for the following subtokens to -100. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "label_all_tokens = True\n",
    "def tokenize_and_align_labels(tokenizer, examples, tags):\n",
    "    tokenized_inputs = tokenizer(examples, truncation=True, is_split_into_words=False, padding='max_length', max_length=512)\n",
    "    labels = []\n",
    "    for i, label in enumerate(tags):\n",
    "        word_ids = tokenized_inputs.word_ids(batch_index=i)\n",
    "        previous_word_idx = None\n",
    "        label_ids = []\n",
    "        for word_idx in word_ids:\n",
    "            # Special tokens have a word id that is None. We set the label to -100 so they are automatically\n",
    "            # ignored in the loss function.\n",
    "            if word_idx is None:\n",
    "                label_ids.append(-100)\n",
    "            # We set the label for the first token of each word.\n",
    "            elif word_idx != previous_word_idx:\n",
    "                label_ids.append(label[word_idx])\n",
    "            # For the other tokens in a word, we set the label to either the current label or -100, depending on\n",
    "            # the label_all_tokens flag.\n",
    "            else:\n",
    "                label_ids.append(label[word_idx] if label_all_tokens else -100)\n",
    "            previous_word_idx = word_idx\n",
    "\n",
    "        labels.append(label_ids)\n",
    "\n",
    "    tokenized_inputs[\"labels\"] = labels\n",
    "    return tokenized_inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have tokenized inputs, you can create train and test datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = tokenize_and_align_labels(tokenizer, df_data['content'].values.tolist(), tags)\n",
    "train_dataset = tf.data.Dataset.from_tensor_slices((\n",
    "    test['input_ids'],\n",
    "    test['labels']\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-100,\n",
       " 9,\n",
       " 9,\n",
       " 9,\n",
       " 9,\n",
       " 9,\n",
       " 9,\n",
       " 5,\n",
       " 5,\n",
       " 5,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 2,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 5,\n",
       " 5,\n",
       " 8,\n",
       " 0,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 5,\n",
       " 5,\n",
       " 5,\n",
       " 5,\n",
       " 5,\n",
       " 8,\n",
       " 11,\n",
       " 11,\n",
       " 11,\n",
       " 11,\n",
       " 11,\n",
       " 11,\n",
       " 11,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 11,\n",
       " 11,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 11,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " 8,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100,\n",
       " -100]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test['labels'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4 - Optimization\n",
    "\n",
    "Now you can finally feed your data into a pretrained 🤗 model. You will optimize a DistilBERT model, which matches the tokenizer you used to preprocess your data. Try playing around with the different hyperparamters to improve your results.\n",
    "\n",
    "Before running the cell below, combine the model parts on your local computer using the Linux/macOS command `cat tf_model.h5.* > tf_model.h5` in the `model` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "All model checkpoint layers were used when initializing TFDistilBertForTokenClassification.\n",
      "\n",
      "All the layers of TFDistilBertForTokenClassification were initialized from the model checkpoint at model/.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertForTokenClassification for predictions without further training.\n"
     ]
    }
   ],
   "source": [
    "from transformers import TFDistilBertForTokenClassification\n",
    "\n",
    "model = TFDistilBertForTokenClassification.from_pretrained('model/', num_labels=len(unique_tags))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/3\n",
      "WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).WARNING:tensorflow:AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7feae4152ee8>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: <cyfunction Socket.send at 0x7feae73d6430> is not a module, class, method, function, traceback, frame, or code object\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
      "WARNING: AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7feae4152ee8>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: <cyfunction Socket.send at 0x7feae73d6430> is not a module, class, method, function, traceback, frame, or code object\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
      "\n",
      "WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n",
      "WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n",
      "WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n",
      "14/14 [==============================] - 4s 303ms/step - loss: 1.6655 - accuracy: 0.5826\n",
      "Epoch 2/3\n",
      "14/14 [==============================] - 4s 304ms/step - loss: 0.5930 - accuracy: 0.7537\n",
      "Epoch 3/3\n",
      "14/14 [==============================] - 4s 303ms/step - loss: 0.5240 - accuracy: 0.7537\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fe980475198>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)\n",
    "model.compile(optimizer=optimizer, loss=model.compute_loss, metrics=['accuracy']) # can also use any keras loss fn\n",
    "model.fit(train_dataset.shuffle(1000).batch(16),\n",
    "          epochs=3, \n",
    "          batch_size=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Things to remember\n",
    "\n",
    "- Named entity recognition (NER) detects and classifies named entities, and can help process resumes, customer reviews, browsing histories, etc. \n",
    "- You must preprocess text data with the corresponding tokenizer to the pretrained model before feeding your input into your Transformer model"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}