07_3_Creating_an_NLP_Data_Loader__M.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network\" target=\"_blank\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\">\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating an NLP Data Loader \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **60** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an AI engineer working on a cutting-edge language translation project, you are tasked with bridging the communication gap between speakers of different languages. Translating languages is no small feat, especially given the intricacies, nuances, and cultural contexts embedded within them. Central to the success of this endeavor is the data - large corpora of bilingual sentences that serve as the bedrock of your models.\n",
    "\n",
    "![Sample Image](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-AI0205EN-SkillsNetwork/Screenshot%202023-10-24%20at%209.54.36%E2%80%AFAM.png)\n",
    "In PyTorch, the data loader plays an indispensable role in managing this vast amount of data. For natural language processing (NLP) tasks like yours, data often comes in variable lengths due to differing sentence structures and lengths across languages. The data loader efficiently batches these variable-length sequences, ensuring that your models are trained on diverse examples in every iteration. This batching is crucial for harnessing the power of parallel computation on GPUs, thus expediting the training process.\n",
    "\n",
    "Furthermore, the data loader aids in shuffling the data set, which is vital for preventing models from memorizing the sequence of training data and promoting better generalization. Especially for NLP tasks, where data might be ordered or clustered by topics, shuffling ensures that the model remains robust and doesn't develop biases based on the order of input.\n",
    "\n",
    "Lastly, in the world of NLP, preprocessing steps such as tokenization, padding, and numericalization are paramount. The data loader in PyTorch provides hooks that allow us to seamlessly integrate these preprocessing steps, ensuring that the raw textual data is transformed into a format that's amenable for deep learning models.\n",
    "In PyTorch, the data loader plays an indispensable role in managing this vast amount of data.\n",
    "\n",
    "In this lab, you will cover the whole process of loading and collating text data using PyTorch.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# __Table of Contents__\n",
    "\n",
    "<ol>\n",
    "    <li>\n",
    "        <a href=\"#Setup\">Setup</a>\n",
    "        <ol>\n",
    "            <li><a href=\"#Installing-required-libraries\">Installing required libraries</a></li>\n",
    "            <li><a href=\"#Importing-required-libraries\">Importing required libraries</a></li>\n",
    "        </ol>\n",
    "    </li>\n",
    "    <li>\n",
    "        <a href=\"#Data-set\">Data set</a>\n",
    "        <ol>\n",
    "            <li><a href=\"#Data-loader\">Data loader</a></li>\n",
    "        </ol>\n",
    "    </li>\n",
    "    <li><a href=\"#Custom-data-set-and-data-loader-in-PyTorch\">Custom data set and data loader in PyTorch</a>\n",
    "        <ol>\n",
    "            <li><a href=\"#Creating-tensors-for-custom-data-set\">Creating tensors for custom data set</a></li>\n",
    "            <li><a href=\"#Custom-collate-function\">Custom collate function</a></li>\n",
    "        </ol>\n",
    "    </li>\n",
    "    <li><a href=\"#Exercise\">Exercise</a>\n",
    "    </li>\n",
    "    <li><a href=\"#[Optional]-Data-loader-for-German-English-translation-task\">[Optional] Data loader for German-English translation task</a>\n",
    "         <ol>\n",
    "            <li><a href=\"#Translation-data-set\">Translation data set</a></li>\n",
    "            <li><a href=\"#Tokenizer-setup\">Tokenizer setup</a></li>\n",
    "            <li><a href=\"#Special-symbols\">Special symbols</a></li>\n",
    "            <li><a href=\"#Tokens-to-indices-transformation-(Vocab)\">Tokens to indices transformation (Vocab)</a></li>\n",
    "        </ol>\n",
    "    </li>\n",
    "    <li><a href=\"#Processing-data-in-batches\">Processing data in batches</a></li>\n",
    "</ol>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup\n",
    "## Installing required libraries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: nltk in /opt/conda/lib/python3.11/site-packages (3.9.1)\n",
      "Requirement already satisfied: click in /opt/conda/lib/python3.11/site-packages (from nltk) (8.1.7)\n",
      "Requirement already satisfied: joblib in /opt/conda/lib/python3.11/site-packages (from nltk) (1.4.2)\n",
      "Requirement already satisfied: regex>=2021.8.3 in /opt/conda/lib/python3.11/site-packages (from nltk) (2024.11.6)\n",
      "Requirement already satisfied: tqdm in /opt/conda/lib/python3.11/site-packages (from nltk) (4.66.4)\n",
      "Requirement already satisfied: transformers in /opt/conda/lib/python3.11/site-packages (4.46.3)\n",
      "Requirement already satisfied: filelock in /opt/conda/lib/python3.11/site-packages (from transformers) (3.16.1)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /opt/conda/lib/python3.11/site-packages (from transformers) (0.26.3)\n",
      "Requirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.11/site-packages (from transformers) (1.24.0)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.11/site-packages (from transformers) (24.0)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.11/site-packages (from transformers) (6.0.1)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.11/site-packages (from transformers) (2024.11.6)\n",
      "Requirement already satisfied: requests in /opt/conda/lib/python3.11/site-packages (from transformers) (2.31.0)\n",
      "Requirement already satisfied: tokenizers<0.21,>=0.20 in /opt/conda/lib/python3.11/site-packages (from transformers) (0.20.3)\n",
      "Requirement already satisfied: safetensors>=0.4.1 in /opt/conda/lib/python3.11/site-packages (from transformers) (0.4.5)\n",
      "Requirement already satisfied: tqdm>=4.27 in /opt/conda/lib/python3.11/site-packages (from transformers) (4.66.4)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in /opt/conda/lib/python3.11/site-packages (from huggingface-hub<1.0,>=0.23.2->transformers) (2024.10.0)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/lib/python3.11/site-packages (from huggingface-hub<1.0,>=0.23.2->transformers) (4.12.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.11/site-packages (from requests->transformers) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.11/site-packages (from requests->transformers) (3.7)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.11/site-packages (from requests->transformers) (2.2.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.11/site-packages (from requests->transformers) (2024.8.30)\n",
      "Requirement already satisfied: sentencepiece in /opt/conda/lib/python3.11/site-packages (0.2.0)\n",
      "Requirement already satisfied: spacy in /opt/conda/lib/python3.11/site-packages (3.8.2)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /opt/conda/lib/python3.11/site-packages (from spacy) (3.0.12)\n",
      "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /opt/conda/lib/python3.11/site-packages (from spacy) (1.0.5)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /opt/conda/lib/python3.11/site-packages (from spacy) (1.0.11)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /opt/conda/lib/python3.11/site-packages (from spacy) (2.0.10)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /opt/conda/lib/python3.11/site-packages (from spacy) (3.0.9)\n",
      "Requirement already satisfied: thinc<8.4.0,>=8.3.0 in /opt/conda/lib/python3.11/site-packages (from spacy) (8.3.2)\n",
      "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /opt/conda/lib/python3.11/site-packages (from spacy) (1.1.3)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /opt/conda/lib/python3.11/site-packages (from spacy) (2.4.8)\n",
      "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /opt/conda/lib/python3.11/site-packages (from spacy) (2.0.10)\n",
      "Requirement already satisfied: weasel<0.5.0,>=0.1.0 in /opt/conda/lib/python3.11/site-packages (from spacy) (0.4.1)\n",
      "Requirement already satisfied: typer<1.0.0,>=0.3.0 in /opt/conda/lib/python3.11/site-packages (from spacy) (0.14.0)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /opt/conda/lib/python3.11/site-packages (from spacy) (4.66.4)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /opt/conda/lib/python3.11/site-packages (from spacy) (2.31.0)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /opt/conda/lib/python3.11/site-packages (from spacy) (2.10.0)\n",
      "Requirement already satisfied: jinja2 in /opt/conda/lib/python3.11/site-packages (from spacy) (3.1.3)\n",
      "Requirement already satisfied: setuptools in /opt/conda/lib/python3.11/site-packages (from spacy) (69.5.1)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.11/site-packages (from spacy) (24.0)\n",
      "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /opt/conda/lib/python3.11/site-packages (from spacy) (3.5.0)\n",
      "Requirement already satisfied: numpy>=1.19.0 in /opt/conda/lib/python3.11/site-packages (from spacy) (1.24.0)\n",
      "Requirement already satisfied: language-data>=1.2 in /opt/conda/lib/python3.11/site-packages (from langcodes<4.0.0,>=3.2.0->spacy) (1.3.0)\n",
      "Requirement already satisfied: annotated-types>=0.6.0 in /opt/conda/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy) (0.7.0)\n",
      "Requirement already satisfied: pydantic-core==2.27.0 in /opt/conda/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy) (2.27.0)\n",
      "Requirement already satisfied: typing-extensions>=4.12.2 in /opt/conda/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy) (4.12.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy) (3.7)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy) (2.2.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy) (2024.8.30)\n",
      "Requirement already satisfied: blis<1.1.0,>=1.0.0 in /opt/conda/lib/python3.11/site-packages (from thinc<8.4.0,>=8.3.0->spacy) (1.0.1)\n",
      "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /opt/conda/lib/python3.11/site-packages (from thinc<8.4.0,>=8.3.0->spacy) (0.1.5)\n",
      "Collecting numpy>=1.19.0 (from spacy)\n",
      "  Using cached numpy-2.0.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)\n",
      "Requirement already satisfied: click>=8.0.0 in /opt/conda/lib/python3.11/site-packages (from typer<1.0.0,>=0.3.0->spacy) (8.1.7)\n",
      "Requirement already satisfied: shellingham>=1.3.0 in /opt/conda/lib/python3.11/site-packages (from typer<1.0.0,>=0.3.0->spacy) (1.5.4)\n",
      "Requirement already satisfied: rich>=10.11.0 in /opt/conda/lib/python3.11/site-packages (from typer<1.0.0,>=0.3.0->spacy) (13.9.4)\n",
      "Requirement already satisfied: cloudpathlib<1.0.0,>=0.7.0 in /opt/conda/lib/python3.11/site-packages (from weasel<0.5.0,>=0.1.0->spacy) (0.20.0)\n",
      "Requirement already satisfied: smart-open<8.0.0,>=5.2.1 in /opt/conda/lib/python3.11/site-packages (from weasel<0.5.0,>=0.1.0->spacy) (7.0.5)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.11/site-packages (from jinja2->spacy) (2.1.5)\n",
      "Requirement already satisfied: marisa-trie>=1.1.0 in /opt/conda/lib/python3.11/site-packages (from language-data>=1.2->langcodes<4.0.0,>=3.2.0->spacy) (1.2.1)\n",
      "Requirement already satisfied: markdown-it-py>=2.2.0 in /opt/conda/lib/python3.11/site-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy) (3.0.0)\n",
      "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /opt/conda/lib/python3.11/site-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy) (2.18.0)\n",
      "Requirement already satisfied: wrapt in /opt/conda/lib/python3.11/site-packages (from smart-open<8.0.0,>=5.2.1->weasel<0.5.0,>=0.1.0->spacy) (1.17.0)\n",
      "Requirement already satisfied: mdurl~=0.1 in /opt/conda/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy) (0.1.2)\n",
      "Using cached numpy-2.0.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.5 MB)\n",
      "Installing collected packages: numpy\n",
      "  Attempting uninstall: numpy\n",
      "    Found existing installation: numpy 1.24.0\n",
      "    Uninstalling numpy-1.24.0:\n",
      "      Successfully uninstalled numpy-1.24.0\n",
      "Successfully installed numpy-2.0.2\n",
      "Collecting numpy==1.24\n",
      "  Using cached numpy-1.24.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)\n",
      "Using cached numpy-1.24.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)\n",
      "Installing collected packages: numpy\n",
      "  Attempting uninstall: numpy\n",
      "    Found existing installation: numpy 2.0.2\n",
      "    Uninstalling numpy-2.0.2:\n",
      "      Successfully uninstalled numpy-2.0.2\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "blis 1.0.1 requires numpy<3.0.0,>=2.0.0, but you have numpy 1.24.0 which is incompatible.\n",
      "thinc 8.3.2 requires numpy<2.1.0,>=2.0.0; python_version >= \"3.9\", but you have numpy 1.24.0 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0mSuccessfully installed numpy-1.24.0\n",
      "Collecting en-core-web-sm==3.8.0\n",
      "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl (12.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.8/12.8 MB\u001b[0m \u001b[31m125.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25h\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the package via spacy.load('en_core_web_sm')\n",
      "Collecting de-core-news-sm==3.8.0\n",
      "  Downloading https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-3.8.0/de_core_news_sm-3.8.0-py3-none-any.whl (14.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.6/14.6 MB\u001b[0m \u001b[31m113.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25h\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the package via spacy.load('de_core_news_sm')\n",
      "Requirement already satisfied: torch==2.0.1 in /opt/conda/lib/python3.11/site-packages (2.0.1)\n",
      "Requirement already satisfied: torchtext==0.15.2 in /opt/conda/lib/python3.11/site-packages (0.15.2)\n",
      "Requirement already satisfied: filelock in /opt/conda/lib/python3.11/site-packages (from torch==2.0.1) (3.16.1)\n",
      "Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.11/site-packages (from torch==2.0.1) (4.12.2)\n",
      "Requirement already satisfied: sympy in /opt/conda/lib/python3.11/site-packages (from torch==2.0.1) (1.13.3)\n",
      "Requirement already satisfied: networkx in /opt/conda/lib/python3.11/site-packages (from torch==2.0.1) (3.4.2)\n",
      "Requirement already satisfied: jinja2 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.1) (3.1.3)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.1) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.1) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.1) (11.7.101)\n",
      "Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.1) (8.5.0.96)\n",
      "Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.1) (11.10.3.66)\n",
      "Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.1) (10.9.0.58)\n",
      "Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.1) (10.2.10.91)\n",
      "Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.1) (11.4.0.1)\n",
      "Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.1) (11.7.4.91)\n",
      "Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.1) (2.14.3)\n",
      "Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.1) (11.7.91)\n",
      "Requirement already satisfied: triton==2.0.0 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.1) (2.0.0)\n",
      "Requirement already satisfied: tqdm in /opt/conda/lib/python3.11/site-packages (from torchtext==0.15.2) (4.66.4)\n",
      "Requirement already satisfied: requests in /opt/conda/lib/python3.11/site-packages (from torchtext==0.15.2) (2.31.0)\n",
      "Requirement already satisfied: numpy in /opt/conda/lib/python3.11/site-packages (from torchtext==0.15.2) (1.24.0)\n",
      "Requirement already satisfied: torchdata==0.6.1 in /opt/conda/lib/python3.11/site-packages (from torchtext==0.15.2) (0.6.1)\n",
      "Requirement already satisfied: setuptools in /opt/conda/lib/python3.11/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1) (69.5.1)\n",
      "Requirement already satisfied: wheel in /opt/conda/lib/python3.11/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1) (0.43.0)\n",
      "Requirement already satisfied: urllib3>=1.25 in /opt/conda/lib/python3.11/site-packages (from torchdata==0.6.1->torchtext==0.15.2) (2.2.1)\n",
      "Requirement already satisfied: cmake in /opt/conda/lib/python3.11/site-packages (from triton==2.0.0->torch==2.0.1) (3.31.1)\n",
      "Requirement already satisfied: lit in /opt/conda/lib/python3.11/site-packages (from triton==2.0.0->torch==2.0.1) (18.1.8)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.11/site-packages (from jinja2->torch==2.0.1) (2.1.5)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.11/site-packages (from requests->torchtext==0.15.2) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.11/site-packages (from requests->torchtext==0.15.2) (3.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.11/site-packages (from requests->torchtext==0.15.2) (2024.8.30)\n",
      "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/lib/python3.11/site-packages (from sympy->torch==2.0.1) (1.3.0)\n",
      "Requirement already satisfied: portalocker in /opt/conda/lib/python3.11/site-packages (3.0.0)\n",
      "Requirement already satisfied: numpy in /opt/conda/lib/python3.11/site-packages (1.24.0)\n",
      "Requirement already satisfied: pandas in /opt/conda/lib/python3.11/site-packages (2.2.3)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.11/site-packages (from pandas) (2.9.0)\n",
      "Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.11/site-packages (from pandas) (2024.1)\n",
      "Requirement already satisfied: tzdata>=2022.7 in /opt/conda/lib/python3.11/site-packages (from pandas) (2024.2)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)\n",
      "Requirement already satisfied: numpy in /opt/conda/lib/python3.11/site-packages (1.24.0)\n",
      "Requirement already satisfied: scikit-learn in /opt/conda/lib/python3.11/site-packages (1.5.2)\n",
      "Requirement already satisfied: scipy>=1.6.0 in /opt/conda/lib/python3.11/site-packages (from scikit-learn) (1.14.1)\n",
      "Requirement already satisfied: joblib>=1.2.0 in /opt/conda/lib/python3.11/site-packages (from scikit-learn) (1.4.2)\n",
      "Requirement already satisfied: threadpoolctl>=3.1.0 in /opt/conda/lib/python3.11/site-packages (from scikit-learn) (3.5.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install nltk\n",
    "!pip install transformers\n",
    "!pip install sentencepiece\n",
    "!pip install spacy\n",
    "!pip install numpy==1.24\n",
    "!python -m spacy download en_core_web_sm\n",
    "!python -m spacy download de_core_news_sm\n",
    "!pip install torch==2.0.1 torchtext==0.15.2\n",
    "!pip install portalocker\n",
    "!pip install numpy pandas \n",
    "!pip install numpy scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check the installed version of each package to make sure you have set the right environment.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing required libraries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.15.2+cpu\n"
     ]
    }
   ],
   "source": [
    "import torchtext\n",
    "print(torchtext.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "from torchtext.data.utils import get_tokenizer\n",
    "from torchtext.vocab import build_vocab_from_iterator\n",
    "from torchtext.datasets import multi30k, Multi30k\n",
    "from typing import Iterable, List\n",
    "from torch.nn.utils.rnn import pad_sequence\n",
    "from torch.utils.data import DataLoader\n",
    "from torchdata.datapipes.iter import IterableWrapper, Mapper\n",
    "import torchtext\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "import numpy as np\n",
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Data set**\n",
    "\n",
    "A data set in **PyTorch** is an object that represents a collection of data samples. Each data sample typically consists of one or more input features and their corresponding target labels. You can also use your data set to transform your data as needed.\n",
    "\n",
    "## **Data loader**\n",
    "\n",
    "A data loader in **PyTorch** is responsible for efficiently loading and batching data from a data set. It abstracts away the process of iterating over a data set, shuffling, and dividing it into batches for training. In NLP applications, the data loader is used to process and transform your text data, rather than just the data set.\n",
    "\n",
    "Data loaders have several key parameters, including the data set to load from, batch size (determining how many samples per batch), shuffle (whether to shuffle the data for each epoch), and more. Data loaders also provide an iterator interface, making it easy to iterate over batches of data during training.\n",
    "\n",
    "Now, you may ask, '**What is an iterator?**'\n",
    "\n",
    "An iterator is an object that can be looped over. It contains elements that can be iterated through and typically includes two methods, `__iter__()` and `__next__()`. When there are no more elements to iterate over, it raises a **`StopIteration`** exception.\n",
    "\n",
    "Iterators are commonly used to traverse large data sets without loading all elements into memory simultaneously, making the process more memory-efficient. In PyTorch, not all data sets are iterators, but all data loaders are.\n",
    "\n",
    "In PyTorch, the data loader processes data in batches, loading and processing one batch at a time into memory efficiently. The batch size, which you specify when creating the data loader, determines how many samples are processed together in each batch. The data loader's purpose is to convert input data and labels into batches of tensors with the same shape for deep learning models to interpret.\n",
    "\n",
    "Finally, a data loader can be used for tasks such as tokenizing, sequencing, converting your samples to the same size, and transforming your data into tensors that your model can understand.\n",
    "\n",
    "--- \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Custom data set and data loader in PyTorch**\n",
    "In this code snippet, you will see how to create a custom data set and use the DataLoader class in PyTorch. The data set consists of a list of random sentences, and the objective is to create batches of sentences for further processing, such as training a neural network model.\n",
    "\n",
    "You will begin by defining a custom data set called CustomDataset. This data set inherits from the `torch.utils.data.Dataset` class and is initialized with a list of sentences. The data set comprises two essential methods:\n",
    "\n",
    "- __init__(self, sentences): Initializes the data set with a list of sentences.\n",
    "- __getitem__(self, idx): Retrieves an item (in this case, a sentence) at a specific index, idx.\n",
    "\n",
    "Next, you create an instance of your custom data set (custom_dataset) by passing in the list of sentences. Additionally, you can specify a batch size (batch_size), which determines how many sentences will be grouped together in each batch during data loading.\n",
    "\n",
    "You will then create a DataLoader (dataloader) by providing your custom data set and batch size to the torch.utils.data.DataLoader class. Furthermore, you set shuffle=True, indicating that the sentences will be randomly shuffled before being divided into batches. This shuffling is particularly useful for training deep learning models, as it prevents the model from learning patterns based on the order of the data.\n",
    "\n",
    "Finally, you iterate through the DataLoader to demonstrate how data is loaded in batches. In this code, you will see that batch size is set to 2, meaning that each batch will contain two sentences. The DataLoader efficiently manages the loading of data in batches, making it suitable for training deep learning models.\n",
    "\n",
    "During iteration, the sentences in each batch are printed to illustrate how the DataLoader groups and presents the data. This code snippet provides a fundamental example of how to set up a custom data set and data loader in PyTorch, which is a common practice in deep learning workflows.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['It is our choices, Harry, that show what we truly are, far more than our abilities.', 'Soon we must all face the choice between what is right and what is easy.']\n",
      "['You are awesome!', 'Youth can not know how age thinks and feels. But old men are guilty if they forget what it was to be young.']\n",
      "[\"Fame's a fickle friend, Harry.\", \"If you want to know what a man's like, take a good look at how he treats his inferiors, not his equals.\"]\n"
     ]
    }
   ],
   "source": [
    "sentences = [\n",
    "    \"If you want to know what a man's like, take a good look at how he treats his inferiors, not his equals.\",\n",
    "    \"Fame's a fickle friend, Harry.\",\n",
    "    \"It is our choices, Harry, that show what we truly are, far more than our abilities.\",\n",
    "    \"Soon we must all face the choice between what is right and what is easy.\",\n",
    "    \"Youth can not know how age thinks and feels. But old men are guilty if they forget what it was to be young.\",\n",
    "    \"You are awesome!\"\n",
    "]\n",
    "\n",
    "# Define a custom dataset\n",
    "class CustomDataset(Dataset):\n",
    "    def __init__(self, sentences):\n",
    "        self.sentences = sentences\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.sentences)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        return self.sentences[idx]\n",
    "\n",
    "# Create an instance of your custom dataset\n",
    "custom_dataset = CustomDataset(sentences)\n",
    "\n",
    "# Define batch size\n",
    "batch_size = 2\n",
    "\n",
    "# Create a DataLoader\n",
    "dataloader = DataLoader(custom_dataset, batch_size=batch_size, shuffle=True)\n",
    "\n",
    "# Iterate through the DataLoader\n",
    "for batch in dataloader:\n",
    "    print(batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown above, the data is organized into batches of 2 sentences each. It's important to note that deep learning models can only comprehend numerical data, and words are meaningless to them. Therefore, your next step is to convert these sentences into tensors. Let's see how to do this.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating tensors for custom data set\n",
    "\n",
    "In this code example, you will see the creation of a custom data set for natural language processing (NLP) tasks using PyTorch. The data set consists of a list of sentences, and your goal is to preprocess these sentences, tokenize them, and convert them into tensors of token indices for use in NLP models. Let's break down the code step by step.\n",
    "\n",
    "The sentences and the CustomDataset class are used in the same way as in the previous code snippet. The changes made to the CustomDataset class are as follows:\n",
    "\n",
    "- __init__: The constructor takes a list of sentences, a tokenizer function, and a vocabulary (vocab) as input.\n",
    "- __len__: This method returns the total number of samples in the data set.\n",
    "- __getitem__: This method is responsible for processing a single sample. It tokenizes the sentence using the provided tokenizer and then converts the tokens into tensor indices using the vocabulary.\n",
    "\n",
    "You can define a tokenizer using the `get_tokenizer` function with the `basic_english` option. Tokenization is the process of splitting a text into individual tokens or words. Next, you build a vocabulary from the sentences. You use the `build_vocab_from_iterator` function to construct the vocabulary from the tokenized sentences.\n",
    "\n",
    "You can create an instance of your custom data set, passing in the sentences, tokenizer, and vocabulary. Finally, you print the length of the custom data set and sample items from the data set for illustration.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Custom Dataset Length: 6\n",
      "Sample Items:\n",
      "Item 1: tensor([11, 19, 63, 17, 13,  2,  3, 47,  6, 16, 45,  0, 55,  3, 41, 46, 24, 10,\n",
      "        43, 61,  9, 44,  0, 14,  9, 33,  1])\n",
      "Item 2: tensor([35,  6, 16,  3, 38, 40,  0,  8,  1])\n",
      "Item 3: tensor([12,  5, 15, 31,  0,  8,  0, 57, 53,  2, 18, 62,  4,  0, 36, 49, 56, 15,\n",
      "        21,  1])\n",
      "Item 4: tensor([54, 18, 50, 23, 34, 58, 30, 27,  2,  5, 52,  7,  2,  5, 32,  1])\n",
      "Item 5: tensor([66, 29, 14, 13, 10, 22, 60,  7, 37,  1, 28, 51, 48,  4, 42, 11, 59, 39,\n",
      "         2, 12, 64, 17, 26, 65,  1])\n",
      "Item 6: tensor([19,  4, 25, 20])\n"
     ]
    }
   ],
   "source": [
    "sentences = [\n",
    "    \"If you want to know what a man's like, take a good look at how he treats his inferiors, not his equals.\",\n",
    "    \"Fame's a fickle friend, Harry.\",\n",
    "    \"It is our choices, Harry, that show what we truly are, far more than our abilities.\",\n",
    "    \"Soon we must all face the choice between what is right and what is easy.\",\n",
    "    \"Youth can not know how age thinks and feels. But old men are guilty if they forget what it was to be young.\",\n",
    "    \"You are awesome!\"\n",
    "]\n",
    "\n",
    "# Define a custom data set\n",
    "class CustomDataset(Dataset):\n",
    "    def __init__(self, sentences, tokenizer, vocab):\n",
    "        self.sentences = sentences\n",
    "        self.tokenizer = tokenizer\n",
    "        self.vocab = vocab\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.sentences)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        tokens = self.tokenizer(self.sentences[idx])\n",
    "        # Convert tokens to tensor indices using vocab\n",
    "        tensor_indices = [self.vocab[token] for token in tokens]\n",
    "        return torch.tensor(tensor_indices)\n",
    "\n",
    "# Tokenizer\n",
    "tokenizer = get_tokenizer(\"basic_english\")\n",
    "\n",
    "# Build vocabulary\n",
    "vocab = build_vocab_from_iterator(map(tokenizer, sentences))\n",
    "\n",
    "# Create an instance of your custom data set\n",
    "custom_dataset = CustomDataset(sentences, tokenizer, vocab)\n",
    "\n",
    "print(\"Custom Dataset Length:\", len(custom_dataset))\n",
    "print(\"Sample Items:\")\n",
    "for i in range(6):\n",
    "    sample_item = custom_dataset[i]\n",
    "    print(f\"Item {i + 1}: {sample_item}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please go ahead and uncomment the following code to apply the data loader and observe the results:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Youth can not know how age thinks and feels. But old men are guilty if they forget what it was to be young.', \"If you want to know what a man's like, take a good look at how he treats his inferiors, not his equals.\"]\n",
      "['It is our choices, Harry, that show what we truly are, far more than our abilities.', 'You are awesome!']\n",
      "['Soon we must all face the choice between what is right and what is easy.', \"Fame's a fickle friend, Harry.\"]\n"
     ]
    }
   ],
   "source": [
    "# \"\"\"\n",
    "# Create an instance of your custom data set\n",
    "custom_dataset = CustomDataset(sentences, tokenizer, vocab)\n",
    "\n",
    "# Define batch size\n",
    "batch_size = 2\n",
    "\n",
    "# Create a data loader\n",
    "#dataloader = DataLoader(custom_dataset, batch_size=batch_size, shuffle=True)\n",
    "\n",
    "# Iterate through the data loader\n",
    "for batch in dataloader:\n",
    "    print(batch)\n",
    "# \"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will encounter an error when attempting to create batches for the tensors. This error arises because the tensor batches have unequal lengths. The data loader is using the default `collate_function`, which requires tensors to have equal lengths. You can define your own `collate_function` and pass the data into it to establish your own rules. Typically, to address the issue of unequal tensor lengths, you employ data padding. This will be demonstrated in the following section.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom collate function\n",
    "\n",
    "A collate function is employed in the context of data loading and batching in machine learning, particularly when dealing with variable-length data, such as sequences (e.g., text, time series, sequences of events). Its primary purpose is to prepare and format individual data samples (examples) into batches that can be efficiently processed by machine learning models.\n",
    "\n",
    "You will begin by defining a custom collate function named `collate_fn`. This function plays a crucial role when handling sequences of varying lengths, such as sentences in NLP. Its purpose is to pad sequences within a batch to have equal lengths, which is a common preprocessing step in NLP tasks.\n",
    "\n",
    "`pad_sequence`: This function is a part of PyTorch and is utilized to pad sequences in a batch, ensuring uniform length. It takes a batch of sequences as input and pads them to match the length of the longest sequence. The `padding_value=0` argument specifies the value to use for padding.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a custom collate function\n",
    "def collate_fn(batch):\n",
    "    # Pad sequences within the batch to have equal lengths\n",
    "    padded_batch = pad_sequence(batch, batch_first=True, padding_value=0)\n",
    "    return padded_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above cell, when padding the sequences, you set `batch_first=True`. When `batch_first=True`, output will be in [batch_size x seq_len] shape, otherwise it will be in [seq_len x batch_size] shape. Some models accept input with [batch_size x seq_len] shape while some other models need the input to be of [seq_len x batch_size] shape. Keep in mind that this parameter takes care of putting the input in the desired shape. \n",
    "\n",
    "Let's see how it actually affects the shape of curated batches. First, you create a data loader from a collate function with `batch_first=True`:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['if', 'you', 'want', 'to', 'know', 'what', 'a', 'man', \"'\", 's', 'like', ',', 'take', 'a', 'good', 'look', 'at', 'how', 'he', 'treats', 'his', 'inferiors', ',', 'not', 'his', 'equals', '.']\n",
      "['fame', \"'\", 's', 'a', 'fickle', 'friend', ',', 'harry', '.', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',']\n",
      "['it', 'is', 'our', 'choices', ',', 'harry', ',', 'that', 'show', 'what', 'we', 'truly', 'are', ',', 'far', 'more', 'than', 'our', 'abilities', '.']\n",
      "['soon', 'we', 'must', 'all', 'face', 'the', 'choice', 'between', 'what', 'is', 'right', 'and', 'what', 'is', 'easy', '.', ',', ',', ',', ',']\n",
      "['youth', 'can', 'not', 'know', 'how', 'age', 'thinks', 'and', 'feels', '.', 'but', 'old', 'men', 'are', 'guilty', 'if', 'they', 'forget', 'what', 'it', 'was', 'to', 'be', 'young', '.']\n",
      "['you', 'are', 'awesome', '!', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',']\n"
     ]
    }
   ],
   "source": [
    "# Create a data loader with the custom collate function with batch_first=True,\n",
    "dataloader = DataLoader(custom_dataset, batch_size=batch_size, collate_fn=collate_fn)\n",
    "\n",
    "# Iterate through the data loader\n",
    "for batch in dataloader: \n",
    "    for row in batch:\n",
    "        for idx in row:\n",
    "            words = [vocab.get_itos()[idx] for idx in row]\n",
    "        print(words)\n",
    "       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking into the result, you can see that the first dimension is the batch. For example, first batch is the first sentence: \"['if', 'you', 'want', 'to', 'know', 'what', 'a', 'man', \"'\", 's', 'like', ',', 'take', 'a', 'good', 'look', 'at', 'how', 'he', 'treats', 'his', 'inferiors', ',', 'not', 'his', 'equals', '.']\". \n",
    "\n",
    "Now, you can try `batch_first=False` which is the DEFAULT value:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a custom collate function\n",
    "def collate_fn_bfFALSE(batch):\n",
    "    # Pad sequences within the batch to have equal lengths\n",
    "    padded_batch = pad_sequence(batch, padding_value=0)\n",
    "    return padded_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you look into the curated data:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['if', 'fame']\n",
      "['you', \"'\"]\n",
      "['want', 's']\n",
      "['to', 'a']\n",
      "['know', 'fickle']\n",
      "['what', 'friend']\n",
      "['a', ',']\n",
      "['man', 'harry']\n",
      "[\"'\", '.']\n",
      "['s', ',']\n",
      "['like', ',']\n",
      "[',', ',']\n",
      "['take', ',']\n",
      "['a', ',']\n",
      "['good', ',']\n",
      "['look', ',']\n",
      "['at', ',']\n",
      "['how', ',']\n",
      "['he', ',']\n",
      "['treats', ',']\n",
      "['his', ',']\n",
      "['inferiors', ',']\n",
      "[',', ',']\n",
      "['not', ',']\n",
      "['his', ',']\n",
      "['equals', ',']\n",
      "['.', ',']\n",
      "['it', 'soon']\n",
      "['is', 'we']\n",
      "['our', 'must']\n",
      "['choices', 'all']\n",
      "[',', 'face']\n",
      "['harry', 'the']\n",
      "[',', 'choice']\n",
      "['that', 'between']\n",
      "['show', 'what']\n",
      "['what', 'is']\n",
      "['we', 'right']\n",
      "['truly', 'and']\n",
      "['are', 'what']\n",
      "[',', 'is']\n",
      "['far', 'easy']\n",
      "['more', '.']\n",
      "['than', ',']\n",
      "['our', ',']\n",
      "['abilities', ',']\n",
      "['.', ',']\n",
      "['youth', 'you']\n",
      "['can', 'are']\n",
      "['not', 'awesome']\n",
      "['know', '!']\n",
      "['how', ',']\n",
      "['age', ',']\n",
      "['thinks', ',']\n",
      "['and', ',']\n",
      "['feels', ',']\n",
      "['.', ',']\n",
      "['but', ',']\n",
      "['old', ',']\n",
      "['men', ',']\n",
      "['are', ',']\n",
      "['guilty', ',']\n",
      "['if', ',']\n",
      "['they', ',']\n",
      "['forget', ',']\n",
      "['what', ',']\n",
      "['it', ',']\n",
      "['was', ',']\n",
      "['to', ',']\n",
      "['be', ',']\n",
      "['young', ',']\n",
      "['.', ',']\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Create a data loader with the custom collate function with batch_first=True,\n",
    "dataloader_bfFALSE = DataLoader(custom_dataset, batch_size=batch_size, collate_fn=collate_fn_bfFALSE)\n",
    "\n",
    "# Iterate through the data loader\n",
    "for seq in dataloader_bfFALSE:\n",
    "    for row in seq:\n",
    "        #print(row)\n",
    "        words = [vocab.get_itos()[idx] for idx in row]\n",
    "        print(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can be seen that the first dimension is now the sequence instead of batch, which means sentences will break so that each row includes a token from each sequence. For example the first row, (['if', 'fame']), includes the first tokens of all the sequences in that batch. You need to be aware of this standard to avoid any confusion when working with recurrent neural networks (RNNs) and transformers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[11, 19, 63, 17, 13,  2,  3, 47,  6, 16, 45,  0, 55,  3, 41, 46, 24, 10,\n",
      "         43, 61,  9, 44,  0, 14,  9, 33,  1],\n",
      "        [35,  6, 16,  3, 38, 40,  0,  8,  1,  0,  0,  0,  0,  0,  0,  0,  0,  0,\n",
      "          0,  0,  0,  0,  0,  0,  0,  0,  0]])\n",
      "Length of sequences in the batch: 27\n",
      "tensor([[12,  5, 15, 31,  0,  8,  0, 57, 53,  2, 18, 62,  4,  0, 36, 49, 56, 15,\n",
      "         21,  1],\n",
      "        [54, 18, 50, 23, 34, 58, 30, 27,  2,  5, 52,  7,  2,  5, 32,  1,  0,  0,\n",
      "          0,  0]])\n",
      "Length of sequences in the batch: 20\n",
      "tensor([[66, 29, 14, 13, 10, 22, 60,  7, 37,  1, 28, 51, 48,  4, 42, 11, 59, 39,\n",
      "          2, 12, 64, 17, 26, 65,  1],\n",
      "        [19,  4, 25, 20,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,\n",
      "          0,  0,  0,  0,  0,  0,  0]])\n",
      "Length of sequences in the batch: 25\n"
     ]
    }
   ],
   "source": [
    "# Iterate through the data loader with batch_first = TRUE\n",
    "for batch in dataloader:    \n",
    "    print(batch)\n",
    "    print(\"Length of sequences in the batch:\",batch.shape[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will see that each batch has a fixed size for all the sequences within the batch.\n",
    "\n",
    "You also have the option to utilize the collate function for tasks such as tokenization, converting tokenized indices, and transforming the result into a tensor. It's important to note that the original data set remains untouched by these transformations.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a custom data set\n",
    "class CustomDataset(Dataset):\n",
    "    def __init__(self, sentences):\n",
    "        self.sentences = sentences\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.sentences)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        return self.sentences[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "custom_dataset=CustomDataset(sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have the raw text:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"If you want to know what a man's like, take a good look at how he treats his inferiors, not his equals.\""
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "custom_dataset[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You create the new ```collate_fn```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def collate_fn(batch):\n",
    "    # Tokenize each sample in the batch using the specified tokenizer\n",
    "    tensor_batch = []\n",
    "    for sample in batch:\n",
    "        tokens = tokenizer(sample)\n",
    "        # Convert tokens to vocabulary indices and create a tensor for each sample\n",
    "        tensor_batch.append(torch.tensor([vocab[token] for token in tokens]))\n",
    "\n",
    "    # Pad sequences within the batch to have equal lengths using pad_sequence\n",
    "    # batch_first=True ensures that the tensors have shape (batch_size, max_sequence_length)\n",
    "    padded_batch = pad_sequence(tensor_batch, batch_first=True)\n",
    "    \n",
    "    # Return the padded batch\n",
    "    return padded_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a data loader with the custom collate function.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a data loader for the custom dataset\n",
    "dataloader = DataLoader(\n",
    "    dataset=custom_dataset,   # Custom PyTorch Dataset containing your data\n",
    "    batch_size=batch_size,     # Number of samples in each mini-batch\n",
    "    shuffle=True,              # Shuffle the data at the beginning of each epoch\n",
    "    collate_fn=collate_fn      # Custom collate function for processing batches\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will see that the result is a tensor of the same shape for each sample in the batch.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[19,  4, 25, 20,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0],\n",
      "        [54, 18, 50, 23, 34, 58, 30, 27,  2,  5, 52,  7,  2,  5, 32,  1]])\n",
      "shape of sample 2\n",
      "tensor([[66, 29, 14, 13, 10, 22, 60,  7, 37,  1, 28, 51, 48,  4, 42, 11, 59, 39,\n",
      "          2, 12, 64, 17, 26, 65,  1],\n",
      "        [12,  5, 15, 31,  0,  8,  0, 57, 53,  2, 18, 62,  4,  0, 36, 49, 56, 15,\n",
      "         21,  1,  0,  0,  0,  0,  0]])\n",
      "shape of sample 2\n",
      "tensor([[11, 19, 63, 17, 13,  2,  3, 47,  6, 16, 45,  0, 55,  3, 41, 46, 24, 10,\n",
      "         43, 61,  9, 44,  0, 14,  9, 33,  1],\n",
      "        [35,  6, 16,  3, 38, 40,  0,  8,  1,  0,  0,  0,  0,  0,  0,  0,  0,  0,\n",
      "          0,  0,  0,  0,  0,  0,  0,  0,  0]])\n",
      "shape of sample 2\n"
     ]
    }
   ],
   "source": [
    "for batch in dataloader:\n",
    "    print(batch)\n",
    "    print(\"shape of sample\",len(batch))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result, batches of tensors with equal lengths have been successfully created.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a data loader with a collate function that processes batches of French text (provided below). Sort the data set on sequences length. Then tokenize, numericalize and pad the sequences. Sorting the sequences will minimize the number of `<PAD>`tokens added to the sequences, which enhances the model's performance. Prepare the data in batches of size 4 and print them.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[19,  4, 25, 20,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,\n",
      "          0,  0,  0,  0,  0,  0,  0],\n",
      "        [66, 29, 14, 13, 10, 22, 60,  7, 37,  1, 28, 51, 48,  4, 42, 11, 59, 39,\n",
      "          2, 12, 64, 17, 26, 65,  1]])\n",
      "tensor([[12,  5, 15, 31,  0,  8,  0, 57, 53,  2, 18, 62,  4,  0, 36, 49, 56, 15,\n",
      "         21,  1],\n",
      "        [35,  6, 16,  3, 38, 40,  0,  8,  1,  0,  0,  0,  0,  0,  0,  0,  0,  0,\n",
      "          0,  0]])\n",
      "tensor([[54, 18, 50, 23, 34, 58, 30, 27,  2,  5, 52,  7,  2,  5, 32,  1,  0,  0,\n",
      "          0,  0,  0,  0,  0,  0,  0,  0,  0],\n",
      "        [11, 19, 63, 17, 13,  2,  3, 47,  6, 16, 45,  0, 55,  3, 41, 46, 24, 10,\n",
      "         43, 61,  9, 44,  0, 14,  9, 33,  1]])\n"
     ]
    }
   ],
   "source": [
    "corpus = [\n",
    "    \"Ceci est une phrase.\",\n",
    "    \"C'est un autre exemple de phrase.\",\n",
    "    \"Voici une troisième phrase.\",\n",
    "    \"Il fait beau aujourd'hui.\",\n",
    "    \"J'aime beaucoup la cuisine française.\",\n",
    "    \"Quel est ton plat préféré ?\",\n",
    "    \"Je t'adore.\",\n",
    "    \"Bon appétit !\",\n",
    "    \"Je suis en train d'apprendre le français.\",\n",
    "    \"Nous devons partir tôt demain matin.\",\n",
    "    \"Je suis heureux.\",\n",
    "    \"Le film était vraiment captivant !\",\n",
    "    \"Je suis là.\",\n",
    "    \"Je ne sais pas.\",\n",
    "    \"Je suis fatigué après une longue journée de travail.\",\n",
    "    \"Est-ce que tu as des projets pour le week-end ?\",\n",
    "    \"Je vais chez le médecin cet après-midi.\",\n",
    "    \"La musique adoucit les mœurs.\",\n",
    "    \"Je dois acheter du pain et du lait.\",\n",
    "    \"Il y a beaucoup de monde dans cette ville.\",\n",
    "    \"Merci beaucoup !\",\n",
    "    \"Au revoir !\",\n",
    "    \"Je suis ravi de vous rencontrer enfin !\",\n",
    "    \"Les vacances sont toujours trop courtes.\",\n",
    "    \"Je suis en retard.\",\n",
    "    \"Félicitations pour ton nouveau travail !\",\n",
    "    \"Je suis désolé, je ne peux pas venir à la réunion.\",\n",
    "    \"À quelle heure est le prochain train ?\",\n",
    "    \"Bonjour !\",\n",
    "    \"C'est génial !\"\n",
    "]\n",
    "for batch in dataloader:\n",
    "    print(batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting fr-core-news-sm==3.8.0\n",
      "  Downloading https://github.com/explosion/spacy-models/releases/download/fr_core_news_sm-3.8.0/fr_core_news_sm-3.8.0-py3-none-any.whl (16.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m16.3/16.3 MB\u001b[0m \u001b[31m104.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25h\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the package via spacy.load('fr_core_news_sm')\n"
     ]
    }
   ],
   "source": [
    "!python -m spacy download fr_core_news_sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def collate_fn_fr(batch):\n",
    "    # Pad sequences within the batch to have equal lengths\n",
    "    tensor_batch=[]\n",
    "    for sample in batch:\n",
    "        tokens = tokenizer(sample)\n",
    "        tensor_batch.append(torch.tensor([vocab[token] for token in tokens]))\n",
    "         \n",
    "    padded_batch = pad_sequence(tensor_batch,batch_first=True)\n",
    "    return padded_batch\n",
    "\n",
    "# Build tokenizer\n",
    "tokenizer = get_tokenizer('spacy', language='fr_core_news_sm')\n",
    "\n",
    "# Build vocabulary\n",
    "vocab = build_vocab_from_iterator(map(tokenizer, corpus))\n",
    "\n",
    "# Sort sentences based on their length\n",
    "sorted_data = sorted(corpus, key=lambda x: len(tokenizer(x)))\n",
    "#print(sorted_data)\n",
    "dataloader = DataLoader(sorted_data, batch_size=4, shuffle=False, collate_fn=collate_fn_fr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>Click here for the solution</summary>\n",
    "\n",
    "```python\n",
    "\n",
    "def collate_fn_fr(batch):\n",
    "    # Pad sequences within the batch to have equal lengths\n",
    "    tensor_batch=[]\n",
    "    for sample in batch:\n",
    "        tokens = tokenizer(sample)\n",
    "        tensor_batch.append(torch.tensor([vocab[token] for token in tokens]))\n",
    "         \n",
    "    padded_batch = pad_sequence(tensor_batch,batch_first=True)\n",
    "    return padded_batch\n",
    "\n",
    "# Build tokenizer\n",
    "tokenizer = get_tokenizer('spacy', language='fr_core_news_sm')\n",
    "\n",
    "# Build vocabulary\n",
    "vocab = build_vocab_from_iterator(map(tokenizer, corpus))\n",
    "\n",
    "# Sort sentences based on their length\n",
    "sorted_data = sorted(corpus, key=lambda x: len(tokenizer(x)))\n",
    "#print(sorted_data)\n",
    "dataloader = DataLoader(sorted_data, batch_size=4, shuffle=False, collate_fn=collate_fn_fr)\n",
    "\n",
    "```\n",
    "</details>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 27,   2,   0],\n",
      "        [ 26,  45,   2],\n",
      "        [ 35,   8,   2],\n",
      "        [ 25, 101,   2]])\n",
      "tensor([[  1, 105,  41,   0],\n",
      "        [  1,   3,  76,   0],\n",
      "        [  1,   3,  82,   0],\n",
      "        [ 11,   4,  74,   2]])\n",
      "tensor([[ 28,   4,  10,   9,   0],\n",
      "        [ 38,  10, 107,   9,   0],\n",
      "        [ 12,  69,  51,  49,   0],\n",
      "        [  1,  16, 103,  17,   0]])\n",
      "tensor([[  1,   3,  14, 100,   0,   0],\n",
      "        [ 37,   4,  19,  92,  95,   7],\n",
      "        [ 33,  71, 122, 117,  52,   2],\n",
      "        [ 32,  85,  42,  80,  87,   0]])\n",
      "tensor([[ 30,  18,  19,  88,  21,   2,   0],\n",
      "        [ 31,  43,   8,  15,  57,  73,   0],\n",
      "        [ 36,  62,  90, 110,  60,  83,   0],\n",
      "        [ 34, 112, 104, 106, 108,  56,   0]])\n",
      "tensor([[ 11,   4, 111,  50,  68,   5,   9,   0],\n",
      "        [  1, 113,  55,   6,  86,  53,  47,   0],\n",
      "        [  1,   3,  98,   5, 116,  99,  66,   2],\n",
      "        [120,  97,  75,   4,   6,  93,  20,   7]])\n",
      "tensor([[  1,   3,  14,  20,  58,  44,   6,  72,   0,   0],\n",
      "        [  1,  63,  40,  13,  89,  67,  13,  79,   0,   0],\n",
      "        [  1,   3,  70,  46,  10,  81,  78,   5,  21,   0],\n",
      "        [ 12, 119,  39,   8,   5,  84,  59,  54, 115,   0]])\n",
      "tensor([[ 29,  24,  96, 109,  48,  61,  94,  18,   6, 118,  23,  65,   7],\n",
      "        [  1,   3,  64,  22,  77,  16,  91,  17, 114, 121,  15, 102,   0]])\n"
     ]
    }
   ],
   "source": [
    "for batch in dataloader:\n",
    "    print(batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>Click here for the solution</summary>\n",
    "\n",
    "```python\n",
    "for batch in dataloader:\n",
    "    print(batch)\n",
    "\n",
    "```\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Optional] Data loader for German-English translation task\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section sets the stage for German-English machine translation using the torchtext and spaCy libraries. It adjusts data set URLs for the Multi30k data set, configures tokenizers for both languages, and establishes vocabularies with special tokens. This foundation is crucial for building and training effective translation models.\n",
    "\n",
    "- **Data set configuration and language definition**\n",
    "  - Default URLs for the Multi30k dataset are modified to fix broken links.\n",
    "  - Source (`de` for German) and target (`en` for English) languages are defined.\n",
    "\n",
    "- **Tokenizer setup**\n",
    "  - Tokenizers for both languages are set up using `spaCy`.\n",
    "\n",
    "- **Token generation**\n",
    "  - A helper function, `yield_tokens`, is created to generate tokens from the data set\n",
    "\n",
    "- **Special symbols**\n",
    "  - Special symbols (e.g., `<unk>`, `<pad>`) and their indices are defined.\n",
    "\n",
    "- **Vocabulary building**\n",
    "  - Vocabularies for both source and target languages are built from the **training** data of the Multi30k dataset converting tokens to unique indices (numbers)\n",
    "\n",
    "- **Default token handling**\n",
    "  - A default index (`UNK_IDX`) is set for tokens not found in the vocabulary.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Translation data set\n",
    "In this section, you fetch a language translation data set called Multi30k. You will modify its default training and validation URLs, and then retrieve and print the first pair of German-English sentences from the training set. First, you will override the default URLs:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You would modify the URLs for the data set since the links to the original data set are broken\n",
    "\n",
    "multi30k.URL[\"train\"] = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-AI0205EN-SkillsNetwork/training.tar.gz\"\n",
    "multi30k.URL[\"valid\"] = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-AI0205EN-SkillsNetwork/validation.tar.gz\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the source language as German ('de') and target language as English ('en'). In Python, global variables are variables defined outside of a function, accessible both inside and outside of functions. They are often written in all caps as a convention to indicate they are constant, global nature and to differentiate them from regular variables.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "SRC_LANGUAGE = 'de'\n",
    "TGT_LANGUAGE = 'en'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize the training data iterator for the Multi30k dataset with the specified source and target languages:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_iter = Multi30k(split='train', language_pair=(SRC_LANGUAGE, TGT_LANGUAGE))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an iterator for the training data set:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_set = iter(train_iter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can print out the first five pairs of source and target sentences from the training data set:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sample 1\n",
      "Source (de): Zwei junge weiße Männer sind im Freien in der Nähe vieler Büsche.\n",
      "Target (en): Two young, White males are outside near many bushes.\n",
      "sample 2\n",
      "Source (de): Mehrere Männer mit Schutzhelmen bedienen ein Antriebsradsystem.\n",
      "Target (en): Several men in hard hats are operating a giant pulley system.\n",
      "sample 3\n",
      "Source (de): Ein kleines Mädchen klettert in ein Spielhaus aus Holz.\n",
      "Target (en): A little girl climbing into a wooden playhouse.\n",
      "sample 4\n",
      "Source (de): Ein Mann in einem blauen Hemd steht auf einer Leiter und putzt ein Fenster.\n",
      "Target (en): A man in a blue shirt is standing on a ladder cleaning a window.\n",
      "sample 5\n",
      "Source (de): Zwei Männer stehen am Herd und bereiten Essen zu.\n",
      "Target (en): Two men are at the stove preparing food.\n"
     ]
    }
   ],
   "source": [
    "for n in range(5):\n",
    "    # Getting the next pair of source and target sentences from the training data set\n",
    "    src, tgt = next(data_set)\n",
    "\n",
    "    # Printing the source (German) and target (English) sentences\n",
    "    print(f\"sample {str(n+1)}\")\n",
    "    print(f\"Source ({SRC_LANGUAGE}): {src}\\nTarget ({TGT_LANGUAGE}): {tgt}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenizer setup\n",
    "\n",
    "The tokenizer, set up using spaCy, breaks down text into smaller units or tokens, facilitating precise language processing and ensuring that words and punctuations are appropriately segmented for the translation task. Let's use the following example samples:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source German (de): Ein Mann in grün hält eine Gitarre, während der andere Mann sein Hemd ansieht.\n",
      "Target English  (en): A man in green holds a guitar while the other man observes his shirt.\n"
     ]
    }
   ],
   "source": [
    "german, english = next(data_set)\n",
    "print(f\"Source German ({SRC_LANGUAGE}): {german}\\nTarget English  ({TGT_LANGUAGE}): { english }\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the```get_tokenizer``` utility function from ```torchtext``` to obtain tokenizers for language processing:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchtext.data.utils import get_tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize the German and English tokenizers using spaCy's 'de_core_news_sm' model:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Making a placeholder dict to store both tokenizers\n",
    "token_transform = {}\n",
    "\n",
    "token_transform[SRC_LANGUAGE] = get_tokenizer('spacy', language='de_core_news_sm')\n",
    "token_transform[TGT_LANGUAGE] = get_tokenizer('spacy', language='en_core_web_sm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The line ```token_transform['de'](german)``` will tokenize the German string (or text) using the previously defined ```token_transform['de']``` for the German language.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Ein',\n",
       " 'Mann',\n",
       " 'in',\n",
       " 'grün',\n",
       " 'hält',\n",
       " 'eine',\n",
       " 'Gitarre',\n",
       " ',',\n",
       " 'während',\n",
       " 'der',\n",
       " 'andere',\n",
       " 'Mann',\n",
       " 'sein',\n",
       " 'Hemd',\n",
       " 'ansieht',\n",
       " '.']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_transform['de'](german)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same thing for English:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['A',\n",
       " 'man',\n",
       " 'in',\n",
       " 'green',\n",
       " 'holds',\n",
       " 'a',\n",
       " 'guitar',\n",
       " 'while',\n",
       " 'the',\n",
       " 'other',\n",
       " 'man',\n",
       " 'observes',\n",
       " 'his',\n",
       " 'shirt',\n",
       " '.']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_transform['en'](english)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Special symbols\n",
    "In a typical NLP  context, the tokens `['<unk>', '<pad>', '<bos>', '<eos>']` have specific meanings:\n",
    "\n",
    "1. `<unk>`: This token represents \"unknown\" or \"out-of-vocabulary\" words. It is used when a word in the input text is not found in the vocabulary or when dealing with words that are rare or unseen during training. When the model encounters an unknown word, it replaces it with the `<unk>` token.\n",
    "\n",
    "2. `<pad>`: This token represents padding. In sequences of text data, such as sentences or documents, sequences may have different lengths. To create batches of data with uniform dimensions, shorter sequences are often padded with this `<pad>` token to match the length of the longest sequence in the batch.\n",
    "\n",
    "3. `<bos>`: This token represents the \"beginning of sequence.\" It is used to indicate the start of a sentence or sequence of tokens. It helps the model understand the beginning of a text sequence.\n",
    "\n",
    "4. `<eos>`: This token represents the \"end of sequence.\" It is used to indicate the end of a sentence or sequence of tokens. It helps the model recognize the end of a text sequence.\n",
    "\n",
    " Define special symbols and indices\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define special symbols and indices\n",
    "UNK_IDX, PAD_IDX, BOS_IDX, EOS_IDX = 0, 1, 2, 3\n",
    "# Make sure the tokens are in order of their indices to properly insert them in vocab\n",
    "special_symbols = ['<unk>', '<pad>', '<bos>', '<eos>']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokens to indices transformation (Vocab)\n",
    "The code initializes a dictionary vocab_transform and then builds vocabularies for both German (de) and English (en) languages from the ```train_iter dataset``` using the helper ```function yield_tokens```. These vocabularies are then stored in the vocab_transform dictionary. The vocabularies are built with certain constraints like a minimum frequency for tokens and the inclusion of special symbols at the beginning.\n",
    "\n",
    "Initialize a dictionary to store vocabularies for the two languages:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "#place holder dict for 'en' and 'de' vocab transforms\n",
    "vocab_transform = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will create a yield_tokens function that processes a given data set iterator (data_iter), and for each sample, tokenizes the data for the specified language (language). It uses a predefined mapping (token_transform) of languages to their corresponding tokenizers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def yield_tokens(data_iter: Iterable, language: str) -> List[str]:\n",
    "    # Define a mapping to associate the source and target languages\n",
    "    # with their respective positions in the data samples.\n",
    "    language_index = {SRC_LANGUAGE: 0, TGT_LANGUAGE: 1}\n",
    "\n",
    "    # Iterate over each data sample in the provided dataset iterator\n",
    "    for data_sample in data_iter:\n",
    "        # Tokenize the data sample corresponding to the specified language\n",
    "        # and yield the resulting tokens.\n",
    "        yield token_transform[language](data_sample[language_index[language]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You build and store the German and English vocabularies from the **training** data set only. You can use the helper function ```yield_tokens``` to tokenize data. Include tokens that appear at least once (min_freq=1) and add special symbols (like <pad>, <unk>, etc.) at the beginning of the vocabulary:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "for ln in [SRC_LANGUAGE, TGT_LANGUAGE]:\n",
    "    # Training data iterator\n",
    "    train_iterator = Multi30k(split='train', language_pair=(SRC_LANGUAGE, TGT_LANGUAGE))\n",
    "    #To decrease the number of padding tokens, you sort data on the source length to batch similar-length sequences together\n",
    "    sorted_dataset = sorted(train_iterator, key=lambda x: len(x[0].split()))\n",
    "    # Create torchtext's Vocab object\n",
    "    vocab_transform[ln] = build_vocab_from_iterator(yield_tokens(sorted_dataset, ln),\n",
    "                                                    min_freq=1,\n",
    "                                                    specials=special_symbols,\n",
    "                                                    special_first=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Set ```UNK_IDX``` as the default index. This index is returned when the token is not found.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If not set, it throws ``RuntimeError`` when the queried token is not found in the Vocabulary.\n",
    "for ln in [SRC_LANGUAGE, TGT_LANGUAGE]:\n",
    "  vocab_transform[ln].set_default_index(UNK_IDX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You take the  English/German text string, tokenize it into words or subwords, and then convert these tokens into their corresponding indices from the vocabulary, resulting in a sequence of integers seq_en that can be used for further processing in a model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "English text string: A man in green holds a guitar while the other man observes his shirt.\n",
      " English sequence: [6, 12, 7, 51, 144, 4, 126, 29, 8, 75, 12, 1748, 27, 23, 5]\n",
      "German text string: Ein Mann in grün hält eine Gitarre, während der andere Mann sein Hemd ansieht.\n",
      " German sequence: [5, 12, 7, 657, 39, 18, 133, 8, 37, 16, 105, 12, 136, 41, 1779, 4]\n"
     ]
    }
   ],
   "source": [
    "seq_en=vocab_transform['en'](token_transform['en'](english))\n",
    "print(f\"English text string: {english}\\n English sequence: {seq_en}\")\n",
    "\n",
    "seq_de=vocab_transform['de'](token_transform['de'](german))\n",
    "print(f\"German text string: {german}\\n German sequence: {seq_de}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```tensor_transform_s``` function adds a beginning-of-sequence (BOS) token at the start, flips the sequence to reverse the order of token IDs and adds an end-of-sequence (EOS) token at the end of a given sequence of token IDs, then returns the concatenated result as a PyTorch tensor, this will be used as an input to our model:\n",
    "\n",
    "```tensor_transform_t``` function does the similar operations except the flip operation. It is a good practice to reverse the order of source sentence in order for the LSTM to perform better.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to add BOS/EOS, flip source sentence and create tensor for input sequence indices\n",
    "def tensor_transform_s(token_ids: List[int]):\n",
    "    return torch.cat((torch.tensor([BOS_IDX]),\n",
    "                      torch.flip(torch.tensor(token_ids), dims=(0,)),\n",
    "                      torch.tensor([EOS_IDX])))\n",
    "\n",
    "# function to add BOS/EOS and create tensor for input sequence indices\n",
    "def tensor_transform_t(token_ids: List[int]):\n",
    "    return torch.cat((torch.tensor([BOS_IDX]),\n",
    "                      torch.tensor(token_ids),\n",
    "                      torch.tensor([EOS_IDX])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([   2,    5,   23,   27, 1748,   12,   75,    8,   29,  126,    4,  144,\n",
       "          51,    7,   12,    6,    3])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seq_en=tensor_transform_s(seq_en)\n",
    "seq_en"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([   2,    5,   12,    7,  657,   39,   18,  133,    8,   37,   16,  105,\n",
       "          12,  136,   41, 1779,    4,    3])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seq_de=tensor_transform_t(seq_de)\n",
    "seq_de"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have defined the transform function, you create a ```sequestial_transforms``` function to put all the transformations together in the correct order.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# helper function to club together sequential operations\n",
    "def sequential_transforms(*transforms):\n",
    "    def func(txt_input):\n",
    "        for transform in transforms:\n",
    "            txt_input = transform(txt_input)\n",
    "        return txt_input\n",
    "    return func\n",
    "\n",
    "# ``src`` and ``tgt`` language text transforms to convert raw strings into tensors indices\n",
    "text_transform = {}\n",
    "\n",
    "text_transform[SRC_LANGUAGE] = sequential_transforms(token_transform[SRC_LANGUAGE], #Tokenization\n",
    "                                            vocab_transform[SRC_LANGUAGE], #Numericalization\n",
    "                                            tensor_transform_s) # Add BOS/EOS and create tensor\n",
    "\n",
    "text_transform[TGT_LANGUAGE] = sequential_transforms(token_transform[TGT_LANGUAGE], #Tokenization\n",
    "                                            vocab_transform[TGT_LANGUAGE], #Numericalization\n",
    "                                            tensor_transform_t) # Add BOS/EOS and create tensor\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processing data in batches\n",
    "The collate_fn function builds upon the utilities you established earlier. It performs the text_transform to a batch of raw data. Furthermore, it ensures consistent sequence lengths within the batch through padding. This transformation readies the data for input to a transformer model designed for language translation tasks.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to collate data samples into batch tensors\n",
    "def collate_fn(batch):\n",
    "    src_batch, tgt_batch = [], []\n",
    "    for src_sample, tgt_sample in batch:\n",
    "        src_sequences = text_transform[SRC_LANGUAGE](src_sample.rstrip(\"\\n\"))\n",
    "        src_sequences = torch.tensor(src_sequences, dtype=torch.int64)\n",
    "        tgt_sequences = text_transform[TGT_LANGUAGE](tgt_sample.rstrip(\"\\n\"))\n",
    "        tgt_sequences = torch.tensor(tgt_sequences, dtype=torch.int64)\n",
    "        src_batch.append(src_sequences)\n",
    "        tgt_batch.append(tgt_sequences)\n",
    "\n",
    "    src_batch = pad_sequence(src_batch, padding_value=PAD_IDX,batch_first=True)\n",
    "    tgt_batch = pad_sequence(tgt_batch, padding_value=PAD_IDX,batch_first=True)\n",
    "    \n",
    "    return src_batch.to(device), tgt_batch.to(device)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You establish a training data iterator using the Multi30k data set and configure a data loader with a batch size of 4. This leverages the predefined collate_fn to efficiently curate and ready batches for training your transformer model. Your primary aim is to delve deeper into the intricacies of the RNN encoder and decoder components.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_441/4285918835.py:6: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  src_sequences = torch.tensor(src_sequences, dtype=torch.int64)\n",
      "/tmp/ipykernel_441/4285918835.py:8: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  tgt_sequences = torch.tensor(tgt_sequences, dtype=torch.int64)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(tensor([[    2,     3,     1,     1,     1],\n",
       "         [    2,  5510,     3,     1,     1],\n",
       "         [    2,  5510,     3,     1,     1],\n",
       "         [    2,  1701,     8, 12642,     3]]),\n",
       " tensor([[   2,    3,    1,    1,    1,    1,    1,    1,    1,    1,    1],\n",
       "         [   2, 6650, 4623,  259,  172, 9953,  115,  692, 3428,    5,    3],\n",
       "         [   2,  216,  110, 3913, 1650, 3823,   71, 2808, 2187,    5,    3],\n",
       "         [   2,    6, 3398,  202,  109,   37,    3,    1,    1,    1,    1]]))"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "BATCH_SIZE = 4\n",
    "\n",
    "train_iterator = Multi30k(split='train', language_pair=(SRC_LANGUAGE, TGT_LANGUAGE))\n",
    "sorted_train_iterator = sorted(train_iterator, key=lambda x: len(x[0].split()))\n",
    "train_dataloader = DataLoader(sorted_train_iterator, batch_size=BATCH_SIZE, collate_fn=collate_fn,drop_last=True)\n",
    "\n",
    "valid_iterator = Multi30k(split='valid', language_pair=(SRC_LANGUAGE, TGT_LANGUAGE))\n",
    "sorted_valid_dataloader = sorted(valid_iterator, key=lambda x: len(x[0].split()))\n",
    "valid_dataloader = DataLoader(sorted_valid_dataloader, batch_size=BATCH_SIZE, collate_fn=collate_fn,drop_last=True)\n",
    "\n",
    "\n",
    "src, trg = next(iter(train_dataloader))\n",
    "src,trg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Congratulations! You have completed the lab.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Joseph Santarcangelo](https://www.linkedin.com/in/joseph-s-50398b136/) has a Ph.D. in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD.\n",
    "\n",
    "[Roodra Kanwar](https://www.linkedin.com/in/roodrakanwar/) is completing his MS in CS specializing in big data from Simon Fraser University. He has previous experience working with machine learning and as a data engineer.\n",
    "\n",
    "[Fateme Akbari](https://www.linkedin.com/in/fatemeakbari/) is a PhD candidate in Information Systems at McMaster University with demonstrated research experience in Machine Learning and NLP.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "© Copyright IBM Corporation. All rights reserved.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{## Change Log}\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{|Date (YYYY-MM-DD)|Version|Changed By|Change Description|}\n",
    "```\n",
    "```{|-|-|-|-|}\n",
    "```\n",
    "```{|2023-10-24|0.1|Roodra|Created Lab Template|}\n",
    "```\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  },
  "prev_pub_hash": "364ea50d4bbf7c18b3b99977a94452c1aafb00d94ce0d1569478e5734e235447"
 },
 "nbformat": 4,
 "nbformat_minor": 4
}