before_optimization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example : Common Tensorflow workflow\n",
    "\n",
    "## Summary\n",
    "This example is common tensorflow training sequence.\n",
    "\n",
    "On this example, we will train hand-writing number classification model with [MNIST dataset](https://en.wikipedia.org/wiki/MNIST_database).\n",
    "- - -\n",
    "### Import pacakges"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This package will use for install uninstalled package\n",
    "import subprocess\n",
    "\n",
    "# Import TensorFlow and install if not installed\n",
    "try:\n",
    "    import tensorflow as tf\n",
    "except(ModuleNotFoundError):\n",
    "    subprocess.run([\"pip3\", \"install\", \"tensorflow\"])\n",
    "    import tensorflow as tf\n",
    "\n",
    "# Import NumPy and install if not installed\n",
    "try:\n",
    "    import numpy as np\n",
    "except(ModuleNotFoundError):\n",
    "    subprocess.run([\"pip3\", \"install\", \"numpy\"])\n",
    "    import numpy as np\n",
    "\n",
    "# Import OpenCV and install if not installed\n",
    "try:\n",
    "    import cv2\n",
    "except(ModuleNotFoundError):\n",
    "    subprocess.run([\"pip3\", \"install\", \"opencv-python\"])\n",
    "    import cv2\n",
    "\n",
    "# Import Python internal modules\n",
    "import os\n",
    "import glob\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What those packages do?\n",
    "* [TensorFlow](https://www.tensorflow.org/) : Define and training model.\n",
    "* [Numpy](https://numpy.org/) : Preprocess image array data for training data.\n",
    "* [cv2 (OpenCV)](https://opencv.org/) : Read image file and return it to numpy array.\n",
    "* [os](https://docs.python.org/3/library/os.html) : Get label and join splited path to one.\n",
    "* [glob](https://docs.python.org/3/library/glob.html) : Get all image files absolute path.\n",
    "* [pathlib](https://docs.python.org/3/library/pathlib.html) : type-hinting.\n",
    "- - -\n",
    "### Define dataset and dataloader\n",
    "We will assume dataset is infinite or It can only stored partial dataset in [RAM](https://en.wikipedia.org/wiki/Random-access_memory). so we won't cache **All** dataset and preprocessed data in RAM.\n",
    "[Every data will processed by inline](https://www.tensorflow.org/guide/data_performance).\n",
    "\n",
    "dataflow will be like picture below: \n",
    "\n",
    "<img src=\"./imgs/without_gpudirect_storage.png\" width='425px' height='450px'>\n",
    "\n",
    "#### Preprocsesing step\n",
    "1. Read image file from NVMe SSD\n",
    "2. Decode image to NumPy arary\n",
    "3. Normalize image by divide array to 255\n",
    "4. Transform 'HW' array into 'CHW' array by Add Channel dimension to index 0 on array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def path_to_label(file_path: Path):\n",
    "    \"\"\"Return label from path\n",
    "\n",
    "    Arg:\n",
    "        file_path (Path): path of file\n",
    "\n",
    "    Return:\n",
    "        label (np.int32): label of file\n",
    "\n",
    "    \"\"\"\n",
    "    \n",
    "    _, label = os.path.split(os.path.dirname(file_path))\n",
    "    return label\n",
    "\n",
    "def path_to_data(file_path: Path):\n",
    "    \"\"\"Return Preprocessed data and from path\n",
    "\n",
    "    Arg:\n",
    "        file_path (Path): path of file\n",
    "\n",
    "    Return:\n",
    "        data (np.int32): preprocessed data of file\n",
    "\n",
    "    \"\"\"\n",
    "\n",
    "    # Read image data and return Numpy Array\n",
    "    data = cv2.imread(file_path, cv2.IMREAD_GRAYSCALE)\n",
    "    # Normalize image\n",
    "    data = data/255.\n",
    "    # Expand dim 0 for batch training\n",
    "    data = np.expand_dims(data, axis=0)\n",
    "\n",
    "    return data\n",
    "\n",
    "class MNISTDataset(tf.data.Dataset):\n",
    "    def _generator(dir_path: Path):\n",
    "        \"\"\"Return data loading generator function\n",
    "\n",
    "        Arg:\n",
    "            dir_path (Path): Directory of dataset\n",
    "\n",
    "        Yields:\n",
    "            data (np.float32): preprocessed data\n",
    "            label (np.int32): label of data\n",
    "        \"\"\"\n",
    "\n",
    "        file_list = glob.glob(os.path.join(dir_path, '*/*.png'))\n",
    "        for file_path in file_list:\n",
    "            data = path_to_data(file_path)\n",
    "            label = path_to_label(file_path)\n",
    "            \n",
    "            yield (data, label)\n",
    "    \n",
    "    def __new__(cls, image_dir:Path):\n",
    "        \"\"\"Return TensorFlow Dataset object from generator function\n",
    "\n",
    "        Arg:\n",
    "            cls (tf.data.Dataset): dataset classitselves\n",
    "            image_dir (Path): Path to images\n",
    "\n",
    "        Return:\n",
    "            dataset (tf.data.Dataset) : dataset from generator\n",
    "        \"\"\"\n",
    "\n",
    "        dataset = tf.data.Dataset.from_generator(\n",
    "            lambda: cls._generator(image_dir),\n",
    "            output_signature=(\n",
    "            tf.TensorSpec(shape=(1, 28, 28), dtype=tf.float32),\n",
    "            tf.TensorSpec(shape=(), dtype=tf.int32),\n",
    "            ),\n",
    "        )\n",
    "        return dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define model, optimizer, loss function\n",
    "\n",
    "This example Task is 'Multi labels classification'. so model would like below.\n",
    "\n",
    "* Model is simple model Based on [Convolutional Layers](https://arxiv.org/abs/1511.08458).\n",
    "* Loss function will be [sparse categorical crossentropy](https://datascience.stackexchange.com/questions/41921/sparse-categorical-crossentropy-vs-categorical-crossentropy-keras-accuracy).\n",
    "* Optimizer will be [AdamW](https://arxiv.org/abs/1711.05101).\n",
    "\n",
    "For convenience, model's performance would be only measured by train set accuracy.\n",
    "\n",
    "#### Model architecture\n",
    "<img src=\"./imgs/model_architecture.png\" width=\"300px\" height=\"500px\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define model\n",
    "model = tf.keras.Sequential([\n",
    "    tf.keras.layers.Conv2D(32, kernel_size=(3,3), input_shape=(1, 28, 28), activation='relu', data_format='channels_first'),\n",
    "    tf.keras.layers.Conv2D(64, kernel_size=(3,3), activation='relu'),\n",
    "    tf.keras.layers.MaxPool2D(pool_size=(2,2)),\n",
    "    tf.keras.layers.Flatten(),\n",
    "    tf.keras.layers.Dense(512, activation='relu'),\n",
    "    tf.keras.layers.Dense(128)\n",
    "])\n",
    "\n",
    "optimizer = tf.keras.optimizers.AdamW(learning_rate=0.001)\n",
    "loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n",
    "metrics = ['accuracy']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compile model and start training\n",
    "\n",
    "Current Training Environment is like below:\n",
    "\n",
    "|Precision|Batch preprocssing|Batch caching|GPU select|GPU memory strategy|\n",
    "|---|---|---|---|---|\n",
    "|[TF32 (TensorFloat-32)](https://blogs.nvidia.com/blog/2020/05/14/tensorfloat-32-precision-format/)|Inline<br>Compute by CPU|Cache only next batch<br>Stored in RAM|Automatically Selected by TensorFlow|As much as Possible<br>[(TensorFlow Default GPU memory strategy)](https://www.tensorflow.org/guide/gpu#limiting_gpu_memory_growth)|\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define epochs, batch size, dataset path\n",
    "epochs = 100\n",
    "batch_size = 2560\n",
    "images_path = './mnist_png/training'\n",
    "\n",
    "# Define dataset for training\n",
    "dataset = MNISTDataset(image_dir=images_path).batch(batch_size).prefetch(tf.data.AUTOTUNE)\n",
    "\n",
    "# Compile model and sent to GPU memory\n",
    "model.compile(optimizer=optimizer,\n",
    "            loss=loss_fn,\n",
    "            metrics=metrics)\n",
    "\n",
    "# Training model\n",
    "model.fit(dataset, epochs=epochs)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tensorflow",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}