doc/getting_started/tutorials/05.persistent-reductions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "79426a2f11e6c3cb",
   "metadata": {},
   "source": [
    "# Persistent reductions and broadcast in Lazy Expressions\n",
    "\n",
    "In this tutorial, we’ll explore Blosc2’s capabilities for lazy computation in Python. We’ll create arrays of various dimensions, operate them using operations like reduction, addition and multiplication, and demonstrate how lazy expressions defer computations to optimize performance.\n",
    "\n",
    "The lazy expression technique is efficient because it postpones the computation of the expression until it is actually needed, removing the need for large temporaries and hence, optimizing memory usage and processing.\n",
    "\n",
    "However, reductions are kind of an exception in computing lazy expressions, as they are always computed eagerly when using regular Python expressions with Blosc2 operands. Fortunately, we can avoid eager computations by using a string version of the expression in combination with the `blosc2.lazyexpr` function. We will show how to create and save a lazy expression, and then compute it to obtain the desired results.\n",
    "\n",
    "We’ll also see how resizing operand arrays is reflected in the results, highlighting the flexibility of lazy expressions.\n",
    "\n",
    "Without further ado, let’s dive into lazy computation, reductions and broadcasting with Blosc2!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8c69fe846b1e13d",
   "metadata": {},
   "source": [
    "## Operands as arrays of different shape\n",
    "\n",
    "We will now create the operands, using different shape for each of them, just for flexing the broadcasting capabilities of lazy expressions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "initial_id",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-01T11:28:01.519612Z",
     "start_time": "2024-11-01T11:28:01.408633Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Array a slice: [[[1 1 1 1]\n",
      "  [1 1 1 1]]\n",
      "\n",
      " [[1 1 1 1]\n",
      "  [1 1 1 1]]]\n",
      "Array b slice: [[2 2 2 2]\n",
      " [2 2 2 2]]\n",
      "Array c slice: [3 3 3 3]\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "import blosc2\n",
    "\n",
    "# Define dimensions of arrays\n",
    "dim_a = (200, 300, 400)  # 3D array\n",
    "dim_b = (200, 400)  # 2D array\n",
    "dim_c = 400  # 1D array\n",
    "\n",
    "# Create arrays with specific dimensions and values\n",
    "a = blosc2.full(dim_a, 1, urlpath=\"a.b2nd\", mode=\"w\")\n",
    "b = blosc2.full(dim_b, 2, urlpath=\"b.b2nd\", mode=\"w\")\n",
    "c = blosc2.full(dim_c, 3, urlpath=\"c.b2nd\", mode=\"w\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a6a6d076255afaf",
   "metadata": {},
   "source": [
    "## Creating, saving and loading a lazy expression\n",
    "\n",
    "First, let's build a string expression that sums the contents of array `a` and operates with the values of `b` by `c`. In this context, creating a string version of the expression is critical; otherwise, reductions should be computed eagerly.\n",
    "\n",
    "Let's see how this works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b8f05b87b99d38ec",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-01T11:28:33.735543Z",
     "start_time": "2024-11-01T11:28:33.679956Z"
    }
   },
   "outputs": [],
   "source": [
    "# Expression that sums all elements of 'a' and multiplies 'b' by 'c'\n",
    "expression = \"a.sum() + b * c\"\n",
    "# Define the operands for the expression\n",
    "operands = {\"a\": a, \"b\": b, \"c\": c}\n",
    "# Create a lazy expression\n",
    "lazy_expression = blosc2.lazyexpr(expression, operands)\n",
    "# Save the lazy expression to the specified path\n",
    "url_path = \"my_expr.b2nd\"\n",
    "lazy_expression.save(urlpath=url_path, mode=\"w\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6226bf8d46ac7c32",
   "metadata": {},
   "source": [
    "In the code above, an expression combining the arrays `a`, `b`, and `c` is expressed in string form: `a.sum() + b ∗ c`.  Then, one builds a lazy expression and save it for later. The expression chosen illustrates how operations automatically adapt to the dimensions of the operands through the concept of broadcasting.\n",
    "\n",
    "**Broadcasting** allows arrays of different shapes (dimensions) to align for mathematical operations, such as addition or multiplication, without the need to enlarge operands by replicating data. The main idea is that smaller dimensions are \"stretched\" to larger dimensions in such a way that allows the operation to be performed consistently.\n",
    "\n",
    "<img src=\"images/lazyexpr-broadcast.png\" width=\"1000\"/>\n",
    "\n",
    "See [NumPy docs on broadcasting](https://numpy.org/doc/stable/user/basics.broadcasting.html) for more information.\n",
    "\n",
    "Now that we have saved the expression, we can open and compute it to obtain the result. Let's see how this is done."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "86b48c7707cea2a7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-01T11:28:43.942911Z",
     "start_time": "2024-11-01T11:28:43.845071Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a.sum() + b * c\n",
      "(200, 400)\n",
      "Time to get shape:0.00004\n",
      "Time to compute:0.05476\n",
      "Result of the operation (slice):\n",
      "[[24000006 24000006 24000006 24000006]\n",
      " [24000006 24000006 24000006 24000006]]\n"
     ]
    }
   ],
   "source": [
    "lazy_expression = blosc2.open(urlpath=url_path)\n",
    "# Print the lazy expression and its shape\n",
    "print(lazy_expression)\n",
    "t1 = time.time()\n",
    "print(lazy_expression.shape)\n",
    "t2 = time.time()\n",
    "print(f\"Time to get shape:{t2 - t1:.5f}\")\n",
    "t1 = time.time()\n",
    "result1 = lazy_expression.compute()\n",
    "t2 = time.time()\n",
    "print(f\"Time to compute:{t2 - t1:.5f}\")\n",
    "print(\"Result of the operation (slice):\")\n",
    "print(result1[:2, :4])  # Print a small slice of the result for demonstration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "362cfd5eb88b9bb6",
   "metadata": {},
   "source": [
    "As we can observe when printing the lazy expression and its shape, the time required to get the `shape` is significantly shorter. This is because `lazy_expression.shape` does not need to compute all the elements of the expression; instead, it only accesses the **metadata** of the operands, from which it is inferred the basic information about the dimensions and type of the result.\n",
    "\n",
    "Thanks to this metadata, if we know the dimensions of the arrays involved in the operation (such as in the case of `a.sum() + b * c`), Blosc2 can **quickly infer the resulting shape** without performing intensive calculations. This allows for fast access to structural information (like the `shape` and `dtype`) without operating on the actual data.\n",
    "\n",
    "In contrast, when we call `lazy_expression.compute()`, all the necessary operations to calculate the final result are executed. Here is where the real computation takes place, and as we can see from the time, this process is significantly longer."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a19ba0d14053d1a0",
   "metadata": {},
   "source": [
    "## Resizing operands of persisted lazy expressions\n",
    "\n",
    "In this section, we will see how persisted lazy expressions automatically adapt to changes in the dimensions and values of the original operands, such as arrays `a` and `b`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "61bcd7d60ec69004",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-01T11:28:52.867743Z",
     "start_time": "2024-11-01T11:28:52.700439Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(300, 400)\n",
      "Time to get shape:0.00010\n",
      "Time to compute:0.06103\n",
      "Result of the operation (slice):\n",
      "[[60000006 60000006 60000006 60000006]\n",
      " [60000006 60000006 60000006 60000006]]\n"
     ]
    }
   ],
   "source": [
    "# Resizing arrays and updating values to see changes in the expression result\n",
    "a.resize((300, 300, 400))\n",
    "a[200:300] = 3\n",
    "b.resize((300, 400))\n",
    "b[200:300] = 5\n",
    "# Open the saved file\n",
    "lazy_expression = blosc2.open(urlpath=url_path)\n",
    "t1 = time.time()\n",
    "print(lazy_expression.shape)\n",
    "t2 = time.time()\n",
    "print(f\"Time to get shape:{t2 - t1:.5f}\")\n",
    "t1 = time.time()\n",
    "result2 = lazy_expression.compute()\n",
    "t2 = time.time()\n",
    "print(f\"Time to compute:{t2 - t1:.5f}\")\n",
    "print(\"Result of the operation (slice):\")\n",
    "print(result2[:2, :4])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d82492bf518c5a39",
   "metadata": {},
   "source": [
    "After increasing the dimensions of the original arrays by modifying the values of `a` and `b`, the lazy expression is reopened. This step is crucial as it allows us to observe how the computation of the expression adapts to the new dimensions. Upon re-opening the expression, we can check that the results now accurately reflect these changes in the dimensions of the array operands. Moreover, see how obtaining the structural information (the `shape`) of the expression is a quick process, requiring only a fraction of the time it takes for the complete computation.\n",
    "\n",
    "This behavior highlights the ability of lazy expressions to adjust to operands using **metadata**, eliminating the need to re-compute each operation from the beginning. Thanks to this approach, notable flexibility and efficiency are achieved in handling arrays of various shapes and sizes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "776fbc7e82d5477f",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "The dynamic adaptation of lazy expressions to changes in the dimensions of array operands illustrates the power of deferred computations in Blosc2. By deferring the computation of expressions until necessary, Blosc2 can quickly access structural information like the `shape` and `dtype`, even when operands **change** on disk, without performing intensive calculations.\n",
    "\n",
    "Also, broadcasting support facilitates working with arrays of different sizes, making the process more powerful and intuitive.\n",
    "\n",
    "Understanding how operations are managed in this context enables developers and data scientists to make the most of reduction and broadcasting capabilities, thereby enhancing the efficiency and effectiveness of their analyses and calculations. The beauty of lazy expressions lies in its ability to simplify the complex and empower our creativity!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6476e6046b5ea76e",
   "metadata": {},
   "source": ""
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}