jupyter/PathFindingTypescript.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2f0eabc4",
   "metadata": {},
   "source": [
    "# Path Finding for Typescript\n",
    "\n",
    "This notebook demonstrates different ways on how path finding algorithms can be utilized for code analysis. \n",
    "\n",
    "Path algorithms in Graphs are famous for e.g. finding the fastest way from one place to another. How can these be applied to static code analysis and how can the results be interpreted?\n",
    "\n",
    "One promising algorithm is [All Pairs Shortest Path](https://neo4j.com/docs/graph-data-science/current/algorithms/all-pairs-shortest-path). It shows dependencies from a different perspective and provides an overview on how directly or indirectly dependencies are connected to each other. The longest shortest path has an additional meaning: It is also known as the [**Graph Diameter**](https://mathworld.wolfram.com/GraphDiameter.html) and is very useful as a metric for the complexity of the Graph (or Subgraphs). The longest path (for directed acyclic graphs) can uncover the longest existing (worst case) dependency chains as long as there are no cycles in the Graph.\n",
    "\n",
    "<br>\n",
    "\n",
    "### References\n",
    "- [jqassistant](https://jqassistant.org)\n",
    "- [Neo4j Python Driver](https://neo4j.com/docs/api/python-driver/current)\n",
    "- [All Pairs Shortest Path](https://neo4j.com/docs/graph-data-science/current/algorithms/all-pairs-shortest-path)\n",
    "- [Longest Path for DAG (neo4j)](https://neo4j.com/docs/graph-data-science/current/algorithms/dag/longest-path)\n",
    "- [Graph Diameter](https://mathworld.wolfram.com/GraphDiameter.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec6baa64",
   "metadata": {},
   "source": [
    "## What GPT-4 has to say about it\n",
    "\n",
    "### All pairs shortest path\n",
    "\n",
    "Interpreting the results of the \"all pairs shortest path\" algorithm on a graph of statically analyzed code modules and their dependencies involves understanding the structure and implications of the paths between nodes (modules) in the graph. Here are some specific steps and insights to consider:\n",
    "\n",
    "1. **Graph Structure**: Each node represents a code module, and edges indicate dependencies. A directed edge from module A to module B implies that A depends on B.\n",
    "\n",
    "2. **Shortest Paths**: The results will give you the shortest path lengths between all pairs of modules. This helps identify:\n",
    "  - **Direct Dependencies**: A length of 1 indicates a direct dependency.\n",
    "  - **Transitive Dependencies**: A length greater than 1 shows indirect dependencies. For example, if the path from A to C is 2, it could mean A → B → C, indicating A indirectly depends on C via B.\n",
    "\n",
    "3. **Module Isolation**: If a module has very long paths to others, it might be more isolated. This could signal potential issues in the code structure, suggesting that module might be overly complex or decoupled from the rest of the system.\n",
    "\n",
    "4. **Critical Paths**: Identify the shortest paths that connect key modules (e.g., core functionalities). These paths can highlight the most crucial dependencies that, if modified, might have extensive impacts on the system.\n",
    "\n",
    "5. **Cycle Detection**: If any pairs have paths that loop back to themselves with a length not equal to 0, it indicates a cycle. Cycles can complicate dependency management, potentially leading to recursive dependencies, which can be problematic in terms of maintainability.\n",
    "\n",
    "6. **Refactoring Opportunities**: By examining the lengths of paths, you might identify modules that could benefit from refactoring to decrease dependency complexity. For example, a module that has dependencies on many others (with longer path lengths) might be a candidate for breaking into smaller, more manageable components.\n",
    "\n",
    "7. **Performance Considerations**: In large systems, long paths could impact performance. If certain modules are far from frequently accessed modules, consider whether they can be optimized for speed.\n",
    "\n",
    "8. **Visual Representation**: Creating a visual representation of the graph with the shortest paths highlighted can be immensely helpful. Tools like Graphviz or D3.js can illustrate these relationships clearly, aiding in your analysis.\n",
    "\n",
    "By focusing on these aspects, you can glean actionable insights from the results of the all pairs shortest path algorithm in the context of your statically analyzed code modules and their dependencies.\n",
    "\n",
    "### Graph diameter (shortest longest path)\n",
    "\n",
    "The longest shortest path in a dependency graph (often referred to in graph theory as the \"diameter\" of the graph) represents the maximum distance (in terms of the number of edges or dependencies) between any two nodes (modules) in the graph. Here’s how you can interpret this metric in the context of statically analyzed code modules and their dependencies:\n",
    "\n",
    "1. **Network Complexity**: The longest shortest path indicates the overall complexity of the network of dependencies. A longer path suggests a more complicated interrelationship among modules. For example, a longest shortest path of 6 could indicate that there is at least one pair of modules in your system that rely on a chain of 6 other modules to communicate or function together.\n",
    "\n",
    "2. **Potential Bottlenecks**: If the longest shortest path is significant, it may suggest potential bottlenecks in your architecture. For instance, if a core module at the beginning of a long path is slow or error-prone, it could affect numerous other modules dependent on it, resulting in systemic performance issues.\n",
    "\n",
    "3. **Critical Communication Points**: The endpoints of the longest shortest path can be seen as critical communication points within your codebase. Understanding these connections can help identify which modules should be prioritized for testing and monitoring, especially during changes.\n",
    "\n",
    "4. **Isolation and Coupling**: A long longest shortest path might indicate that some modules are isolated and far removed from others, which can suggest low cohesion. This can be a sign that the architecture might benefit from refactoring to reduce unnecessary dependencies or to improve modularity.\n",
    "\n",
    "5. **Refactoring Opportunities**: If the longest shortest path is disproportionately long, it may highlight areas in the codebase where modules are too tightly coupled. This situation presents an opportunity for refactoring to create more independent modules or components that can interact with fewer dependencies.\n",
    "\n",
    "6. **Impact of Changes**: Modules that lie along or are endpoints of the longest shortest paths are likely to have a significant impact on the overall system. Changes to them should be approached with caution and accompanied by rigorous testing.\n",
    "\n",
    "7. **Cycle Detection**: In some cases, a long shortest path can indicate the presence of cycles in the graph. If there are paths that seem to loop back on themselves, it suggests potential design flaws that could lead to recursion or infinite loops, complicating maintenance.\n",
    "\n",
    "8. **Architectural Decisions**: The longest shortest path can inform architectural decisions by providing insights into which dependencies might need to be revised or eliminated. For instance, if certain modules are consistently part of the longest path, it could justify investing resources in redesigning their interactions.\n",
    "\n",
    "In summary, interpreting the longest shortest path provides a comprehensive view of the interdependencies among modules in your system, focusing on complexity, potential bottlenecks, and opportunities for improvement in architecture and design.\n",
    "\n",
    "### Longest path\n",
    "\n",
    "1. **Complex Dependencies**: Longest paths indicate modules that have extensive dependencies before reaching another module. For instance, if you find a path like A → B → C → D with a length of 4, it highlights a complex chain of dependencies. This can suggest that changes in module A might have far-reaching implications across the system.\n",
    "\n",
    "2. **Potential Bottlenecks**: Modules located at the beginning of long paths could be performance bottlenecks. If a frequently used module is several layers deep in dependencies (e.g., A → B → C), it may slow down the entire system. Optimizing or refactoring these modules could improve performance.\n",
    "\n",
    "3. **Maintenance Challenges**: Long paths may indicate parts of the code that are difficult to maintain or understand. For example, if module A requires multiple intermediary modules (B, C, D) for its functionality, developers may struggle to trace how changes propagate, leading to potential bugs.\n",
    "\n",
    "4. **Risk of Change**: A module with a long dependency path can be more risky to modify. If A has dependencies on several modules down the line (e.g., A → B → C → D → E), any changes to A could inadvertently affect E, which might be critical or sensitive. This insight can help prioritize testing and reviews around such modules.\n",
    "\n",
    "5. **Decoupling Opportunities**: Identifying the longest paths can highlight areas where you might want to break up dependencies. If there’s a long chain that could be simplified (e.g., by creating intermediary modules or interfaces), it may lead to a more modular and maintainable architecture.\n",
    "\n",
    "6. **Redundancy**: Long paths may also reveal redundancy in dependencies. For instance, if multiple modules depend on a series of others (A → B → C → D and A → B → E), it could indicate unnecessary coupling that can be streamlined.\n",
    "\n",
    "7. **System Understanding**: Long paths can help map out the architecture of your codebase. Understanding which modules are pivotal in long chains can provide insights into the overall design and help inform architectural decisions.\n",
    "\n",
    "8. **Documentation & Knowledge Transfer**: If certain modules consistently appear in the longest paths, they may require better documentation. Ensuring that their roles and the reasons for their lengthy dependencies are well understood can facilitate knowledge transfer among team members.\n",
    "\n",
    "By focusing on the longest paths in your dependency graph, you can uncover areas requiring attention for optimization, maintenance, and improved system architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d19447e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from IPython.display import display\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plot\n",
    "import typing as typ\n",
    "import numpy as np\n",
    "from neo4j import GraphDatabase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "807bba03",
   "metadata": {},
   "outputs": [],
   "source": [
    "#The following cell uses the build-in %html \"magic\" to override the CSS style for tables to a much smaller size.\n",
    "#This is especially needed for PDF export of tables with multiple columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb92ef74",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%html\n",
    "<style>\n",
    "/* CSS style for smaller dataframe tables. */\n",
    ".dataframe th {\n",
    "    font-size: 8px;\n",
    "}\n",
    ".dataframe td {\n",
    "    font-size: 8px;\n",
    "}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "648e2a5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Main Colormap\n",
    "main_color_map = 'nipy_spectral'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e49ca888",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pandas DataFrame Display Configuration\n",
    "pd.set_option('display.max_colwidth', 300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c5dab37",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Please set the environment variable \"NEO4J_INITIAL_PASSWORD\" in your shell \n",
    "# before starting jupyter notebook to provide the password for the user \"neo4j\". \n",
    "# It is not recommended to hardcode the password into jupyter notebook for security reasons.\n",
    "\n",
    "driver = GraphDatabase.driver(uri=\"bolt://localhost:7687\", auth=(\"neo4j\", os.environ.get(\"NEO4J_INITIAL_PASSWORD\")))\n",
    "driver.verify_connectivity()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1db254b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cypher_query_from_file(filename):\n",
    "    with open(filename) as file:\n",
    "        return ' '.join(file.readlines())\n",
    "    \n",
    "\n",
    "def query_cypher_to_data_frame(filename, parameters_: typ.Optional[typ.Dict[str, typ.Any]] = None):\n",
    "    records, summary, keys = driver.execute_query(get_cypher_query_from_file(filename),parameters_=parameters_)\n",
    "    return pd.DataFrame([r.values() for r in records], columns=keys)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d2e62d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_directed_unweighted_projection(parameters: dict) -> bool: \n",
    "    \"\"\"\n",
    "    Creates an directed homogenous unweighted in-memory Graph projection for/with Neo4j Graph Data Science Plugin.\n",
    "    It returns True if there is data available for the given parameter and False otherwise.\n",
    "    Parameters\n",
    "    ----------\n",
    "    dependencies_projection : str\n",
    "        The name prefix for the in-memory projection for dependencies. Example: \"typescript-module-path-finding-notebook\"\n",
    "    dependencies_projection_node : str\n",
    "        The label of the nodes that will be used for the projection. Example: \"Module\"\n",
    "    dependencies_projection_weight_property : str\n",
    "        The name of the node property that contains the dependency weight. Example: \"lowCouplingElement25PercentWeight\"\n",
    "    dependencies_projection_embedding_dimension : str\n",
    "        The number of the dimensions and therefore size of the resulting array of floating point numbers\n",
    "    \"\"\"\n",
    "    \n",
    "    is_data_missing=query_cypher_to_data_frame(\"../cypher/Dependencies_Projection/Dependencies_0_Check_Projectable.cypher\", parameters).empty\n",
    "    if is_data_missing: return False\n",
    "\n",
    "    query_cypher_to_data_frame(\"../cypher/Dependencies_Projection/Dependencies_1_Delete_Projection.cypher\", parameters)\n",
    "    query_cypher_to_data_frame(\"../cypher/Dependencies_Projection/Dependencies_2_Delete_Subgraph.cypher\", parameters)\n",
    "    # To exclude the direction of the relationships use the following line to create the projection:\n",
    "    # query_cypher_to_data_frame(\"../cypher/Dependencies_Projection/Dependencies_4_Create_Undirected_Projection.cypher\", parameters)\n",
    "    query_cypher_to_data_frame(\"../cypher/Dependencies_Projection/Dependencies_3_Create_Projection.cypher\", parameters)\n",
    "    query_cypher_to_data_frame(\"../cypher/Dependencies_Projection/Dependencies_5_Create_Subgraph.cypher\", parameters)\n",
    "    return True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f2e905c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def empty_projection_result() -> pd.DataFrame:\n",
    "    \"\"\"\n",
    "    Creates an empty result and is meant to be used in case there is no data for the projection.\n",
    "    \"\"\"\n",
    "    \n",
    "    print(\"No projected data for path finding available\")\n",
    "    return pd.DataFrame(columns=['totalCost', 'sourceProject', 'sourceScan', 'sourceRootProject', 'isDifferentTargetProject', 'isDifferentTargetScan', 'isDifferentTargetRootProject', 'distance', 'distanceTotalPairCount', 'distanceTotalSourceCount', 'distanceTotalTargetCount', 'nodeCount', 'pairCount'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2d60597",
   "metadata": {},
   "outputs": [],
   "source": [
    "def query_if_data_available(is_data_available: bool, cypher_file_name: str, parameters: dict) -> pd.DataFrame: \n",
    "    \"\"\"\n",
    "    Runs the cypher query in the file given as cypherFileName with the given parameters if the data is available,\n",
    "    or returns an empty result if the projection failed due to missing data.\n",
    "    \n",
    "    is_data_available\n",
    "    ----------\n",
    "    Pass the returned boolean from the projection creation here. Will return an empty result if false and otherwise execute the query.\n",
    "\n",
    "    cypher_file_name\n",
    "    ----------\n",
    "    Name of the file containing the Cypher query that executes node embeddings procedure.\n",
    "\n",
    "    parameters\n",
    "    ----------\n",
    "    dependencies_projection : str\n",
    "        The name prefix for the in-memory projection for dependencies. Example: \"typescript-module-path-finding-notebook\"\n",
    "    dependencies_projection_node : str\n",
    "        The label of the nodes that will be used for the projection. Example: \"Module\"\n",
    "    dependencies_projection_weight_property : str\n",
    "        The name of the node property that contains the dependency weight. Example: \"lowCouplingElement25PercentWeight\"\n",
    "    \"\"\"\n",
    "    \n",
    "    return query_cypher_to_data_frame(cypher_file_name, parameters) if is_data_available else empty_projection_result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ef848fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_total_distance_distribution(data_frame: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"\n",
    "    Returns the DataFrame only containing the overall/total distances, their pair count and the count of the distinct source and target nodes\n",
    "    \n",
    "    data_frame\n",
    "    ----------\n",
    "    Contains the path algorithm result including the columns \"distance\", \"distanceTotalPairCount\", \"distanceTotalSourceCount\" and \"distanceTotalTargetCount\"\n",
    "    \"\"\"\n",
    "    \n",
    "    return data_frame[['distance', 'distanceTotalPairCount', 'distanceTotalSourceCount', 'distanceTotalTargetCount']].drop_duplicates().reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e84f2736",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_longest_path_for_column(column : str, data_frame: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"\n",
    "    Returns the DataFrame grouped by the given column containing their max/longest distance.\n",
    "    \n",
    "    column\n",
    "    ----------\n",
    "    Name of the column to group by. Example: \"sourceProject\".\n",
    "    \n",
    "    data_frame\n",
    "    ----------\n",
    "    Contains the path algorithm result including the columns \"distance\" and the chosen column parameter.\n",
    "    \"\"\"\n",
    "    \n",
    "    return data_frame.groupby(column)['distance'].max().sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "036264ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_distance_distribution_for_each(column : str, data_frame: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"\n",
    "    Returns the transposed pivot of the DataFrame grouped by the given column.\n",
    "    \n",
    "    column\n",
    "    ----------\n",
    "    Name of the column to group by. Example: \"sourceProject\".\n",
    "    \n",
    "    data_frame\n",
    "    ----------\n",
    "    Contains the path algorithm result including the columns \"distance\", \"pairCount\" and the chosen column parameter.\n",
    "    \"\"\"\n",
    "\n",
    "    if data_frame.empty:\n",
    "        return empty_projection_result()\n",
    "    \n",
    "    data_frame = data_frame.copy()\n",
    "\n",
    "    # If not already grouped, group by the given column and the distance and sum up the pair count (=number of paths)\n",
    "    data_frame = data_frame.groupby([column, \"distance\"], as_index=False)[\"pairCount\"].apply(sum)\n",
    "\n",
    "    # The rows of the parameter \"column\" contain the source project (e.g. Java artifact or Typescript project) and their path count.\n",
    "    # The columns contain the distances (length of the paths).\n",
    "    data_frame = data_frame.pivot(index='distance', columns=column, values='pairCount')\n",
    "\n",
    "    # Sort by column sum and then take only the first 40 columns\n",
    "    sum_of_pair_count = data_frame.sum()\n",
    "    data_frame = data_frame[sum_of_pair_count.sort_values(ascending=False).index[:40]]\n",
    "\n",
    "    # Fill missing values with zeroes\n",
    "    data_frame = data_frame.fillna(0)\n",
    "\n",
    "    # Transpose the table (flip columns and rows) to have a row for every column (e.g. \"sourceProject\")\n",
    "    data_frame = data_frame.transpose()\n",
    "\n",
    "    return data_frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de2e71ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "def normalize_distance_distribution_for_each_row(data_frame: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"\n",
    "    Returns the normalized data in percentage of the DataFrame for each row\n",
    "    \n",
    "    data_frame\n",
    "    ----------\n",
    "    Pivot dataframe with source projects in rows, distances in columns and pair count values\n",
    "    \"\"\"\n",
    "    return data_frame.div(data_frame.sum(axis=1), axis=0) * 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27a583e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_total_distances_bar_chart(data_frame: pd.DataFrame, title: str, ylabel: str):\n",
    "    \"\"\"\n",
    "    Plots the total number of pairs for each distance.\n",
    "    \n",
    "    data_frame\n",
    "    ----------\n",
    "    Contains the path algorithm result including the columns \"distance\" and \"distanceTotalPairCount\"\n",
    "\n",
    "    title\n",
    "    ----------\n",
    "    Title of the plot. Example: \"All pairs shortest path for Typescript module dependencies in total (bar)\"\n",
    "\n",
    "    ylabel\n",
    "    ----------\n",
    "    Y-axis label. Example: \"total number of Typescript module pairs\"\n",
    "    \"\"\"\n",
    "    \n",
    "    if data_frame.empty:\n",
    "        print(\"No data to plot '\" + title + \"'\")\n",
    "    else:\n",
    "        data_frame.plot(\n",
    "            title=title,\n",
    "            kind='bar', \n",
    "            grid=True,\n",
    "            x='distance',\n",
    "            y='distanceTotalPairCount',\n",
    "            xlabel='path length',\n",
    "            ylabel=ylabel,\n",
    "            figsize=(9,7),\n",
    "            legend=False,\n",
    "            fontsize=8,\n",
    "            cmap=main_color_map\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e63ddc97",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_total_distances_pie_chart(data_frame: pd.DataFrame, title: str, ylabel: str):\n",
    "    \"\"\"\n",
    "    Plots the total number of pairs for each distance.\n",
    "    \n",
    "    data_frame\n",
    "    ----------\n",
    "    Contains the path algorithm result including the columns \"distance\" and \"distanceTotalPairCount\"\n",
    "\n",
    "    title\n",
    "    ----------\n",
    "    Title of the plot. Example: \"All pairs shortest path for Typescript module dependencies in total (pie)\".\n",
    "\n",
    "    ylabel\n",
    "    ----------\n",
    "    Y-axis label. Example: \"Total number of Typescript module pairs\"\n",
    "    \"\"\"\n",
    "\n",
    "    if data_frame.empty:\n",
    "        print(\"No data to plot '\" + title + \"'\")\n",
    "    else:\n",
    "        explode_all=np.full(data_frame.shape[0], 0.01)\n",
    "        total_number_of_pairs=data_frame['distanceTotalPairCount'].sum()\n",
    "        def custom_auto_percentage_format(percentage):\n",
    "            return '{:1.2f}% ({:.0f})'.format(percentage, total_number_of_pairs*percentage/100)\n",
    "\n",
    "        plot.figure();\n",
    "        data_frame.plot(\n",
    "            title=title, \n",
    "            kind='pie', \n",
    "            labels=data_frame['distance'],\n",
    "            y='distanceTotalPairCount', \n",
    "            ylabel=ylabel,\n",
    "            legend=True,\n",
    "            labeldistance=None,\n",
    "            autopct=custom_auto_percentage_format,\n",
    "            explode=explode_all,\n",
    "            textprops={'fontsize': 8},\n",
    "            pctdistance=1.2,\n",
    "            figsize=(8,8),\n",
    "            cmap=main_color_map\n",
    "        )\n",
    "        plot.legend(bbox_to_anchor=(1.05, 1), loc='upper left', title='path length')\n",
    "        plot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7243fbfd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_longest_distance_of_each_row(data_frame: pd.DataFrame, title: str, xlabel: str, ylabel: str):\n",
    "    \"\"\"\n",
    "    Plots the longest distance per source project \n",
    "    \n",
    "    data_frame\n",
    "    ----------\n",
    "    Contains the path algorithm result pivot table including the columns \"distance\"\n",
    "\n",
    "    title\n",
    "    ----------\n",
    "    Title of the plot. Example: 'Longest shortest path (\"diameter\") for Typescript module dependencies per project'\n",
    "\n",
    "    xlabel\n",
    "    ----------\n",
    "    X-axis label. Example: \"Project\".\n",
    "    \n",
    "    ylabel\n",
    "    ----------\n",
    "    Y-axis label. Example: \"Typescript module pairs\".    \n",
    "    \"\"\"\n",
    "\n",
    "    if data_frame.empty:\n",
    "        print(\"No data to plot '\" + title + \"'\")\n",
    "    else:\n",
    "        plot.figure();\n",
    "        axes = data_frame.transpose().plot(\n",
    "            title=title, \n",
    "            kind='bar', \n",
    "            grid=True,\n",
    "            xlabel=xlabel,\n",
    "            ylabel=ylabel,\n",
    "            figsize=(10,6),\n",
    "            legend=False,\n",
    "            cmap=main_color_map\n",
    "        )\n",
    "        plot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5262a4ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_stacked_distances_for_each_row(data_frame: pd.DataFrame, title: str, xlabel: str, ylabel: str, logy: bool = False):\n",
    "    \"\"\"\n",
    "    Plots each source project (e.g. Java artifact or Typescript project) stacked by distance (number of dependency pairs)\n",
    "    \n",
    "    data_frame\n",
    "    ----------\n",
    "    Contains the output of \"get_distance_distribution_for_each\" with e.g. the \n",
    "    source projects as rows, distances as columns and number of pairs as values.\n",
    "\n",
    "    title\n",
    "    ----------\n",
    "    Title of the plot. Example: \"Longest shortest path (\"diameter\") for Typescript packages \".\n",
    "\n",
    "    xlabel\n",
    "    ----------\n",
    "    X-axis label. Example: \"Project\".\n",
    "    \n",
    "    ylabel\n",
    "    ----------\n",
    "    Y-axis label. Example: \"Typescript module pairs\".\n",
    "    \"\"\"\n",
    "\n",
    "    if data_frame.empty:\n",
    "        print(\"No data to plot '\" + title + \"'\")\n",
    "    else:\n",
    "        plot.figure();\n",
    "        axes = data_frame.plot(\n",
    "            title=title, \n",
    "            kind='bar', \n",
    "            grid=True,\n",
    "            stacked=True,\n",
    "            logy=logy,\n",
    "            xlabel=xlabel,\n",
    "            ylabel=ylabel,\n",
    "            figsize=(10,6),\n",
    "            legend=True,\n",
    "            cmap=main_color_map\n",
    "        ).legend(bbox_to_anchor=(1.0, 1.0), title=\"path length\")\n",
    "        plot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6afb912b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def group_to_others_below_threshold(data_frame : pd.DataFrame, value_column : str, name_column: str, threshold: float) -> pd.DataFrame:    \n",
    "    \"\"\"\n",
    "    Adds a new percentage column for the value column and \n",
    "    groups all values below the given threshold to \"others\" in the name column.\n",
    "\n",
    "    Parameters:\n",
    "    - data_frame (pd.DataFrame): Input pandas DataFrame\n",
    "    - value_column (str): Name of the column that contains the numeric value\n",
    "    - name_column (str): Name of the column that contains the group name that will be replaced by \"others\" for small values\n",
    "    - threshold (float): Threshold in % that is used to group values below it into the \"others\" group\n",
    "\n",
    "    Returns:\n",
    "    int:Returning value\n",
    "\n",
    "    \"\"\"\n",
    "    result_data_frame = data_frame.copy();\n",
    "\n",
    "    percent_column_name = value_column + 'Percent';\n",
    "\n",
    "    # Add column with the name given in \"percent_column_name\" with the percentage of the value column.\n",
    "    result_data_frame[percent_column_name] = result_data_frame[value_column] / result_data_frame[value_column].sum() * 100.0;\n",
    "\n",
    "    # Change the external module name to \"others\" if it is called less than the specified threshold\n",
    "    result_data_frame.loc[result_data_frame[percent_column_name] < threshold, name_column] = 'others';\n",
    "\n",
    "    # Group external module name (foremost the new \"others\" entries) and sum their percentage\n",
    "    result_data_frame = result_data_frame.groupby(name_column)[percent_column_name].sum();\n",
    "\n",
    "    # Sort by values descending\n",
    "    return result_data_frame.sort_values(ascending=False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40653fe5",
   "metadata": {},
   "outputs": [],
   "source": [
    "def group_to_others_below_threshold(data_frame : pd.DataFrame, value_column : str, name_column: str, threshold: float) -> pd.DataFrame:    \n",
    "    \"\"\"Adds a new percentage column for the value column and \n",
    "    groups all values below the given threshold to \"others\" in the name column.\n",
    "\n",
    "    Parameters:\n",
    "    - data_frame (pd.DataFrame): Input pandas DataFrame\n",
    "    - value_column (str): Name of the column that contains the numeric value\n",
    "    - name_column (str): Name of the column that contains the group name that will be replaced by \"others\" for small values\n",
    "    - threshold (float): Threshold in % that is used to group values below it into the \"others\" group\n",
    "\n",
    "    Returns:\n",
    "    int:Returning value\n",
    "\n",
    "    \"\"\"\n",
    "    result_data_frame = data_frame.copy();\n",
    "\n",
    "    percent_column_name = value_column + 'Percent';\n",
    "\n",
    "    # Add column with the name given in \"percent_column_name\" with the percentage of the value column.\n",
    "    result_data_frame[percent_column_name] = result_data_frame[value_column] / result_data_frame[value_column].sum() * 100.0;\n",
    "\n",
    "    # Change the external module name to \"others\" if it is called less than the specified threshold\n",
    "    result_data_frame.loc[result_data_frame[percent_column_name] < threshold, name_column] = 'others';\n",
    "\n",
    "    # Group external module name (foremost the new \"others\" entries) and sum their percentage\n",
    "    result_data_frame = result_data_frame.groupby(name_column)[percent_column_name].sum();\n",
    "\n",
    "    # Sort by values descending\n",
    "    return result_data_frame.sort_values(ascending=False);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b42163d",
   "metadata": {},
   "source": [
    "## 1. Typescript Modules"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e1ce698",
   "metadata": {},
   "source": [
    "## 1.1 All pairs shortest path\n",
    "\n",
    "Use \"[All Pairs Shortest Path](https://neo4j.com/docs/graph-data-science/current/algorithms/all-pairs-shortest-path)\" algorithm to get the shortest path distance between all pairs of dependent Typescript packages. It shows how many Packages have a direct dependency (distance 1), how many are reachable with one dependency in between (distance 2), and so on..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b468bae",
   "metadata": {},
   "source": [
    "### 1.1.1 Create a projection of all Typescript module dependencies\n",
    "\n",
    "Creates a in-memory projection of \"TS:Module\" nodes and their \"DEPENDS_ON\" relationships as a preparation to run the Graph algorithms. The weight property is not used for now (September 2024) but may be needed for other algorithms/variants some time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ecc41b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "module_path_finding_parameters={\n",
    "    \"dependencies_projection\": \"typescript-module-path-finding-notebook\",\n",
    "    \"dependencies_projection_node\": \"Module\",\n",
    "    \"dependencies_projection_weight_property\": \"lowCouplingElement25PercentWeight\",\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b637ce2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the directed and unweighted projection with the parameters defined directly above\n",
    "is_module_data_available=create_directed_unweighted_projection(module_path_finding_parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaf7a594",
   "metadata": {},
   "source": [
    "#### Projected Graph statistics for Typescript module dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf414643",
   "metadata": {},
   "outputs": [],
   "source": [
    "module_projection_statistics=query_cypher_to_data_frame(\"../cypher/Dependencies_Projection/Dependencies_12_Get_Projection_Statistics.cypher\", module_path_finding_parameters)\n",
    "module_projection_statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "924c9b26",
   "metadata": {},
   "source": [
    "### 1.1.2 All pairs shortest path in total\n",
    "\n",
    "First, we'll have a look at the overall/total result of the all pairs shortest path algorithm for all dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62f50f28",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Execute algorithm \"All pairs shortest path\" and query overall and project specific results\n",
    "all_pairs_shortest_paths_distribution_per_project_and_root_project=query_if_data_available(is_module_data_available, \"../cypher/Path_Finding/Path_Finding_5_All_pairs_shortest_path_distribution_per_project.cypher\", module_path_finding_parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6f93160",
   "metadata": {},
   "source": [
    "#### All pairs shortest path in total - Longest shortest path (Graph diameter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce1621b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "module_dependencies_graph_diameter=all_pairs_shortest_paths_distribution_per_project_and_root_project['distance'].max()\n",
    "print('The diameter (longest shortest path) of the projected module dependencies Graph is:', module_dependencies_graph_diameter)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e502afa1",
   "metadata": {},
   "source": [
    "#### All pairs shortest path in total - Path count per length - Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9ebcaa9",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_pairs_shortest_paths_distribution_per_project_in_total=get_total_distance_distribution(all_pairs_shortest_paths_distribution_per_project_and_root_project)\n",
    "all_pairs_shortest_paths_distribution_per_project_in_total.head(50)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df87dc06",
   "metadata": {},
   "source": [
    "#### All pairs shortest path in total - Path count per length - Bar chart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fee79fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_total_distances_bar_chart(\n",
    "    data_frame=all_pairs_shortest_paths_distribution_per_project_in_total, \n",
    "    title='All pairs shortest path for Typescript module dependencies in total (bar)',\n",
    "    ylabel='total number of Typescript module paths'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf4b2733",
   "metadata": {},
   "source": [
    "#### All pairs shortest path in total - Path count per length - Pie chart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab444978",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_total_distances_pie_chart(\n",
    "    data_frame=all_pairs_shortest_paths_distribution_per_project_in_total,\n",
    "    # data_frame=all_pairs_shortest_paths_distribution_per_project_in_total_significant,\n",
    "    title='All pairs shortest path for Typescript module dependencies in total (pie)',\n",
    "    ylabel='total number of Typescript module paths'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1129f77",
   "metadata": {},
   "source": [
    "### 1.1.3 All pairs shortest path in detail\n",
    "\n",
    "The following table shows the first 10 rows with all details of the query above. It contains the results of the \"all pairs shortest path\" algorithm including the project the source node belong to and if the target node is the same or not. The main intuition here is to show how the data is structured. It provides the basis for tables and charts shown in following sections below, that filter and group the data accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64ab393b",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_pairs_shortest_paths_distribution_per_project_and_root_project.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae9d8868",
   "metadata": {},
   "source": [
    "### 1.1.4 All pairs shortest path for each project\n",
    "\n",
    "In this section we'll focus only on pairs of nodes that both belong to the same project, filtering out every line that has `isDifferentTargetProject==False`. The first ten rows are shown in a table followed by charts that show the distribution of shortest path distances across different projects in stacked bar charts (absolute and normalized).\n",
    "\n",
    "**Note:** It is possible that a (shortest) path could have nodes in between that belong to different projects. Therefore, the data of each project isn't perfectly isolated. However, it shows how the dependencies interact across projects \"in real life\" while still providing a decent isolation of each project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f84af34",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_pairs_shortest_paths_distribution_per_project_isolated=all_pairs_shortest_paths_distribution_per_project_and_root_project.query('isDifferentTargetProject == False')\n",
    "all_pairs_shortest_paths_distribution_per_project_isolated.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c400fc8c",
   "metadata": {},
   "source": [
    "#### All pairs shortest path for each project - Longest shortest path (Diameter) for each project\n",
    "\n",
    "Shows the top 20 projects with the longest shortest path (=Graph Diameter)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bc3b27d",
   "metadata": {},
   "outputs": [],
   "source": [
    "graph_diameter_per_project = get_longest_path_for_column('sourceProject', all_pairs_shortest_paths_distribution_per_project_isolated)\n",
    "graph_diameter_per_project.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c65fd16",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_longest_distance_of_each_row(\n",
    "    data_frame=graph_diameter_per_project,\n",
    "    title='Longest shortest path (\"diameter\") for Typescript module dependencies per project',\n",
    "    xlabel='Project',\n",
    "    ylabel='longest path length'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1138de0c",
   "metadata": {},
   "source": [
    "#### All pairs shortest path for each project - Bar chart (absolute)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b90cfbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_pairs_shortest_paths_distribution_per_project_isolated_pivot = get_distance_distribution_for_each('sourceProject', all_pairs_shortest_paths_distribution_per_project_isolated)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebeaf685",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_stacked_distances_for_each_row(\n",
    "    data_frame=all_pairs_shortest_paths_distribution_per_project_isolated_pivot,\n",
    "    title='All pairs shortest path for Typescript module dependencies stacked per project (absolute, logarithmic)',\n",
    "    xlabel='Project',\n",
    "    ylabel='Typescript module paths',\n",
    "    logy=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5f8f817",
   "metadata": {},
   "source": [
    "#### All pairs shortest path for each project - Bar chart (normalized)\n",
    "\n",
    "Shows the top 50 projects with the highest number of dependency paths stacked by their length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "533d1b9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Normalize data (percent of sum pairs)\n",
    "all_pairs_shortest_paths_distribution_per_project_isolated_normalized_pivot=normalize_distance_distribution_for_each_row(all_pairs_shortest_paths_distribution_per_project_isolated_pivot)\n",
    "all_pairs_shortest_paths_distribution_per_project_isolated_normalized_pivot.head(50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2721c41d",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_stacked_distances_for_each_row(\n",
    "    data_frame=all_pairs_shortest_paths_distribution_per_project_isolated_normalized_pivot.head(50),\n",
    "    title='All pairs shortest path for Typescript module dependencies stacked per project (normalized in %)',\n",
    "    xlabel='Project',\n",
    "    ylabel='Typescript module paths'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f76616b7",
   "metadata": {},
   "source": [
    "### 1.1.5 All pairs shortest path for each root project\n",
    "\n",
    "In this section we'll focus only on pairs of nodes that both belong to the same root project, filtering out every line that has `isDifferentTargetRootProject==False`. The first ten rows are shown in a table followed by charts that show the distribution of shortest path distances across different root projects in stacked bar charts (absolute and normalized).\n",
    "\n",
    "**Note:** It is possible that a (shortest) path could have nodes in between that belong to different root projects. Therefore, the data of each root project isn't perfectly isolated. However, it shows how the dependencies interact across root projects \"in real life\" while still providing a decent isolation of each root project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e551ada6",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_pairs_shortest_paths_distribution_per_root_project_isolated=all_pairs_shortest_paths_distribution_per_project_and_root_project.query('isDifferentTargetRootProject == False')\n",
    "\n",
    "all_pairs_shortest_paths_distribution_per_root_project_isolated.\\\n",
    "    groupby([\"sourceRootProject\", \"distance\"], as_index=False)\\\n",
    "    [[\"pairCount\", \"sourceNodeCount\",\"targetNodeCount\"]].\\\n",
    "    apply(max).head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f9defa3",
   "metadata": {},
   "source": [
    "#### All pairs shortest path for each root project - Longest shortest path (Diameter) for each root project\n",
    "\n",
    "Shows the top 20 root projects with the longest shortest path (=Graph Diameter)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9c57989",
   "metadata": {},
   "outputs": [],
   "source": [
    "graph_diameter_per_root_project = get_longest_path_for_column('sourceRootProject', all_pairs_shortest_paths_distribution_per_root_project_isolated)\n",
    "graph_diameter_per_root_project.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd33364f",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_longest_distance_of_each_row(\n",
    "    data_frame=graph_diameter_per_root_project,\n",
    "    title='Longest shortest path (\"diameter\") for Typescript module dependencies per root project',\n",
    "    xlabel='root project',\n",
    "    ylabel='longest path length'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0551c259",
   "metadata": {},
   "source": [
    "#### All pairs shortest path for each root project - Bar chart (absolute)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64f86f9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_pairs_shortest_paths_distribution_per_root_project_isolated_pivot = get_distance_distribution_for_each('sourceRootProject', all_pairs_shortest_paths_distribution_per_root_project_isolated)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d9e4e55",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_stacked_distances_for_each_row(\n",
    "    data_frame=all_pairs_shortest_paths_distribution_per_root_project_isolated_pivot,\n",
    "    title='All pairs shortest path for Typescript module dependencies stacked per root project (absolute, logarithmic)',\n",
    "    xlabel='Root project',\n",
    "    ylabel='Typescript module paths',\n",
    "    logy=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bb61fa8",
   "metadata": {},
   "source": [
    "#### All pairs shortest path for each root project - Bar chart (normalized)\n",
    "\n",
    "Shows the top 50 root projects with the highest number of dependency paths stacked by their length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d09379a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Normalize data (percent of sum pairs)\n",
    "all_pairs_shortest_paths_distribution_per_root_project_isolated_normalized_pivot=normalize_distance_distribution_for_each_row(all_pairs_shortest_paths_distribution_per_project_isolated_pivot)\n",
    "all_pairs_shortest_paths_distribution_per_root_project_isolated_normalized_pivot.head(50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a222ac47",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_stacked_distances_for_each_row(\n",
    "    data_frame=all_pairs_shortest_paths_distribution_per_root_project_isolated_normalized_pivot.head(50),\n",
    "    title='All pairs shortest path for Typescript module dependencies stacked per root project (normalized in %)',\n",
    "    xlabel='root project',\n",
    "    ylabel='Typescript module paths'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37da9d90",
   "metadata": {},
   "source": [
    "## 1.2 Longest path\n",
    "\n",
    "Use [Longest Path](https://neo4j.com/docs/graph-data-science/current/algorithms/dag/longest-path) algorithm to get the longest paths between Typescript packages. It is typically higher than the longest shortest path (diameter) and helps together with it to get a good overview of the complexity.\n",
    "\n",
    "**Note:** This algorithm requires a Directed Acyclic Graph (DAG) and will lead to inaccurate results when the Graph contains cycles."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9f9b8bb",
   "metadata": {},
   "source": [
    "### 1.2.1 Longest path in total\n",
    "\n",
    "First, we'll have a look at the overall/total result of the longest path algorithm for all dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b43628b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Execute algorithm \"longest path (for directed acyclic graphs)\" and query overall and project specific results\n",
    "longest_paths_distribution_per_project=query_if_data_available(is_module_data_available, \"../cypher/Path_Finding/Path_Finding_6_Longest_paths_distribution_per_project.cypher\", module_path_finding_parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaa7ae83",
   "metadata": {},
   "source": [
    "#### Longest path in total - Max longest path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6aef9c47",
   "metadata": {},
   "outputs": [],
   "source": [
    "module_dependencies_max_longest_path=longest_paths_distribution_per_project['distance'].max()\n",
    "print('The max. longest path of the projected module dependencies is:', module_dependencies_max_longest_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cf188a8",
   "metadata": {},
   "source": [
    "#### Longest path in total - Paths per length - Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4141d7d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, display only the overall/total distances, their pair count and the count of the distinct source and target nodes\n",
    "longest_paths_distribution_per_project_in_total=get_total_distance_distribution(longest_paths_distribution_per_project)\n",
    "longest_paths_distribution_per_project_in_total.head(50)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46544c70",
   "metadata": {},
   "source": [
    "#### Longest path in total - Path count per length - Bar chart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9db65656",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_total_distances_bar_chart(\n",
    "    data_frame=longest_paths_distribution_per_project_in_total, \n",
    "    title='Longest path for Typescript module dependencies in total (bar)',\n",
    "    ylabel='total number of Typescript module paths'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4962464",
   "metadata": {},
   "source": [
    "#### Longest path in total - Path count per length - Pie chart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf2efa87",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_total_distances_pie_chart(\n",
    "    data_frame=longest_paths_distribution_per_project_in_total,\n",
    "    title='Longest path for Typescript module dependencies in total (pie)',\n",
    "    ylabel='total number of Typescript module paths'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "904db536",
   "metadata": {},
   "source": [
    "### 1.2.2 Longest path in detail\n",
    "\n",
    "The following table shows the first 10 rows with all details of the query above. It contains the results of the \"longest path\" algorithm including the project the source node belongs to and if the target node is in the same project or not. The main intuition is to show how the data is structured. It provides the basis for tables and charts shown in following sections below, that filter and group the data accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f13cf0e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "longest_paths_distribution_per_project.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57aee73a",
   "metadata": {},
   "source": [
    "### 1.2.3 Longest path for each project\n",
    "\n",
    "In this section we'll focus only on pairs of nodes that both belong to the same project, filtering out every line that has `isDifferentTargetProject==False`. The first ten rows are shown in a table followed by charts that show the distribution of longest path distances across different projects in stacked bar charts (absolute and normalized).\n",
    "\n",
    "**Note:** It is possible that a (longest) path could have nodes in between that belong to different projects. Therefore, the data of each project isn't perfectly isolated. However, it shows how the dependencies interact across projects \"in real life\" while still providing a decent isolation of each project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "964bff35",
   "metadata": {},
   "outputs": [],
   "source": [
    "longest_paths_distribution_per_project_isolated=longest_paths_distribution_per_project.query('isDifferentTargetProject == False')\n",
    "longest_paths_distribution_per_project_isolated.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34ec57d5",
   "metadata": {},
   "source": [
    "#### Longest path for each project - Max. longest path for each project\n",
    "\n",
    "Shows the top 20 projects with their max. longest path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c09f2926",
   "metadata": {},
   "outputs": [],
   "source": [
    "longest_path_per_project = get_longest_path_for_column('sourceProject', longest_paths_distribution_per_project_isolated)\n",
    "longest_path_per_project.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "874f20b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_longest_distance_of_each_row(\n",
    "    data_frame=longest_path_per_project,\n",
    "    title='Max. longest path for Typescript module dependencies per project',\n",
    "    xlabel='Project',\n",
    "    ylabel='max. longest path length'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0c2d8ba",
   "metadata": {},
   "source": [
    "#### Longest path for each project - Bar chart (absolute)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47bd1a08",
   "metadata": {},
   "outputs": [],
   "source": [
    "longest_paths_distribution_per_project_isolated_pivot = get_distance_distribution_for_each('sourceProject', longest_paths_distribution_per_project_isolated)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b9c50a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_stacked_distances_for_each_row(\n",
    "    data_frame=longest_paths_distribution_per_project_isolated_pivot,\n",
    "    title='Longest path for Typescript module dependencies stacked per project (absolute, logarithmic)',\n",
    "    xlabel='Project',\n",
    "    ylabel='Typescript module paths',\n",
    "    logy=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc4c2036",
   "metadata": {},
   "source": [
    "#### Longest path for each project - Bar chart (normalized)\n",
    "\n",
    "Shows the top 50 projects with the highest number of dependency paths stacked by their length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9219023",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Normalize data (percent of sum pairs)\n",
    "longest_paths_distribution_per_project_isolated_normalized_pivot=normalize_distance_distribution_for_each_row(longest_paths_distribution_per_project_isolated_pivot)\n",
    "longest_paths_distribution_per_project_isolated_normalized_pivot.head(50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e2e15eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_stacked_distances_for_each_row(\n",
    "    data_frame=longest_paths_distribution_per_project_isolated_normalized_pivot.head(50),\n",
    "    title='Longest path for Typescript module dependencies stacked per project (normalized in %)',\n",
    "    xlabel='Project',\n",
    "    ylabel='Typescript module paths'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b76e3a9c",
   "metadata": {},
   "source": [
    "### 1.2.4 Longest path for each root project\n",
    "\n",
    "In this section we'll focus only on pairs of nodes that both belong to the same root project, filtering out every line that has `isDifferentTargetRootProject==False`. The first ten rows are shown in a table followed by charts that show the distribution of longest path distances across different root projects in stacked bar charts (absolute and normalized).\n",
    "\n",
    "**Note:** It is possible that a (longest) path could have nodes in-between that belong to different root projects. Therefore, the data of each root project isn't perfectly isolated. However, it shows how the dependencies interact across root projects \"in real life\" while still providing a decent amount of isolation of each root project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45439d79",
   "metadata": {},
   "outputs": [],
   "source": [
    "longest_paths_distribution_per_root_project_isolated=longest_paths_distribution_per_project.query('isDifferentTargetRootProject == False')\n",
    "longest_paths_distribution_per_root_project_isolated.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6c0e349",
   "metadata": {},
   "source": [
    "#### Longest path for each root project - Max. longest path for each root project\n",
    "\n",
    "Shows the top 20 root projects with their max. longest path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "236ec07d",
   "metadata": {},
   "outputs": [],
   "source": [
    "longest_path_per_root_project = get_longest_path_for_column('sourceRootProject', longest_paths_distribution_per_root_project_isolated)\n",
    "longest_path_per_root_project.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37215d21",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_longest_distance_of_each_row(\n",
    "    data_frame=longest_path_per_root_project,\n",
    "    title='Max. longest path for Typescript module dependencies per root project',\n",
    "    xlabel='Module',\n",
    "    ylabel='max. longest path length'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8622d0f2",
   "metadata": {},
   "source": [
    "#### Longest path for each root project - Bar chart (absolute)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f70894c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "longest_paths_distribution_per_root_project_isolated_pivot = get_distance_distribution_for_each('sourceRootProject', longest_paths_distribution_per_root_project_isolated)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "675b71b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_stacked_distances_for_each_row(\n",
    "    data_frame=longest_paths_distribution_per_root_project_isolated_pivot,\n",
    "    title='Longest path for Typescript module dependencies stacked per root project (absolute, logarithmic)',\n",
    "    xlabel='Module',\n",
    "    ylabel='Typescript module paths',\n",
    "    logy=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47bb4916",
   "metadata": {},
   "source": [
    "#### Longest path for each root project - Bar chart (normalized)\n",
    "\n",
    "Shows the top 50 root projects with the highest number of dependency paths stacked by their length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4426a2f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Normalize data (percent of sum pairs)\n",
    "longest_paths_distribution_per_root_project_isolated_normalized_pivot=normalize_distance_distribution_for_each_row(longest_paths_distribution_per_root_project_isolated_pivot)\n",
    "longest_paths_distribution_per_root_project_isolated_normalized_pivot.head(50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4d85859",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_stacked_distances_for_each_row(\n",
    "    data_frame=longest_paths_distribution_per_root_project_isolated_normalized_pivot.head(50),\n",
    "    title='Longest path for Typescript module dependencies stacked per root project (normalized in %)',\n",
    "    xlabel='Root project',\n",
    "    ylabel='Typescript module paths'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82025919",
   "metadata": {},
   "source": [
    "## 3. Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "701bcb1c",
   "metadata": {},
   "source": [
    "### 3.1 Typescript modules summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6d16c65",
   "metadata": {},
   "outputs": [],
   "source": [
    "module_path_algorithm_summary_values = {\n",
    "    'count': module_projection_statistics.nodeCount[0] if not module_projection_statistics.empty else 0, \n",
    "    'degree density': module_projection_statistics.density[0] if not module_projection_statistics.empty else 0, \n",
    "    'degree median': module_projection_statistics['degreeDistribution.p50'][0] if not module_projection_statistics.empty else 0,\n",
    "    'degree max': module_projection_statistics['degreeDistribution.max'][0] if not module_projection_statistics.empty else 0,\n",
    "    'longest shortest path (diameter)': module_dependencies_graph_diameter,\n",
    "    'max. longest path': module_dependencies_max_longest_path\n",
    "}\n",
    "module_path_algorithm_summary = pd.DataFrame.from_records([module_path_algorithm_summary_values])\n",
    "module_path_algorithm_summary"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "JohT"
   }
  ],
  "code_graph_analysis_pipeline_data_validation": "ValidateTypescriptModuleDependencies",
  "kernelspec": {
   "display_name": "codegraph",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  },
  "title": "Path finding algorithms for Typescript module dependencies with Neo4j"
 },
 "nbformat": 4,
 "nbformat_minor": 5
}