Reorganize folders of DataScienceTutorials.jl

README.md

@@ -6,7 +6,7 @@ This repository contains the source code for a [set of tutorials](https://juliaa
 
 You can read the tutorials [online](https://juliaai.github.io/DataScienceTutorials.jl/).
 
-You can find a runnable script for each tutorial at the top of each tutorial page along with a `Project.toml` and a `Manifest.toml` you can use to re-create the exact environment that was used to run the tutorial.
+You can find a runnable script for each tutorial linked at the top of each tutorial page along with a `Project.toml` and a `Manifest.toml` you can use to re-create the exact environment that was used to run the tutorial.
 
 To do so, save both files in an appropriate folder, start Julia, `cd` to the folder and
 
@@ -23,57 +23,85 @@ Pkg.instantiate()
 The rest of these instructions assume that you've cloned the package and have `cd` to it.
 
 ### Structure
-
-All tutorials correspond to a Literate script that's in `_literate/`.
+The following are the folders relevant to pages on the website:
+```
+├── _literate
+│   ├── data             # has "Data Basics" tutorials
+│   ├── getting-started  # has "Getting Started" tutorials
+│   ├── isl              # has "Introduction to Statistical Learning" tutorials
+│   ├── end-to-end       # has "End-to-End" tutorials
+│   └── advanced         # has "Advanced" tutorials
+├── data                 # This and the four folders below import content from "_literate" to the website 
+├── getting-started
+├── isl
+├── end-to-end
+├── advanced
+├── info                 # has markdown files corresponding to info pages
+├── index.md             # has markdown for the landing page
+├── search.md            # has markdown for the search page
+├── routes.json          # has all the navigation bar data
+├── collapse-script.jl   # script that adds dropdowns to tutorials
+├── deploy.jl            # deployment script
+├── Manifest.toml
+└── Project.toml         # Project's environment
+```
+To understand the rest of the structure which could help you change styles with CSS or add interaction with JavaScript read the relevant page on [Franklin's documentation](https://franklinjl.org/workflow/)
 
 ### Fixing an existing tutorial
 
-Find the corresponding script, fix it in a PR.
+Find the corresponding Julia script in `_literate` and fix it in a PR.
 
 ### Add a new tutorial
 
-* Duplicate the folder `EX-wine`.
-* Change its name:
-  * `EX-somename` for an "end-to-end" tutorial `somename`
-  * `A-somename` for a "getting started" tutorial `somename`
-  * `D0-somename` for a "data" tutorial `somename`
-  * `ISL-lab-x` for an "Introduction to Statistical Learning" tutorial
+* Go to the appropriate folder inside `_literate` depending on the category of the tutorial as described above
+* Duplicate one of the tutorials as a starting point.
 * Remove `Manifest.toml` and `Project.toml`
-* Activate that folder and add the packages that you'll need (MLJ, ...)
+* Create and activate an environment in that folder and add the packages that you'll need (MLJ, ...)
 * Write your tutorial following the blueprint
+* Run `julia collapse-script.jl` to add necessary Franklin syntax to your tutorial to make sections in it collapsible like other tutorials
+
+**Note**: your tutorial **must** "just work" otherwise it will be ignored, in other words, any Julia user should be able to just copy the folder containing your `.jl` and `.toml` files, and run it without having to do anything special.
 
-**Note**: your tutorial **must** "just work" otherwise it will be ignored, in other words, we should be able to just copy the folder containing your `.jl` and `.toml` files, and run it without having to do anything special.
+Once all that's done, the remaining things to do are to create the HTML page and a link in the appropriate location. Let's assume you wanted to add an E2E tutorial "Dinosaurs" then this implies that `_literate/end-to-end/dinosaurs.jl` exists and you would:
 
-Once all that's done, the remaining things to do are to create the HTML page and a link in the appropriate location. Let's assume you wanted to add an E2E tutorial "Dinosaurs" then in the previous step you'd have `EX-dinosaurs` and you would
+* Create a file `dinosaurs.md` in the top-level folder `end-to-end/` by duplicating the `end-to-end/wine.md` and changing the reference in it to `\tutorial{end-to-end/dinosaurs}`
+* Add a link pointing to that tutorial in `routing.json` following the template so your tutorial shows in the navigation bar
+* Ensure your tutorials renders correctly by running the following:
+```julia
+cd("path/to/DataScienceTutorials")
 
-* create a file `dinosaurs.md` in `end-to-end/` by duplicating the `end-to-end/wine.md` and changing the reference in it to `\tutorial{EX-dinosaurs}`
-* add a link pointing to that tutorial in `_libs/nav/head.js` following the template so your tutorial shows in the navigation bar
-* lastly, to make sections in your tutorial collapsible like other tutorials run the `collapse-script.jl` file via `julia collapse-script.jl`
+using Pkg
+Pkg.activate(".")
+Pkg.instantiate()
 
+using Franklin
+serve()                 # serve the website locally
+```
+This may generate some files under `__site`. Don't push them in your PR.
 
 ### Publishing updates
 
 **Assumptions**:
 
-* you have a PR with changes, someone has reviewed them and they got merged into the main branch
+* you have a PR with changes, someone has reviewed them and they got merged into the master branch
 
-* Be sure the version of Julia declared near the top of `index.md`
+* Be sure the version of Julia declared [here](https://juliaai.github.io/DataScienceTutorials.jl/how-to-run-code/)
   matches the version used to generate the web-site (which should
   match the version declared in each tutorial's Manifest.toml file)
 
 
-**Once the changes are in the main branch**:
-
-* run `cd("path/to/DataScienceTutorials"); using Franklin` to launch Franklin
-* run `serve(single=true, verb=true)` to ensure no issues generating the relevant html pages with code block evaluations, and then run `serve()` (after restarting) to serve the pages live on a local browser for viewing
-* run `include("deploy.jl")` to re-generate the LUNR index and push the changes to GitHub.
+**Once the changes are in the master branch:**
 
-The second step requires you have `lunr` and `cheerio` installed, if not:
-
-```
+* Run `cd("path/to/DataScienceTutorials"); using Franklin` to launch Franklin
+* In case you don't have `lunr` and `cheerio` installed already, also do:
+```julia
 using NodeJS
 run(`sudo $(npm_cmd()) i lunr cheerio`)
 ```
+* Run `serve(single=true, verb=true)` to ensure no issues generating the relevant html pages with code block evaluations
+* Run `serve()` (after restarting) to serve the pages live on a local browser for viewing
+* run `include("deploy.jl")` which re-generates the LUNR index and automatically pushes the changes to GitHub.
+
 
 This should take ≤ 15 seconds to complete.
 
@@ -139,9 +167,9 @@ serve(; eval_all=true)
 
 note that this will take a while.
 
-#### Merge conflicts
+#### Merge conflicts or Missing Styles
 
-If you get merge conflicts, do
+If you get merge conflicts or have styles that seem to be missing after `serve()`, do
 
 ```julia
 cleanpull()

_layout/foot.html

@@ -5,7 +5,7 @@
 <script src="/libs/collapse/collapse.js"></script>
 <script src="/libs/pure/ui.min.js"></script>
 <!-- head and footer-nav -->
-<script src="/libs/nav/head.js"></script>
+<script src="/libs/nav/head.js"  type="module"></script>
 <!-- landing page -->
 <script src="/libs/landing/landing.js"></script>
 <!-- navigation bar -->

_libs/landing/landing.js

@@ -51,6 +51,6 @@ document.addEventListener("DOMContentLoaded", function() {
     // Add click event listener to the div element
     element.addEventListener('click', function(event) {
         // Change the location to "/how-to-run-code"
-        window.location.href = (hosted) ? origin + "/DataScienceTutorials.jl" + "/how-to-run-code" : "/how-to-run-code";
+        window.location.href = (hosted) ? origin + "/DataScienceTutorials.jl" + "/info/how-to-run-code" : "/info/how-to-run-code";
     });
 });

_libs/nav/head.js

@@ -1,80 +1,4 @@
-
-const navItems = [
-  { name: 'Home', href: '/', sections: [], sectionItemWidth: '', id: 'home' },
-  {
-    name: 'Data Basics',                                    // Category name to be shown in navigation bar
-    id: 'data',                                             // id to be manipulated with js
-    href: '/data',                                             // in case it should link anywhere
-    sections: [                                             // list items to be shown in its dropdown and where they link to
-      { name: 'Loading Data', href: '/data/loading/' },
-      { name: 'Data Frames', href: '/data/dataframe/' },
-      { name: 'Categorical Arrays', href: '/data/categorical/' },
-      { name: 'Scientific Type', href: '/data/scitype/' },
-      { name: 'Data processing', href: '/data/processing/' },
-    ],
-    sectionItemWidth: 'short-item'
-  },
-  {
-    name: 'Getting Started',
-    id: 'getting-started',
-    href: '/getting-started',
-    sections: [
-      { name: 'Choosing a model', href: '/getting-started/choosing-a-model/' },
-      { name: 'Fit, predict, transform', href: '/getting-started/fit-and-predict/' },
-      { name: 'Model tuning', href: '/getting-started/model-tuning/' },
-      { name: 'Ensembles', href: '/getting-started/ensembles/' },
-      { name: 'Ensembles (2)', href: '/getting-started/ensembles-2/' },
-      { name: 'Composing models', href: '/getting-started/composing-models/' },
-      { name: 'Stacking', href: '/getting-started/stacking/' },
-    ],
-    sectionItemWidth: 'medium-item'
-  },
-  {
-    name: 'Intro to Stats Learning',
-    id: 'stats-learning',
-    href: '/isl',
-    sections: [
-      { name: 'Lab 2', href: '/isl/lab-2/' },
-      { name: 'Lab 3', href: '/isl/lab-3/' },
-      { name: 'Lab 4', href: '/isl/lab-4/' },
-      { name: 'Lab 5', href: '/isl/lab-5/' },
-      { name: 'Lab 6b', href: '/isl/lab-6b/' },
-      { name: 'Lab 8', href: '/isl/lab-8/' },
-      { name: 'Lab 9', href: '/isl/lab-9/' },
-      { name: 'Lab 10', href: '/isl/lab-10/' },
-    ],
-    sectionItemWidth: 'long-item'
-  },
-
-  {
-    name: 'End to End',
-    id: 'end-to-end',
-    href: '/end-to-end',
-    sections: [
-      { name: 'Telco Churn', href: '/end-to-end/telco/' },
-      { name: 'AMES', href: '/end-to-end/AMES/' },
-      { name: 'Wine', href: '/end-to-end/wine/' },
-      { name: 'Crabs (XGB)', href: '/end-to-end/crabs-xgb/' },
-      { name: 'Horse', href: '/end-to-end/horse/' },
-      { name: 'King County Houses', href: '/end-to-end/HouseKingCounty/' },
-      { name: 'Airfoil', href: '/end-to-end/airfoil' },
-      { name: 'Boston (lgbm)', href: '/end-to-end/boston-lgbm' },
-      { name: 'Using GLM.jl', href: '/end-to-end/glm/' },
-      { name: 'Power Generation', href: '/end-to-end/powergen/' },
-      { name: 'Boston (Flux)', href: '/end-to-end/boston-flux' },
-      { name: 'Breast Cancer', href: '/end-to-end/breastcancer' },
-      { name: 'Credit Fraud', href: '/end-to-end/creditfraud' },
-    ],
-    sectionItemWidth: 'long-item'
-  },
-  {
-    name: 'Advanced',
-    id: 'advanced',
-    href: '#!',
-    sections: [{ name: 'Ensembles (3)', href: '/advanced/ensembles-3' }],
-    sectionItemWidth: 'medium-item'
-  },
-];
+import navItems from '../../routes.json' assert {type: 'json'};
 
 // first get info on whether hosted or not
 const origin = window.location.origin;
@@ -130,7 +54,7 @@ navItems.forEach((item) => {
 
 // add a final li as searchform
 let formAction = (hosted) ? origin + "/DataScienceTutorials.jl" + "/search/index.html" : "/search/index.html";
-searchForm = `
+let searchForm = `
   <li>
     <form id="lunrSearchForm" name="lunrSearchForm" style="margin-left: 1.5rem; margin-right: -2rem;">
       <input class="search-input" name="q" placeholder="Search..." type="text">
@@ -192,7 +116,7 @@ generateSidebar(navItems);
 // Flatten the nav items so we can easily iterate through them
 function flattenNavItems(items) {
   return items.reduce((acc, item) => {
-    const mainHrefs = ["/data", "/end-to-end", "/getting-started", "/isl", "#!"];
+    const mainHrefs = ["/info/data", "/info/end-to-end", "/info/getting-started", "/info/isl", "#!"];
     if (!mainHrefs.includes(item.href)) {      
       acc.push(item);
     }

advanced/ensembles-3.md

@@ -3,4 +3,4 @@
 
 # Ensemble models 3 (learning networks)
 
-\tutorial{ADV-ensembles-3}
+\tutorial{advanced/ensembles-3}

collapse-script.jl

@@ -71,22 +71,24 @@ function read_tutorials(tutorials_dir)
 
     # Iterate over all files named "tutorial.js" in subdirectories
     for subdir in readdir(tutorials_dir)
-        file_path = joinpath(tutorials_dir, subdir, "tutorial.jl")
-        if isfile(file_path)
-            try
-                file = open(file_path, "r")
-                content = read(file, String)
-                close(file)
-                file = open(file_path, "w")
-                modified_content = introduce_dropdowns(content)
-                write(file, modified_content)
-                close(file)
-                # Store content in the dictionary
-            catch e
-                throw("Error reading file '$file_path': $(e)")
+        for tutorial_subdir in readdir(joinpath(tutorials_dir, subdir))
+            file_path = joinpath(tutorials_dir, subdir, tutorial_subdir, "tutorial.jl")
+            if isfile(file_path)
+                try
+                    file = open(file_path, "r")
+                    content = read(file, String)
+                    close(file)
+                    file = open(file_path, "w")
+                    modified_content = introduce_dropdowns(content)
+                    write(file, modified_content)
+                    close(file)
+                    # Store content in the dictionary
+                catch e
+                    throw("Error reading file '$file_path': $(e)")
+                end
             end
         end
-    end
+    end 
 end
 
 

data/categorical.md

@@ -3,4 +3,4 @@
 
 # Handling categorical data
 
-\tutorial{D0-categorical}
+\tutorial{data/categorical}

data/dataframe.md

@@ -2,4 +2,4 @@
 
 # Manipulating a DataFrame
 
-\tutorial{D0-dataframe}
+\tutorial{data/dataframe}

data/loading.md

@@ -3,4 +3,4 @@
 
 # Loading and elementary processing of data
 
-\tutorial{D0-loading}
+\tutorial{data/loading}

data/processing.md

@@ -3,4 +3,4 @@
 
 # Further data processing
 
-\tutorial{D0-processing}
+\tutorial{data/processing}

data/scitype.md

@@ -3,4 +3,4 @@
 
 # Data interpretation: Scientific Types
 
-\tutorial{D0-scitype}
+\tutorial{data/scitype}

end-to-end/AMES.md

@@ -3,4 +3,4 @@
 
 # AMES
 
-\tutorial{EX-AMES}
+\tutorial{end-to-end/AMES}

end-to-end/HouseKingCounty.md

@@ -4,4 +4,4 @@
 
 # House King County
 
-\tutorial{EX-housekingcounty}
+\tutorial{end-to-end/housekingcounty}

end-to-end/airfoil.md

@@ -3,4 +3,4 @@
 
 # Airfoil
 
-\tutorial{EX-airfoil}
+\tutorial{end-to-end/airfoil}

end-to-end/boston-flux.md

@@ -3,4 +3,4 @@
 
 # Boston with Flux
 
-\tutorial{EX-boston-flux}
+\tutorial{end-to-end/boston-flux}

end-to-end/boston-lgbm.md

@@ -3,4 +3,4 @@
 
 # Boston with LightGBM
 
-\tutorial{EX-boston-lgbm}
+\tutorial{end-to-end/boston-lgbm}

end-to-end/breastcancer.md

@@ -3,4 +3,4 @@
 
 # Breast Cancer Wisconsin(Diagnostic)
 
-\tutorial{EX-breastcancer}
+\tutorial{end-to-end/breastcancer}

end-to-end/crabs-xgb.md

@@ -3,4 +3,4 @@
 
 # Crabs with XGBoost
 
-\tutorial{EX-crabs-xgb}
+\tutorial{end-to-end/crabs-xgb}

end-to-end/creditfraud.md

@@ -3,4 +3,4 @@
 
 # Credit Card Fraud
 
-\tutorial{EX-creditfraud}
+\tutorial{end-to-end/creditfraud}

end-to-end/glm.md

@@ -2,4 +2,4 @@
 
 # Using GLM.jl
 
-\tutorial{EX-GLM}
+\tutorial{end-to-end/GLM}

end-to-end/horse.md

@@ -3,4 +3,4 @@
 
 # Horse colic data
 
-\tutorial{EX-horse}
+\tutorial{end-to-end/horse}

end-to-end/powergen.md

@@ -2,4 +2,4 @@
 
 # Power Generation
 
-\tutorial{EX-powergen}
+\tutorial{end-to-end/powergen}