download subtitles and presentations and handle groups

.gitignore

@@ -59,3 +59,6 @@ target/
 bin/
 default_out_path/
 _browser_user_data_dir/
+
+.vscode/
+_browser_persistent_session/

README.md

@@ -15,7 +15,7 @@ See it in action:
     <img width="700" height="auto" src="docs/images/demo.gif" alt="echo360 demo" />
 </p>
 
-**NEWS:** It now works with `echo360.org` platform as well. Special thanks to [*@cloudrac3r*](https://github.com/cloudrac3r) and *Emma* for their kind offering of providing sources and helped debugging it. Read [FAQ](#echo360-cloud) for details.
+**NEWS:** It now works with `echo360.org` platform as well. Special thanks to [_@cloudrac3r_](https://github.com/cloudrac3r) and _Emma_ for their kind offering of providing sources and helped debugging it. Read [FAQ](#echo360-cloud) for details.
 
 # Getting Started
 
@@ -44,7 +44,7 @@ echo360-downloader COURSE_URL  # where COURSE_URL is your course url
 
 ### Optional
 
--   ffmpeg (for transcoding ts file to mp4 file) See [here (windows)](https://www.easytechguides.com/install-ffmpeg/) or [here](https://github.com/adaptlearning/adapt_authoring/wiki/Installing-FFmpeg) for a brief instructions of installing it in different OS.
+- ffmpeg (for transcoding ts file to mp4 file) See [here (windows)](https://www.easytechguides.com/install-ffmpeg/) or [here](https://github.com/adaptlearning/adapt_authoring/wiki/Installing-FFmpeg) for a brief instructions of installing it in different OS.
 
 ## Manual
 
@@ -62,9 +62,9 @@ python echo360.py
 
 ### Operating System
 
--   Linux
--   OS X
--   Windows
+- Linux
+- OS X
+- Windows
 
 # Usage
 
@@ -78,6 +78,7 @@ python echo360.py
 ```
 
 ### Script args
+
 ```
 >>> usage: echo360.py [-h] [--output OUTPUT_PATH]
                   [--after-date AFTER_DATEYYYY-MM-DD)]
@@ -135,6 +136,8 @@ optional arguments:
                         downloader will also try to download the second
                         video, which could be the alternative feed. Might
                         only work on some 'echo360.org' hosts.
+  --subtitles, -s       Download VTT subtitles for each video feed.
+  --dump-json           Download JSON representation of course to output directory.
   --debug               Enable extensive logging.
   --auto                Only effective for 'echo360.org' host. When set, this
                         script will attempts to automatically redirects after
@@ -145,6 +148,7 @@ optional arguments:
                         default behaviour and exists only for backward
                         compatibility reason.
 ```
+
 # Examples
 
 ```shell
@@ -208,11 +212,15 @@ This is first built for the echo system in the University of Sydney, and then va
 ```shell
 https://$(hostname)/ess/portal/section/$(UUID)
 ```
+
 or
+
 ```shell
 https://echo360.org[.xx]/
 ```
+
 or with a dot net variant
+
 ```shell
 https://echo360.net[.xx]/
 ```
@@ -252,56 +260,66 @@ Echo360 cloud refers to websites in the format of `https://echo360.org[.xx]`. Th
 This method requires you to setup SSO credentials, therefore, it needs to open up a browser for you to setup your own university's SSO credentials.
 
 To download videos, run:
+
 ```shell
 ./run.sh https://echo360.<org|net>[.xx]/section/$(UUID)/home
 ```
-where `[.xx]` is an optional country flag specific to your echo360 platform and `$(UUID)` is the unique identifier for your course. This should the url that you can retrieve from your course's *main page* like the following.
+
+where `[.xx]` is an optional country flag specific to your echo360 platform and `$(UUID)` is the unique identifier for your course. This should the url that you can retrieve from your course's _main page_ like the following.
 
 <img height="auto" src="docs/images/echo360cloud_course-page.png" alt="echo360 cloud course main page" />
 
 Note that this implies `setup-credential` option and will use chrome-webdriver by default. If you don't have chrome or prefer to use firefox, run it with the ` --firefox` flag like so:
+
 ```shell
 ./run.sh https://echo360.<org|net>[.xx]/section/$(UUID)/home --firefox
 ```
 
 After running the command, it will opens up a browser instance, most likely with a login page. You should then login with your student's credentials like what you would normally do. After you have successfully logged in, the module should automatically redirects you and continues. If the script hangs (e.g. failed to recognises that you have logged in), feel free to let me know.
 
-
 ### I'm not sure of how to run it?
 
 First, you'd need to install [Python](https://www.python.org/downloads/) in your system. Then, you can follow the youtube tutorial videos to get an idea of how to use the module.
 
 - For [Windows users](https://www.youtube.com/watch?v=Lv1wtjnCcwI) (and showcased how to retrieve actual echo360 course url)
-[![](docs/images/youtube_win_tutorial.jpg)](https://www.youtube.com/watch?v=Lv1wtjnCcwI)
+  [![](docs/images/youtube_win_tutorial.jpg)](https://www.youtube.com/watch?v=Lv1wtjnCcwI)
 
 ### My credentials does not work?
 
 You can setup any credentials need with manually logging into websites, by running the script with:
+
 ```sh
 ./run.sh ECHO360_URL --setup-credential
 ```
+
 This will open up a chrome instance that allows you to log into your website as you normally do. Afterwards, simply type 'continue' into your shell and press enter to continue to proceeds with the rest of the script.
 
 ### My credentials does not work (echo360.org)?
 
 For echo360.org, the default behaviour is it will always require you to setup-credentials, and the module will automatically detect login token and proceed the download process. For some institutions, this seems to be not sufficient (#29).
 
 You can disable such behaviour with
+
 ```sh
 ./run.sh ECHO360_ORG_URL --manual
 ```
+
 for manual setup; and once you had logged in, type
+
 ```sh
 continue
 ```
+
 in your terminal to continue.
 
 ### How do I download only individual video(s)?
 
 You are in luck! It is now possible to pick a subset of videos to download from (instead of needing to download everything like before). Just pass the interactive argument like this:
+
 ```sh
 ./run.sh ECHO360_URL --interactive  # or ./run.sh ECHO360_URL -i
 ```
+
 ...and it shall presents an interactive screen for you to pick each individual video(s) that you want to download, like the screenshot as shown below.
 
 <img src="/docs/images/pick_individual_videos_screenshot.png" width="650" height="auto" >

echo360/course.py

@@ -1,24 +1,26 @@
+import functools
 import json
-import re
 import sys
 
 import requests
 import selenium
 import logging
 
+from .utils import strip_illegal_path
 from .videos import EchoVideos, EchoCloudVideos
 
 _LOGGER = logging.getLogger(__name__)
 
 
 class EchoCourse(object):
-    def __init__(self, uuid, hostname=None, alternative_feeds=False):
+    def __init__(self, uuid, hostname=None, alternative_feeds=False, subtitles=False):
         self._course_id = None
         self._course_name = None
         self._uuid = uuid
         self._videos = None
         self._driver = None
         self._alternative_feeds = alternative_feeds
+        self._subtitles = subtitles
         if hostname is None:
             self._hostname = "https://view.streaming.sydney.edu.au:8443"
         else:
@@ -139,7 +141,11 @@ def get_videos(self):
                 course_data_json = self._get_course_data()
                 videos_json = course_data_json["data"]
                 self._videos = EchoCloudVideos(
-                    videos_json, self._driver, self.hostname, self._alternative_feeds
+                    videos_json,
+                    self._driver,
+                    self.hostname,
+                    self._alternative_feeds,
+                    self._subtitles,
                 )
             # except KeyError as e:
             #     print("Unable to parse course videos from JSON (course_data)")
@@ -173,20 +179,29 @@ def course_id(self):
         return self._course_id
 
     @property
+    @functools.lru_cache
     def course_name(self):
-        if self._course_name is None:
-            # try each available video as some video might be special has contains
-            # no information about the course.
-            for v in self.course_data["data"]:
-                try:
-                    self._course_name = v["lesson"]["video"]["published"]["courseName"]
-                    break
-                except KeyError:
-                    pass
-            if self._course_name is None:
-                # no available course name found...?
-                self._course_name = "[[UNTITLED]]"
-        return self._course_name
+        cookies = {
+            cookie["name"]: cookie["value"] for cookie in self._driver.get_cookies()
+        }
+        response = requests.get(
+            "https://echo360.net.au/user/enrollments", cookies=cookies
+        )
+        if response.status_code == 200:
+            course_list = response.json()["data"]
+            for sections_parts in course_list:
+                matching = [
+                    x
+                    for x in sections_parts["userSections"]
+                    if x["sectionId"] == self._uuid
+                ]
+                if len(matching) > 0:
+                    course = matching[0]
+                    return strip_illegal_path(
+                        f"{course['courseCode']} - {course['sectionName']} {course['courseName']}"
+                    )
+
+        return "[[UNTITLED]]"
 
     @property
     def nice_name(self):

echo360/downloader.py

@@ -1,3 +1,5 @@
+from datetime import datetime
+import json
 import dateutil.parser
 import os
 import sys
@@ -6,7 +8,7 @@
 
 from .course import EchoCloudCourse
 from .echo_exceptions import EchoLoginError
-from .utils import naive_versiontuple, PERSISTENT_SESSION_FOLDER
+from .utils import naive_versiontuple, PERSISTENT_SESSION_FOLDER, strip_illegal_path
 
 import pip_ensure_version
 from pick import pick
@@ -191,6 +193,7 @@ def __init__(
         webdriver_to_use="phantomjs",
         interactive_mode=False,
         persistent_session=False,
+        dump_json=False,
     ):
         self._course = course
         root_path = os.path.dirname(os.path.abspath(sys.modules["__main__"].__file__))
@@ -200,6 +203,7 @@ def __init__(
         self._date_range = date_range
         self._username = username
         self._password = password
+        self._dump_json = dump_json
         self.interactive_mode = interactive_mode
 
         self.regex_replace_invalid = re.compile(r"[\\\\/:*?\"<>|]")
@@ -342,14 +346,26 @@ def download_all(self):
             self.login()
         sys.stdout.write(">> Retrieving echo360 Course Info... ")
         sys.stdout.flush()
-        videos = self._course.get_videos().videos
-        print("Done!")
+
         # change the output directory to be inside a folder named after the course
-        self._output_dir = os.path.join(
-            self._output_dir, "{0}".format(self._course.nice_name).strip()
-        )
         # replace invalid character for folder
-        self.regex_replace_invalid.sub("_", self._output_dir)
+        if isinstance(self._course, EchoCloudCourse):
+            self._output_dir = os.path.join(
+                self._output_dir,
+                "{0}".format(self._course.nice_name).strip(),
+            )
+        if self._output_dir and not os.path.isdir(self._output_dir):
+            os.makedirs(self._output_dir)
+        if self._dump_json:
+            dump_json_path = os.path.join(
+                self._output_dir,
+                f"course_{datetime.now().replace(microsecond=0).isoformat().replace(':','_')}.json",
+            )
+            with open(dump_json_path, "w") as f:
+                f.write(json.dumps(self._course._get_course_data()))
+
+        videos = self._course.get_videos().videos
+        print("Done!")
 
         filtered_videos = [video for video in videos if self._in_date_range(video.date)]
         videos_to_be_download = []