WIP: Topic/abstract storage

.gitignore

@@ -21,10 +21,13 @@ credentials/**
 .env
 
 # other data
-data/**
 tags*
 logfile.log
 
 blacklists/**
 whitelists/**
 config/**
+
+data/demo/3video/dancingonmyown.mov
+
+data/demo/3video/info.json

commands.py

@@ -12,6 +12,7 @@
 
 def __run(cmd, cli_args, *args):
     if cli_args.dry:
+        print(" ".join(cmd))
         return cmd
     try:
         returncode = sp.call(cmd)
@@ -184,6 +185,17 @@ def develop(args):
     CONT_NAME = "mtriage_developer"
     TAG_NAME = "{}-gpu".format(args.tag) if args.gpu else args.tag
 
+    volumes = [
+        "-v",
+        "{}:/mtriage".format(DIR_PATH),
+        "-v",
+        "{}/.config/gcloud:/root/.config/gcloud".format(HOME_PATH),
+    ]
+
+    if args.yaml is not None:
+        yaml_path = os.path.abspath(args.yaml)
+        volumes += ["-v", "{}:/run_args.yaml".format(yaml_path)]
+
     # --runtime only exists on nvidia docker, so we pass a bubblegum flag when not available
     # so that the call arguments are well formed.
     return __run(
@@ -199,10 +211,7 @@ def develop(args):
             "BASE_DIR=/mtriage",
             get_env_config(),
             "--privileged",
-            "-v",
-            "{}:/mtriage".format(DIR_PATH),
-            "-v",
-            "{}/.config/gcloud:/root/.config/gcloud".format(HOME_PATH),
+            *volumes,
             "{}:{}".format(NAME, TAG_NAME),
             "/bin/bash",
         ],
@@ -248,7 +257,7 @@ def run(args):
         "-v",
         "{}/media:/mtriage/media".format(DIR_PATH),
         "-v",
-        "{}/credentials:/mtriage/credentials".format(DIR_PATH),
+        "{}/data:/mtriage/data".format(DIR_PATH),
         "-v",
         "{}:/run_args.yaml".format(yaml_path),
         "-v",
@@ -274,6 +283,9 @@ def run(args):
         args,
     )
 
+    if not args.persist:
+        clean(args)
+
 
 def parse_args(cli_args):
     parser = argparse.ArgumentParser(description="mtriage dev scripts")
@@ -285,6 +297,7 @@ def parse_args(cli_args):
     run_p.add_argument("--gpu", action="store_true")
     run_p.add_argument("--dry", action="store_true")
     run_p.add_argument("--dev", action="store_true")
+    run_p.add_argument("--persist", action="store_true")
 
     dev_p = subparsers.add_parser("dev")
     dev_p.add_argument("--whitelist")
@@ -293,6 +306,7 @@ def parse_args(cli_args):
     dev_p.add_argument("--gpu", action="store_true")
     dev_p.add_argument("--dry", action="store_true")
     dev_p.add_argument("--verbose", action="store_true")
+    dev_p.add_argument("--yaml", type=str2yamlfile)
     dev_p.add_argument(
         "command",
         choices=["develop", "build", "test", "clean"],

credentials/.gitignore → data/.gitignore

@@ -1,2 +1,3 @@
 **/*
 !.gitignore
+!demo/

data/demo/1local/1.txt

@@ -0,0 +1 @@
+This is a simple text file.

data/demo/1local/2.md

@@ -0,0 +1,3 @@
+# Markdown example
+
+The __tiniest__ bit less simple than a txt file.

docs/components/youtube.md

@@ -1,8 +1,11 @@
-# Selector: `youtube`
+# Configuring the Youtube selector 
 
-In order to run the youtube selector, mtriage requires a Google Cloud Platform service account.
+In order to run the Youtube selector, mtriage requires a Google Cloud Platform
+API key.
 
-1. Create a service account and download a 'credentials.json' from the [credentials page](https://console.cloud.google.com/apis/credentials) on the Google Cloud console by creating a service account, and downloading a JSON version of the service acount key.
-2. Move the downloaded JSON file to 'credentials/google.json. This file is
-   gitignored, and so will not be pushed to any remotes.
-3. In the service account API settings on Google Cloud Console in the browser, enable the "Youtube Data v3 API".
+1. Create a new project in GCP, and in the [credentials
+   page](https://console.cloud.google.com/apis/credentials), enable the
+   'Youtube Data V3' API.
+2. Create a new API key, ensuring that it has access to the Youtube V3 API. 
+3. In the '.env' file in mtriage's root folder, add the line
+   `GOOGLE_API_KEY=xxxxx`, replacing 'xxxxx' with your downloaded API key. 

docs/install.md

@@ -0,0 +1,34 @@
+# Install 
+
+Mtriage is a tool developed at [Forensic Architecture](https://forensic-architecture.org) to orchestrate complex workflows that download media of various kinds, analyse them, and visualise results. To understand what mtriage can do, this tutorial will briefly outline the different components of an mtriage workflow, and then show you how to create one that analyses Youtube videos frame-by-frame with a Resnet50 object detection classifer pretrained on ImageNet. To conclude, we'll briefly touch on how much more mtriage is capable of, and show how it is easy to extend mtriage to analyse other kinds of media. 
+
+## Architecture
+Mtriage has three kinds of components: selectors, analysers, and viewers. Each component manages a different stage of a workflow:
+* **Selector** - indexes and then downloads media from a source, such as Youtube. Selectors can implement their own web scraping techniques, or simply leverage the search functionality of online platforms in order to return results. The Youtube selector, for example, takes as input a search terms and two dates (start and end), returning all the videos that Youtube returns for the search term that was uploaded between the two given dates.
+* **Analyser** - analyses media that has been downloaded from a selector, producing *derived* media that contain the analysis results. An analyser may produce media that is of a different kind than its input media. The frames analyser, for example, takes a video as input and produces a set of images (one frame for each second, say) as output.
+* **Viewer** - visualises derived media in an interactive website. Viewers make mtriage an end-to-end tool for analysing media, as they mean that you can present results, and even create interactive workstations, directly from analysis results.
+
+## Downloading Mtriage
+Start by cloning the source code:
+
+```bash
+git clone https://github.com/forensic-architecture/mtriage.git
+```
+
+Mtriage has two primary dependencies: [Python](https://www.python.org/) 3 and [Docker CE](https://docs.docker.com/install/). Mtriage will _probably_ work with Python 2.x as well, but it's untested. If you have a CUDA GPU, you can use [Nvidia Docker](https://github.com/NVIDIA/nvidia-docker) instead of Docker to make certain analysers more performant.
+
+Once you have Python and Docker installed, install the three dependencies in requirements.txt. Two of these are for testing (pytest and black); the only runtime dependency is [pyyaml](https://pyyaml.org/). 
+```
+cd mtriage
+pip install -r requirements.txt
+```
+
+Run the test suite to ensure that everython is working. This command may take a while, as the first time you run mtriage it will download the [latest Docker image](https://cloud.docker.com/u/forensicarchitecture/repository/docker/forensicarchitecture/mtriage). Mtriage commands will run much faster after this first one:
+
+```bash
+./mtriage dev test
+```
+
+Assuming this command completed and all the tests passed, you are now ready to run mtriage workflows! 
+
+

docs/overview.md

@@ -8,4 +8,4 @@ We developed mtriage to address the insufficiency in machine learning tooling fo
 
 Mtriage is open source and in active development. This means that everyone can not only use mtriage in their own research, but also that community contributions (of a new classifier, or a new media source) can potentially be made available to all other users as upstream contributions.
 
-To get started with mtriage, check out [Getting Started](docs/getting-started.md).
+To get started with mtriage, check out [Getting Started](docs/getting-started.md).

docs/testing.md

@@ -11,7 +11,7 @@ using the locally installed Python environment.
 
 To run all tests, use the following command:
 ```
-./mtriage run test
+./mtriage dev test
 ```
 
 See docs/custom-components.md for more information on how to write tests for

docs/tutorial/1/1a.yaml

@@ -0,0 +1,6 @@
+folder: media/demo_official/1
+select:
+  name: Local
+  config:
+    source: data/demo/1local
+    # aggregate: true

docs/tutorial/1/1b.yaml

@@ -0,0 +1,9 @@
+folder: media/demo_official/1
+elements_in:
+  - Local
+analyse:
+  name: ExtractTypes
+  config:
+    exts:
+      - txt
+      - md

docs/tutorial/1/1c.yaml

@@ -0,0 +1,11 @@
+folder: media/demo_official/1
+select:
+  name: Local
+  config:
+    source: data/demo/1local
+analyse:
+  name: ExtractTypes
+  config:
+    exts:
+      - txt
+      - md