-
Notifications
You must be signed in to change notification settings - Fork 4
Help: Wizard ‐ Challenge ‐ Problem
A challenge problem is (in principle) made of data and code. Pages 1 and 2 allowed you to supply the dataset and its split into 3 subsets (training, validation, and test). Page 3 allows you to supply the code, a.k.a. "ingestion program".
An ingestion program is a piece of code, which is executed when a challenge participant make a submission. It "ingests" it and execute "something" to help processing it. There are several possible use cases:
- Parsing the submission and deciding how to process it, e.g. the organizers may allow the participants to submit either results, code or data.
- Allowing submission of source code or libraries. The ingestion program can then call functions supplied by the participants and executed them on the input data. Advantage: the input data is read by the same reader for everybody, the participants are not penalized if their code fails to read the data and/or if it takes time to read the data. Other advantage: the organizers can run cross-validation experiments with not possible cheating of the participants.
- Allowing time series predictions, active learning or query learning. The ingestion program can "serve" data on demand to the code supplied by the participants. In fact, the ingestion program can even generate artificial data, if needed! However you may not need it, read on:
If you are organizing a challenge with RESULT submission (no participant-supplied code executed on the challenge platform), you should not supply an ingestion program. If your template has one already, delete it. If there is none, just click "Update", then "Next".
If your participants must supply executables, you do not necessarily need to supply an ingestion program. The challenge platform will execute any submission that comes with a metadata file. This file need to include the command to be executed, e.g.:
command: python $program/run.py $input $output
You must supply a so-called "Ingestion Program" that reads data and participants' submissions if you ask the participants for instance to supply a python class model.py that is NOT executable. Your ingestion program will then read the data and call that class to train and test the predictive model. We provide an example of ingestion program for the Iris challenge.