Create surveys feature (#147)

RapidPro contact fields have a name and a key.
In the UI, users enter a name, and it is converted to a key (which is all
lowercase and has spaces converted to underscores).
The key is used everywhere when we want to reference a variable, e.g. via
`@fields.key`. The name is used for display, but otherwise is pretty useless.
Yet, RapidPro enforces that the name may not contain underscores, and thus it
must differ from the key if we want to have underscores in the key.
This is cumbersome.

Therefore, in flow definitions, we now allow specifying a `save_name` for
`save_value` rows which may contain underscores. Thus the exact same string can
be used both in this column and when referencing the field via `@fields.key`.
A name is autogenerated by replacing the underscores with spaces (but for a
flow author, the name is irrelevant anyway).

---------

Co-authored-by: Ian Stride <[email protected]>

docs/sheets.md

@@ -223,7 +223,7 @@ def sheet_to_list_of_nested_dict(sheet, user_model):
 	rather than List[RowModel]).
 	'''
     row_parser = RowParser(user_model, CellParser())
-    sheet_parser = SheetParser(row_parser, sheet.table)
+    sheet_parser = SheetParser(sheet.table, row_parser)
     data_rows = sheet_parser.parse_all()  # list of row model
     return [row.dict() for row in data_rows]
     # Below is what the content index parser does:

docs/surveys.md

@@ -0,0 +1,124 @@
+# Surveys
+
+Surveys can be created by defining a data sheet of questions, indexing it in the content index and adding a `survey` row in the content index.
+
+A basic usage example can be found in `TestSurveyParser.test_basic_survey` in `tests/test_surveyparser.py`.
+
+
+## The question data sheet
+
+Each survey consist of questions. Questions have an underlying data model `SurveyQuestionRowModel`. This consists of the fields defined in `SurveyQuestionModel` in `src/rpft/parsers/creation/surveymodels.py` and an additional `ID` field.
+
+Each question consists of the question text, an associated variable that the user input is stored in, and a variety of other fields.
+
+### Basic question fields
+
+These are the basic fields of a question definition (can be used as column headers for question data sheets).
+
+- `ID`: Identifier, used for flow and variable name generation.
+- `type`: Question type. Pre-defined types include `text`, `mcq`, ..., but custom ones can be used if the specific templates are defined by the user.
+- `messages`: The question text. This is a list of multiple messages, each message having a `text` and optional `image`/`audio`/`video` attachment fields, as well as a list `attachments` of generic attachments.
+	- `question`: Shorthand for `messages.1.text`; you may use this instead of `messages` if none of your questions send more than 1 message.
+	- `attachment`: Shorthand for `messages.1.attachment`; you may use this instead of `messages` if none of your questions send more than 1 message.
+	- Note that these shorthands can NOT be used within template definitions.
+- `variable`: Variable to store the user input in. If blank, generated from the question ID as `sq_{survey_id}_{question_id}`. The survey_id/question_id is the survey's name/question ID, **in all lowercase with non-alphanumeric characters removed**
+- `completion_variable`: Variable indicating whether question has been completed. If blank, generated from the variable as `{variable}_complete`
+- `choices`: For multiple choice questions: a list of choices
+- `expiration.message`: Message that gets send when the user doesn't respond in a long time
+- `expiration.time`: [not implemented]
+
+It is possible to reuse questions across multiple surveys (see `tags` below). In that case, we need to make sure that each copy of a question gets a unique name for its variables. Auto-generating the variable names from the question ID solves the question of creating unique variable names, however, we also need a way to refer to these variable names independent of the `surveyid` which is used for this.
+
+Therefore we have the following shorthands, which can be used within any field of a question:
+
+- `@answer` is short for `@fields.{variable}`. This is useful even without reusing questions, e.g. within confirmation/validation/stop conditions (see below).
+- `@answerid` is short for `{variable}`. This can be used when defining new variables (in postprocessing steps) whose names should depend on the variable in the question.
+- `@prefix` is short for `@fields.sq_{surveyid}`. This is useful when referencing variables from previous questions of the survey, by using e.g. `@prefix_{questionid}`
+- `@prefixid` is short for `sq_{surveyid}`. Similar to above
+
+
+### Special question fields
+
+These are the more complex fields of a question definition (can be used as column headers for question data sheets).
+
+#### `tags`: Tags for filtering
+
+Data sheets can be created by filtering an existing data sheet by a condition (e.g. `'my_tag' in tags`), so that only rows fulfilling the condition are included. This way, the same pool of questions can be used for multiple surveys, by selecting questions via a survey-specific tag.
+
+#### `relevant`: Omit a question based on previous answers
+
+If any of the given conditions does not hold, skip the question. These conditions will commonly depend on previous answers.
+
+#### `confirmation`: Conditional Answer confirmation
+
+If one of the conditions holds, print the confirmation message associated with that condition, with options Yes/No. If user enters No, repeat the question.
+
+Example:
+
+- Do you confirm that you're under 18? [if @answer < 18]
+- Please confirm your input @answer [Unconditional confirmation can be realized by specifying a condition that is always true]
+
+#### `stop`: Conditional premature end of survey (later: forward skip?)
+
+If one of the conditions holds, send the message associated with the condition and end the survey.
+
+Example:
+
+- user's age is less than 18
+- user is not a parent
+- user does not live in the target region
+
+#### `validation`: Validation / conditional repetition of question
+
+If one of the conditions holds, send the message associated with the condition and repeat the question.
+
+Example:
+
+- Your name is too short. Please enter again.
+
+#### `postprocessing`: Variable postprocessing
+
+Postprocessing to do after a user's answer is successfully stored. This could be an assignment (of the same or another variable), or a flow that is triggered.
+
+Examples:
+
+- take the user's entered name and capitalize it (stored in the same variable)
+- create a new age_bucket variable based on the user's age input. If the age variable is called `sq_sid_age`, specifying the new variable in the assignment to be `@answerid_bucket` with create a variable `sq_sid_age_bucket`
+
+#### `skipoption`: Optional questions
+
+A way for the user to skip the question by typing in a specific phrase.
+
+## Content index rows
+
+After creating a data sheet with questions, in the content index, you can create a row of type `data_sheet` and specify the `data_model` as `SurveyQuestionRowModel`. This is a global model that does not need to be defined by the user in a custom module.
+
+Then, create a row of type `survey`. For this, the following columns are relevant:
+
+- `data_sheet`: A data sheet with questions
+- `new_name`: Name of the survey. If not provided, the name of the `data_sheet` is used.
+- `config`: A SurveyConfig object, see `src/rpft/parsers/creation/surveymodels.py`
+    - `variable_prefix`: Prefix to apply to all RapidPro variables that are created by the survey. For each `SurveyQuestion`, this is the `variable`, `completion_variable` and `postprocessing.assignments.*.variable`. Ideally, avoid this feature in favor of using auto-generated variable names, `@answer`, `@answerid` and `@prefix`.
+    - `expiration_message`: Message to send when a question flow expires. If a question does not specify an expiration message, this message is used by default.
+- `template arguments`: Template arguments to be passed down to the survey template
+
+This will create one flow for each question, named `survey - {survey name} - question - {question ID}`, as well as a survey flow `survey - {survey name}` that invokes each question via `start_new_flow`. This is achieved via templating. The templates can be customized if needed.
+
+
+## Survey templates
+
+We define global templates that are used by surveys. These templates can be found in `src/rpft/parsers/creation/survey_templates/`. They are as follows:
+
+- `template_survey_wrapper`: Flow rendering all the questions.
+	- Receives the following context variables that can be used in the template:
+		- `questions`: a list of `SurveyQuestionRowModel`
+		- `survey_name`: Name of the survey
+		- `survey_id`: ID of the survey (generated from name)
+	- In the content index, a `survey` row can have `template_arguments`. If present, these are passed to the `template_survey_wrapper` template when creating a survey.
+- `template_survey_question_wrapper`: Question functionality that is common to all input types. Invoked by the survey via `start_new_flow`
+	- Receives the fields of the `SurveyQuestionRowModel` as its context variables
+	- Currently, it is not possible to pass template arguments to this template.
+- `template_survey_question_block_{type}`: For each question input type `{type}`, there is a template to read the user data. These are included into the `template_survey_question_wrapper` via `insert_as_block`
+	- Because this template is inserted as a block, any context that is available in `template_survey_question_wrapper` (in particular, `question`) is also available in this template.
+
+The user can overwrite these by defining a template of the same name in the content index, thereby using their own custom templates. There is no constraint on what `{type}` can be, therefore the user can also create their own question types.

src/rpft/parsers/common/rowparser.py

@@ -5,6 +5,8 @@
 
 from pydantic.v1 import BaseModel
 
+from rpft.parsers.common.cellparser import CellParser
+
 
 class RowParserError(Exception):
     pass
@@ -122,10 +124,10 @@ class RowParser:
     TYPE_ANNOTATION_SEPARATOR = ":"
     DEFAULT_VALUE_SEPARATOR = "="
 
-    def __init__(self, model, cell_parser):
+    def __init__(self, model, cell_parser=None):
         self.model = model
         self.output = None  # Gets reinitialized with each call to parse_row
-        self.cell_parser = cell_parser
+        self.cell_parser = cell_parser or CellParser()
 
     def try_assign_as_kwarg(self, field, key, value, model):
         # If value can be interpreted as a (field, field_value) pair for a field of

src/rpft/parsers/common/sheetparser.py

@@ -1,20 +1,39 @@
 import copy
 from rpft.parsers.common.rowdatasheet import RowDataSheet
+from rpft.parsers.common.rowparser import RowParser
 from rpft.logger.logger import get_logger, logging_context
 
 LOGGER = get_logger()
 
 
 class SheetParser:
-    def __init__(self, row_parser, table, context={}):
+    def parse_sheet(table, row_model):
         """
         Args:
+            table: Tablib Dataset representing the table to be parsed.
+            row_model: Data model to convert rows of the sheet into.
+
+        Returns:
+            RowDataSheet instance containing a list of row_model instances
+        """
+
+        sheet_parser = SheetParser(table, row_model)
+        return sheet_parser.get_row_data_sheet()
+
+    def __init__(self, table, row_model=None, row_parser=None, context={}):
+        """
+        Either a row_parser or a row_model need to be provided.
+
+        Args:
+            table: Tablib Dataset representing the table to be parsed.
+            row_model: Data model to convert rows of the sheet into.
             row_parser: parser to convert flat dicts to RowModel instances.
             context: context used for template parsing
-            table: Tablib Dataset representing the table to be parsed.
         """
 
-        self.row_parser = row_parser
+        if not (row_parser or row_model):
+            raise ValueError("SheetParser: needs either row_parser or row_model")
+        self.row_parser = row_parser or RowParser(row_model)
         self.bookmarks = {}
         self.input_rows = []
         for row_idx, row in enumerate(table):

src/rpft/parsers/creation/__init__.py

@@ -0,0 +1,61 @@
+from rpft.logger.logger import get_logger
+from rpft.parsers.creation.models import TemplateSheet
+
+LOGGER = get_logger()
+
+
+def map_template_arguments(template: TemplateSheet, args, context, data_sheets) -> dict:
+    """
+    Map template arguments, which are positional, to the arguments from the template
+    definition, and add the values of the arguments to the context with the appropriate
+    variable name (from the definition).
+    """
+    arg_defs = template.argument_definitions
+
+    if arg_defs and len(args) > len(arg_defs):
+        # Once the row parser is cleaned up to eliminate trailing '' entries, this
+        # won't be necessary
+        extra_args = args[len(arg_defs) :]
+        non_empty_extra_args = [ea for ea in extra_args if ea]
+
+        if non_empty_extra_args:
+            LOGGER.warning(
+                "Too many template arguments provided, "
+                + str(
+                    {
+                        "template": template.name,
+                        "extra": non_empty_extra_args,
+                        "definition": arg_defs,
+                        "arguments": args,
+                    }
+                )
+            )
+
+        args = args[: len(arg_defs)]
+
+    args_padding = [""] * (len(arg_defs) - len(args))
+
+    for arg_def, arg in zip(arg_defs, args + args_padding):
+        value = arg if arg != "" else arg_def.default_value
+
+        if value == "":
+            LOGGER.critical(f'Required template argument "{arg_def.name}" not provided')
+
+        value = data_sheets[value].rows if arg_def.type == "sheet" else value
+
+        if arg_def.name in context and value != context[arg_def.name]:
+            LOGGER.warn(
+                "Template argument reassigned, "
+                + str(
+                    {
+                        "template": template.name,
+                        "name": arg_def.name,
+                        "before": context[arg_def.name],
+                        "after": value,
+                    }
+                )
+            )
+
+        context[arg_def.name] = value
+
+    return context