Minimal infrastructure for server and admin UI?

[Onurbon]

Context and scope

We would like to help non-technical users run the TttC pipeline and generate reports for their own data.
This RFC is to discuss what the MVP might be in terms of infrastructure and choice of technologies.

Note that we want all the new complexity to remain optional , meaning that developers should still be able to just pull this repo and generate reports locally using the current pipeline (without having to work inside docker or kick of a server or anything like that).

Potential MVP

We could build a simple website where users can upload a CSV file, chose a name and url (slug) for their report, potentially change a few options (e.g. number of clusters and language to translate to) and then click a button "generate report".

The users could then refresh the page from time to time to check the status of the job (as currently tracked in status.json) and when the report generation is completed, they could find it at the appropriate url.

Simplifying assumptions

For the very first iteration, we might want to make the following assumptions:

1. the first users will be dealing with relatively small CSV files (< 1000 raws)
2. the pipeline jobs will be relatively fast (< 30min)
3. the pipeline jobs will be relatively reliable (crashing less than 5% of the time)
4. user management is not yet necessary (job names and configs can be stored on local storage)
5. we don't need to worry about AI credits just yet (server providers will put spending limits on their API keys)

Minimal server API specs

We could start implementing a server with the following endpoints:

• POST :api/input/{file-name} to upload a new csv files and save data under /pipeline/inputs
• POST :api/config/{project} to define a project's config file and save it under /pipeline/configs
• POST :api/run/{project} to kick off a job (spawn a child process python main.py ./configs/{project}.json -skip-interaction)
• GET :api/{project}/status to check the status of a job (read content of status.json)
• GET :api/{project}/report to serve the html report when ready

When posting a new project config, the server could return a preview of which steps will need to (re)run, similar to what the CLI shows when running in interactive mode.

Choice of technologies

We could consider:

• FastAPI for the server
• NextJS for the admin UI
• Docker to simplify deployment

[lightningorb]

Since FastAPI is the proposed library for running backend services, it may be worth investigating background tasks https://fastapi.tiangolo.com/tutorial/background-tasks/ as a candidate for long-running tasks not requiring the use of workers & AMQP.

[mashbean]

If there is a sample CSV file, may help non-technical users to understand what kind of dataset could compatible with TttC.