Skip to content

Commit 91c1dd0

Browse files
Brandon-PerryBrandon-Perry
committed
updated README
1 parent 343ff80 commit 91c1dd0

File tree

1 file changed

+140
-99
lines changed

1 file changed

+140
-99
lines changed

‎README.md

+140-99
Original file line numberDiff line numberDiff line change
@@ -1,69 +1,174 @@
1-
# tttc-light-js-server
1+
# Talk to the City
22

3-
A backend API for turbo pipeline.
3+
[Talk to the City (TTTC)](https://ai.objectives.institute/talk-to-the-city) Talk to the City is an open-source LLM interface for improving collective deliberation and decision-making by analyzing detailed, qualitative data. It aggregates responses and arranges similar arguments into clusters.
44

5-
## Running the pipeline locally
5+
This repo will allow you to setup your own instance of TTTC. The basic workflow is
66

7-
The current version has a local pipeline that uploads reports to a Google Cloud Storage instance upon completion.
7+
1. Submit a csv or google sheet with your survey data, either through the NextJS client or the Express API.
8+
2. The backend app will use an LLM to parse your data.
9+
3. The backend app will upload a JSON file to a Google Cloud Storage Bucket that you provide.
10+
4. Your report can be viewed by going to `http://[next client url]/report/[encoded url for your JSON file]`.
811

9-
Create a `.env` file with your own OpenAI key.
12+
If you want to use Talk to the City without any setup, you can go to our website at TBD and follow the instructions on [how to use TTTC](#usage)
1013

11-
Optionally, you can also set a password. If you give this password to a friend, they'll be allowed to use your server by putting this password in the OPENAI_API_KEY field in their own .env, instead of copying your OpenAI key directly.
14+
## Setup
1215

13-
```
14-
export OPENAI_API_KEY=sk-something-something
15-
export OPENAI_API_KEY_PASSWORD=some-password
16-
```
16+
Clone the repo to your local computer:
17+
18+
`git clone https://github.com/AIObjectives/tttc-light-js.git`
19+
20+
or if you have git ssh
21+
22+
`git clone [email protected]:AIObjectives/tttc-light-js.git`
23+
24+
### Google Cloud Storage and Services
25+
26+
TTTC currently only supports using Google Cloud for storing report data out of the box.
27+
28+
First create a new storage bucket:
29+
30+
- Create a Google Cloud project and a Google Cloud Storage bucket
31+
- Make sure this bucket has public access so that anyone can read from it (to
32+
make your reports accessible from your browser): - Turn off the "Prevent public access" protect - In the "Permissions" tab, click "Grant access." Add the principal `allUsers`
33+
and assign the role `Storage Object User`.
34+
35+
Then create a service account for this bucket:
36+
37+
- In the "IAM & Admin" view, select "Service Accounts" from the left menu, and
38+
then click "Create service account"
39+
- Give this account the "Editor" role
40+
- Create keys for this account and download them as a json file:
41+
- Save this file as `./google-credentials.json`
42+
- Encode this using by running the command `base64 -i ./google-credentials.json`
43+
- You will put this encoding into your .env file later
44+
45+
Set up gcloud SDK on your machine
46+
47+
- install `gcloud` (see https://cloud.google.com/sdk/docs/install-sdk)
48+
- `gcloud auth login`
49+
- `gcloud config set project your-project-name`
50+
- `gcloud auth configure-docker`
1751

18-
Likewise for Claude:
52+
### .env
53+
54+
You will need to add two .env files, and optionally a third
55+
56+
#### express-pipeline/.env
1957

2058
```
59+
export GCLOUD_STORAGE_BUCKET=some-bucket-name
60+
export GOOGLE_CREDENTIALS_ENCODED=some-alphanumeric-string-from-previous-step
61+
export CLIENT_BASE_URL=http://wherever-your-client-is
62+
63+
# either
64+
export OPENAI_API_KEY=sk-something-something
65+
export OPENAI_API_KEY_PASSWORD=some-password
66+
# or
2167
export ANTHROPIC_API_KEY=sk-something-something
2268
export ANTHROPIC_API_KEY_PASSWORD=some-password
2369
```
2470

25-
If a team member has already set up a Google Cloud Storage project, add the keys for that project to the same `.env` file. Otheriwse,
26-
see the "Setting up a Google Cloud instance" section for how to set up a project.
71+
#### next-client/.env
2772

2873
```
29-
export GCLOUD_STORAGE_BUCKET=some-bucket-name
30-
export GOOGLE_CREDENTIALS_ENCODED=some-alphanumeric-string
74+
export PIPELINE_EXPRESS_URL=http://whereever-youre-hosting-client/generate
3175
```
3276

33-
Then run:
77+
#### examples/.env (optional)
78+
79+
Combine all the env files and place one in there. Only useful if you want to try our examples.
80+
81+
### Local Instance
82+
83+
If you want to run a local version of the app that's not publically accessable:
84+
85+
1. Open up your terminal and navigate to the repo folder (i.e. /Desktop/tttc-light-js)
86+
2. Run `npm run build`.
87+
3. This should open up a server for the next-client and express-pipeline on localhost:3000 and localhost:8080 respectively.
88+
4. This build will be optimized for production.
89+
90+
#### Using docker locally (not recommended)
91+
92+
There should be no need to use docker for local development, except when working on the Dockerfile or testing something docker-related, in which case you might want to use these scripts:
3493

3594
```
36-
npm i
37-
npm run build
38-
npm start
95+
./bin/docker-build-local.sh # build
96+
./bin/docker-run.sh # run
3997
```
4098

41-
Keep the npm server running, and provide input through the page at localhost:8080 (to upload a data file),
42-
or through one of the scripts in the `examples/` folder (to reference a Google Sheet).
99+
### Remote Instance
100+
101+
#### Add a Google Run Service
102+
103+
- Go to the Google Run console
104+
- Create a new service
105+
- In the field "Container image URL", specify the Image Source by clicking the "Select" button and choosing the container image for your project in the "Artifact Registry" tab. The Image Source should look something like `gcr.io/your-project-id/tttc-light-js-app@...`. If you don't see your image listed, make sure you've run the Docker build script above without errors.
106+
- Allow unauthorised access ("unauthenticated invocations" -- like for public APIs).
107+
- Select "CPU is only allocated during request processing" (no need to keep this running at all time)
108+
109+
Note: the first deploy with fail if you haven't set the .env variables as described above.
110+
111+
Your cloud instance is now ready to use!
112+
113+
To upload a new image (e.g. after a fresh git pull), deploy a new docker image to the same instance:
114+
115+
- Run `./bin/docker-build-gcloud.sh`
116+
- Open the google console and search for gloud cloud run.
117+
- Find your project and the `tttc-light-js` app.
118+
- Click "EDIT AND DEPLOY NEW VERSION".
119+
- Find the new docker image that you just pushed from the list of available images.
120+
121+
#### Host Next Client
43122

44-
## API docs
123+
See here for [how to deploy your NextJS app](https://nextjs.org/docs/pages/building-your-application/deploying).
124+
125+
## Usage
126+
127+
### Next Client
128+
129+
This process should work, regardless of whether the app is built locally or remotely. However, the Next client can only take csvs. To submit a Google Sheet, use the API directly.
130+
131+
You can generate your data and view your reports using the Next client.
132+
133+
To do so:
134+
135+
1. Navigate to wherever your Next client is being hosted.
136+
2. On the homepage you should see a form to submit your data.
137+
3. Enter the title, your [api key](#api-key), and add the csv file.
138+
4. Optionally: You can click on the advanced settings for further customization.
139+
5. Click submit. It should soon after give you the url for where your JSON data will be stored and the url to view the report.
140+
6. Depending on how large your file was, it should take anywhere from 30 seconds to 15 minutes to finish. So bookmark the link and check on it periodically.
141+
7. Once the JSON data has been uploaded to Google Cloud, you can view the report by going to` http://[client url]/report/[encoded uri for your stored JSON object]`. You can then save the report from your browser or add it to your webpage using an iframe.
142+
143+
Note: The backend Express app does not save your API keys on the server. The Next app will save your API keys (and other inputs) in your browser's session storage, which should be secure, and will delete when you close your browser tab.
144+
145+
### API
146+
147+
You can submit your data directly to the Express API using whatever method you're comfortable with.
148+
Note: You must have the Next client running to view your report. Otherwise, it will just generate the JSON data.
45149

46150
The enpoint to generate reports is `POST /generate` and it expects a JSON body of the following type:
47151

48152
```
49153
export type Options = {
50-
apiKey?: string; // a valid OpenAI key with gpt-4 access (or a password to use the server's key)
154+
model?: optional;
155+
apiKey: string; // a valid OpenAI key with gpt-4 access (or a password to use the server's key)
51156
data: SourceRow[]; // input data in JSON format, see next section for SourceRow definition
52157
title: string; // title for the report, defaults to ""
53158
question: string; // the question asked to participants, defaults to ""
159+
pieCharts?: {title:string, items: {label:string, count:number}[]}[]; // optional array if you want pie charts in your report
54160
description: string; // intro or abstract to include at the start of the report, defaults to ""
55161
batchSize?: number; // max number of parrallel calls for gpt-4, defaults to 5
56162
filename?: string; // where to store the report on gcloud (it generate a name if none is provided)
57163
systemInstructions?: string; // optional additional instructions for system prompt
58164
clusteringInstructions?: string; // optional additional instructions for clustering step
59165
extractionInstructions?: string; // optional additional instructions for extraction step
60166
dedupInstructions?: string; // optional additional instructions for deduplication step
167+
googleSheet?: {url: string, pieChartColumns?:string[], filterEmails?: string[], onSubmissionPerEmail: boolean} // optional data input using google sheets
61168
};
62169
```
63170

64-
The enpoint is implemented in `./src/server.ts` and you can look at the client's file `./public/index.js` (line 108) for an example of how to use it.
65-
66-
## Data format
171+
The enpoint is implemented in `express-pipeline/src/server.ts`. You can see an example of it being used in `next-client/src/features/actions/SubmitAction.ts` and in the `examples/` folder.
67172

68173
The data field must contain an array of objects of the following type:
69174

@@ -79,80 +184,16 @@ export type SourceRow = {
79184
};
80185
```
81186

82-
## Provided client
83-
84-
Open `localhost:8080/` to see the example of client provided.
85-
The client is written in plain html/css/js in the `public` folder.
86-
87-
## Setting up a Google Cloud instance
88-
89-
### Set up Google Cloud storage & services
90-
91-
First create a new storage bucket:
92-
93-
- Create a Google Cloud project and a Google Cloud Storage bucket
94-
- Add `GCLOUD_STORAGE_BUCKET=name-of-your-bucket` to your `.env`
95-
- Make sure this bucket has public access so that anyone can read from it (to
96-
make your reports accessible from your browser): - Turn off the "Prevent public access" protect - In the "Permissions" tab, click "Grant access." Add the principal `allUsers`
97-
and assign the role `Storage Object User`.
98-
99-
Then create a service account for this bucket:
100-
101-
- In the "IAM & Admin" view, select "Service Accounts" from the left menu, and
102-
then click "Create service account"
103-
- Give this account the "Editor" role
104-
- Create keys for this account and download them as a json file:
105-
- Save this file as `./google-credentials.json`
106-
- Encode this using by running the command `base64 -i ./google-credentials.json`
107-
- Put this in a variable `GOOGLE_CREDENTIALS_ENCODED` in your `.env`
108-
109-
Your .env file should now look like this:
110-
111-
```
112-
export OPENAI_API_KEY=sk-something-something
113-
export OPENAI_API_KEY_PASSWORD=optional-password
114-
export GCLOUD_STORAGE_BUCKET=name-of-your-bucket
115-
export GOOGLE_CREDENTIALS_ENCODED=some-long-encoded-string
116-
```
117-
118-
### Set up gcloud SDK on your machine
119-
120-
- install `gcloud` (see https://cloud.google.com/sdk/docs/install-sdk)
121-
- `gcloud auth login`
122-
- `gcloud config set project your-project-name`
123-
- `gcloud auth configure-docker`
124-
125-
### Deploy Docker instance to Google Cloud Storage
126-
127-
Use the provided script to build and push the docker image:
128-
129-
- `./bin/docker-build-gcloud.sh`
130-
131-
### Add a Google Run Service
132-
133-
- Go to the Google Run console
134-
- Create a new service
135-
- In the field "Container image URL", specify the Image Source by clicking the "Select" button and choosing the container image for your project in the "Artifact Registry" tab. The Image Source should look something like `gcr.io/your-project-id/tttc-light-js-app@...`. If you don't see your image listed, make sure you've run the Docker build script above without errors.
136-
- Allow unauthorised access ("unauthenticated invocations" -- like for public APIs)
137-
- Select "CPU is only allocated during request processing" (no need to keep this running at all time)
187+
## API Key
138188

139-
Note: the first deploy with fail if you haven't set the .env variables as described above
189+
If your organization does not have a key for ChatGPT or Claude, you can obtain one by:
140190

141-
Your cloud instance is now ready to use!
142-
143-
To upload a new image (e.g. after a fresh git pull), deploy a new docker image to the same instance:
191+
1. Go to [OpenAI's website](https://openai.com/) or [Anthropic's Website](https://anthropic.com) and signup or login.
192+
2. Navigate to the API section, accessible from the dashboard or user menu.
193+
3. Choose a plan if you are not already subscribed.
194+
4. Go the API keys section and press the button to generate a new API key.
195+
5. Save this key and not share it.
144196

145-
- Run `./bin/docker-build-gcloud.sh`
146-
- Open the google console and search for gloud cloud run
147-
- Find your project and the `tttc-light-js` app
148-
- Click "EDIT AND DEPLOY NEW VERSION"
149-
- Find the new docker image that you just pushed from the list of available images
197+
## Development and Contributing
150198

151-
## Using docker locally (not recommended)
152-
153-
There should be no need to use docker for local development, except when working on the Dockerfile or testing something docker-related, in which case you might want to use these scripts:
154-
155-
```
156-
./bin/docker-build-local.sh # build
157-
./bin/docker-run.sh # run
158-
```
199+
See DEV.md

0 commit comments

Comments
 (0)