Skip to content

Commit e9c38e6

Browse files
diofeherdiofeher
authored
fix: docker compose setup (#293)
* fix: docker compose setup Signed-off-by: Diógenes Fernandes <[email protected]> * reoder Signed-off-by: Diógenes Fernandes <[email protected]> * add client and docs Signed-off-by: Diógenes Fernandes <[email protected]> * Adding docs Signed-off-by: Diógenes Fernandes <[email protected]> * remove unused code Signed-off-by: Diógenes Fernandes <[email protected]> * docs for feed script Signed-off-by: Diógenes Fernandes <[email protected]> * better docs Signed-off-by: Diogenes Fernandes <[email protected]> * refactoring client Signed-off-by: Diogenes Fernandes <[email protected]> * improving docs Signed-off-by: Diogenes Fernandes <[email protected]> * using postgres adapted Signed-off-by: Diogenes Fernandes <[email protected]> * adding Makefile Signed-off-by: Diogenes Fernandes <[email protected]> * adding comments Signed-off-by: Diogenes Fernandes <[email protected]> * update docs Signed-off-by: Diógenes Fernandes <[email protected]> --------- Signed-off-by: Diógenes Fernandes <[email protected]> Signed-off-by: Diogenes Fernandes <[email protected]>
1 parent b457ba9 commit e9c38e6

File tree

15 files changed

+155
-31
lines changed

15 files changed

+155
-31
lines changed

‎.gitignore

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
/work
22
/docs
33
/registry
4-
.idea
4+
.idea
5+
.DS_Store

‎Makefile

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# generate-registry runs `go generate` to build the registry files and saves them to /tmp/registry
2+
generate-registry:
3+
@cd "$(CURDIR)/backend" && go generate ./... && go run ./cmd/generate/ --licenses-file ../licenses.json --destination-dir /tmp/registry
4+
5+
# load-registry feed the data from /tmp/registry into the local R2 bucket (search/worker/.wrangler/state/r2) folder
6+
load-registry:
7+
@cd "$(CURDIR)/search/worker" && npm run feed-data
8+
9+
# index-search downloads search data from api.opentofu.org and feeds that data into the postgres database used for searching
10+
index-search:
11+
@cd "$(CURDIR)/search/pg-indexer" && PG_CONNECTION_STRING=postgres://postgres:secret@localhost:5432/postgres?sslmode=disable go run .
12+
13+
# after docker-compose us running, run this command to feed data into the application
14+
feed-data:
15+
make generate-registry
16+
make load-registry
17+
make index-search

‎docker-compose.yml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,5 +29,3 @@ services:
2929
- source: ./search/worker
3030
target: /work
3131
type: bind
32-
environment:
33-
- DATABASE_URL=postgresql://posgres:secret@posgres/postgres

‎search/pg-indexer/schema.sql

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,3 +28,6 @@ CREATE TABLE IF NOT EXISTS import_jobs
2828

2929
ALTER TABLE entities ADD COLUMN popularity INT DEFAULT 0;
3030
ALTER TABLE entities ADD COLUMN warnings INT DEFAULT 0;
31+
32+
-- pg_trgm is used for similarity function support
33+
CREATE EXTENSION IF NOT EXISTS pg_trgm;

‎search/worker/README.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Search worker
2+
3+
The Search worker is used as the backend that route requests on our Cloudflare Worker. Static files are served from R2 directly and search goes to our serverless Neon Database. There's a cache in Cloudflare worker as well that is handled by this application.
4+
5+
## Installation
6+
7+
To setup your local environment for this project:
8+
9+
1. Copy the `.example.dev.vars` to a new `.dev.vars` file in the same folder.
10+
2. When running docker-compose, the whole folder will be copied, so you can overwrite `.dev.vars` with your environment variables.
11+
12+
## Feeding data
13+
14+
There's a `Makefile` folder on the root with commands to setup the data in this application.

‎search/worker/example.dev.vars

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
DATABASE_URL = "postgresql://postgres:secret@postgres/postgres"
2+
ENVIRONMENT = "dev"

‎search/worker/package-lock.json

Lines changed: 20 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎search/worker/package.json

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
"dev": "wrangler dev",
88
"start": "wrangler dev",
99
"test": "vitest",
10+
"feed-data": "node scripts/feed-data-r2.js",
1011
"cf-typegen": "wrangler types"
1112
},
1213
"devDependencies": {
@@ -17,6 +18,7 @@
1718
"wrangler": "^3.60.3"
1819
},
1920
"dependencies": {
20-
"@neondatabase/serverless": "^0.9.4"
21+
"@neondatabase/serverless": "^0.9.4",
22+
"postgres": "^3.4.5"
2123
}
2224
}

‎search/worker/scripts/feed-data-r2.js

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
// This script is used to retrieve data generated by `go run generate.go` at backend folder.
2+
// It will generate registry data that will be used by this script to feed into wrangler dev folder, mimicking a R2 bucket.
3+
// This script expects data will be at /tmp/registry.
4+
const fs = require('fs');
5+
const path = require('path');
6+
const { exec } = require('node:child_process');
7+
8+
const directoryPath = '/tmp/registry';
9+
10+
fs.readdir(directoryPath, { recursive: true }, (err, files) => {
11+
if (err) {
12+
console.error('Error reading directory:', err);
13+
return;
14+
}
15+
16+
files.forEach(fileName => {
17+
const realFilePath = path.join(directoryPath, fileName);
18+
const r2Name = realFilePath.replace(directoryPath, "");
19+
20+
const cmd = `npx wrangler r2 object put registry-ui-api${r2Name} --file ${realFilePath} --local`
21+
console.log(cmd)
22+
exec(cmd, function (err, stdout, stderr) {
23+
console.log(stdout);
24+
console.error(stderr);
25+
});
26+
27+
});
28+
});

‎search/worker/src/client.ts

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
import { Client } from '@neondatabase/serverless';
2+
import { DBClient } from "./types";
3+
import postgres from "postgres";
4+
5+
// PGClient is used to connect to the local postgres instance. In production, we use a neon serverless database.
6+
// This adapter is created in order to interact and maintain the same API neon serverless uses.
7+
export class PGClient {
8+
db: postgres.Sql
9+
constructor(connection: string) {
10+
this.db = postgres(connection)
11+
}
12+
13+
connect() {}
14+
end() : Promise<any> {
15+
return Promise.resolve();
16+
}
17+
18+
query(query: string, queryParams: string[]): any {
19+
// Adapting to neonserverless/db return of rows
20+
// unsafe usage should be good since we are using this adapter only locally
21+
return { rows: this.db.unsafe(query, queryParams)}
22+
}
23+
}
24+
25+
function getClientInstance(environment: string, databaseUrl: string): DBClient {
26+
if (environment == "dev") {
27+
return new PGClient(databaseUrl);
28+
} else {
29+
return new Client(databaseUrl);
30+
}
31+
}
32+
33+
export async function getClient(environment: string, databaseUrl: string): Promise<DBClient> {
34+
if (databaseUrl === undefined) {
35+
throw new Error('DATABASE_URL is required');
36+
}
37+
38+
const now = performance.now();
39+
const client = getClientInstance(environment, databaseUrl);
40+
await client.connect();
41+
console.log('Connected to database in', performance.now() - now, 'ms');
42+
return client;
43+
}

0 commit comments

Comments
 (0)