Skip to content
This repository was archived by the owner on Mar 12, 2025. It is now read-only.

Commit 5fc74de

Browse files
callmekatootiecallmekatootie
committed
Initial commit
0 parents  commit 5fc74de

17 files changed

+4159
-0
lines changed

‎.gitignore

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
node_modules
2+
3+
.env

‎README.md

+97
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# UBahn - Backend Job
2+
3+
Queries U-bahn's database for users and posts them to kafka to be picked up by the u-bahn-skill processor
4+
5+
## Dependencies
6+
7+
- Nodejs(v12+)
8+
- docker
9+
- Redis
10+
- PostgreSQL
11+
12+
## Configuration
13+
14+
Configuration for the skill processor trigger is at `config/default.js`.
15+
The following parameters can be set in config files or in env variables:
16+
17+
- LOG_LEVEL: the log level; default value: 'debug'
18+
- DB_NAME: the database name
19+
- DB_USERNAME: the database username
20+
- DB_PASSWORD: the database password
21+
- DB_HOST: the database host
22+
- DB_PORT: the database port
23+
- AUTH0_URL: Auth0 URL, used to get TC M2M token
24+
- AUTH0_AUDIENCE: Auth0 audience, used to get TC M2M token
25+
- TOKEN_CACHE_TIME: Auth0 token cache time, used to get TC M2M token
26+
- AUTH0_CLIENT_ID: Auth0 client id, used to get TC M2M token
27+
- AUTH0_CLIENT_SECRET: Auth0 client secret, used to get TC M2M token
28+
- AUTH0_PROXY_SERVER_URL: Proxy Auth0 URL, used to get TC M2M token
29+
- BUSAPI_URL: Topcoder Bus API URL
30+
- KAFKA_ERROR_TOPIC: The error topic at which bus api will publish any errors
31+
- KAFKA_MESSAGE_ORIGINATOR: The originator value for the kafka messages
32+
- SKILL_SYNC_TOPIC: the sync skill Kafka message topic, default value is 'backgroundjob.sync.user.skills'
33+
- REDIS_HOST: the redis host, default value is 'localhost',
34+
- REDIS_PORT: the redis port, default value is 6379,
35+
- REDIS_PASSWORD: the redis password,
36+
- REDIS_USER: the redis user,
37+
- USER_RECORD_OFFSET: the record offset, default value is 0,
38+
- USER_RECORD_LIMIT: the record limit, default value is 10,
39+
- OFFSET_REDIS_KEY: the redis of offset key, default value is 'USER_RECORD_OFFSET',
40+
- SLEEP_TIME: The pause time between two create operations, default value: 1000 ms
41+
42+
## Local install with Docker
43+
44+
- Navigate to the directory `docker-redis-db`
45+
- Run the command `docker-compose up -d`
46+
47+
## Local deployment
48+
49+
1. From the project root directory, run the following command to install the dependencies
50+
51+
```bash
52+
npm install
53+
```
54+
55+
2. To run linters if required
56+
57+
```bash
58+
npm run lint
59+
```
60+
61+
To fix possible lint errors:
62+
63+
```bash
64+
npm run lint:fix
65+
```
66+
67+
3. Start the processor and health check dropin
68+
69+
```bash
70+
npm start
71+
```
72+
73+
## Local Deployment with Docker
74+
75+
To run the Skill Processor Trigger using docker, follow the below steps
76+
77+
1. Navigate to the directory `docker`
78+
79+
2. Rename the file `sample.api.env` to `api.env`
80+
81+
3. Set the auth0 config in the file `api.env`
82+
83+
4. Once that is done, run the following command
84+
85+
```bash
86+
docker-compose up
87+
```
88+
89+
5. When you are running the application for the first time, It will take some time initially to download the image and install the dependencies
90+
91+
## Verification
92+
93+
1. config `AUTH0_CLIENT_ID`, `AUTH0_CLIENT_SECRET`
94+
2. run `npm run insert-data` to load test data to db
95+
3. run `npm start` to post event when there is not offset key in redis
96+
4. run `npm start` to post event when there is an offset key in redis
97+
6. watch the app console, it will show the offset and post message

‎build.sh

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
#!/bin/bash
2+
set -eo pipefail
3+
APP_NAME=$1
4+
UPDATE_CACHE=""
5+
echo "" > docker/api.env
6+
docker-compose -f docker/docker-compose.yml build $APP_NAME
7+
docker create --name app $APP_NAME:latest
8+
9+
if [ -d node_modules ]
10+
then
11+
mv package-lock.json old-package-lock.json
12+
docker cp app:/$APP_NAME/package-lock.json package-lock.json
13+
set +eo pipefail
14+
UPDATE_CACHE=$(cmp package-lock.json old-package-lock.json)
15+
set -eo pipefail
16+
else
17+
UPDATE_CACHE=1
18+
fi
19+
20+
if [ "$UPDATE_CACHE" == 1 ]
21+
then
22+
docker cp app:/$APP_NAME/node_modules .
23+
fi

‎config/default.js

+35
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
/**
2+
* The default configuration file.
3+
*/
4+
5+
module.exports = {
6+
LOG_LEVEL: process.env.LOG_LEVEL || 'debug',
7+
8+
BUSAPI_URL: process.env.BUSAPI_URL || 'https://api.topcoder-dev.com/v5',
9+
KAFKA_ERROR_TOPIC: process.env.KAFKA_ERROR_TOPIC || 'common.error.reporting',
10+
KAFKA_MESSAGE_ORIGINATOR: process.env.KAFKA_MESSAGE_ORIGINATOR || 'backgroundjob.service',
11+
12+
SKILL_SYNC_TOPIC: process.env.SKILL_SYNC_TOPIC || 'backgroundjob.sync.user.skills',
13+
14+
AUTH0_URL: process.env.AUTH0_URL || 'https://topcoder-dev.auth0.com/oauth/token', // Auth0 credentials
15+
AUTH0_AUDIENCE: process.env.AUTH0_TOPCODER_AUDIENCE || 'https://m2m.topcoder-dev.com/',
16+
TOKEN_CACHE_TIME: process.env.TOKEN_CACHE_TIME,
17+
AUTH0_CLIENT_ID: process.env.AUTH0_CLIENT_ID,
18+
AUTH0_CLIENT_SECRET: process.env.AUTH0_CLIENT_SECRET,
19+
AUTH0_PROXY_SERVER_URL: process.env.AUTH0_PROXY_SERVER_URL,
20+
21+
DB_NAME: process.env.DB_NAME || 'ubahn-db',
22+
DB_USERNAME: process.env.DB_USER || 'postgres',
23+
DB_PASSWORD: process.env.DB_PASSWORD || 'password',
24+
DB_HOST: process.env.DB_HOST || 'localhost',
25+
DB_PORT: process.env.DB_PORT || 5432,
26+
27+
REDIS_HOST: process.env.REDIS_HOST || 'localhost',
28+
REDIS_PORT: parseInt(process.env.REDIS_PORT || 6379),
29+
REDIS_PASSWORD: process.env.REDIS_PASSWORD,
30+
REDIS_USER: process.env.REDIS_USER,
31+
USER_RECORD_OFFSET: parseInt(process.env.USER_RECORD_OFFSET || 0),
32+
USER_RECORD_LIMIT: parseInt(process.env.USER_RECORD_LIMIT || 10),
33+
OFFSET_REDIS_KEY: process.env.OFFSET_REDIS_KEY || 'USER_RECORD_OFFSET',
34+
SLEEP_TIME: parseInt(process.env.SLEEP_TIME || 1000)
35+
}

‎docker-redis-db/docker-compose.yml

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
version: '3'
2+
services:
3+
postgres:
4+
image: "postgres:12.4"
5+
ports:
6+
- "5432:5432"
7+
environment:
8+
POSTGRES_PASSWORD: password
9+
POSTGRES_USER: postgres
10+
POSTGRES_DB: ubahn-db
11+
redis:
12+
image: "redis:6.2"
13+
ports:
14+
- "6379:6379"

‎docker/Dockerfile

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Use the base image with Node.js 12.16.3
2+
FROM node:12.16.3
3+
4+
# Copy the current directory into the Docker image
5+
COPY . /ubahn-skill-processor-trigger
6+
7+
# Set working directory for future use
8+
WORKDIR /ubahn-skill-processor-trigger
9+
10+
# Install the dependencies from package.json
11+
RUN npm install
12+
CMD npm start

‎docker/docker-compose.yml

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
version: '3'
2+
services:
3+
ubahn-skill-processor-trigger:
4+
image: ubahn-skill-processor-trigger:latest
5+
build:
6+
context: ../
7+
dockerfile: docker/Dockerfile
8+
env_file:
9+
- api.env
10+
network_mode: "host"

‎docker/sample.api.env

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
AUTH0_CLIENT_ID=<Auth0 Client ID>
2+
AUTH0_CLIENT_SECRET=<Auth0 Client Secret>

0 commit comments

Comments
 (0)