Skip to content

Commit c00a10c

Browse files
ammsalmedependabot[bot]natlibfi-jonollil
authored
Dev (#136)
* Update dependabot [MRA-641] * Dependency & maintenance update (#134) * Bump http-status from 1.7.0 to 1.7.3 (#133) Bumps [http-status](https://github.com/adaltas/node-http-status) from 1.7.0 to 1.7.3. - [Changelog](https://github.com/adaltas/node-http-status/blob/master/CHANGELOG.md) - [Commits](adaltas/node-http-status@v1.7.0...v1.7.3) --- updated-dependencies: - dependency-name: http-status dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump @natlibfi/marc-record-validate from 8.0.1 to 8.0.3 (#132) Bumps [@natlibfi/marc-record-validate](https://github.com/natlibfi/marc-record-validate) from 8.0.1 to 8.0.3. - [Release notes](https://github.com/natlibfi/marc-record-validate/releases) - [Commits](NatLibFi/marc-record-validate@v8.0.1...v8.0.3) --- updated-dependencies: - dependency-name: "@natlibfi/marc-record-validate" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump @natlibfi/melinda-commons from 13.0.6 to 13.0.7 (#131) Bumps [@natlibfi/melinda-commons](https://github.com/natlibfi/melinda-commons-js) from 13.0.6 to 13.0.7. - [Release notes](https://github.com/natlibfi/melinda-commons-js/releases) - [Commits](NatLibFi/melinda-commons-js@v13.0.6...v13.0.7) --- updated-dependencies: - dependency-name: "@natlibfi/melinda-commons" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump @natlibfi/marc-record-validators-melinda from 10.13.0 to 10.14.0 (#130) Bumps [@natlibfi/marc-record-validators-melinda](https://github.com/natlibfi/marc-record-validators-melinda) from 10.13.0 to 10.14.0. - [Release notes](https://github.com/natlibfi/marc-record-validators-melinda/releases) - [Commits](NatLibFi/marc-record-validators-melinda@v10.13.0...v10.14.0) --- updated-dependencies: - dependency-name: "@natlibfi/marc-record-validators-melinda" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump @babel/traverse from 7.22.20 to 7.23.2 (#135) Bumps [@babel/traverse](https://github.com/babel/babel/tree/HEAD/packages/babel-traverse) from 7.22.20 to 7.23.2. - [Release notes](https://github.com/babel/babel/releases) - [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md) - [Commits](https://github.com/babel/babel/commits/v7.23.2/packages/babel-traverse) --- updated-dependencies: - dependency-name: "@babel/traverse" dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Feature get logs with filter (#137) * Unit testing mongo operators started * fix empty array logitemtype * Add test etc. (#139) * add tests * improve old tests * add test to bad date * Dev+hotfix (#140) * Use subfieldValues: false for toMarcRecord * Fix tests * More tests * Update marc-record:v8.0.0 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joni Ollila <[email protected]>
1 parent 7700e12 commit c00a10c

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

75 files changed

+14290
-9109
lines changed

‎.github/dependabot.yml

+9-4
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ updates:
1010
interval: "daily"
1111
time: "06:30"
1212
timezone: "Europe/Helsinki"
13-
target-branch: "dependencies"
13+
target-branch: "next"
1414

1515
# Minor updates to npm production dependencies daily
1616
- package-ecosystem: "npm"
@@ -20,9 +20,14 @@ updates:
2020
time: "06:45"
2121
timezone: "Europe/Helsinki"
2222
versioning-strategy: lockfile-only
23-
allow:
24-
- dependency-type: "production"
25-
target-branch: "dependencies"
23+
target-branch: "next"
24+
pull-request-branch-name:
25+
separator: "-"
26+
groups:
27+
production-dependencies:
28+
dependency-type: "production"
29+
development-dependencies:
30+
dependency-type: "development"
2631

2732
# Major updates to npm dependencies weekly @tuesday
2833
# Not possible yet https://github.com/dependabot/dependabot-core/issues/1778

‎package-lock.json

+9,465-8,964
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎package.json

+131-119
Original file line numberDiff line numberDiff line change
@@ -1,123 +1,135 @@
11
{
2-
"name": "@natlibfi/melinda-rest-api-commons",
3-
"description": "Shared modules for microservices of Melinda rest api import system",
4-
"author": {
5-
"name": "The National Library of Finland"
6-
},
7-
"keywords": [],
8-
"homepage": "https://github.com/natlibfi/melinda-rest-api-commons",
9-
"bugs": {
10-
"url": "https://github.com/natlibfi/melinda-rest-api-commons/issues"
11-
},
12-
"repository": {
13-
"type": "git",
14-
"url": "[email protected]:natlibfi/melinda-rest-api-commons.git"
15-
},
16-
"license": "AGPL-3.0+",
17-
"version": "4.0.15",
18-
"main": "./dist/index.js",
19-
"engines": {
20-
"node": ">=18"
21-
},
22-
"publishConfig": {
23-
"access": "public"
24-
},
25-
"scripts": {
26-
"prepare": "npm run build",
27-
"lint": "eslint ./src",
28-
"lint:dev": "eslint ./src --fix",
29-
"test": "npm run lint && npm run test:base -- src/**/*.spec.js",
30-
"test:dev": "npm run lint:dev && npm run test:base -- src/**/*.spec.js",
31-
"test:base": "cross-env NODE_ENV=test nyc mocha --require @babel/register --exclude src/amqp.health.spec.js --exclude src/amqp.tests.spec.js",
32-
"test:local": "cross-env NODE_ENV=test nyc mocha --require @babel/register -- src/amqp.*.spec.js",
33-
"test:healthcheck": "cross-env NODE_ENV=test nyc mocha --require @babel/register -- src/amqp.health.spec.js",
34-
"coverage": "npm run cover:unit && npm run cover:report",
35-
"cover:unit": "nyc --silent npm run test",
36-
"cover:report": "nyc report --reporter=lcov --reporter=text",
37-
"dev": "NODE_ENV=development cross-env DEBUG='@natlibfi/*' nodemon",
38-
"build": "babel src --source-maps --copy-files --delete-dir-on-start --out-dir=dist"
39-
},
40-
"dependencies": {
41-
"@natlibfi/marc-record": "^7.3.1",
42-
"@natlibfi/marc-record-serializers": "^10.1.1",
43-
"@natlibfi/marc-record-validate": "^8.0.1",
44-
"@natlibfi/marc-record-validators-melinda": "^10.13.0",
45-
"@natlibfi/melinda-backend-commons": "^2.2.1",
46-
"@natlibfi/melinda-commons": "^13.0.6",
47-
"amqplib": "^0.10.3",
48-
"debug": "^4.3.4",
49-
"http-status": "^1.7.0",
50-
"moment": "^2.29.4",
51-
"mongo-sanitize": "^1.1.0",
52-
"mongodb": "^4.17.1"
53-
},
54-
"devDependencies": {
55-
"@babel/cli": "^7.22.15",
56-
"@babel/core": "^7.22.20",
57-
"@babel/plugin-transform-runtime": "7.22.15",
58-
"@babel/preset-env": "^7.22.20",
59-
"@natlibfi/eslint-config-melinda-backend": "^3.0.2",
60-
"@natlibfi/fixugen": "^2.0.1",
61-
"@natlibfi/fixura": "^3.0.1",
62-
"babel-plugin-istanbul": "^6.1.1",
63-
"babel-plugin-rewire": "^1.2.0",
64-
"chai": "^4.3.8",
65-
"cross-env": "^7.0.3",
66-
"eslint": "^8.49.0",
67-
"mocha": "^10.2.0",
68-
"nodemon": "^3.0.1",
69-
"nyc": "^15.1.0"
70-
},
71-
"eslintConfig": {
72-
"extends": "@natlibfi/melinda-backend"
73-
},
74-
"babel": {
75-
"presets": [
76-
[
77-
"@babel/preset-env",
78-
{
79-
"targets": "maintained node versions"
80-
}
81-
]
82-
],
83-
"plugins": [
84-
[
85-
"@babel/transform-runtime"
86-
]
87-
],
88-
"env": {
89-
"test": {
90-
"plugins": [
91-
"istanbul",
92-
"rewire"
93-
]
94-
}
2+
"name": "@natlibfi/melinda-rest-api-commons",
3+
"description": "Shared modules for microservices of Melinda rest api import system",
4+
"author": {
5+
"name": "The National Library of Finland"
6+
},
7+
"keywords": [],
8+
"homepage": "https://github.com/natlibfi/melinda-rest-api-commons",
9+
"bugs": {
10+
"url": "https://github.com/natlibfi/melinda-rest-api-commons/issues"
11+
},
12+
"repository": {
13+
"type": "git",
14+
"url": "[email protected]:natlibfi/melinda-rest-api-commons.git"
15+
},
16+
"license": "AGPL-3.0+",
17+
"version": "4.0.17-alpha.3",
18+
"main": "./dist/index.js",
19+
"engines": {
20+
"node": ">=18"
21+
},
22+
"publishConfig": {
23+
"access": "public"
24+
},
25+
"scripts": {
26+
"prepare": "npm run build",
27+
"lint": "eslint ./src",
28+
"lint:dev": "eslint ./src --fix",
29+
"test": "npm run lint && npm run test:base -- src/**/*.spec.js src/*.spec.js",
30+
"test:dev": "npm run lint:dev && npm run test:base -- src/**/*.spec.js",
31+
"test:base": "cross-env NODE_ENV=test nyc mocha --package ./package.json --exclude src/amqp.health.spec.js --exclude src/amqp.tests.spec.js",
32+
"test:local": "cross-env NODE_ENV=test nyc mocha --package ./package.json -- src/amqp.*.spec.js",
33+
"test:healthcheck": "cross-env NODE_ENV=test nyc mocha --require @babel/register -- src/amqp.health.spec.js",
34+
"coverage": "npm run cover:unit && npm run cover:report",
35+
"cover:unit": "nyc --silent npm run test",
36+
"cover:report": "nyc report --reporter=lcov --reporter=text",
37+
"dev": "NODE_ENV=development cross-env DEBUG='@natlibfi/*' nodemon",
38+
"build": "babel src --source-maps --copy-files --delete-dir-on-start --out-dir=dist"
39+
},
40+
"dependencies": {
41+
"@natlibfi/marc-record": "^8.0.0",
42+
"@natlibfi/marc-record-serializers": "^10.1.1",
43+
"@natlibfi/marc-record-validate": "^8.0.1",
44+
"@natlibfi/marc-record-validators-melinda": "^10.13.0",
45+
"@natlibfi/melinda-backend-commons": "^2.2.1",
46+
"@natlibfi/melinda-commons": "^13.0.6",
47+
"amqplib": "^0.10.3",
48+
"debug": "^4.3.4",
49+
"http-status": "^1.7.0",
50+
"moment": "^2.29.4",
51+
"mongo-sanitize": "^1.1.0",
52+
"mongodb": "^4.17.1"
53+
},
54+
"devDependencies": {
55+
"@babel/cli": "^7.22.15",
56+
"@babel/core": "^7.22.20",
57+
"@babel/plugin-transform-runtime": "^7.22.15",
58+
"@babel/preset-env": "^7.22.20",
59+
"@natlibfi/eslint-config-melinda-backend": "^3.0.2",
60+
"@natlibfi/fixugen": "^2.0.1",
61+
"@natlibfi/fixura": "^3.0.1",
62+
"@natlibfi/fixura-mongo": "^2.0.10-alpha.1",
63+
"babel-plugin-istanbul": "^6.1.1",
64+
"babel-plugin-rewire": "^1.2.0",
65+
"chai": "^4.3.8",
66+
"cross-env": "^7.0.3",
67+
"eslint": "^8.49.0",
68+
"mocha": "^10.2.0",
69+
"nodemon": "^3.0.1",
70+
"nyc": "^15.1.0"
71+
},
72+
"eslintConfig": {
73+
"extends": "@natlibfi/melinda-backend"
74+
},
75+
"babel": {
76+
"presets": [
77+
[
78+
"@babel/preset-env",
79+
{
80+
"targets": "maintained node versions"
9581
}
96-
},
97-
"nyc": {
98-
"exclude": [
99-
"**/*.spec.js"
100-
],
101-
"reporter": [
102-
"lcov",
103-
"html"
104-
],
105-
"require": [
106-
"@babel/register"
107-
],
108-
"sourceMap": false,
109-
"instrument": false,
110-
"lines": 80,
111-
"statements": 80,
112-
"functions": 80,
113-
"branches": 80
114-
},
115-
"nodemonConfig": {
116-
"exec": "npm run test:dev",
117-
"watch": [
118-
"src/*"
119-
],
120-
"ext": "js",
121-
"delay": 2.5
82+
]
83+
],
84+
"plugins": [
85+
[
86+
"@babel/transform-runtime"
87+
]
88+
],
89+
"env": {
90+
"test": {
91+
"plugins": [
92+
"istanbul",
93+
"rewire"
94+
]
95+
}
12296
}
97+
},
98+
"mocha": {
99+
"spec": [
100+
"src/*.spec.js"
101+
],
102+
"require": [
103+
"@babel/register"
104+
],
105+
"inline-diffs": true,
106+
"maxDiffSize": 25000,
107+
"bail": true,
108+
"exit": true
109+
},
110+
"nyc": {
111+
"exclude": [
112+
"**/*.spec.js"
113+
],
114+
"reporter": [
115+
"text"
116+
],
117+
"require": [
118+
"@babel/register"
119+
],
120+
"sourceMap": false,
121+
"instrument": false,
122+
"lines": 80,
123+
"statements": 80,
124+
"functions": 80,
125+
"branches": 80
126+
},
127+
"nodemonConfig": {
128+
"exec": "npm run test:dev",
129+
"watch": [
130+
"src/*"
131+
],
132+
"ext": "js",
133+
"delay": 2.5
134+
}
123135
}

‎src/amqp.js

+2-2
Original file line numberDiff line numberDiff line change
@@ -276,8 +276,8 @@ export default async function (AMQP_URL, runHealthCheck = false) {
276276

277277
return messages.map(message => {
278278
const content = JSON.parse(message.content.toString());
279-
// Should we have here validationOptions
280-
return new MarcRecord(content.data);
279+
// Use subfieldValues: false validationOption here
280+
return new MarcRecord(content.data, {subfieldValues: false});
281281
});
282282
}
283283

‎src/mongoLog.js

+50-10
Original file line numberDiff line numberDiff line change
@@ -35,14 +35,14 @@ import httpStatus from 'http-status';
3535
import sanitize from 'mongo-sanitize';
3636
import {LOG_ITEM_TYPE} from './constants';
3737

38-
export default async function (MONGO_URI) {
38+
export default async function (MONGO_URI, dbName = 'rest-api') {
3939
const logger = createLogger();
4040

4141
// Connect to mongo (MONGO)
4242
const client = await MongoClient.connect(MONGO_URI, {useNewUrlParser: true, useUnifiedTopology: true});
43-
const db = client.db('rest-api');
43+
const db = client.db(dbName);
4444
const collection = 'logs';
45-
return {addLogItem, query, queryById, getListOfLogs, getExpandedListOfLogs, protect, remove, removeBySequences};
45+
return {addLogItem, query, queryById, getListOfLogs, getListOfCatalogers, getExpandedListOfLogs, protect, remove, removeBySequences};
4646

4747
async function addLogItem(logItem) {
4848
const time = moment().toDate();
@@ -52,12 +52,14 @@ export default async function (MONGO_URI) {
5252
protected: false
5353
};
5454
try {
55+
// console.log(newLogItem); // eslint-disable-line
5556
checkLogItemType(logItem.logItemType, false);
5657
const result = await db.collection(collection).insertOne(newLogItem);
5758
if (result.acknowledged) {
5859
const {blobSequence, blobSequenceStart, blobSequenceEnd} = logItem;
5960
const itemString = blobSequenceStart && blobSequenceEnd ? `${blobSequenceStart} - ${blobSequenceEnd}` : `${blobSequence}`;
6061
logger.info(`*** New ${logItem.logItemType} added for ${logItem.correlationId}, blobSequence(s): ${itemString}. ***`);
62+
// console.log('addLogItem done', result); // eslint-disable-line
6163
return;
6264
}
6365
throw new ApiError(httpStatus.INTERNAL_SERVER_ERROR);
@@ -107,20 +109,41 @@ export default async function (MONGO_URI) {
107109
return {status: result.length > 0 ? httpStatus.OK : httpStatus.NOT_FOUND, payload: result.length > 0 ? result : 'No logs found'};
108110
}
109111

112+
async function getListOfCatalogers() {
113+
logger.debug(`Getting list of Catalogers`);
114+
115+
const result = await db.collection(collection) // eslint-disable-line functional/immutable-data
116+
.distinct('cataloger');
117+
118+
return result;
119+
}
120+
110121
// getExpandedListOfLogs returns groped MERGE_LOGs and MATCH_LOGs
111-
async function getExpandedListOfLogs() {
122+
async function getExpandedListOfLogs({logItemTypes = [LOG_ITEM_TYPE.MERGE_LOG, LOG_ITEM_TYPE.MATCH_LOG], catalogers = [], dateBefore = new Date(), dateAfter = new Date('2000-01-01')}) {
123+
logger.debug(`commons: logItemTypes: ${JSON.stringify(logItemTypes)}, dateAfter: ${dateAfter}, dateBefore: ${dateBefore}}, catalogers: ${JSON.stringify(catalogers)}`);
124+
logger.debug(JSON.stringify(generateMatchObject(logItemTypes, catalogers, dateBefore, dateAfter))); // eslint-disable-line
112125
//checkLogItemType(logItemType, false, false);
113126
logger.debug(`Getting expanded list of logs`);
114127
const pipeline = [
115128
// currently return only MERGE_LOG and MATCH_LOG
116-
{'$match': {'logItemType': {'$in': ['MERGE_LOG', 'MATCH_LOG']}}},
117-
{'$sort':
118-
{'correlationId': 1, 'logItemType': 1, 'creationTime': 1}},
119-
{'$group':
120-
{'_id': {'correlationId': '$correlationId', 'logItemType': '$logItemType'},
129+
generateMatchObject(logItemTypes, catalogers, dateBefore, dateAfter),
130+
{
131+
'$sort':
132+
{'correlationId': 1, 'logItemType': 1, 'creationTime': 1}
133+
},
134+
{
135+
'$group':
136+
{
137+
'_id': {'correlationId': '$correlationId', 'logItemType': '$logItemType'},
121138
'creationTime': {'$first': '$creationTime'},
122139
'cataloger': {'$first': '$cataloger'},
123-
'logCount': {'$sum': 1}}}
140+
'logCount': {'$sum': 1}
141+
}
142+
},
143+
{
144+
'$sort':
145+
{'correlationId': 1, 'logItemType': 1, 'creationTime': 1}
146+
}
124147
];
125148

126149
const result = await db.collection(collection) // eslint-disable-line functional/immutable-data
@@ -138,6 +161,23 @@ export default async function (MONGO_URI) {
138161

139162
logger.debug(`Query result: ${fixedResult.length > 0 ? `Found ${fixedResult.length} log items!` : 'Not found!'}`);
140163
return {status: fixedResult.length > 0 ? httpStatus.OK : httpStatus.NOT_FOUND, payload: fixedResult.length > 0 ? fixedResult : 'No logs found'};
164+
165+
function generateMatchObject(logItemTypes, catalogers, dateBefore, dateAfter) {
166+
const dateBeforeIso = new Date(dateBefore).toISOString();
167+
const dateAfterIso = new Date(dateAfter).toISOString();
168+
const matchOptions = {
169+
'$match': {
170+
'logItemType': logItemTypes.length > 0 ? {'$in': logItemTypes} : /.*/ui,
171+
'cataloger': catalogers.length > 0 ? {'$in': catalogers} : /.*/ui,
172+
'creationTime': {
173+
'$gte': dateAfterIso,
174+
'$lte': dateBeforeIso
175+
}
176+
}
177+
};
178+
179+
return matchOptions;
180+
}
141181
}
142182

143183
async function protect(correlationId, blobSequence) {

0 commit comments

Comments
 (0)