Staging Changes (#1769)

* [DAPS-1630] - Bug core metrics update (#1632)

* refactor: remove system secret from http params.

* fix: metricsUpdateMsgCounts and newJsonMetricParse methods.

* feat: add test_DatabaseAPI test to core server CMakeLists.txt

* [DAPS-1609] Move dependency build to submodule (#1624)

* refactor: move base image check to separate stage and file.
* refactor: add .gitlab-ci.yml and stage_image_check.yml files.
* refactor: create custom script for base image check job.
* refactor: add compressed file extensions to gitignore.
* fix: correct url from pipeline to trigger/pipeline
* fix: update docker dependencies file to latest commit which includes package.json.
* refactor: rely on dependencies from downstream pipeline.
* refactor: add common library that was missing from repo, foxx and core.
* refactor: fix build-arg for intermediate build.
* refactor: switch gcs image to using correct dependencies image.
* fix: remove compose build of old Dockefile.dependencies.
* fix: add ability to make changes in config folder.
* refactor: remove common from foxx docker container build, it should not be needed.
* fix: add missing LD_LIBRARY_PATH before curl command in install_foxx.sh script.


---------

Co-authored-by: nedvedba <[email protected]>

* [DAPS-1625] - user id map for session key josh (#1644) (#1647)

* [DAPS-1625]  fix bug in session and transient keys.

---------

Co-authored-by: AronPerez <[email protected]>

* Update devel from staging (#1649)

* Update issue templates

* Update issue templates

* Update pull_request_template.md (#1433)

* [Release] v3.13.01 Resolve Master Branch Diff (#1485)

* Address formatting of jquery file
* add more descriptive posix path error message
* Remove redundant password file creation command
* Add eslint plugin for jsdoc to github action
* Update esling.config.js to error if invalid jsdoc
* Add missing eslint jsdoc plugin
* Remove non existant rule
* Fix jsdoc issues in data_router
* Fix JSDOc issues in process.js fiel
* Fix JSDOc issues in repo_router.js
* Address JSDoc in tasks.js
* Address JSDoc linting issues
* Switch to ignores statement
* Address more eslint jsdoc items
* Be more selective about eslint rules
* Remove requrie description because seems to be faulty

* Address finicky jsdoc linter

* Fix js formatting

* Feature DLT 1109 foxx new token data (#1194)

* Stub logic for updating related edge in router function for token/set

* Include logic for conditionally updating token data and pushing data to edge.

* Change token_key to remove illegal characters

* Fix object attribute call

* Formally add query params to function. Extract query params near start of action. Check both exist before taking action on edge. Update comments, notes, TODOs

* Stub testing for user_router

* Fix naming of user_router test so it will be picked up with sane specification. Add test to CMakeLists

* Add some fixtures for creating related user to allow for integration testing

* Fix manifest json. Hard code user collection check. Set time variable.

* Use correct path for router to user endpoints. Extract some query params for validation.

* Remove intentionally failing test. Break now working test into logical sections. Update comments.

* Add test for new features

* Add further validation of no unexpected side-effects

* Remove extraneous user document checks and comments

* More testing for branches in introduced logic. Additional check on newly inserted globus_coll

* assert -> expect

* Work around multiple access and ordering issue by utilizing more test user fixtures.

* Rearrange key for token edge, add suggested fields to documents

* Pull user_doc from existing logic. Use object destructuring. Add error condition. Introduce protobuf enum to backend. Add parser for other_token_data string. Update document creation functions

* Extract fixture build call to separate script. Add script as fixture in cmake

* Remove some extraneous logic, remove password file overwrite.

* Remove unused zeromq flag

* Adjust testing to include enum, expect error on invalid input combination, typo, naming

* Adjust scope string in user_router test. Add validations to parseOtherTokenData. Add tests for positive and error cases of parseOtherTokenData.

* Change chai expects in support.test.js to use callable. Update perms on fixture setup script. Use URL encoded pipe character in user_router.test.js

* Group OR in user_router.js

* Adjust user_router.test.js integration tests to access deconstructed other_token_data

* Add baseline jsdoc for parse method. Remove some unwanted data from globus_token doc insert

* Add GLOBUS_DEFAULT to AccessTokenType enum. Add default to user_router endpoint param. Update logic to switch based on enum. Test new change by specifying default type and other_token_data. Fix name of test.

* Move query parameter validations earlier in function. Decrease nesting of switch logic.

* Cleanup some whitespace changes, add desired comments on assumptions.

* Formatting

* Fix naming for support test so they will run.

* Update CHANGELOG.md with changes in minor feature. Update comments on user_router.js for saving a new globus_coll

* Add comment about more fields in support.js::parseOtherTokenData

* Fix issue for formatter

* Add expiration check

* Commit for CI

* Formatting

---------

Co-authored-by: Anthony Ramirez <[email protected]>

* Add ci jobs to improve security posture

* Change to unit stage for testing

* Remove sast job

* Attempt to fix semgrep sast job failure

* Debug jobs

* Add test stage for sast template job

* Add missing test stage

* Add dependency scanning

* Split sast into separate test stage

* Remove commented out line

* Remove extra comment that is unused

* Debugging log failure

* Address jsdoc and typo in description

* Remove outdated comments

* Make comments JSDoc compliant

* Address formatting inconsistency

* Account for multiple images returned in log

* Update .gitlab/common.yml

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>

* Debugging log container matching

* Fix typo in gitlab anchor

* Debug escape anchor

* Debug

* Debug anchor 2

* Debug anchor 3

* Debug anchor 4

* Debug anchor 5

* Debug 6

* Debug 7

* Debug 8

* Fix names of containers from gcs-authz to gcs

* Cleanup add comment back in

* Update common.yml

* Remove Exports

* Update data_router.js

Remove nested docs.

* Update repo_router.js

* Update task_router.js

* Update eslint.config.js

* Update eslint.config.js

* fix jsdoc comment in posix path

* Address JSDOc complaints

* Address JSDoc errors

* Address linter

* Address linter errors

* Fix order of parameters

* Apply formatting

* Add missing argument form strategy method call

* Apply prettier

* Update posix_path.js

* Address ai suggestion about better error message

* Debugging log failure

* Account for multiple images returned in log

* Update .gitlab/common.yml

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>

* Debugging log container matching

* Fix typo in gitlab anchor

* Debug escape anchor

* Debug

* Debug anchor 2

* Debug anchor 3

* Debug anchor 4

* Debug anchor 5

* Debug 6

* Debug 7

* Debug 8

* Fix names of containers from gcs-authz to gcs

* Cleanup add comment back in

* Update common.yml

* Remove Exports

* Cleanup comments and apply formatting

* Update nlhoman json version

* Update protobuf versions

* Add tests for authz router

* Specify ubuntu 20.04 explicitly

* Revert dependency versions

* Fix problems with path consistent method

* Cleanup javascript

* Format javascript

* Feature DLT 1127 relay new token data (#1217)

* Add additional params to DatabaseAPI::userSetAccessToken to pass message with addtional info on to database API.

* Add convenience method with old type signature of DatabaseAPI::userSetAccessToken for compatibility

* Conditionally add other token options param when calling dbGetRaw.

* Pass token type along to Database API.

* Pass token type from message.

* Extract logic for building URL from dbPost and dbGetRaw into separate function for easier testing and debugging

* Extract url building logic from dbGet

* Add some comments for moved code, address a TODO, add new TODO for AccessToken Message type field

* Address default on token type, formatting

* Add new AccessTokenType entries for a sentinel value and Globus default in SDMS.proto, use as desired in DatabaseAPI.cpp

* Change query param to match backend changes

* Restore formatting of SDMS.proto

* Make built URL const where appropriate in DatabaseAPI.cpp and DatabaseAPI.hpp

* Change default AccessTokenType for UserSetAccessTokenRequest in SDMS_Auth.proto.

* Remove logging of potentially senitive data

* Adjust AccessTokenType ACCESS_SENTINEL to have value of uint8 max.

---------

Co-authored-by: Anthony Ramirez <[email protected]>

* [DLT-1110] Update files with prettier
[DLT-1110] Add controller unit tests
[DLT-1110] Remove debug
[DLT-1110] Correct import statements, endpoint list
[DLT-1110] Refactor out into MVC
[DLT-1110] Add debug
[DLT-1110] Functioning modal, needs refactoring
[DLT-1110] Refactor dlg start transfer using OOO programming. Should be MVC cased on what I'm seeing but we'll see

* [DLT-1110] Split branches

* [DLT-1110] Update model, make logic private, remove controller form transfer request when update or get

* [DLT-1110] Add commnets to logic

* [DLT-1110] Correct HTML escape, francy tree init

* [DLT-1110] Pull out template HTML

* [DLT-1110] Fix update bug

* [DLT-1110] Dependency injection

* [DLT-1110] Update if statements

* [DLT-1110] Update style

* Revert "[DLT-1110] Mapped Collection Endpoint Browse (1/4)"

* Address eslint

* Address remaining eslint items

* Fix bug on task_router.js abort function (#1234)

Co-authored-by: Anthony Ramirez <[email protected]>

* Address Aaron feedback

* applied potential fix to deprecation warning

* Reorder createRecord

* Apply formatting

* Throw error if OpenStack error code returned from API

* Add dependency between logs

* Add dependencies on end to end jobs as well

* Job rule is not supported

* Remove always run log

* add end to end signal job

* Fix tag of signal

* Feature DLT 1120 retrieve transfer token (#1228)

* Add conditional switch and stub some logic for handling other resources in callbacks

* Fix order of logging. Add redirect

* Debug logging for web service. Attempt to set token.

* Remove incorrect method call to data, use data attribute.

* Add some TODOs and logging

* Give setAccessToken ability to accept additional params to pass to protobuf

* Add additional params to DatabaseAPI::userSetAccessToken to pass message with addtional info on to database API.

* Add convenience method with old type signature of DatabaseAPI::userSetAccessToken for compatibility

* Conditionally add other token options param when calling dbGetRaw.

* Pass token type along to Database API.

* Pass token type from message.

* Stub logic for updating related edge in router function for token/set

* Include logic for conditionally updating token data and pushing data to edge.

* Change token_key to remove illegal characters

* Fix object attribute call

* Prettier formatting for datafed-ws.js

* Roll back changes to DatabaseAPI.hpp and DatabaseAPI.cpp that are covered in #1127

* Refactor new inclusions in datafed-ws.js to use as much existing code as possible, only diverge where necessary.

* datafed-ws.js change scopes to scope

* datafed-ws.js log user ID array

* datafed-ws.js clarify uid assignment at ui/authn endpoint, add note on error cases, remove incorrect comment about uid, add appropriate error if collection_id not present

* datafed-ws.js Extract AccessTokenType enum, add comment about fetching from protobuf; remove unnecessary commented code

* datafed-ws.js Extract transfer set logic when calling setAccessToken to improve readability of router.

* datafed-ws.js Create function for resolving token type; create function to handle logic in building optional_data for setAccessToken; reduce unneccessary nesting in ui/authn endpoint

* datafed-ws.js Fix spacing, fix typo in variable name

* datafed-ws.js Fix bug referring to request session

* datafed-ws.js Nesting was necessary.

* datafed-ws.js Address some TODOs, set token type with more context

* datafed-ws.js Remove some additional extraneous logging

* datafed-ws.js Formatting

* Add comment about token set

* datafed-ws.js Update deprecated substring method, Fix order for error case when new user receives transfer token

* TokenHandler.js Refactor token handling logic out of web server main. datafed-ws.js User new OAuthTokenHandler class to handle token logic.

* TokenHandler.js Implement token validator; move OAuthTransferToken def to top. TokenHandler.test.js Minimal testing of token handler

* TokenHandler.test.js Add tests for case when resource server is auth

* TokenHandler.test.js Add testing for Globus transfer resource server

* TokenHandler.js Implement validation for existence of required keys. TokenHandler.test.js Adjust other_tokens fixtures to be nested according to incoming data; formatting.

* datafed-ws.js formatting

* TokenHandler.js Update error messages, remove unnecessary code in getTokenType, returns in resolveTokenType

* datafed-ws.js Add error handling around token_handler construction; add errors and handling for setAccessToken, JSDoc.

* datafed-ws.js Remove hard coded collection_id.

* datafed-ws.js Remove thrown errors in nested functions; single redirect.

---------

Co-authored-by: Anthony Ramirez <[email protected]>

* Bug daps 1243 datafed web logging (#1251)

* datafed-ws.js Reverse order of LogLevel object properties

* datafed-ws.js Throw error so stack trace is adequately populated

* Revert line number

* TokenHandler.js Fix linting errors.

---------

Co-authored-by: Anthony Ramirez <[email protected]>

* [DLT-1110] Mapped Collection Endpoint Browse (1/4) (#1240)

* [DLT-1110] Update style
[DLT-1110] Update if statements
[DLT-1110] Dependency injection
[DLT-1110] Fix update bug
[DLT-1110] Pull out template HTML
[DLT-1110] Correct HTML escape, francy tree init
[DLT-1110] Add commnets to logic
[DLT-1110] Update model, make logic private, remove controller form transfer request when update or get
[DLT-1110] Split branches
[DLT-1110] Update files with prettier
[DLT-1110] Add controller unit tests
[DLT-1110] Remove debug
[DLT-1110] Correct import statements, endpoint list
[DLT-1110] Refactor out into MVC
[DLT-1110] Add debug
[DLT-1110] Functioning modal, needs refactoring
[DLT-1110] Refactor dlg start transfer using OOO programming. Should be MVC cased on what I'm seeing but we'll see

* [DLT-1110] Mapped Collection Endpoint Browse Tests (2/4) (#1207)

* [DLT-1110] Add tests

* [DLT-1110] Update chai with mocha, attempt usage of mock-import

* [DLT-1110] Add mocha config, correct prettier

* [DLT-1110] Update tests, correct logic

* [DLT-1110] Update tests, add fixtures, update setup, lock in packages

* [DLT-1110] Update test

* [DLT-1110] Config

* [DLT-1110] Upgrade node v14 -> v23

* [DLT-1110] Upgrade node v23 -> v18, better jump

* [DLT-1110] Correct ver .sh file

* Revert "[DLT-1110] Correct ver .sh file"

This reverts commit 5c78c59.

* Revert "[DLT-1110] Upgrade node v23 -> v18, better jump"

This reverts commit 9d66847.

* Revert "[DLT-1110] Upgrade node v14 -> v23"

This reverts commit fb8b93d.

* Revert "[DLT-1110] Config"

This reverts commit 82715f0.

* Revert "[DLT-1110] Mapped Collection Endpoint Browse Tests (2/4) (#1207)"

This reverts commit 22545fa.

* added updated version t0 requirements.txt

* Fix edge case in GridFTP Authz where '/' is used

* cpp-py-formatter

* Add changelog comment

* Address prettier formatting

* Add method for testing config, fix bug in global config, fix redundant log statements

* Make error detection clearer in bool

* Add mock core server

* Add integration and, liveness test for mock

* Address mock integration setup and consolidate

* Apply formatting

* Remove unused timeout variable from mock

* Address JSDoc complaints

* Apply prettier

* Fix formatting and eslint prettier issues

* Address codacy feedback

* Add changelog comment

* Clean up grammar in comment

* Add note in Config.h

* Apply clang formatting

* Address codacy issues

* Add env var

* Update Arangodb packaged version to using 3.12.4

* Fix confusion in key uses in end to end tests

* Add change log comment

* Add changelog comment

* corrected imported member

* Add compose entries for foxx tests

* Update the schema documentation to improve the usability.

* Add example schema with a reference

* Add notebook for linting jupyter

* Using working directory option

* Update .github/workflows/jupyternotebook-lint.yml

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>

* Update python notebooks

* Attempt to fix jupyternote book linting errors

* Format with black

* Specify line length with black to be consistent with flake8

* Add a Changelog comment

* Change line length to 88

* Add black formatters

* Make formatting consistent across project

* Fix linter

* Temporarily keep line length in black to 79 and address in separate issue

* Change line length to 88 for now
:

* updated changelog

* Fix bug in CI scripts associated with repo and gcs image build

* Merge hotfixes into devel (#1326)

* [DAPS-DEPS] Update cpp-py-formatter to 0.3.0

* [DAPS-DEPS] Update cpp-py-formatter to 0.3.1

* [DAPS-DEPS] Update ver, add push

* updated versions for release

* Update project.rst

* Update format-check.yml

* Pull in version numbers appropriately

* Avoid build cache

* Aggressive prune

* Trigger rebuild

* generate_repo_config.sh Accept env variable for datafed server from repo service instead of hard coding

* Allow core config thread options to be read in from env

* Add changelog comment

* Adjust deployment version number

* Fix Version month

* Update CHANGELOG.md

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: Aaron Perez <[email protected]>
Co-authored-by: Blake Nedved <[email protected]>
Co-authored-by: Joshua S Brown <[email protected]>
Co-authored-by: JoshuaSBrown <[email protected]>
Co-authored-by: Anthony Ramirez <[email protected]>
Co-authored-by: nedvedba <[email protected]>
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>

* Feature DAPS 1215 foxx UserGetAccessTokenRequest mapped collection support (#1284)

* Begin prototyping required code for mapped collection token retrieval.

* SDMS_Auth.proto Add optional fields to UserGetAccessTokenRequest and UserAccessTokenReply to support finding correct tokens

* user_router.js Add baseline query params and checks. DatabaseAPI.cpp & DatabaseAPI.hpp Use updated names and types. ClientWorker.cpp Update comments.

* user_router.js Add logic for finding token based on collection; add token match and missing match cases

* Return necessary data from Database to refresh tokens. Mock data on dir/list to test out changes.

* user_router.js Add check for existence of Globus collection

* user_router.js Remove existence check, move to filter method; Add notes on covered cases; add notes on code

* datafed-ws.js Temporary handling of need consent response.

* user_router.js Add queryParam validation for new params via joilib. ClientWorker.cpp Add refresh token try/catch to send through flow if refresh fails

* datafed-ws.js Use correct converted casing for protobuf messages; redirect to error when needs_consent

* datafed-ws.js No redirect, show error directly in list.

* datafed-ws.js Early return to prevent further requests

* support.js Update AccessTokenType enum. user_token.js New class introduced for handling logic for validating params when getting tokens and building GetAccessToken responses. user_router.js Replace logic with calls to new lib

* user_token.js Add jsdoc definition for return type and format function. user_router.js Remove unnecessary comment.

* user_token.js More jsdoc definition; Add validation for collection token document required fields. user_token.test.js Add preliminary unit tests for static methods.

* CMakeLists.txt Add new unit tests to build. user_token.test.js Modify naming to match convention.

* CMakeLists.txt Fix variable get. user_token.js Fix export.

* user_fixture.js Add new user for get/token tests. user_router.test.js Add simple tests for get/token endpoint. user_token.js Fix naming for token document scopes field. user_token.test.js Changes for scope field bugfix.

* user_router.js Formatting; Throw error if more than one token matches, add note about token type.

* ClientWorker.cpp Address some TODOs and extraneous comments. DatabaseAPI.cpp Remove TODOs. SDMS_Auth.proto Remove unused field.

* user_token.js Update formatUserToken to always return object values. DatabaseAPI.cpp Add note on possibly missing field. user_token.test.js Update tests to check for new expected values.

* user_router.js Clean up some comments for clarity. datafed-ws.js Formatting.

* ClientWorker.cpp Rewrite comment regarding mapped token refresh.

* user_token.js Formatting.

* CHANGELOG.md update for feature

* user_router.js Re-introduce accidentally removed `globus_collection` object.

* CMakeLists.txt Remove comment about necessity

* Update token lookup logic to use `byExample`, add test case for missing Globus collection. Add globus_coll fixture.

* SDMS_Auth.proto Change UserAccessTokenReply needs_consent to optional field.

* user_router.js Update token/get to default to start consent flow when collection tokens are present. user_router.test.js Update test from error to needs_consent.

* Formatting changes.

---------

Co-authored-by: Anthony Ramirez <[email protected]>

* [DLT-1110] Implement Consent Required Action (3/4 & 4/4) (#1242)

* [DLT-1110] Refactor browse into component and use CSS

* [DAPS-1110] Update

* [DAPS-1110] Update web server

* [DAPS-1110] add refactors

* [DAPS-1110] Prettier

* [DAPS-1110] Refactor tests

* [DAPS-1110] Remove .. in ep ls, remove logs

* [DAPS-1110] Add sinon, remove dead code

* [DAPS-1110] Address transfer start button bug

* [DAPS-1110] eslint, update api.test.js to reflect usage

* [DAPS-1110] format

* [DAPS-1110] Correct bug

* JSON parse error fix (#1328)

* DatabaseAPI.cpp Pull in json dependency, use json object to serialize request payload

* Replace json serialization in more locations where parseSearchRequest is being called.

* Add comments around locations that will need json serialization

* Convert missed body append to serialization format

* typo

* Refactoring DatabaseAPI.cpp to incorporate json serialization

* More serialization work

* Pull in changes from 1214 for dataPut

* Bring in changes from 1214 for dataGet

* DatabaseAPI.cpp Fix error with missing curly brace. Some formatting that is likely wrong. Serialize up to original point. Need to validate.

* Add missing scope resolution

* More typos

* More typos. Scoping

* Remove non-existant vars

* declare body

* Formatting

* Formatting and verification of updates

* Replace most places

* Formatting

* Remove redeclaration

* DatabaseAPI.cpp Remove some comments.

* DatabaseAPI.cpp Add missing push_back

* Adding braces

* Prevent Double Escaping string, by using nlohmann json parse to read in value.

* remove escapeJSON calls, redundant with nlohmann json dump

* Allow parsing for metric data to be run in parallel for now, output messages when old serialization is used.

* Ensure ASCII setting.

* Upgrade debug to warning.

---------

Co-authored-by: Anthony Ramirez <[email protected]>
Co-authored-by: Austin Hampton <[email protected]>
Co-authored-by: Joshua S Brown <[email protected]>

* Feature - Mapped Collection Token Support (#1334)

* Begin prototyping required code for mapped collection token retrieval.

* SDMS_Auth.proto Add optional fields to UserGetAccessTokenRequest and UserAccessTokenReply to support finding correct tokens

* user_router.js Add baseline query params and checks. DatabaseAPI.cpp & DatabaseAPI.hpp Use updated names and types. ClientWorker.cpp Update comments.

* user_router.js Add logic for finding token based on collection; add token match and missing match cases

* Return necessary data from Database to refresh tokens. Mock data on dir/list to test out changes.

* user_router.js Add check for existence of Globus collection

* user_router.js Remove existence check, move to filter method; Add notes on covered cases; add notes on code

* datafed-ws.js Temporary handling of need consent response.

* user_router.js Add queryParam validation for new params via joilib. ClientWorker.cpp Add refresh token try/catch to send through flow if refresh fails

* datafed-ws.js Use correct converted casing for protobuf messages; redirect to error when needs_consent

* datafed-ws.js No redirect, show error directly in list.

* datafed-ws.js Early return to prevent further requests

* support.js Update AccessTokenType enum. user_token.js New class introduced for handling logic for validating params when getting tokens and building GetAccessToken responses. user_router.js Replace logic with calls to new lib

* user_token.js Add jsdoc definition for return type and format function. user_router.js Remove unnecessary comment.

* user_token.js More jsdoc definition; Add validation for collection token document required fields. user_token.test.js Add preliminary unit tests for static methods.

* CMakeLists.txt Add new unit tests to build. user_token.test.js Modify naming to match convention.

* CMakeLists.txt Fix variable get. user_token.js Fix export.

* user_fixture.js Add new user for get/token tests. user_router.test.js Add simple tests for get/token endpoint. user_token.js Fix naming for token document scopes field. user_token.test.js Changes for scope field bugfix.

* user_router.js Formatting; Throw error if more than one token matches, add note about token type.

* ClientWorker.cpp Address some TODOs and extraneous comments. DatabaseAPI.cpp Remove TODOs. SDMS_Auth.proto Remove unused field.

* user_token.js Update formatUserToken to always return object values. DatabaseAPI.cpp Add note on possibly missing field. user_token.test.js Update tests to check for new expected values.

* user_router.js Clean up some comments for clarity. datafed-ws.js Formatting.

* ClientWorker.cpp Rewrite comment regarding mapped token refresh.

* user_token.js Formatting.

* CHANGELOG.md update for feature

* user_router.js Re-introduce accidentally removed `globus_collection` object.

* CMakeLists.txt Remove comment about necessity

* Add collection_type and collection_id to dat/put task state

* support.js Update getAccessToken method to work based on collection information, JSDoc. tasks.js Include context required in call to getAccessToken, add comment on refresh. TaskWorker.cpp Identify where changes will be reflected.

* Update token lookup logic to use `byExample`, add test case for missing Globus collection. Add globus_coll fixture.

* SDMS_Auth.proto Change UserAccessTokenReply needs_consent to optional field.

* user_router.js Update token/get to default to start consent flow when collection tokens are present. user_router.test.js Update test from error to needs_consent.

* Formatting changes.

* SDMS_Auth.proto Add new fields to DataGetRequest

* Begin refactor of token decision logic.

* Address bugs in fetching and mapping data, make unit tests pass.

* globus_token.js Match other models for get

* tasks.js Pull in new logic for determining tokens, replace tokens; Needs to be verified. user_token.js Add new format function for compatibility with transfer doc replacement.

* DatabaseAPI.cpp Sane serialization of json body for taskInitDataPut. TaskWorker.cpp Comments around work items.

* DatabaseAPI.cpp Sane serialization of json body for taskInitDataGet.

* ClientWorker.cpp Stub out required changes for consent flow when creating tasks

* ClientWorker.cpp More thorough consent flow for procDataPutRequest, stub on needs_consent

* tasks.js Throw error when initializing data put

* tasks.js Format error throwing. ClientWorker.cpp comment out potentially unnecessary code.

* user_token.js Add exists method to abstract check. tasks.js User exists method when checking token.

* user_token.js JSDoc typing

* tasks.js Add more required fields to taskRunDataPut for refresh in TaskWorker.

* tasks.js Add collection_id to params in taskRunDataPut for refresh in TaskWorker. TaskWorker.cpp Refresh conditionals

* DatabaseAPI.cpp Update merged devel changes to address some leftover code.

* DatabaseAPI.cpp Add collection specification to get request.

* DatabaseAPI.cpp whitespace. ClientWorker.cpp Clean up unused code. user_router.js Switch to checking token existence through standard API.

* user_token.js Add comment about type return. data_router.js Do validations at router level for user collection token relationship. tasks.js Remove validation from logic level; make dataGet mirror dataPut; Updates to naming and params structure.

* ClientWorker.cpp Revert to minimize diff. TaskWorker.cpp Remove unnecessary TODO; move set client for refresh in transfer tasks.

* models/ JSDocs, formatting, make token fetches follow defined model. user_token.js JSDocs and typing.

* user_token.js Cover case where collection does not exist.

* Small changes for UI detection of collection type

* fix epview call for UI changes, add list view change to react appropriately to needing consent according to current code

* extract ep view logic to function for either path to utilize

* small changes and notes for UI updates

* UI search based on path, not ID. camelCasing

* TaskWorker.cpp small fix to only enforce requirement of additional refresh fields when token is transfer token.

* add params for dat/get on UI

* task.js Fix bug where client is incorrectly referenced.

* Add user model tests. Extract DataFedOAuthToken.js from user_token.js for reusability.

* Add globus collection model tests.

* Add globus token tests. Add fixtures to support globus token testing. Make all model properties static for iteration purposes. Utilize constructors for typing.

* Add user token tests. Make non-existent user throw an error when building user tokens.

* Change error to be not found

* Formatting changes

* Remove unused changes in support.js getAccessToken

* Changelog update, update comments, remove log.

---------

Co-authored-by: Anthony Ramirez <[email protected]>

* Add small fix for storeCollectionId (#1341)

* Add small fix for storeCollectionId

* move todo comment

---------

Co-authored-by: Anthony Ramirez <[email protected]>

* Fix bug with playwright (#1343)

* Fix bug with playwright install for end to end web test

---------

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>

* [DAPS-1330] Address Download Extension Transfer (#1340)

* [DAPS-1337] Refactor GET and Put mode

* [DAPS-1337] Clear consent issue

* [DAPS-1337] Correct bug

* [DAPS-1337] Add Dir mode path logic update

* Parse md json strings (#1347)

* Parse md json strings

* parse execQuery params

* move to direct only

* make params an object

* remove console log

---------

Co-authored-by: Anthony Ramirez <[email protected]>

* [DAPS-1349] Provenance Capture Update Bug (#1350)

* [DAPS-1349] Return logic to original check

* [DAPS-1349] version

* [Release] February 2025 (#1352)

* updated versions for release

* Trigger rebuild

* Adjust deployment version number

* Fix Version month

* Update Version.cmake

* removed leading zero

---------

Co-authored-by: JoshuaSBrown <[email protected]>
Co-authored-by: Joshua S Brown <[email protected]>

* Add .venv/ to gitignore (#1335)

* [DAPS-1388] Add GCP Support (#1392)

* [DAPS-1388] Add endpoint model, Add endpoint model, unify seperate collection logic, endpoint validation check, Add endpoint query validations, next is token

* [DAPS-1388] Add tests for model

* [DAPS-1388] PR comments

* Update web/static/components/transfer/transfer-endpoint-manager.js

Co-authored-by: Joshua S Brown <[email protected]>

---------

Co-authored-by: Joshua S Brown <[email protected]>

* Add missing python module (#1404)

* Set reasonable default that will help with filling in repo form for core t… (#1400)

* Set reasonable default that will help filling in repo form for core to repo communication in compose file

* Update scripts/compose_generate_env.sh

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>

* Set appropriate default when building compose image

---------

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>

* Add documentation describing how to test the foxx microservices unit … (#1393)

* Add documentation describing how to test the foxx microservices unit tests

* Update testing.rst

* Note on database version, variable definition and usage when running arangodb version described, and steps to run unit test numbered.

* Fix ci build (#1390)

* updated requirements

* updated requirements

* updated protobuf submodule

* update unit test ci version

* updated protobuf version

* updated setuptools version

* fixed condition

* changed where packages are installed

* used python3.11 to create the virtual environment instead of latest

* reverted protobuf version

* testing specificying version

* removed pyvenv in ci job for consistency

* installed python3.11 in dependencies docker file

* added missing -y flag

* removed unification of install scripts

* added missing python3.11 install

* updated ci job to install python3.11

* changed installed python version to 3.9 since protobuf does not officially support 3.11

* changed installed python version to 3.9 since protobuf does not officially support 3.11

* missed version

* updated includes for newer gcc versions

* updated versions

* updated nlohmann json

* updated json schema version

* changed library search path

* hopefully fixed json dep issues

* updating function signature to match newer version of base class

* updated protobuf

* updated protobuf and cmake

* downgraded cmake

* fixed protobuf version

* fixed protobuf version

* 1398 pin python version (#1405)

* Pin python version

* fixed incorrect path

* Switch to using .tar file install of libsodium (#1414)

* 1413 libsodium build refactor (#1415)

* Switch to using .tar file install of libsodium

* Switch bad option in wget command from -C to -P

* Update scripts/dependency_install_functions.sh

* Update scripts/dependency_install_functions.sh

* Update dependency_install_functions.sh

Libsodium folder version number.

* Update dependency_install_functions.sh

Make paths explicit.

* cleaned up comments

* Update scripts/dependency_install_functions.sh

Co-authored-by: Joshua S Brown <[email protected]>

* Update cmake/JSONSchema.cmake

Co-authored-by: Joshua S Brown <[email protected]>

* Update scripts/dependency_install_functions.sh

Co-authored-by: Joshua S Brown <[email protected]>

* Update cmake/JSONSchema.cmake

Co-authored-by: Joshua S Brown <[email protected]>

* add changes from review

* Changed install scripts to be consistent with new python installation method (#1417)

* changed install scripts to be consistent

* added sudo check

* add apt sources check

---------

Co-authored-by: Blake Nedved <[email protected]>
Co-authored-by: nedvedba <[email protected]>

* Fix regression (#1428)

* Fix docker logs output in end-to-end tests (#1426)

* Improve serialization randomization (#1427)

* Increase serialization randomization pause heuristic - 6 seconds is not long enough to register running pipelines.

* improve serialization

* Upgrade formatter (#1440)

* Update action for CPP and Python formatter

* 1437 fix zmq inproc bind order (#1438)

* Reorder zmq bind and connect calls to prevent undefined behavior when using INPROC

* Make sure zmq messages are closed when exception is encountered

* Make correction to socket name

* Initialize cmd call

* Add missing include

* Allow looping over containers if there is more than one. (#1446)

* 1445 docker logs multiple containers (#1447)

* Allow looping over containers if there is more than one.

* Fix multipline comment

* Fix conflicting dependency install flags (#1444)

* [DAPS-1408] Provenance Visual Management Lint (1/2) (#1431)

* [DAPS-14xx] Move files over and move out svg

* [DAPS-1408] Provenance Visual Management (2/2) (#1419)

* [DAPS-14xx] Base

* [DAPS-14xx] Add styling and tooltips

* [DAPS-14xx] Add node and label customization

* [DAPS-14xx] Add editor modal on right-click. Draggable model

* [DAPS-14xx] Revert

* [DAPS-14xx] Seperate files, update styles

* [DAPS-14xx] simplify feat

* [DAPS-14xx] Expand mocha tests, prettier

* [DAPS-14xx] Update checkout version

* [DAPS-14xx] Add todo

* [DAPS-14xx] Update tests, consts, fix tooltip

* [DAPS-14xx] Base:

[DAPS-14xx] Address collapse, expand, hide

* [DAPS-14xx] THEME, address comments

* [DAPS-14xx] Zoom

* [DAPS-14xx] JSDOC

* Update web/static/jquery-ui-dark/datafed.css

Co-authored-by: Joshua S Brown <[email protected]>

* Update web/static/jquery-ui-dark/datafed.css

Co-authored-by: Joshua S Brown <[email protected]>

* Update web/static/jquery-ui-dark/datafed.css

Co-authored-by: Joshua S Brown <[email protected]>

---------

Co-authored-by: Joshua S Brown <[email protected]>

* [DAPS-1395] Address Memory Leaks in Core Server (#1424)

* Add missing virtual destructors

* Fix ZeroMQ INPROC and TCP context

* Add memory tests to CMakeLists.txt as an option

* Add protobuf library shutdown to unit tests for memory cleanup

* [TASK] Update tests and fixture (#1454)

* [DAPS-1331] Collection ID Updated on restart (#1468)

* [DAPS-1331] Modified the script to only delete and recreate guest collections when the base path actually changes

* [DAPS-1331] Add base path logic

* [DAPS-1331] Update guest collection file

* [DAPS-1331] Fix base path caching logic in should_recreate_guest_collection function to recreate by default when no cached path is available

* [DAPS-1331] Add tests

* [DAPS-1331] Revert Gitlab CI

* [DAPS-RELEASE]

* [TASK] Remove icons from labels (#1489)

---------

Co-authored-by: Anthony <[email protected]>
Co-authored-by: Anthony Ramirez <[email protected]>
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
Co-authored-by: Aaron Perez <[email protected]>
Co-authored-by: AronPerez <[email protected]>
Co-authored-by: Blake Nedved <[email protected]>
Co-authored-by: par-hermes <[email protected]>
Co-authored-by: nedvedba <[email protected]>
Co-authored-by: Austin Hampton <[email protected]>
Co-authored-by: Theo Beers <[email protected]>

* [DAPS-1625] - user id map for session key josh (#1644)

* [DAPS-1625]  fix bug in session and transient keys.

---------

Co-authored-by: AronPerez <[email protected]>

* [TASK] Correct 2 spaces to 4 for YML (#1640)

* fix: merge conflict changes that were missed.

---------

Co-authored-by: Aaron Perez <[email protected]>
Co-authored-by: Anthony <[email protected]>
Co-authored-by: Anthony Ramirez <[email protected]>
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
Co-authored-by: Aaron Perez <[email protected]>
Co-authored-by: Blake Nedved <[email protected]>
Co-authored-by: par-hermes <[email protected]>
Co-authored-by: nedvedba <[email protected]>
Co-authored-by: Austin Hampton <[email protected]>
Co-authored-by: Theo Beers <[email protected]>

* [DAPS-1585] [Dependencies] update: upgrade ssl dependency. 3.2.5 (#1646)

* Staging to devel update (#1657)

* Update devel from staging

* [DAPS-1605] fix install_foxx.sh - split ssl_args (#1623)

* fix(install_foxx): remove ssl_args argument it is not a valid flag.

* Update devel from staging (#1649)

* [DAPS-1585] [Dependencies] update: upgrade ssl dependency. 3.2.5 (#1646)

* Staging to devel update (#1657)

* Update devel from staging

* [DAPS-1605] fix install_foxx.sh - split ssl_args (#1623)

* fix(install_foxx): remove ssl_args argument it is not a valid flag.

* [DAPS-1651] bug compose env generator (#1656) (#1658)

* [DAPS-1651] fixed bug in compose env generator (#1656), compose_generate_env.sh script was not reading existing values correctly.

* Adding the logger functions for future PR's (#1675)

* feature: Implemented logging functions

* [DAPS-1659] remove dependencies install scripts (#1660)

* refactor: remove all scripts related to dependencies from DataFed repo
* feature: update DataFedDependencies submodule and move ground truth of requirements.txt to DataFedDependencies.

* 1669 feature support core service protobuf message with repo types (#1670)

* 1671 daps feature foxx add repository and execution strategy types (#1672)

* update: DataFedDependencies to 57483e1cd4eac9d84162dd7202e72fe353728361

* Update datafed dependencies (#1677)

* refactor: remove remaining occurences of zeromq system secret. (#1661)

* [DAPS-1678] refactor split error codes from foxx support.js and move to lib/error_codes.js (#1679)

* [DAPS-1522] User router logging improvements (#1629)

* style: Removed non helpful logs from tasks.js

* fix: reverted protobuf to match devel (a9b006)

* refactor: Refactoered the user_router to have a first pass of
standardized logging (needs more testing)

* style:Fomratted with prettier

* refactor: changed correlation id within logs to use
req.headers['x-correlation-id'] instead of just having placeholder text

* chore: Auto-format JavaScript files with Prettier

* refactor: First pass of logging within user_router.js

* refactor: last minute changes

* chore: Auto-format JavaScript files with Prettier

* Refactor: Added second pass of logging to user router

* Forgot to also add the new file

* chore: Auto-format JavaScript files with Prettier

* chore: Auto-format JavaScript files with Prettier

* refactor:Made final adjustments to logging within user router

* chore: Auto-format JavaScript files with Prettier

* refactor: replaced undefined with null for appropriate areas; made
slight changes to some log output to obfuscate sensitive data

* refactor: Fixed spacing within logs; Removed unnecessary logging

* chore: Auto-format JavaScript files with Prettier

* chore: Auto-format JavaScript files with Prettier

* style: Removed non helpful logs from tasks.js

* fix: reverted protobuf to match devel (a9b006)

* chore: Auto-format JavaScript files with Prettier

* refactor: Fixed spacing within logs; Removed unnecessary logging

* chore: Auto-format JavaScript files with Prettier

* chore: Auto-format JavaScript files with Prettier

* Style: Corrected status' on logs

Co-authored-by: Joshua S Brown <[email protected]>

* refactor:Second pass of user_router logging

* refactor: Removed temporary logger file

* refactor:removed external/protobuf

* refactor: Final touches for logging

* chore: Auto-format JavaScript files with Prettier

* refactor: Cleaning up PII issues

* refactor:Final checks, changed /view's extra output

* Submodule update

* refactor: Commited all reviewed changes

* chore: Auto-format JavaScript files with Prettier

---------

Co-authored-by: Joshua S Brown <[email protected]>

* refactor: adjust validation.js swap g_lib with error_code require rem… (#1691)

* refactor: adjust validation.js swap g_lib with error_code require remove circular dependency from validation.js by removing validateRepositorySupportsDataOperations

Co-authored-by: Aaron Perez <[email protected]>

* [DAPS-1692] - CI, End-to-end web tests, fix flaky test (#1693)

* [DAPS-1694] refactor - Foxx move permissions functions from support.js to lib/permissions (#1695)

* [DAPS-1685] Feature compose enable arangodb ssl (#1687)

* [DAPS-1700] - [CI] - fix: limit arangodb job output to last 3 hours. (#1701)

* [DAPS-1676] Feature - foxx arango add factory for repositories for metadata and globus (#1697)

Co-authored-by: Aaron Perez <[email protected]>

* [DAPS-1718] - Web, Core, Python Client - Protobuf ExecutionMethod enum, add RepoAllocationCreateResponse (#1719)

* [DAPS-1713] - Core, Web, Python Client, refactor: protobuf, allow optional fields when creating repo to support metadat… (#1714)

* [DAPS-1715] - Core, refactor: make path, pub_key, address, endpoint optional in repoCreateRequest (#1716)

* [DAPS-1705] - Foxx, feature: integrate metadata globus factory repo router create (#1706)

Co-authored-by: Aaron Perez <[email protected]>

* [DAPS-1688] - Update dependencies, Crypto libssl switched to version 3 globus_sdk version pinned (#1689)

* [1729-DAPS] - bug ci, downstream datafed dependencies pipelines are building the container image from incorrect sha (#1732)

* [DAPS-1711] - feature foxx standardize repo response schema (#1712)

* [DAPS-1725] refactor: remove confusing apache conf file. (#1728)

* [1707] Update Web Dependencies Before Install (#1709)

Co-authored-by: Joshua S Brown <[email protected]>
Co-authored-by: JoshuaSBrown <[email protected]>

* [DAPS-1522] - feature, foxx, task router logging improvements (#1648)

Co-authored-by: Joshua S Brown <[email protected]>
Co-authored-by: JoshuaSBrown <[email protected]>

* [DAPS-1522] - feature, foxx, query router logging improvements (#1627)

Co-authored-by: Joshua S Brown <[email protected]>
Co-authored-by: JoshuaSBrown <[email protected]>

* [DAPS-1735] bug: remove duplicate user_router test (#1736)

* [DAPS-1731] - Feature, scripts, compose - add scripts to generate globus credentials for web service (#1731)

Co-authored-by: Joshua S Brown <[email protected]>
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>

* [DAPS-1725] - refactor tests mock core server centralized (#1726)

* [DAPS-1741] Update: native client id in intialize_globus_endpoint and globus_clea… (#1741)

* [DAPS-1745]  - scripts fix: account for nested client credentials. (#1746)

* [DAPS-1725-2] - tests, fix - centralized mock core service libraries fixed (part 2) (#1747)

* [DAPS-1742] - refactor script replace os.path.join with urllib.parse.urljoin (#1744)

* [DAPS-1749] refactor: set cmake policy to silence noisy warning. (#1750)

* [DAPS-1522] - foxx feature tag router logging improvements (#1734)


Co-authored-by: Joshua S Brown <[email protected]>

* [DAPS-1378] - web, bug: fix mapping of multiple globus accounts. (#1753)

* [DAPS-1756] - scripts, foxx, fix: add retries and connection check to install_foxx.sh script. (#1757)

* [DAPS-1522] Version Router Logging Improvements (#1758)

* [DAPS-1737] - compose, refactor compose cleanup arango ssl env variables (#1765)

* [DAPS-1766] - fix python client provisioning job

* [DAPS-1663] - Feature, Core Service, adding Correlation ID to Logging (#1704)

Co-authored-by: JoshuaSBrown <[email protected]>
Co-authored-by: Joshua S Brown <[email protected]>

---------

Co-authored-by: Joshua S Brown <[email protected]>
Co-authored-by: AronPerez <[email protected]>
Co-authored-by: Anthony <[email protected]>
Co-authored-by: Anthony Ramirez <[email protected]>
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
Co-authored-by: Aaron Perez <[email protected]>
Co-authored-by: par-hermes <[email protected]>
Co-authored-by: Austin Hampton <[email protected]>
Co-authored-by: Theo Beers <[email protected]>
Co-authored-by: JoshuaSBrown <[email protected]>
Co-authored-by: Austin Hampton <[email protected]>
Co-authored-by: Polina Shpilker <[email protected]>

Author: 10 people

.github/workflows/build-docs.yml

@@ -40,7 +40,7 @@ jobs:
                   ./scripts/generate_datafed.sh
             - name: Install documentation build dependencies
               run: |
-                  sudo ./scripts/install_docs_dependencies.sh
+                  sudo ./external/DataFedDependencies/scripts/install_docs_dependencies.sh
             - name: Build documentation
               run: |
                   cmake -S. -B build -DBUILD_AUTHZ=OFF -DBUILD_CORE_SERVER=OFF -DBUILD_COMMON=OFF -DBUILD_DOCS=ON -DBUILD_FOXX=OFF -DBUILD_REPO_SERVER=OFF -DBUILD_PYTHON_CLIENT=ON -DBUILD_TESTS=OFF -DBUILD_WEB_SERVER=OFF -DENABLE_UNIT_TESTS=OFF

.github/workflows/unit-tests.yml

@@ -10,9 +10,12 @@ jobs:
             - name: Update debian
               run: apt update
             - name: Install dependencies
+              with:
+                  submodules: recursive
+                  fetch-depth: 0
               run: |
                   ./scripts/generate_datafed.sh
-                  ./scripts/install_core_dependencies.sh
+                  ./external/DataFedDependencies/scripts/install_core_dependencies.sh
             - name: Build
               run: |
                   /opt/datafed/dependencies/bin/cmake -S. -B build -DCMAKE_BUILD_TYPE=Debug -DBUILD_WEB_SERVER=OFF

.gitignore

@@ -53,7 +53,8 @@ scripts/admin_datafed_backup.sh
 scripts/admin_refresh_certs.sh
 scripts/globus/__pycache__
 services/
-tmp/
+tests/mock_core/Version.hpp
+tmp
 web/SDMS.proto
 web/SDMS_Anon.proto
 web/SDMS_Auth.proto
@@ -65,6 +66,8 @@ web/package-lock.json
 web/static/datafed-core-key.pub
 *.swp
 *.swo
+*.tar.gz
+*.tgz
 
 # for web tests
 tests/end-to-end/web-UI/node_modules/

.gitlab-ci.yml

@@ -18,6 +18,7 @@ include:
     - local: .gitlab/stage_pipeline_serialize.yml
     - local: .gitlab/infrastructure.yml
     - local: .gitlab/stage_clear_cache.yml
+    - local: .gitlab/stage_base_image_check.yml
     - local: .gitlab/stage_build_base.yml
     - local: .gitlab/stage_provision_client.yml
     - local: .gitlab/stage_image_check.yml
@@ -32,6 +33,7 @@ stages:
     - trigger-infrastructure
     - signal
     - clear-docker-cache
+    - base-image-check
     - build-base
     - provision-client
     - image-check
@@ -56,3 +58,4 @@ variables:
     REGISTRY: "camden.ornl.gov"
     DATAFED_DEPENDENCIES_INSTALL_PATH: "/shared/install"
     DOCKER_TLS_CERTDIR: "" # Required for running docker in docker
+    GIT_SUBMODULE_STRATEGY: recursive

.gitlab/build/build_core_image.yml

@@ -1,53 +1,53 @@
 ---
 stages:
-  - build
+    - build
 
 include:
-  - local: .gitlab/common.yml
+    - local: .gitlab/common.yml
 
 build-core:
-  extends: .docker_build_script
-  stage: build
-  variables:
-    PROJECT: "datafed"
-    COMPONENT: "core"
-    GIT_STRATEGY: clone
-    DOCKER_FILE_PATH: "core/docker/Dockerfile"
-    DATAFED_HARBOR_REGISTRY: "$REGISTRY" # needed by c_harbor_artifact_count
-    BUILD_INTERMEDIATE: "FALSE"
-  tags:
-    - ci-datafed-core
-    - docker
-  rules:
-    - changes:
-        - docker/**/*
-        - scripts/**/*
-        - core/**/*
-        - common/**/*
-        - CMakeLists.txt
-        - cmake/**/*
-        - .gitlab-ci.yml
-      when: on_success
+    extends: .docker_build_script
+    stage: build
+    variables:
+        PROJECT: "datafed"
+        COMPONENT: "core"
+        GIT_STRATEGY: clone
+        DOCKER_FILE_PATH: "core/docker/Dockerfile"
+        DATAFED_HARBOR_REGISTRY: "$REGISTRY" # needed by c_harbor_artifact_count
+        BUILD_INTERMEDIATE: "FALSE"
+    tags:
+        - ci-datafed-core
+        - docker
+    rules:
+        - changes:
+              - docker/**/*
+              - scripts/**/*
+              - core/**/*
+              - common/**/*
+              - CMakeLists.txt
+              - cmake/**/*
+              - .gitlab-ci.yml
+          when: on_success
 
 retag-image:
-  extends: .docker_retag_image
-  stage: build
-  variables:
-    PROJECT: "datafed"
-    COMPONENT: "core"
-    GIT_STRATEGY: clone
-    DATAFED_HARBOR_REGISTRY: "$REGISTRY" # needed by c_harbor_artifact_count
-    BUILD_INTERMEDIATE: "FALSE"
-  tags:
-    - docker
-  rules:
-    - changes:
-        - docker/**/*
-        - scripts/**/*
-        - core/**/*
-        - common/**/*
-        - CMakeLists.txt
-        - cmake/**/*
-        - .gitlab-ci.yml
-      when: never
-    - when: on_success
+    extends: .docker_retag_image
+    stage: build
+    variables:
+        PROJECT: "datafed"
+        COMPONENT: "core"
+        GIT_STRATEGY: clone
+        DATAFED_HARBOR_REGISTRY: "$REGISTRY" # needed by c_harbor_artifact_count
+        BUILD_INTERMEDIATE: "FALSE"
+    tags:
+        - docker
+    rules:
+        - changes:
+              - docker/**/*
+              - scripts/**/*
+              - core/**/*
+              - common/**/*
+              - CMakeLists.txt
+              - cmake/**/*
+              - .gitlab-ci.yml
+          when: never
+        - when: on_success

.gitlab/build/build_foxx_image.yml

@@ -1,56 +1,56 @@
 ---
 stages:
-  - build
+    - build
 
 include:
-  - local: .gitlab/common.yml
+    - local: .gitlab/common.yml
 
 build-foxx:
-  extends: .docker_build_script
-  stage: build
-  variables:
-    PROJECT: "datafed"
-    COMPONENT: "foxx"
-    GIT_STRATEGY: clone
-    DOCKER_FILE_PATH: "docker/Dockerfile.foxx"
-    DATAFED_HARBOR_REGISTRY: "$REGISTRY" # needed by c_harbor_artifact_count
-    BUILD_INTERMEDIATE: "FALSE"
-  tags:
-    - docker
-  rules:
-    - changes:
-        - docker/**/*
-        - scripts/**/*
-        - cmake/**/*
-        - core/database/**/*
-        - core/CMakeLists.txt
-        - common/proto/**/*
-        - .gitlab-ci.yml
-        - .gitlab/**/*
-        - CMakeLists.txt
-      when: on_success
+    extends: .docker_build_script
+    stage: build
+    variables:
+        PROJECT: "datafed"
+        COMPONENT: "foxx"
+        GIT_STRATEGY: clone
+        DOCKER_FILE_PATH: "docker/Dockerfile.foxx"
+        DATAFED_HARBOR_REGISTRY: "$REGISTRY" # needed by c_harbor_artifact_count
+        BUILD_INTERMEDIATE: "FALSE"
+    tags:
+        - docker
+    rules:
+        - changes:
+              - docker/**/*
+              - scripts/**/*
+              - cmake/**/*
+              - core/database/**/*
+              - core/CMakeLists.txt
+              - common/proto/**/*
+              - .gitlab-ci.yml
+              - .gitlab/**/*
+              - CMakeLists.txt
+          when: on_success
 
 retag-image:
-  extends: .docker_retag_image
-  stage: build
-  variables:
-    PROJECT: "datafed"
-    COMPONENT: "foxx"
-    GIT_STRATEGY: clone
-    DATAFED_HARBOR_REGISTRY: "$REGISTRY" # needed by c_harbor_artifact_count
-    BUILD_INTERMEDIATE: "FALSE"
-  tags:
-    - docker
-  rules:
-    - changes:
-        - docker/**/*
-        - scripts/**/*
-        - cmake/**/*
-        - core/database/**/*
-        - core/CMakeLists.txt
-        - common/proto/**/*
-        - .gitlab-ci.yml
-        - .gitlab/**/*
-        - CMakeLists.txt
-      when: never
-    - when: on_success
+    extends: .docker_retag_image
+    stage: build
+    variables:
+        PROJECT: "datafed"
+        COMPONENT: "foxx"
+        GIT_STRATEGY: clone
+        DATAFED_HARBOR_REGISTRY: "$REGISTRY" # needed by c_harbor_artifact_count
+        BUILD_INTERMEDIATE: "FALSE"
+    tags:
+        - docker
+    rules:
+        - changes:
+              - docker/**/*
+              - scripts/**/*
+              - cmake/**/*
+              - core/database/**/*
+              - core/CMakeLists.txt
+              - common/proto/**/*
+              - .gitlab-ci.yml
+              - .gitlab/**/*
+              - CMakeLists.txt
+          when: never
+        - when: on_success