Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

define "invoker" behavior #11

Open
Tracked by #3
keighrim opened this issue May 11, 2023 · 0 comments
Open
Tracked by #3

define "invoker" behavior #11

keighrim opened this issue May 11, 2023 · 0 comments
Milestone

Comments

@keighrim
Copy link
Member

(subtask of #3)

The evaluation invoker is the main player to manage a single evaluation process. Some features are also discussed in clamsproject/aapb-annotations#2 (comment).

At a high level, it should be able to handle

  1. user input to start an evaluation process; this input doesn't have to manual, but can be a github actions dispatch upon a release of a new version of a CLAMS app via app-directory.
  2. collection of AAPB GUIDs based on the evaluation dataset specified in the input
  3. initiation of a CLAMS pipeline based on the GUIDs from the previous component and a pipeline configuration specified in the input
  4. acquisition of result MMIF files once the pipeline is successfully finished (also see persisting machine annotations #10)
  5. invocation of an "evaluator" specified in the input and obtain HTML with the evaluation result
  6. automatic creation of a PR with the HTML file(s) from the previous component into github.io deployment branch/directory for manual review before publication of the evaluation results.
@clams-bot clams-bot added this to infra May 11, 2023
@github-project-automation github-project-automation bot moved this to Todo in infra May 11, 2023
@keighrim keighrim added this to the eval-v1 milestone Jan 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Todo
Development

No branches or pull requests

1 participant