feat(LMES): Support custom template and prompt

[yhwang]

Expand the LMEvalJob CRD to support custom templates and system prompts. This is mainly for custom unitxt task recipes. Now, users can use the template and systemPrompt fields under the taskRecipes to specify the custom template and system prompt.

[yhwang]

Instead of calling mkdir in the Dockerfile.lmes-job, the driver creates the folders for custom unitxt catalogs. This would minimize the changes in the Dockerfile.

[yhwang]

Tried to refactor the testing code in the driver_test.go. I believe it's in better shape now.

[yhwang]

Here is a sample LMEvalJob with a custom template and custom system prompt:

apiVersion: trustyai.opendatahub.io/v1alpha1
kind: LMEvalJob
metadata:
  name: custom-card-template
  namespace: tas
spec:
  allowOnline: true
  allowCodeExecution: true
  model: hf
  modelArgs:
  - name: pretrained
    value: google/flan-t5-base
  taskList:
    taskRecipes:
    - template:
        ref: tp_0
      systemPrompt:
        ref: sp_0
      card:
        name: "cards.wnli"
    custom:
      templates:
      - name: tp_0
        value: |
          {
              "__type__": "input_output_template",
              "input_format": "{text_a_type}: {text_a}\n{text_b_type}: {text_b}",
              "output_format": "{label}",
              "target_prefix": "The {type_of_relation} class is ",
              "instruction": "Given a {text_a_type} and {text_b_type} classify the {type_of_relation} of the {text_b_type} to one of {classes}.",
              "postprocessors": [
                  "processors.take_first_non_empty_line",
                  "processors.lower_case_till_punc"
              ]
          }
      systemPrompts:
      - name: sp_0
        value: "Be concise. At every point give the shortest acceptable answer."
  logSamples: true

The template.ref points to the custom template's name under the taskList.custom.templates. In this way, a custom artifact can be used by multiple taskRecipes.

[ruivieira]

/retest

[kiersten-stokes]

+1 to making the custom elements re-usable. This will be super helpful for future features like custom benchmarks

[openshift-ci]

@kiersten-stokes: changing LGTM is restricted to collaborators

In response to this:

+1 to making the custom elements re-usable. This will be super helpful for future features like custom benchmarks

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kiersten-stokes, ruivieira

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

PR image build and manifest generation completed successfully!

📦 PR image: quay.io/trustyai/trustyai-service-operator-ci:ccd0ea2c1a05e7872831f3503848a6c286b0ad55

📦 LMES driver image: quay.io/trustyai/ta-lmes-driver:ccd0ea2c1a05e7872831f3503848a6c286b0ad55

📦 LMES job image: quay.io/trustyai/ta-lmes-job:ccd0ea2c1a05e7872831f3503848a6c286b0ad55

📦 Guardrails orchestrator image: quay.io/trustyai/ta-guardrails-orchestrator:ccd0ea2c1a05e7872831f3503848a6c286b0ad55

🗂️ CI manifests

devFlags:
  manifests:
    - contextDir: config
      sourcePath: ''
      uri: https://api.github.com/repos/trustyai-explainability/trustyai-service-operator-ci/tarball/operator-ccd0ea2c1a05e7872831f3503848a6c286b0ad55

[openshift-ci]

@yhwang: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/trustyai-service-operator-e2e	ccd0ea2	link	true	/test trustyai-service-operator-e2e

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.