HPCC4J-795 TestGenerator Full Service and Automation improvements by jpmcmu · Pull Request #981 · hpcc-systems/hpcc4j

jpmcmu · 2026-03-23T13:19:03Z

Added support for full service test generation
Added intial support for a Github workflow to generate tests
Various improvements

Signed-off-by: James McMullan James.McMullan@lexisnexis.com

Type of change:

This change is a bug fix (non-breaking change which fixes an issue).
This change is a new feature (non-breaking change which adds functionality).
This change is a breaking change (fix or feature that will cause existing behavior to change).

Custom Platform Testing (Optional)

Custom HPCC-Platform Repository:
repository:

Custom HPCC-Platform Branch:
branch:

Checklist:

Testing:

- Added support for full service test generation - Added intial support for a Github workflow to generate tests - Various improvements Signed-off-by: James McMullan James.McMullan@lexisnexis.com

github-actions · 2026-03-23T13:19:28Z

Jira Issue: https://hpccsystems.atlassian.net/browse/HPCC4J-795

Jirabot Action Result:
Workflow Transition To: Merge Pending
Updated PR

github-actions · 2026-03-23T13:22:08Z

🔄 Upmerge Test Results

Status: ✅ All branches merged successfully
PR: #981 - HPCC4J-795 TestGenerator Full Service and Automation improvements
Base Branch: candidate-10.0.x
Test Time: 2026-03-23 13:19:25 UTC

✅ Successful Branches (3)

candidate-10.0.x
candidate-10.2.x
master
This comment was automatically generated by the upmerge test workflow.

Copilot

Pull request overview

This PR enhances the scripts/wsclient-test-generator tooling to support “full-service” test generation (discovering and generating tests for all business methods in a service), adds multi-environment execution and reporting capabilities, and introduces new JUnit @Category marker interfaces for structured test categorization in HPCC4J.

Changes:

Add full-service orchestration to TestGeneratorAgent.py (Step 0 discovery → per-method analysis/generation → metadata aggregation), plus CI-safe operation and optional Copilot model selection.
Introduce environment configuration/filtering, parallel test execution support, and new/expanded prompt templates (including aggregate reporting and per-issue ticket generation).
Add new test category marker interfaces (CFT/ECT/EHT/CNT) under commons-hpcc.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
scripts/wsclient-test-generator/environments.example.json	Adds example schema for multi-environment connection configuration.
scripts/wsclient-test-generator/UnverifiedServerIssuesReportPrompt.md	Expands reporting instructions and adds per-issue ticket file generation guidance.
scripts/wsclient-test-generator/TestGeneratorAgent.py	Implements full-service mode, CI mode, env config/filtering, parallel execution, and report aggregation support.
scripts/wsclient-test-generator/TestGenerationPrompt.md	Updates generation prompt to require labels/categories and `environmentRequirements` metadata, plus dataset-generation guidance.
scripts/wsclient-test-generator/ServiceAnalysisPrompt.md	New prompt to discover business methods and emit a machine-readable ordered method list.
scripts/wsclient-test-generator/README.md	Documents new modes/options, environment filtering, parallelism, and outputs.
scripts/wsclient-test-generator/MethodAnalysisPrompt.md	Adds CNT category guidance, file-search strategy, and scenario injection.
scripts/wsclient-test-generator/FullServiceModePlan.md	Design/plan document describing full-service mode workflow and risks.
scripts/wsclient-test-generator/FixTestCompilationPrompt.md	Expands compilation-fix prompt with file-search + safer edit strategies.
scripts/wsclient-test-generator/FinalReportPrompt.md	Expands final report requirements (labels, env matrix, server/client issue sections).
scripts/wsclient-test-generator/BatchFailureAnalysisPrompt.md	Expands failure triage instructions, adds dataset remediation guidance.
scripts/wsclient-test-generator/AggregateReportPrompt.md	New prompt to generate cross-environment aggregate reports.
commons-hpcc/src/main/java/org/hpccsystems/commons/annotations/CoreFunctionalityTests.java	Adds `@Category` marker for CFT tests (extends `RemoteTests`).
commons-hpcc/src/main/java/org/hpccsystems/commons/annotations/EdgeCaseTests.java	Adds `@Category` marker for ECT tests (extends `RemoteTests`).
commons-hpcc/src/main/java/org/hpccsystems/commons/annotations/ErrorHandlingTests.java	Adds `@Category` marker for EHT tests (extends `RemoteTests`).
commons-hpcc/src/main/java/org/hpccsystems/commons/annotations/ConnectivityTests.java	Adds `@Category` marker for CNT tests (extends `RemoteTests`).
.github/workflows/test-generator.yml	Adds a manual GitHub Actions workflow to generate/build and run tests across containerized/baremetal/secure envs.
.github/actions/setup-copilot-cli/action.yml	Exports `GH_TOKEN` to the job environment for subsequent steps’ Copilot CLI auth.

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 11 comments.

Copilot · 2026-04-08T13:18:12Z

+# Run tests — parallel or sequential depending on PARALLEL_THREADS
+            print(f"\n🧪 Running {len(tests_to_execute)} test(s)" +
+                  (f" (up to {PARALLEL_THREADS} in parallel)..." if PARALLEL_THREADS > 1 else " sequentially..."))
+            test_results = run_tests_parallel(
+                tests_to_execute, test_class,
+                hpcc_conn, wssql_conn, hpcc_user, hpcc_pass,
+                disable_dataset_generation=disable_dataset_generation,
+                num_threads=PARALLEL_THREADS,
+            )


Line 1244 is aligned at column 0 while the following statements remain indented as if they’re inside the while iteration < ... loop. This will trigger an IndentationError (indented block with no enclosing statement). Indent line 1244 to match the rest of the loop body (or unindent 1245–1252) so the block structure is syntactically valid.

Copilot · 2026-04-08T13:18:12Z

+        List of result dicts in the same order as *tests_to_execute*, each
+        containing ``{"metadata": <test_info>, "result": <run_individual_test result>}``.
+    """
+    ordered_results: List[Dict[str, Any]] = [None] * len(tests_to_execute)  # type: ignore[list-item]


The declared type is List[Dict[str, Any]] but the list is initialized with None elements. This forces a type: ignore that defeats the benefit of the type hints. Use List[Optional[Dict[str, Any]]] (and drop the ignore), or initialize with an empty list and append results while preserving ordering another way.

Suggested change

ordered_results: List[Dict[str, Any]] = [None] * len(tests_to_execute) # type: ignore[list-item]

ordered_results: List[Optional[Dict[str, Any]]] = [None] * len(tests_to_execute)

Copilot · 2026-04-08T13:18:13Z

    Also adds directory context for HPCC4J, HPCC Platform, and tmp directories.
    """
-    cmd = ["copilot", "-p", prompt_text]
+    cmd = ["copilot", "-p", prompt_text, "--no-ask-user"]


build_copilot_cmd() now forces --no-ask-user, but copilot_generate() still prints that Copilot will run interactively and prompts users to review/save output. These conflict and will confuse users (especially in CI). Update the messaging to reflect non-interactive execution, or only add --no-ask-user when --ci is set.

Suggested change

cmd = ["copilot", "-p", prompt_text, "--no-ask-user"]

cmd = ["copilot", "-p", prompt_text]

# Only force non-interactive execution in CI mode so local runs can

# remain consistent with interactive user messaging elsewhere.

ci_mode = bool(getattr(args, "ci", False) or os.environ.get("CI"))

if ci_mode:

cmd.append("--no-ask-user")

Copilot · 2026-04-08T13:18:13Z

@@ -351,24 +520,31 @@ def copilot_generate(prompt_file, output_file, variables=None):
    print("Note: This will run Copilot interactively. Please review the analysis and ensure it's saved to the correct file.")


build_copilot_cmd() now forces --no-ask-user, but copilot_generate() still prints that Copilot will run interactively and prompts users to review/save output. These conflict and will confuse users (especially in CI). Update the messaging to reflect non-interactive execution, or only add --no-ask-user when --ci is set.

Suggested change

print("Note: This will run Copilot interactively. Please review the analysis and ensure it's saved to the correct file.")

print("Note: Copilot will run non-interactively and is expected to create the target file automatically.")

Copilot · 2026-04-08T13:18:13Z

+| Error Handling | `EHT` | `@Category(ErrorHandlingTests.class)` | Invalid inputs, expected error responses, failure scenarios |
+| Connectivity | `CNT` | `@Category(ConnectivityTests.class)` | Service reachability, authentication, endpoint validation |
+
+--


This appears intended to be a Markdown horizontal rule, but -- won’t render as one in standard Markdown. Use --- (or remove the line) to avoid formatting breaking in rendered prompt docs.

Suggested change

--

---

Copilot · 2026-04-08T13:18:14Z


-### Current Version (November 2025)
- ✅ Four-step automated test generation
+### February 2026


The changelog entries conflict: Step 0/service discovery is introduced under February 2026, but the November 2025 entry also claims Step 0 was added then. Reconcile these entries so the changelog reflects a consistent history.

Copilot · 2026-04-08T13:18:14Z

+- ✅ **Updated `TestGenerationPrompt.md`**: Accepts service analysis context for dependency-aware test generation
+
+### November 2025
+- ✅ Five-step automated test generation (Step 0 added for service discovery)


The changelog entries conflict: Step 0/service discovery is introduced under February 2026, but the November 2025 entry also claims Step 0 was added then. Reconcile these entries so the changelog reflects a consistent history.

Suggested change

- ✅ Five-step automated test generation (Step 0 added for service discovery)

- ✅ Four-step automated test generation

Copilot · 2026-04-08T13:18:14Z

+      output_dir:
+        description: "Path (relative to repo root) of TestGeneratorAgent output dir to load metadata from. Leave blank to run all tests in the service's test class."
+        required: false
+        default: ""
+        type: string


The workflow defines an output_dir input and sets OUTPUT_DIR in the environment, but the invoked command does not pass any flag corresponding to this value, and the agent (per the shown diffs) derives OUTPUT_DIR internally rather than reading an env var. Either (a) wire this input into TestGeneratorAgent.py via a supported CLI arg, or (b) remove the input/env var to avoid a misleading no-op control.

Suggested change

output_dir:

description: "Path (relative to repo root) of TestGeneratorAgent output dir to load metadata from. Leave blank to run all tests in the service's test class."

required: false

default: ""

type: string

Copilot · 2026-04-08T13:18:15Z

+      - name: Run tests — containerized environment
+        if: steps.k8s_deploy.outcome == 'success'
+        continue-on-error: true
+        env:


The workflow defines an output_dir input and sets OUTPUT_DIR in the environment, but the invoked command does not pass any flag corresponding to this value, and the agent (per the shown diffs) derives OUTPUT_DIR internally rather than reading an env var. Either (a) wire this input into TestGeneratorAgent.py via a supported CLI arg, or (b) remove the input/env var to avoid a misleading no-op control.

Copilot · 2026-04-08T13:18:15Z

+          SERVICE_NAME: ${{ github.event.inputs.service_name }}
+          METHOD_NAME: ${{ github.event.inputs.method_name }}
+          PARALLEL_THREADS: ${{ github.event.inputs.parallel_threads }}
+          OUTPUT_DIR: ${{ github.event.inputs.output_dir }}


The workflow defines an output_dir input and sets OUTPUT_DIR in the environment, but the invoked command does not pass any flag corresponding to this value, and the agent (per the shown diffs) derives OUTPUT_DIR internally rather than reading an env var. Either (a) wire this input into TestGeneratorAgent.py via a supported CLI arg, or (b) remove the input/env var to avoid a misleading no-op control.

Suggested change

OUTPUT_DIR: ${{ github.event.inputs.output_dir }}

HPCC4J-795 TestGenerator Full Service and Automation improvements

cae1b3f

- Added support for full service test generation - Added intial support for a Github workflow to generate tests - Various improvements Signed-off-by: James McMullan James.McMullan@lexisnexis.com

jpmcmu requested a review from Copilot March 23, 2026 13:27

Copilot started reviewing on behalf of jpmcmu March 23, 2026 13:31 View session

Copilot AI reviewed Mar 23, 2026

View reviewed changes

Workflow fixes

9ece227

jpmcmu marked this pull request as ready for review April 7, 2026 13:08

jpmcmu requested a review from rpastrana April 7, 2026 13:12

Code review changes

995245e

jpmcmu requested a review from Copilot April 8, 2026 13:09

Copilot AI reviewed Apr 8, 2026

View reviewed changes

Copilot started reviewing on behalf of jpmcmu April 8, 2026 14:17 View session

	ordered_results: List[Dict[str, Any]] = [None] * len(tests_to_execute) # type: ignore[list-item]
	ordered_results: List[Optional[Dict[str, Any]]] = [None] * len(tests_to_execute)

-    cmd = ["copilot", "-p", prompt_text, "--no-ask-user"]
+    cmd = ["copilot", "-p", prompt_text]
+    # Only force non-interactive execution in CI mode so local runs can
+    # remain consistent with interactive user messaging elsewhere.
+    ci_mode = bool(getattr(args, "ci", False) or os.environ.get("CI"))
+    if ci_mode:
+        cmd.append("--no-ask-user")

		@@ -351,24 +520,31 @@ def copilot_generate(prompt_file, output_file, variables=None):
		print("Note: This will run Copilot interactively. Please review the analysis and ensure it's saved to the correct file.")

	print("Note: This will run Copilot interactively. Please review the analysis and ensure it's saved to the correct file.")
	print("Note: Copilot will run non-interactively and is expected to create the target file automatically.")

	- ✅ Five-step automated test generation (Step 0 added for service discovery)
	- ✅ Four-step automated test generation

Uh oh!

Conversation

jpmcmu commented Mar 23, 2026

Type of change:

Custom Platform Testing (Optional)

Checklist:

Testing:

Uh oh!

github-actions Bot commented Mar 23, 2026

Uh oh!

github-actions Bot commented Mar 23, 2026

🔄 Upmerge Test Results

✅ Successful Branches (3)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants