Skip to content

Add import command for data loader CLI #2618

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 26 commits into
base: master
Choose a base branch
from

Conversation

inv-jishnu
Copy link
Contributor

Description

In this PR I am adding data loader import command and command options for data import.

Related issues and/or PRs

Please review this PR once the below PRs is merged and latest changes from master are merged to this branch

Changes made

I have added the import command options and import command as a CLI command for data import

Checklist

The following is a best-effort checklist. If any items in this checklist are not applicable to this PR or are dependent on other, unmerged PRs, please still mark the checkboxes after you have read and understood each item.

  • I have commented my code, particularly in hard-to-understand areas.
  • I have updated the documentation to reflect the changes.
  • Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
  • Tests (unit, integration, etc.) have been added for the changes.
  • My changes generate no new warnings.
  • Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

This is the third PR for data loader CLI.

Release notes

NA

@inv-jishnu inv-jishnu added the enhancement New feature or request label Apr 30, 2025
@inv-jishnu inv-jishnu self-assigned this Apr 30, 2025
@inv-jishnu inv-jishnu marked this pull request as draft April 30, 2025 12:15
Base automatically changed from feat/data-loader/cli-export-2 to master May 22, 2025 05:58
@ypeckstadt ypeckstadt requested a review from Copilot May 23, 2025 02:30
@ypeckstadt ypeckstadt marked this pull request as ready for review May 23, 2025 02:32
@ypeckstadt ypeckstadt requested a review from a team as a code owner May 23, 2025 02:32
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a new import command to the data loader CLI, including command options, implementation, and a basic unit test.

  • Introduce ImportCommandOptions with various CLI flags for importing
  • Implement ImportCommand to wire options into ImportManager and handle validation/logging
  • Add ImportCommandTest to verify behavior when required inputs are missing

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File Description
ImportCommandOptions.java Defined new PicoCLI options for the import command
ImportCommand.java Implemented import command logic, validation, and wiring to core import APIs
ImportCommandTest.java Added a test for missing/invalid config and import file
Comments suppressed due to low confidence (2)

data-loader/cli/src/main/java/com/scalar/db/dataloader/cli/command/dataimport/ImportCommand.java:148

  • Typo in variable name scalarDbStorageManger (should be scalarDbStorageManager).
ScalarDbStorageManager scalarDbStorageManger =

data-loader/cli/src/test/java/com/scalar/db/dataloader/cli/command/dataimport/ImportCommandTest.java:50

  • The test expects IllegalArgumentException, but the command may throw a ParameterException or other exception type. Update the assertion to expect the correct exception or add more precise error checking.
assertThrows(IllegalArgumentException.class, () -> importCommand.call());

protected char delimiter;

@CommandLine.Option(
names = {"--header", "-h"},
Copy link
Preview

Copilot AI May 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The short option '-h' typically maps to help; consider choosing a different short flag to avoid clashing with PicoCLI's default help option.

Suggested change
names = {"--header", "-h"},
names = {"--header", "-hdr"},

Copilot uses AI. Check for mistakes.

* @throws ParameterException if the path is invalid
*/
private Optional<ControlFile> parseControlFileFromPath(String controlFilePath) {
if (controlFilePath == null) {
Copy link
Preview

Copilot AI May 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parseControlFileFromPath only checks for null but not blank strings; an empty "" path will cause ObjectMapper to try reading a non-existent file. Consider using StringUtils.isBlank(controlFilePath).

Suggested change
if (controlFilePath == null) {
if (StringUtils.isBlank(controlFilePath)) {

Copilot uses AI. Check for mistakes.

builder.importMode(importMode);
}
if (!splitLogMode) {
builder.logMode(LogMode.SINGLE_FILE);
Copy link
Preview

Copilot AI May 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The splitLogMode flag is only used to set SINGLE_FILE mode; when true, you never set LogMode.SPLIT_BY_DATA_CHUNK. Add an else block to set the split-chunk mode.

Suggested change
builder.logMode(LogMode.SINGLE_FILE);
builder.logMode(LogMode.SINGLE_FILE);
} else {
builder.logMode(LogMode.SPLIT_BY_DATA_CHUNK);

Copilot uses AI. Check for mistakes.


@CommandLine.Command(name = "import", description = "Import data into a ScalarDB table")
@CommandLine.Command(name = "import", description = "import data into a ScalarDB table")
Copy link
Preview

Copilot AI May 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] The command description was changed to start with a lowercase letter; for consistency with other commands, consider capitalizing the first word.

Suggested change
@CommandLine.Command(name = "import", description = "import data into a ScalarDB table")
@CommandLine.Command(name = "import", description = "Import data into a ScalarDB table")

Copilot uses AI. Check for mistakes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants