Add import command for data loader CLI #2618
Conversation
| TableMetadataService tableMetadataService = | ||
| new TableMetadataService(storageFactory.getStorageAdmin()); |
There was a problem hiding this comment.
You should close the DistributedStorageAdmin instance after use.
DistributedStorageAdmin admin = storageFactory.getStorageAdmin();
// ...
admin.close();
If there are other parts that need the same fix, could you take care of them in a separate PR?
There was a problem hiding this comment.
Thank you.
I will update the code to close it and will add necessary changes to close other instances in another PR.
| } | ||
| if (importOptions.getLogMode().equals(LogMode.SPLIT_BY_DATA_CHUNK)) { | ||
| importManager.addListener(new SplitByDataChunkImportLogger(config, logWriterFactory)); | ||
| } else importManager.addListener(new SingleFileImportLogger(config, logWriterFactory)); |
There was a problem hiding this comment.
Small nit:
| } else importManager.addListener(new SingleFileImportLogger(config, logWriterFactory)); | |
| } else { | |
| importManager.addListener(new SingleFileImportLogger(config, logWriterFactory)); | |
| } |
| File configFile = new File(configFilePath); | ||
| StorageFactory storageFactory = StorageFactory.create(configFile); | ||
| TableMetadataService tableMetadataService = | ||
| new TableMetadataService(storageFactory.getStorageAdmin()); |
There was a problem hiding this comment.
Will this DistributedStorageAdmin be closed?
There was a problem hiding this comment.
No, I will update the code to close it once the use is completed.
| importProcessorFactory, | ||
| ScalarDbMode.TRANSACTION, | ||
| null, | ||
| scalarDbTransactionManager.getDistributedTransactionManager()); |
There was a problem hiding this comment.
Will this instance be closed?
There was a problem hiding this comment.
Currently No. I will created another PR to make necessary changes to confirm that both transaction manager and storage are closed once the process is completed. I will make this change in core part as we can confirm the import process completion in the import manager class.
| importOptions, | ||
| importProcessorFactory, | ||
| ScalarDbMode.STORAGE, | ||
| scalarDbStorageManager.getDistributedStorage(), |
There was a problem hiding this comment.
No, I will update it as mentioned in the above comment,
| } | ||
| if (importOptions.getLogMode().equals(LogMode.SPLIT_BY_DATA_CHUNK)) { | ||
| importManager.addListener(new SplitByDataChunkImportLogger(config, logWriterFactory)); | ||
| } else importManager.addListener(new SingleFileImportLogger(config, logWriterFactory)); |
There was a problem hiding this comment.
Can you wrap this with braces?
|
@brfrn169 san, @komamitsu san, |
Co-authored-by: Peckstadt Yves <[email protected]> Co-authored-by: Toshihiro Suzuki <[email protected]>
Co-authored-by: Peckstadt Yves <[email protected]> Co-authored-by: Toshihiro Suzuki <[email protected]>
Co-authored-by: Peckstadt Yves <[email protected]> Co-authored-by: Toshihiro Suzuki <[email protected]>
Co-authored-by: Peckstadt Yves <[email protected]> Co-authored-by: Toshihiro Suzuki <[email protected]>
Co-authored-by: Peckstadt Yves <[email protected]> Co-authored-by: Toshihiro Suzuki <[email protected]>
Description
In this PR I am adding data loader import command and command options for data import.
Related issues and/or PRs
Please review this PR once the below PRs is merged and latest changes from master are merged to this branch
Changes made
I have added the import command options and import command as a CLI command for data import
Checklist
Additional notes (optional)
This is the third PR for data loader CLI.
Release notes
NA