Skip to content

Commit

Permalink
MLE-14507 Renamed options to use --lower-hyphenated
Browse files Browse the repository at this point in the history
No functional changes, just a bunch of search/replace. 

Added a test - `VerifyOptionNamesTest` - that ensures there are no option names containing uppercase characters.
  • Loading branch information
rjrudin committed Jun 10, 2024
1 parent 2305512 commit 3e31c3e
Show file tree
Hide file tree
Showing 91 changed files with 971 additions and 891 deletions.
34 changes: 17 additions & 17 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,32 +140,32 @@ You can cause a failure with MarkLogic that caused the command to stop:

```
./nt/bin/nt import-files --path "new-tool-cli/src/test/resources/mixed-files/*" \
--connectionString "new-tool-user:password@localhost:8000" \
--connection-string "new-tool-user:password@localhost:8000" \
--repartition 1 \
--abortOnWriteFailure \
--abort-on-write-failure \
--permissions "invalid-role,read,new-tool-role,update" \
--uriReplace ".*/mixed-files,'/test'"
--uri-replace ".*/mixed-files,'/test'"
```

You can cause a failure and ask to see the full stacktrace (often noisy and not helpful):

```
./nt/bin/nt import-files --path "new-tool-cli/src/test/resources/mixed-files/*" \
--connectionString "new-tool-user:password@localhost:8000" \
--connection-string "new-tool-user:password@localhost:8000" \
--repartition 1 \
--permissions "invalid-role,read,new-tool-role,update" \
--uriReplace ".*/mixed-files,'/test'" \
--abortOnWriteFailure \
--uri-replace ".*/mixed-files,'/test'" \
--abort-on-write-failure \
--stacktrace
```

You can cause a failure and tell the command to keep executing by not including `--abortOnWriteFailure`:
You can cause a failure and tell the command to keep executing by not including `--abort-on-write-failure`:

```
./nt/bin/nt import-files --path "new-tool-cli/src/test/resources/mixed-files/*" \
--connectionString "new-tool-user:password@localhost:8000" \
--connection-string "new-tool-user:password@localhost:8000" \
--permissions "invalid-role,read,new-tool-role,update" \
--uriReplace ".*/mixed-files,'/test'"
--uri-replace ".*/mixed-files,'/test'"
```

## Testing with a load balancer
Expand All @@ -181,13 +181,13 @@ owned by the performance team. Feel free to adjust this config locally as needed
Example of using the existing config to copy from port 8015 to port 8016 in the performance cluster:

```
./nt/bin/nt copy --connectionString "admin:admin@localhost:8006" \
./nt/bin/nt copy --connection-string "admin:admin@localhost:8006" \
--collections "address_small" \
--batchSize 500 \
--batch-size 500 \
--limit 10000 \
--categories content,metadata \
--outputConnectionString "admin:admin@localhost:8007" \
--outputThreadCount 3 --partitionsPerForest 1 --outputBatchSize 200
--output-connection-string "admin:admin@localhost:8007" \
--output-thread-count 3 --partitions-per-forest 1 --output-batch-size 200
```

## Testing against a separate Spark cluster
Expand Down Expand Up @@ -239,7 +239,7 @@ cluster:
```
$SPARK_HOME/bin/spark-submit --class com.marklogic.newtool.cli.Submit \
--master spark://NYWHYC3G0W:7077 new-tool-cli/build/libs/new-tool-cli-0.2.0-all.jar \
import-files --path /Users/rudin/workspace/new-tool/new-tool-cli/src/test/resources/mixed-files --preview 5 --previewDrop content
import-files --path /Users/rudin/workspace/new-tool/new-tool-cli/src/test/resources/mixed-files --preview 5 --preview-drop content
```

After spark-submit completes, you can refresh <http://localhost:8080> to see evidence of the completed application.
Expand All @@ -253,7 +253,7 @@ to something you can access :
$SPARK_HOME/bin/spark-submit --class com.marklogic.newtool.cli.Submit \
--packages org.apache.hadoop:hadoop-aws:3.3.6,org.apache.hadoop:hadoop-client:3.3.6 \
--master spark://NYWHYC3G0W:7077 new-tool-cli/build/libs/new-tool-cli-0.1-SNAPSHOT-all.jar \
import-files --path "s3a://changeme/*.*" --preview 10 --previewDrop content
import-files --path "s3a://changeme/*.*" --preview 10 --preview-drop content
```

### Testing with AWS EMR
Expand All @@ -279,11 +279,11 @@ Once your cluster is created, you'll add a "Step" in order to run spark-submit:
3. For "Spark-submit options", enter `--class com.marklogic.newtool.cli.Submit`.
4. For "Arguments", enter the CLI command all the args you would normally enter when using the CLI.

If your CLI command will be accessing S3, you most likely should not include `--s3AddCredentials`. The EMR EC2 instance
If your CLI command will be accessing S3, you most likely should not include `--s3-add-credentials`. The EMR EC2 instance
will already have access to the S3 buckets per the "EC2 instance profile" you configured while creating your cluster.

Additionally, if your CLI command is accessing an S3 bucket in a region other than the one that EMR is running in,
you can add `--s3Endpoint s3.us-east-1.amazon.com` as an argument, replacing "us-east-1" with the region that the
you can add `--s3-endpoint s3.us-east-1.amazon.com` as an argument, replacing "us-east-1" with the region that the
S3 buckets is in.

After adding your step, it will run. It typically takes about 30s for the step to run, and it may take a minute or so
Expand Down
58 changes: 29 additions & 29 deletions docs/common-options.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,9 @@ Generally, you must include at least the following information for each command:
- Authentication information.

For the common use case of using digest or basic authentication with a MarkLogic app server, you can use the
`--connectionString` option to specify the host, port, username, and password in a single concise option:
`--connection-string` option to specify the host, port, username, and password in a single concise option:

--connectionString user:password@host:port
--connection-string user:password@host:port

For other authentication mechanisms, you must use the `--host` and `--port` options to define the host and port for
your MarkLogic app server.
Expand All @@ -33,30 +33,30 @@ All available connection options are shown in the table below:

| Option | Description |
| --- | --- |
| --authType | Type of authentication to use. Possible values are `BASIC`, `DIGEST`, `CLOUD`, `KERBEROS`, `CERTIFICATE`, and `SAML`.|
| --basePath | Path to prepend to each call to a MarkLogic REST API app server. |
| --certificateFile | File path for a key store to be used for 'CERTIFICATE' authentication. |
| --certificatePassword | Password for the key store referenced by '--certificateFile'. |
| --connectionString | Defines a connection string as user:password@host:port; only usable when using `DIGEST` or `BASIC` authentication. |
| --cloudApiKey | API key for authenticating with a MarkLogic Cloud cluster. |
| --connectionType | Defines whether connections can be made directly to each host in the MarkLogic cluster. Possible values are `DIRECT` and `GATEWAY`. |
| --auth-type | Type of authentication to use. Possible values are `BASIC`, `DIGEST`, `CLOUD`, `KERBEROS`, `CERTIFICATE`, and `SAML`.|
| --base-path | Path to prepend to each call to a MarkLogic REST API app server. |
| --certificate-file | File path for a keystore to be used for 'CERTIFICATE' authentication. |
| --certificate-password | Password for the keystore referenced by '--certificate-file'. |
| --connection-string | Defines a connection string as user:password@host:port; only usable when using `DIGEST` or `BASIC` authentication. |
| --cloud-api-key | API key for authenticating with a MarkLogic Cloud cluster. |
| --connection-type | Defines whether connections can be made directly to each host in the MarkLogic cluster. Possible values are `DIRECT` and `GATEWAY`. |
| --database | Name of a database to connect if it differs from the one associated with the app server identified by '--port'. |
| --disableGzippedResponses | If included, responses from MarkLogic will not be gzipped. May improve performance when responses are very small.
| --disable-gzipped-responses | If included, responses from MarkLogic will not be gzipped. May improve performance when responses are very small.
| --host | The MarkLogic host to connect to. |
| --kerberosPrincipal | Principal to be used with `KERBEROS` authentication. |
| --keyStoreAlgorithm | Algorithm of the key store identified by '--keyStorePath'; defaults to `SunX509`. |
| --keyStorePassword | Password for the key store identified by '--keyStorePath'. |
| --keyStorePath | File path for a key store for two-way SSL connections. |
| --keyStoreType | Type of the key store identified by '--keyStorePath'; defaults to `JKS`. |
| --kerberos-principal | Principal to be used with `KERBEROS` authentication. |
| --keystore-algorithm | Algorithm of the keystore identified by '--keystore-path'; defaults to `SunX509`. |
| --keystore-password | Password for the keystore identified by '--keystore-path'. |
| --keystore-path | File path for a keystore for two-way SSL connections. |
| --keystore-type | Type of the keystore identified by '--keystore-path'; defaults to `JKS`. |
| --password | Password when using `DIGEST` or `BASIC` authentication. |
| --port | Port of the [REST API app server](https://docs.marklogic.com/guide/rest-dev) to connect to. |
| --samlToken | Token to be used with `SAML` authentication. |
| --sslHostnameVerifier | Hostname verification strategy when connecting via SSL. Possible values are `ANY`, `COMMON`, and `STRICT`. |
| --sslProtocol | SSL protocol to use when the MarkLogic app server requires an SSL connection. If a key store or trust store is configured, defaults to `TLSv1.2`. |
| --trustStoreAlgorithm | Algorithm of the trust store identified by '--trustStorePath'; defaults to `SunX509`. |
| --trustStorePassword | Password for the trust store identified by '--trustStorePath'. |
| --trustStorePath | File path for a trust store for establishing trust with the certificate used by the MarkLogic app server. |
| --trustStoreType | Type of the trust store identified by '--trustStorePath'; defaults to `JKS`. |
| --saml-token | Token to be used with `SAML` authentication. |
| --ssl-hostname-verifier | Hostname verification strategy when connecting via SSL. Possible values are `ANY`, `COMMON`, and `STRICT`. |
| --ssl-protocol | SSL protocol to use when the MarkLogic app server requires an SSL connection. If a keystore or truststore is configured, defaults to `TLSv1.2`. |
| --truststore-algorithm | Algorithm of the truststore identified by '--truststore-path'; defaults to `SunX509`. |
| --truststore-password | Password for the truststore identified by '--truststore-path'. |
| --truststore-path | File path for a truststore for establishing trust with the certificate used by the MarkLogic app server. |
| --truststore-type | Type of the truststore identified by '--truststore-path'; defaults to `JKS`. |
| --username | Username when using `DIGEST` or `BASIC` authentication. |


Expand Down Expand Up @@ -93,17 +93,17 @@ you can preview an import without writing any of the data to MarkLogic:
For commands that read from a source other than MarkLogic, you are not required to specify any MarkLogic connection
information when including `--preview` since no connection needs to be made to MarkLogic.

The number after `--preview` specifies how many records to show. You can use `--previewDrop` to specify potentially
verbose columns to drop from the preview. And you can use `--previewVertical` to see the results a vertical display
The number after `--preview` specifies how many records to show. You can use `--preview-drop` to specify potentially
verbose columns to drop from the preview. And you can use `--preview-vertical` to see the results a vertical display
instead of in a table:

```
./bin/nt import-parquet-files \
--connectionString "nt-user:password@localhost:8004" \
--connection-string "nt-user:password@localhost:8004" \
--path export/parquet \
--preview 10 \
--previewDrop job_title,department
--previewVertical
--preview-drop job_title,department
--preview-vertical
```

Note that in the case of previewing an import, NT will show the data as it has been read, which consists of a set of
Expand All @@ -120,10 +120,10 @@ The following shows an example of only importing the first 10 rows from a delimi
```
./bin/nt import-delimited-files \
--path ../data/employees.csv.gz \
--connectionString "nt-user:password@localhost:8004" \
--connection-string "nt-user:password@localhost:8004" \
--permissions nt-role,read,nt-role,update \
--collections employee \
--uriTemplate "/employee/{id}.json" \
--uri-template "/employee/{id}.json" \
--limit 10
```

Expand Down
14 changes: 7 additions & 7 deletions docs/copy.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,30 +22,30 @@ The following options control which documents are read from MarkLogic:

| Option | Description |
| --- |--- |
| --stringQuery | A string query utilizing MarkLogic's search grammar. |
| --string-query | A string query utilizing MarkLogic's search grammar. |
| --query | A structured, serialized CTS, or combined query expressed as JSON or XML. |
| --options | Name of a REST API search options document; typically used with a string query. |
| --collections | Comma-delimited sequence of collection names. |
| --directory | A database directory for constraining on URIs. |

You must specify at least one of `--stringQuery`, `--query`, `--collections`, or `--directory`. You may specify any
You must specify at least one of `--string-query`, `--query`, `--collections`, or `--directory`. You may specify any
combination of those options as well.

The `copy` command then requires that you specify connection information via for the target database that the documents
will be copied into. Each of the [connection options](common-options.md) can be used for this target database, but with
`output` as a prefix so that they are distinguished from the connections used for the source database. For example,
`--outputConnectionString` is used to specify a connection string for the target database. If you are copying the documents
`--output-connection-string` is used to specify a connection string for the target database. If you are copying the documents
to the same database that they were read from, you can omit output connection options.

The following shows an example of copying documents from a collection to a different database in the same MarkLogic
cluster:

```
./bin/nt copy \
--connectionString "user:password@localhost:8000" \
--connection-string "user:password@localhost:8000" \
--collections "example" \
--outputConnectionString "user:password@localhost:8000" \
--outputDatabase "target-database"
--output-connection-string "user:password@localhost:8000" \
--output-database "target-database"
```

## Controlling what metadata is read
Expand All @@ -62,5 +62,5 @@ following will only read documents and their collections and permissions:

The `copy` command supports many of the same options as the [import commands](import/common-import-features.md) for
writing documents. But similar to the output connection options, each option for controlling how documents are written
is prefixed with `output`. For example, to specify collections for the documents, `--outputCollections` is used instead
is prefixed with `output`. For example, to specify collections for the documents, `--output-collections` is used instead
of `--collections`.
8 changes: 4 additions & 4 deletions docs/export/export-archives.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,13 @@ The following options then control which documents are selected to be exported:

| Option | Description |
| --- |--- |
| --stringQuery | A string query utilizing MarkLogic's search grammar. |
| --string-query | A string query utilizing MarkLogic's search grammar. |
| --query | A structured, serialized CTS, or combined query expressed as JSON or XML. |
| --options | Name of a REST API search options document; typically used with a string query. |
| --collections | Comma-delimited sequence of collection names. |
| --directory | A database directory for constraining on URIs. |

You must specify at least one of `--stringQuery`, `--query`, `--collections`, or `--directory`. You may specify any
You must specify at least one of `--string-query`, `--query`, `--collections`, or `--directory`. You may specify any
combination of those options as well.

You must then use the `--path` option to specify a directory to write archive files to.
Expand All @@ -58,5 +58,5 @@ to each document before it is written to an archive. A transform is configured v
| Option | Description |
| --- | --- |
| --transform | Name of a MarkLogic REST transform to apply to the document before writing it. |
| --transformParams | Comma-delimited list of transform parameter names and values - e.g. param1,value1,param2,value2. |
| --transformParamsDelimiter | Delimiter for `--transformParams`; typically set when a value contains a comma. |
| --transform-params | Comma-delimited list of transform parameter names and values - e.g. param1,value1,param2,value2. |
| --transform-params-delimiter | Delimiter for `--transform-params`; typically set when a value contains a comma. |
20 changes: 10 additions & 10 deletions docs/export/export-documents.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,19 +19,19 @@ The `export-files` command is used to select documents in a MarkLogic database a
You must specify a `--path` option for where files should be written along with connection information for the
MarkLogic database you wish to write to:

./bin/nt export-files --path /path/to/files --connectionString "user:password@localhost:8000"
./bin/nt export-files --path /path/to/files --connection-string "user:password@localhost:8000"

The following options then control which documents are selected to be exported:

| Option | Description |
| --- |--- |
| --stringQuery | A string query utilizing MarkLogic's search grammar. |
| --string-query | A string query utilizing MarkLogic's search grammar. |
| --query | A structured, serialized CTS, or combined query expressed as JSON or XML. |
| --options | Name of a REST API search options document; typically used with a string query. |
| --collections | Comma-delimited sequence of collection names. |
| --directory | A database directory for constraining on URIs. |

You must specify at least one of `--stringQuery`, `--query`, `--collections`, or `--directory`. You may specify any
You must specify at least one of `--string-query`, `--query`, `--collections`, or `--directory`. You may specify any
combination of those options as well.

## Transforming document content
Expand All @@ -42,8 +42,8 @@ to each document before it is written to a file. A transform is configured via t
| Option | Description |
| --- | --- |
| --transform | Name of a MarkLogic REST transform to apply to the document before writing it. |
| --transformParams | Comma-delimited list of transform parameter names and values - e.g. param1,value1,param2,value2. |
| --transformParamsDelimiter | Delimiter for `--transformParams`; typically set when a value contains a comma. |
| --transform-params | Comma-delimited list of transform parameter names and values - e.g. param1,value1,param2,value2. |
| --transform-params-delimiter | Delimiter for `--transform-params`; typically set when a value contains a comma. |

## Compressing content

Expand All @@ -67,7 +67,7 @@ the following command below from the [Getting Started guide](getting-started.md)

```
rm export/*.zip
./bin/nt export-files --connectionString nt-user:password@localhost:8004 \
./bin/nt export-files --connection-string nt-user:password@localhost:8004 \
--collections employee \
--path export --compression zip
```
Expand All @@ -76,15 +76,15 @@ The `./export` directory will have 12 zip files in it. This count is due to how
which involves creating 4 partitions by default per forest in the MarkLogic database. The example application has 3
forests in its content database, and thus 12 partitions are created, resulting in 12 separate zip files.

You can use the `--partitionsPerForest` option to control how many partitions - and thus workers - read documents
You can use the `--partitions-per-forest` option to control how many partitions - and thus workers - read documents
from each forest in your database:

```
rm export/*.zip
./bin/nt export-files --connectionString nt-user:password@localhost:8004 \
./bin/nt export-files --connection-string nt-user:password@localhost:8004 \
--collections employee \
--path export --compression zip \
--partitionsPerForest 1
--partitions-per-forest 1
```

This approach will produce 3 zip files - one per forest.
Expand All @@ -94,7 +94,7 @@ writing data, regardless of how many were used to read the data:

```
rm export/*.zip
./bin/nt export-files --connectionString nt-user:password@localhost:8004 \
./bin/nt export-files --connection-string nt-user:password@localhost:8004 \
--collections employee \
--path export --compression zip \
--repartition 1
Expand Down
Loading

0 comments on commit 3e31c3e

Please sign in to comment.