Merge pull request #202 from marklogic/feature/path-docs

Path option now properly supports wildcards

Author: rjrudin

docs/common-options.md

@@ -40,6 +40,33 @@ Flux beginning with that sequence of letters:
 If Flux cannot uniquely identify the command name, it will print an error and list the command names that match what
 you entered.
 
+## Reading options from a file
+
+Flux supports reading options from a file. In a text file, you can either add each option name and value to a separate
+line:
+
+```
+--host
+localhost
+--port
+8000
+etc...
+```
+
+Or you can put one or more options on the same line:
+
+```
+--host localhost
+--port 8000
+etc...
+```
+
+You then reference the file via the `@` symbol followed by a filename:
+
+    ./bin/flux import-files @my-options.txt
+
+You can reference multiple files this way and also include additional options on the command line.
+
 ## Connecting to MarkLogic
 
 Every command in Flux will need to connect to a MarkLogic database, either for reading data or writing data or both. 
@@ -49,6 +76,8 @@ Generally, you must include at least the following information for each command:
 - The port of a [MarkLogic REST API app server](https://docs.marklogic.com/guide/rest-dev) connected to the database you wish to interact with.
 - Authentication information.
 
+### Using a connection string
+
 For the common use case of using digest or basic authentication with a MarkLogic app server, you can use the 
 `--connection-string` option to specify the host, port, username, and password in a single concise option:
 
@@ -68,6 +97,22 @@ password of `sp@r:k`, you would use the following string:
 For other authentication mechanisms, you must use the `--host` and `--port` options to define the host and port for 
 your MarkLogic app server. 
 
+### Determining the connection type
+
+The `--connection-type` option determines which of the following approaches Flux uses for connecting to MarkLogic:
+
+- `GATEWAY` = the default value; Flux assumes that it cannot directly connect to each host in the MarkLogic cluster, most
+likely due to the value of `--host` or the host value found in `--connection-string` being that of a load balancer that
+controls access to MarkLogic.
+- `DIRECT` = Flux will try to connect to each host in the MarkLogic cluster. 
+
+If you do not have a load balancer in front of MarkLogic, and if Flux is able to connect to each host that hosts one
+or more forests for the database you wish to access, then you can set `--connection-type` to a value of `DIRECT`. This
+will often improve performance as Flux will be able to both connect to multiple hosts, thereby utilizing the app server
+threads available on each host, and also write directly to a forest on the host that it connects to. 
+
+### Connection options
+
 All available connection options are shown in the table below:
 
 | Option | Description | 
@@ -99,33 +144,6 @@ All available connection options are shown in the table below:
 | `--username` | Username when using `DIGEST` or `BASIC` authentication. |
 
 
-## Reading options from a file
-
-Flux supports reading options from a file. In a text file, you can either add each option name and value to a separate
-line:
-
-```
---host
-localhost
---port
-8000
-etc...
-```
-
-Or you can put one or more options on the same line:
-
-```
---host localhost
---port 8000
-etc...
-```
-
-You then reference the file via the `@` symbol followed by a filename:
-
-    ./bin/flux import-files @my-options.txt
-
-You can reference multiple files this way and also include additional options on the command line.
-
 ## Previewing data
 
 The `--preview` option works on every command in Flux. For example, given a set of Parquet files in a directory, 

docs/import/import-files/selecting-files.md

@@ -17,17 +17,32 @@ of specifying paths are described below.
 
 ## Specifying paths
 
-The `--path` option controls where files are read from. You can specify multiple occurrences of `--path`, each with a 
-different value, to import files from many sources in a single command invocation:
+Each command that imports from files requires the use of the `--path` option with at least one value - for example:
+
+    --path path/to/files
+
+You can include multiple values for the `--path` option, which can utilize both relative and absolute file paths:
+
+    --path relative/path/to/files /absolute/path/to/files
 
-    --path relative/path/to/files --path /absolute/path/to/files
 
 The value of the `--path` option can be any directory or file path. You can use wildcards in any part of the path. For
 example, the following, would select every file starting with `example` in any child directory of the root `/data`
 directory:
 
     --path /data/*/example*
 
+## Filtering files
+
+You can restrict which files are read from a directory by specifying a standard
+[glob expression](https://en.wikipedia.org/wiki/Glob_(programming)) via the `--filter` option:
+
+    --path /data/examples --filter "example*.json"
+
+Depending on your shell environment, you may need to include the value of `--filter` in double quotes as shown above to
+ensure that each asterisk is interpreted correctly. However, if you include `--filter` in an options file as 
+described in [Common Options](../../common-options.md), you do not need double quotes around the value. 
+
 ## Reading from S3
 
 Flux can read files from S3 via a path expression of the form `s3a://bucket-name/optional/path`.
@@ -56,10 +71,3 @@ By default, child directories of each directory specified by `--path` are includ
 option:
 
     --recursive-file-lookup false
-
-## Filtering files
-
-You can restrict which files are read from a directory by specifying a standard
-[glob expression](https://en.wikipedia.org/wiki/Glob_(programming)) via the `--filter` option:
-
-    --path /data/examples --filter some*.json

flux-cli/src/main/java/com/marklogic/flux/impl/SparkUtil.java

@@ -22,7 +22,6 @@ public static SparkSession buildSparkSession() {
     public static SparkSession buildSparkSession(String masterUrl) {
         return SparkSession.builder()
             .master(masterUrl)
-            // These can be overridden via the "-C" CLI option.
             .config("spark.ui.showConsoleProgress", "true")
             .config("spark.sql.session.timeZone", "UTC")
             .getOrCreate();

flux-cli/src/main/java/com/marklogic/flux/impl/importdata/ReadFilesParams.java

@@ -16,7 +16,15 @@
 public class ReadFilesParams<T extends ReadFilesOptions> implements ReadFilesOptions<T> {
 
     // "path" is the name so that picocli shows "--path <path>" instead of "--path <paths>".
-    @CommandLine.Option(required = true, names = "--path", description = "Specify one or more path expressions for selecting files to import.")
+    @CommandLine.Option(
+        required = true,
+        names = "--path",
+        description = "One or more path expressions for selecting files to import.",
+        // For a path like "/opt/data*.xml", the user's shell may resolve the asterisk and determine a list of all the
+        // files that match. Each file then becomes a value passed to this List. In order for that to work properly,
+        // we need arity set to allow for unlimited values. See https://picocli.info/#_arity for more information.
+        arity = "1..*"
+    )
     private List<String> path = new ArrayList<>();
 
     @CommandLine.Option(names = "--abort-on-read-failure", description = "Causes the command to abort when it fails to read a file.")

flux-cli/src/test/java/com/marklogic/flux/impl/importdata/ImportAggregateXmlFilesTest.java

@@ -32,7 +32,7 @@ void elementAndPathAreRequired() {
         assertStderrContains(() -> run(
             "import-aggregate-xml-files",
             "--connection-string", makeConnectionString()
-        ), "Missing required options: '--path <path>', '--element <element>'");
+        ), "Missing required options: '--element <element>', '--path <path>'");
     }
 
     @Test

flux-cli/src/test/java/com/marklogic/flux/impl/importdata/ImportFilesTest.java

@@ -42,6 +42,42 @@ void multiplePaths() {
         verifyDocsWereWritten(uris.length, uris);
     }
 
+    @Test
+    void pathAndFilterWithWildcardInOptionsFile() {
+        run(
+            "import-files",
+            "@src/test/resources/options-files/import-people-files.txt",
+            "--connection-string", makeConnectionString(),
+            "--permissions", DEFAULT_PERMISSIONS,
+            "--collections", "people-files",
+            "--uri-replace", ".*/xml-file,''"
+        );
+
+        assertCollectionSize(
+            "Verifying that when --filter is included in an options file, double quotes should not be used for " +
+                "the value of --filter.",
+            "people-files", 2
+        );
+    }
+
+    @Test
+    void pathWithArityofTwo() {
+        run(
+            "import-files",
+            "--path", "src/test/resources/mixed-files/hello.txt", "src/test/resources/mixed-files/hello.xml",
+            "--connection-string", makeConnectionString(),
+            "--permissions", DEFAULT_PERMISSIONS,
+            "--collections", "people-files",
+            "--uri-replace", ".*/xml-file,''"
+        );
+
+        assertCollectionSize(
+            "Verifying that --path accepts multiple values, which is needed in order for an asterisk to work " +
+                "in some shell environments.",
+            "people-files", 2
+        );
+    }
+
     @Test
     void withUsernameAndPasswordAndAuthType() {
         run(

flux-cli/src/test/resources/options-files/import-people-files.txt

@@ -0,0 +1,2 @@
+--path src/test/resources/xml-file/temp
+--filter people*.xml