Merge remote-tracking branch 'origin/candidate-10.0.x' into candidate-10.2.x

Signed-off-by: Gordon Smith <[email protected]>

# Conflicts:
#	commons-hpcc/pom.xml
#	dfsclient/pom.xml
#	pom.xml
#	spark-hpcc/pom.xml
#	wsclient/pom.xml

Author: GordonSmith

spark-hpcc/Examples/PySparkExample.ipynb

@@ -119,12 +119,12 @@
     "- **recordSamplingSeed**: *Optional* The seed that controls the random generation used for sampling. The same seed against the same HPCC cluster and HPCC platform version should result in the same sampling.\n",
     "- **projectList**: *Optional* The columns that should be read from the HPCC Systems dataset.\n",
     "- **useTLK** *Optional* Defaults to false, determines whether or not the TLK (Top Level Key) should be used when reading index files.\n",
-    "- **fileParts** *Optional* List of file parts to read; supports a comma-separated list of files, file part ranges, or a combination of both.\n",,
+    "- **fileParts** *Optional* List of file parts to read; supports a comma-separated list of files, file part ranges, or a combination of both.\n",
     "- **stringProcessing** *Optional* Comma separated list of processing rules to apply to strings available rules: [NONE, TRIM, TRIM_FIXED, EMPTY_TO_NULL]. Default behavior is NONE. TRIM will apply left and right trim all strings, TRIM_FIXED will apply left trim and right trim for only fixed length strings, and EMPTY_TO_NULL will convert empty strings to a null value.\n",
-    "- **unsignedEightToDecimal** *Optional* By default Unsigned8 values in HPCC are read into a Java long, which will result in some values overflowing and becoming negative. If the Unsigned8 value is used a a unique identifier this is acceptable, but for cases where the numeric value is needed the Unsigned8 can be read into a Decimal value which supports the entire Unsigned8 range.\n"
+    "- **unsignedEightToDecimal** *Optional* By default Unsigned8 values in HPCC are read into a Java long, which will result in some values overflowing and becoming negative. If the Unsigned8 value is used a a unique identifier this is acceptable, but for cases where the numeric value is needed the Unsigned8 can be read into a Decimal value which supports the entire Unsigned8 range.\n",
     "\n",
     "---\n",
-    "## Troubleshooting:\n",
+    "## Troubleshooting Reads:\n",
     "- **Empty dataset**: This typically indicates that an error occured on one or more of the worker nodes during the read process, the worker logs should contain more information about the particular failure.\n"
    ]
   },
@@ -357,6 +357,28 @@
     "    for f in futures:\n",
     "        f.result()   # Wait for completion"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a4149a90",
+   "metadata": {},
+   "source": [
+    "\n",
+    "# General Troubleshooting Tips\n",
+    "\n",
+    "---\n",
+    "\n",
+    "- **Issues with individual file parts:** Potential problems such as slow transfer speeds for a particular file part can be diagnosed by using the fileParts option during the read process to selectively read individual file parts and compare transfer speeds.\n",
+    "\n",
+    "```python\n",
+    "        readDf = spark.read.load(format=\"hpcc\",\n",
+    "                                host=\"http://127.0.0.1:8010\",\n",
+    "                                cluster=\"mythor\",\n",
+    "                                path=\"/spark/test/dataset\",\n",
+    "                                fileParts=\"13\")\n",
+    "```\n",
+    "- **Network Congestion:** Reading data between distributed clusters (Spark and HPCC) opens many concurrent TCP connections, which can overwhelm network links or gateways. On the receiving cluster, this appears as slow transfer speeds and file access timeouts; on the sending cluster, it shows up as large TCP send queues and dropped packets. To mitigate this, limit the number of concurrent Spark read tasks, either by reducing executor cores per job or by limiting the number of workers."
+   ]
   }
  ],
  "metadata": {