Skip to content

Commit d2116a9

Browse files
committed
update doc
1 parent 057fdfa commit d2116a9

1 file changed

Lines changed: 11 additions & 0 deletions

File tree

docs/Aurora/Spark job submit tool usage on aurora.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -238,6 +238,10 @@ From here you can change Spark and OAP MLlib configurations. For more Spark conf
238238
| spark.driver.extraClassPath | Extra classpath entries to prepend to the classpath of the driver. OAP MLlib Jar path is added here. |
239239
| spark.eventLog.enabled | Whether to log Spark events. |
240240
| spark.eventLog.dir | Base directory in which Spark events are logged. |
241+
| spark.shuffle.manager | Set the shuffle manager |
242+
| spark.shuffle.daos.pool.uuid | UUID of the DAOS pool to use for shuffle data. |
243+
| spark.shuffle.daos.container.uuid | UUID of the DAOS container to use for shuffle data. |
244+
241245

242246
## Example Configuration
243247
Here is an example that prioritizes making the dense K-Means example most efficient.
@@ -275,3 +279,10 @@ spark.executor.resource.gpu.amount 1 # each executor has 1 GPU
275279
spark.executor.instances 36 # 36 (12 x 3) executors in total
276280
spark.task.resource.gpu.amount 0.125 # Each task uses 1/8 of a GPU. This value should be determined based on your **target task concurrency**. For example, if each node has 12 GPUs and you want 96 concurrent GPU tasks per node, then spark.task.resource.gpu.amount = 12 / 96 = 0.125. In this case, 8 tasks share each GPU, allowing better parallelism for lightweight stages like preprocessing.
277281
```
282+
283+
By default, DAOS shuffle is enabled in env_local.sh. If you want to disable DAOS shuffle, you can remove the following lines.
284+
```shell
285+
spark.shuffle.manager org.apache.spark.shuffle.daos.DaosShuffleManager
286+
spark.shuffle.daos.pool.uuid Intel
287+
spark.shuffle.daos.container.uuid spark_shuffle
288+
```

0 commit comments

Comments
 (0)