You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+5-24Lines changed: 5 additions & 24 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,12 +1,12 @@
1
1
# dask-profiling
2
2
3
-
Profile dask [distributed](https://github.com/dask/distributed) clusters with [py-spy](https://github.com/benfred/py-spy) or [viztracer](https://github.com/gaogaotiantian/viztracer).
3
+
Profile dask [distributed](https://github.com/dask/distributed) clusters with [py-spy](https://github.com/benfred/py-spy).
4
4
5
5
```python
6
6
import dask
7
7
import distributed
8
8
9
-
from dask_profiling import pyspy_on_scheduler, viztrace_scheduler
# ^ Saves a Chrome trace to `viztracer.json` locally
27
23
):
28
24
df.set_index("id").mean().compute()
29
25
```
30
26
31
-
Using `pyspy_on_scheduler`or `viztrace_scheduler`attaches a profiler to the scheduler process, records a profile, and sends the file back to the client.
27
+
Using `pyspy_on_scheduler` attaches a profiler to the scheduler process, records a profile, and sends the file back to the client.
32
28
33
-
By default, py-spy profiles are recorded in [speedscope](https://www.speedscope.app/) format.[Viztracer profiles](https://viztracer.readthedocs.io/en/latest/basic_usage.html#display-report) are recorded in Chrome trace format, viewable with [Perfetto](https://ui.perfetto.dev/) or <chrome://tracing>.
29
+
By default, py-spy profiles are recorded in [speedscope](https://www.speedscope.app/) format.
34
30
35
-
`dask-profiling` (and, transitively, `py-spy`/`viztracer`) must be installed in the environment where the scheduler is running.
31
+
`dask-profiling` (and, transitively, `py-spy`) must be installed in the environment where the scheduler is running.
36
32
37
33
## Installation
38
34
@@ -56,19 +52,6 @@ In a Docker container, `dask-profiling` will "just work" for Docker/moby version
56
52
57
53
On Ubuntu-based containers, ptrace system calls are [further blocked](https://www.kernel.org/doc/Documentation/admin-guide/LSM/Yama.rst): processes are prohibited from ptracing each other even within the same UID. To work around this, `dask-profiling` automatically uses [`prctl(2)`](https://man7.org/linux/man-pages/man2/prctl.2.html) to mark the scheduler process as ptrace-able by itself and any child processes, then launches py-spy as a child process.
58
54
59
-
## Viztracer targeting with `trace_sparse`
60
-
61
-
`viztrace_scheduler` adds a non-standard option to VizTracer: the `trace_sparse` parameter. With this, you basically give a passlist of names (as strings) of functions to trace. The tracer will only run while those functions are active; at all other times, it'll be paused. Names are given in the same way you would to `unittest.mock.patch`.
62
-
63
-
`viztrace_scheduler` adds some magic so that if the function you want to trace happens to be registered as a comm handler on the Scheduler, everything will behave as you'd expect. (Since the handler functions are added into `scheduler.handlers`_before_`viztrace_scheduler` has the chance to patch them, those dicts need to be explicitly updated to hold the patched handlers as well. `viztrace_scheduler` takes care of this for you.)
64
-
65
-
Examples:
66
-
67
-
*`trace_sparse="distributed.Scheduler.handle_task_finished"` - Trace only the `handle_task_finished` function, and everything it calls while running
68
-
*`trace_sparse=["dask.highlevelgraph.HighLevelGraph.__dask_distributed_unpack__", "distributed.Scheduler.handle_task_finished"]` - Trace both high-level-graph unpacking and `handle_task_finished`
69
-
70
-
Note that this doesn't work very well on async functions. `distributed.comm.tcp.TCP.read` doesn't work that well, for example.
71
-
72
55
## Development
73
56
74
57
Install [Poetry](https://python-poetry.org/docs/#installation). To create a virtual environment, install dev dependencies, and install the package for local development:
@@ -83,5 +66,3 @@ There is one very very basic end-to-end test for py-spy. Running it requires Doc
83
66
$ pytest tests
84
67
```
85
68
and wait a long time for the image to build and run.
0 commit comments