Skip to content

Commit 84a2be7

Browse files
MasonProtterMasonProtter
authored
Try to improve UX with the variable boxing errors (#142)
* add `@localize` macro to easily debox variables * add a markdown formated error hint for `BoxedVariableError` * update boxing docpage * markdown weakdep * fix typo * fix error checking test * map -> tmap * relax Markdown compat * update changelog * better lower bound on Markdown * add `@localize` to docs * exempt Markdown from downgrade CI * better pretty printing * bump version * regenerate md file
1 parent 67a19c7 commit 84a2be7

File tree

11 files changed

+334
-35
lines changed

11 files changed

+334
-35
lines changed

Diff for: ‎.github/workflows/downgrade_CI.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,6 @@ jobs:
2323
version: ${{ matrix.version }}
2424
- uses: cjdoris/julia-downgrade-compat-action@v1
2525
with:
26-
skip: Pkg,TOML,Test
26+
skip: Pkg,TOML,Test,Markdown
2727
- uses: julia-actions/julia-buildpkg@v1
2828
- uses: julia-actions/julia-runtest@v1

Diff for: ‎CHANGELOG.md

+6
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,11 @@
11
OhMyThreads.jl Changelog
22
=========================
33

4+
Version 0.8.1
5+
------------
6+
- ![Feature][badge-feature] Added a `@localize` macro which turns `@localize x y expr` into `let x=x, y=y; expr end` ([#142][gh-pr-142])
7+
- ![INFO][badge-info] The error messafe for captured variables now has a longer error hint that displays when the `Markdown` package is loaded (e.g. in the REPL.) ([#142][gh-pr-142])
8+
49
Version 0.8.0
510
-------------
611
- ![BREAKING][badge-breaking] We now detect and throw errors if an `OhMyThreads` parallel function is passed a closure containing a `Box`ed variable. This behaviour can be disabled with the new `@allow_boxed_captures` macro, and re-enabled with `@disallow_boxed_captures`. ([#141][gh-pr-141])
@@ -146,3 +151,4 @@ Version 0.2.0
146151
[gh-pr-121]: https://github.com/JuliaFolds2/OhMyThreads.jl/pull/121
147152
[gh-pr-135]: https://github.com/JuliaFolds2/OhMyThreads.jl/pull/135
148153
[gh-pr-141]: https://github.com/JuliaFolds2/OhMyThreads.jl/pull/141
154+
[gh-pr-142]: https://github.com/JuliaFolds2/OhMyThreads.jl/pull/142

Diff for: ‎Project.toml

+9-2
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
name = "OhMyThreads"
22
uuid = "67456a42-1dca-4109-a031-0a68de7e3ad5"
33
authors = ["Carsten Bauer <[email protected]>", "Mason Protter <[email protected]>"]
4-
version = "0.8.0"
4+
version = "0.8.1"
55

66
[deps]
77
BangBang = "198e06fe-97b7-11e9-32a5-e1d131e6ad66"
@@ -10,13 +10,20 @@ ScopedValues = "7e506255-f358-4e82-b7e4-beb19740aa63"
1010
StableTasks = "91464d47-22a1-43fe-8b7f-2d57ee82463f"
1111
TaskLocalValues = "ed4db957-447d-4319-bfb6-7fa9ae7ecf34"
1212

13+
[weakdeps]
14+
Markdown = "d6f4376e-aef5-505a-96c1-9c027394607a"
15+
16+
[extensions]
17+
MarkdownExt = "Markdown"
18+
1319
[compat]
1420
Aqua = "0.8"
1521
BangBang = "0.3.40, 0.4"
1622
ChunkSplitters = "3"
23+
Markdown = "1"
24+
ScopedValues = "1.3"
1725
StableTasks = "0.1.5"
1826
TaskLocalValues = "0.1"
19-
ScopedValues = "1.3"
2027
Test = "1"
2128
julia = "1.10"
2229

Diff for: ‎docs/src/literate/boxing/boxing.jl

+67-2
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ All multithreading in julia is built around the idea of passing around
55
and executing functions, but often these functions "enclose" data from
66
an outer local scope, making them what's called a "closure".
77
8+
# ## Boxed variables causing race conditions
9+
810
Julia allows functions which capture variables to re-bind those variables
911
to different values, but doing so can cause subtle race conditions in
1012
multithreaded code.
@@ -44,11 +46,26 @@ try
4446
out
4547
end
4648
catch e;
47-
println(e.msg) # show that error
49+
## Show the error
50+
Base.showerror(stdout, e)
4851
end
4952

5053
#====================================
51-
If you really desire to bypass this behaviour, you can use the
54+
In this case, we could fix the race conditon by marking `A` as local:
55+
====================================#
56+
57+
let
58+
out = tmap(1:10) do i
59+
local A = i # Note the use of `local`
60+
sleep(1/100)
61+
A
62+
end
63+
A = 1
64+
out
65+
end
66+
67+
#====================================
68+
If you really desire to bypass this error, you can use the
5269
`@allow_boxed_captures` macro
5370
====================================#
5471

@@ -61,3 +78,51 @@ If you really desire to bypass this behaviour, you can use the
6178
A = 1
6279
out
6380
end
81+
82+
#====================================
83+
## Non-race conditon boxed variables
84+
85+
Any re-binding of captured variables can cause boxing, even when that boxing isn't strictly necessary, like the following example where we do not rebind `A` in the loop:
86+
====================================#
87+
try
88+
let A = 1
89+
if rand(Bool)
90+
## Rebind A, it's now boxed!
91+
A = 2
92+
end
93+
@tasks for i in 1:2
94+
@show A
95+
end
96+
end
97+
catch e;
98+
println("Yup, that errored!")
99+
end
100+
#====================================
101+
This comes down to how julia parses and lowers code. To avoid this, you can use an inner `let` block to localize `A` to the loop:
102+
====================================#
103+
104+
let A = 1
105+
if rand(Bool)
106+
A = 2
107+
end
108+
let A = A # This stops A from being boxed!
109+
@tasks for i in 1:2
110+
@show A
111+
end
112+
end
113+
end
114+
115+
#====================================
116+
OhMyThreads provides a macro `@localize` to automate this process:
117+
====================================#
118+
119+
let A = 1
120+
if rand(Bool)
121+
A = 2
122+
end
123+
## This stops A from being boxed!
124+
@localize A @tasks for i in 1:2
125+
@show A
126+
end
127+
end
128+

Diff for: ‎docs/src/literate/boxing/boxing.md

+150-24
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ All multithreading in julia is built around the idea of passing around
88
and executing functions, but often these functions "enclose" data from
99
an outer local scope, making them what's called a "closure".
1010

11+
# ## Boxed variables causing race conditions
12+
1113
Julia allows functions which capture variables to re-bind those variables
1214
to different values, but doing so can cause subtle race conditions in
1315
multithreaded code.
@@ -29,15 +31,15 @@ end
2931
````
3032
10-element Vector{Int64}:
3133
6
32-
2
33-
3
34-
2
35-
3
36-
2
37-
3
38-
2
39-
3
40-
4
34+
6
35+
6
36+
6
37+
6
38+
6
39+
6
40+
6
41+
6
42+
6
4143
````
4244

4345
You may have expected that to return `[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]`,
@@ -62,20 +64,78 @@ try
6264
out
6365
end
6466
catch e;
65-
println(e.msg) # show that error
67+
# Show the error
68+
Base.showerror(stdout, e)
6669
end
6770
````
6871

6972
````
70-
Attempted to capture and modify outer local variable(s) A, which would be not only slow, but could also cause a race condition. Consider marking these variables as local inside their respective closure, or redesigning your code to avoid the race condition.
73+
Attempted to capture and modify outer local variable: A
7174
72-
If these variables are inside a @one_by_one or @only_one block, consider using a mutable Ref instead of re-binding the variable.
75+
See https://juliafolds2.github.io/OhMyThreads.jl/stable/literate/boxing/boxing/ for a fuller explanation.
7376
74-
This error can be bypassed with the @allow_boxed_captures macro.
77+
Hint
78+
----
7579
80+
Capturing boxed variables can be not only slow, but also cause surprising
81+
and incorrect results.
82+
83+
• If you meant for these variables to be local to each loop
84+
iteration and not depend on a variable from an outer scope, you
85+
should mark them as local inside the closure.
86+
87+
• If you meant to reference a variable from the outer scope, but do
88+
not want access to it to be boxed, you can wrap uses of it in a
89+
let block, like e.g.
90+
91+
function foo(x, N)
92+
rand(Bool) && x = 1 # This rebinding of x causes it to be boxed ...
93+
let x = x # ... Unless we localize it here with the let block
94+
@tasks for i in 1:N
95+
f(x)
96+
end
97+
end
98+
end
99+
100+
• OhMyThreads.jl provides a @localize macro that automates the above
101+
let block, i.e. @localize x f(x) is the same as let x=x; f(x) end
102+
103+
• If these variables are being re-bound inside a @one_by_one or
104+
@only_one block, consider using a mutable Ref instead of
105+
re-binding the variable.
106+
107+
This error can be bypassed with the @allow_boxed_captures macro.
76108
````
77109

78-
If you really desire to bypass this behaviour, you can use the
110+
In this case, we could fix the race conditon by marking `A` as local:
111+
112+
````julia
113+
let
114+
out = tmap(1:10) do i
115+
local A = i # Note the use of `local`
116+
sleep(1/100)
117+
A
118+
end
119+
A = 1
120+
out
121+
end
122+
````
123+
124+
````
125+
10-element Vector{Int64}:
126+
1
127+
2
128+
3
129+
4
130+
5
131+
6
132+
7
133+
8
134+
9
135+
10
136+
````
137+
138+
If you really desire to bypass this error, you can use the
79139
`@allow_boxed_captures` macro
80140

81141
````julia
@@ -92,16 +152,82 @@ end
92152

93153
````
94154
10-element Vector{Int64}:
95-
7
96-
6
97-
7
98-
6
99-
7
100-
6
101-
7
102-
6
103-
7
104-
7
155+
10
156+
10
157+
10
158+
10
159+
10
160+
10
161+
10
162+
10
163+
10
164+
10
165+
````
166+
167+
## Non-race conditon boxed variables
168+
169+
Any re-binding of captured variables can cause boxing, even when that boxing isn't strictly necessary, like the following example where we do not rebind `A` in the loop:
170+
171+
````julia
172+
try
173+
let A = 1
174+
if rand(Bool)
175+
# Rebind A, it's now boxed!
176+
A = 2
177+
end
178+
@tasks for i in 1:2
179+
@show A
180+
end
181+
end
182+
catch e;
183+
println("Yup, that errored!")
184+
end
185+
````
186+
187+
````
188+
Yup, that errored!
189+
190+
````
191+
192+
This comes down to how julia parses and lowers code. To avoid this, you can use an inner `let` block to localize `A` to the loop:
193+
194+
````julia
195+
let A = 1
196+
if rand(Bool)
197+
A = 2
198+
end
199+
let A = A # This stops A from being boxed!
200+
@tasks for i in 1:2
201+
@show A
202+
end
203+
end
204+
end
205+
````
206+
207+
````
208+
A = 1
209+
A = 1
210+
211+
````
212+
213+
OhMyThreads provides a macro `@localize` to automate this process:
214+
215+
````julia
216+
let A = 1
217+
if rand(Bool)
218+
A = 2
219+
end
220+
# This stops A from being boxed!
221+
@localize A @tasks for i in 1:2
222+
@show A
223+
end
224+
end
225+
````
226+
227+
````
228+
A = 1
229+
A = 1
230+
105231
````
106232

107233
---

Diff for: ‎docs/src/refs/api.md

+1
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ CollapsedDocStrings = true
1515
@one_by_one
1616
@allow_boxed_captures
1717
@disallow_boxed_captures
18+
@localize
1819
```
1920

2021
### Functions

0 commit comments

Comments
 (0)