Setting remote values

[r-barnes]

Based on the following tests:

@everywhere using DistributedArrays
ras    = [@spawnat p rand(30,30) for p in workers()[1:4]]
ras    = reshape(ras,(2,2))
D      = DArray(ras)
D[1,1] = 5
m1     = @spawnat 2 D[:L][2,2] = 5 #Works
m2     = @spawnat 2 D[2,2] = 5     #Doesn't work
fetch(m1)
fetch(m2)

It looks as though DArrays offers view-only access to non-local portions of the array and write access only to local portions. Is that correct?

Any thoughts on how to set remote values?

[vchuravy]

Yes see #155 for the reasoning behind it.

I = (2,2)
chidx = locate(D, I...)
idxs = d.indices[chidx...]
localidx = ntuple(i -> (I[i] - first(idxs[i]) + 1), ndims(d))
pid = d.pids[chidx...]

remote_call(pid, D, localidx, data) do D, I, data
     localpart(D)[localidx...] = data
end

Which if your data is not a scalar should be reasonable efficient.
This is based of

DistributedArrays.jl/src/darray.jl

Line 620 in e6915d2

function getindex_tuple(d::DArray{T}, I::Tuple{Vararg{Int}}) where T

and might need some adjustment to deal with slices.

DistributedArrays.jl/src/darray.jl

Line 347 in e6915d2

function makelocal(A::DArray{<:Any, <:Any, AT}, I::Vararg{Any, N}) where {N, AT}

does that and so we should probably factor those parts out.

[r-barnes]

Got it. I'm planning to use DArray with SparseArrays to form a large binary matrix representing the frontier of a breadth-first search, so I can send the index updates en masse after each round provided I know where to send them. I can figure that out based on the snippet you provided.

[vchuravy]

We now have DistributedArrays.allowscalar(false) so I think we could add setindex! in general and optimise the batch case.

[r-barnes]

Per my other post, I'm worried about communication delays, so I'm thinking about bulk exchange of halos now.