Distributed computing using SharedArray

Introduction

SharedArray is a special type of array that allows multiple processes on the same device to have direct access to the same data in memory, without having to copy it between processes.

Unlike regular arrays, which are copied when passed to another process., SharedArray uses system shared memory. This means that all the participating workflows see the same piece of RAM, so any changes made by one process are immediately visible to everyone else.

This paper presents a minimal but illustrative example: distributed summation of elements of a random number matrix by two independent processes. This model problem illustrates the basic operations.:addprocs, @spawnat, wait, SharedArray. It also demonstrates the potential performance benefits of switching from a sequential computing scheme to a parallel one.

Joining libraries

Connecting the necessary libraries:

Distributed– for organizing multiprocessing;
TickTock– to measure the execution time.

for lib in ["Distributed", "TickTock"]
    try
        eval(Meta.parse("using $lib"))
    catch ex
        import Pkg
        Pkg.add(lib)
        eval(Meta.parse("using $lib"))
    end
end

Preparation of two work processes

Let's run two additional workflows with the command addprocs(2) and make sure that they are successfully registered by displaying a list of active workers via workers(). We get a vector, the identifiers of which will be used further for addressing tasks.

addprocs(2)
workers()

2-element Vector{Int64}:
 2
 3

Creating SharedArray objects

Creating two data structures in shared memory:

the first – SA, a two-dimensional 6x2 shared array filled with random numbers; it plays the role of source data.
the second – SR, a common vector of two elements initialized with zeros; it is designed to accumulate results calculated by different processes.

Using SharedArray eliminates the need to copy data between processes.

using SharedArrays
SA = convert(SharedArray{Float64, 2}, rand(6,2))

6×2 SharedMatrix{Float64}:
 0.783205  0.327205
 0.432006  0.914733
 0.390884  0.326474
 0.305342  0.319739
 0.242857  0.0980725
 0.59598   0.954856

SR = SharedArray{Float64,1}(2)

2-element SharedVector{Float64}:
 0.0
 0.0

Distributed Computing Tests

Let's check the basic mechanism of distributed computing: using a macro @spawnat we will send process 2 the task to calculate the sum of the elements of the first column. SA and write the result in SR[1].

@spawnat 2 SR[1] = sum(SA[:,1])

Future(2, 1, 18, ReentrantLock(), nothing)

display(SR)

2-element SharedVector{Float64}:
 0.0
 0.0

We will send a similar task for the second column to process 3. Pay attention to an important detail.: @spawnat returns an object Future and it does not block the main process, calculations are performed asynchronously. Therefore, an immediate check SR it can also show zero values — this is the expected behavior, illustrating the asynchronous nature of distributed tasks.

@spawnat 3 SR[2] = sum(SA[:,2])

Future(3, 1, 19, ReentrantLock(), nothing)

Performance comparison of distributed and single-process computing

Now let's conduct a more difficult experiment with the number of iterations N = 10⁷. First, we will perform calculations sequentially in the main process, then distributed on workflows 2 and 3, waiting for their completion through wait.([f1, f2]). In both cases, we will measure the time using tick() and tock(). A comparison will show that the distributed version is faster: with this amount of work, the overhead of parallel organization is repeatedly offset by the benefits of running two cycles on different cores simultaneously.

N = 10^7

1000000

# one process
SR[1:2] .= 0.0
display(SR)
tick()
SR[1] = begin x = 0.0; for i in 1:N x += sum(SA[:,1]); end; x; end;
SR[2] = begin x = 0.0; for i in 1:N x += sum(SA[:,2]); end; x; end;
display(SR)
tock()

2-element SharedVector{Float64}:
 0.0
 0.0

[ Info:  started timer at: 2026-10-15T05:17:04.778

2-element SharedVector{Float64}:
 2.750272925045633e6
 2.941078833171551e6

[ Info:          6.015979337s: 6 seconds, 15 milliseconds

# parallel computing
SR[1:2] .= 0.0
display(SR)
tick()
f1 = @spawnat 2 SR[1] = begin x = 0.0; for i in 1:N x += sum(SA[:,1]); end; x; end;
f2 = @spawnat 3 SR[2] = begin x = 0.0; for i in 1:N x += sum(SA[:,2]); end; x; end;
@time wait.([f1, f2])
display(SR)
tock()

2-element SharedVector{Float64}:
 0.0
 0.0

[ Info:  started timer at: 2026-10-15T05:17:02.136

  1.802403 seconds (200.93 k allocations: 10.613 MiB, 10.99% compilation time)

2-element SharedVector{Float64}:
 2.750272925045633e6
 2.941078833171551e6

[ Info:          2.172321257s: 2 seconds, 172 milliseconds

Cleaning up resources

We will complete the work processes with the command rmprocs(workers()) and make sure that only the main process remains in the system. Team procs() returns the vector 1. The correct release of resources is an important part of working with distributed systems, since incomplete worker workflows can take up memory and processor time.

rmprocs(workers())
procs()

1-element Vector{Int64}:
 1

Conclusion

This example covers all the main stages of working with distributed computing in Engee: launching workflows, creating shared arrays, asynchronous assignment of tasks, and evaluating acceleration. The bundle of the module Distributed And like SharedArray provides parallel data processing without explicit messaging, and the jobs returned are @spawnat, require synchronization via wait where the order of calculations is important. Even on a trivial summation problem, an almost threefold acceleration was achieved (≈2.17 s versus ≈6.02 s).

The presented scheme underlies a wide class of tasks: parallel aggregation of large data arrays, independent model runs in Monte Carlo methods, hyperparameter enumeration and ensemble training in machine learning, multivariate engineering analysis, as well as solving partial differential equations and genomic analysis in scientific computing.

Thus, the considered methods form a basic toolkit that is directly generalized to more complex scenarios where efficient use of multicore systems and reduction of counting time are required.