2. Examples¶
2.1. Block, Block¶
Assume we have a process grid with 2 rows and 1 column, and we have a 2x10
array a distributed over it. Let a be a two-dimensional array with a
block distribution in both dimensions. Note that since the proc_grid_size
of the first dimension is 1, it is essentially undistributed. Because of
this, having a cyclic dist_type for this dimension would be equivalent.
In process 0:
>>> distbuffer = a0.__distarray__()
>>> distbuffer.keys()
['__version__', 'buffer', 'dim_data']
>>> distbuffer['__version__']
'0.10.0'
>>> distbuffer['buffer']
array([ 0.2, 0.6, 0.9, 0.6, 0.8, 0.4, 0.2, 0.2, 0.3, 0.5])
>>> distbuffer['dim_data']
({'size': 2,
'dist_type': 'b',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'start': 0,
'stop': 1},
{'size': 10,
'dist_type': 'b',
'proc_grid_rank': 0,
'proc_grid_size': 1,
'start': 0,
'stop': 10})
In process 1:
>>> distbuffer = a1.__distarray__()
>>> distbuffer.keys()
['__version__', 'buffer', 'dim_data']
>>> distbuffer['__version__']
'0.10.0'
>>> distbuffer['buffer']
array([ 0.9, 0.2, 1. , 0.4, 0.5, 0. , 0.6, 0.8, 0.6, 1. ])
>>> distbuffer['dim_data']
({'size': 2,
'dist_type': 'b',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'start': 1,
'stop': 2},
{'size': 10,
'dist_type': 'b',
'proc_grid_rank': 0,
'proc_grid_size': 1,
'start': 0,
'stop': 10})
2.2. Block with padding¶
Assume we have a process grid with 2 processes, and we have an 18-element array
a distributed over it. Let a be a one-dimensional array with a
block-padded distribution for its 0th (and only) dimension.
Since the 'padding' for each process is (1, 1), the local array on each
process has one element of padding on the left and one element of padding on
the right. Since each of these processes is at one edge of the process grid
(and the array has no 'periodic' dimensions), the “outside” element on each
local array is an example of “boundary padding”, and the “inside” element on
each local array is an example of “communication padding”. Note that the
'size' of the distributed array is not equal to the combined buffer sizes
of a0 and a1 , since communication padding is not counted toward
'size' (though the boundary padding is).
For this example, the distribution of global indices (with ‘B’ representing boundary padding and ‘C’ representing communication padding) is as follows:
Process 0: B 1 2 3 4 5 6 7 8 C
Process 1: C 9 10 11 12 13 14 15 16 B
The ‘B’ element on process 0 occupies global index 0, and the ‘B’ element on
process 1 occupies global index 17. Each ‘B’ element counts towards the
array’s 'size'. The communication elements on each process overlap with a
data element on the other process to indicate which data elements these
communication elements are meant to communicate with.
The protocol data structure on each process is as follows.
In process 0:
>>> distbuffer = a0.__distarray__()
>>> distbuffer.keys()
['__version__', 'buffer', 'dim_data']
>>> distbuffer['__version__']
'0.10.0'
>>> distbuffer['buffer']
array([ 0.2, 0.6, 0.9, 0.6, 0.8, 0.4, 0.2, 0.2, 0.3, 0.9])
>>> distbuffer['dim_data']
({'size': 18,
'dist_type': 'b',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'start': 0,
'stop': 10,
'padding': (1, 1)})
In process 1:
>>> distbuffer = a1.__distarray__()
>>> distbuffer.keys()
['__version__', 'buffer', 'dim_data']
>>> distbuffer['__version__']
'0.10.0'
>>> distbuffer['buffer']
array([ 0.3, 0.9, 0.2, 1. , 0.4, 0.5, 0. , 0.6, 0.8, 0.6])
>>> distbuffer['dim_data']
({'size': 18,
'dist_type': 'b',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'start': 8,
'stop': 18,
'padding': (1, 1)})
2.3. Unstructured¶
Assume we have a process grid with 3 rows, and we have a size 30 array a
distributed over it. Let a be a one-dimensional unstructured array with 7
elements on process 0, 3 elements on process 1, and 20 elements on process 2.
On all processes:
>>> distbuffer = local_array.__distarray__()
>>> distbuffer.keys()
['__version__', 'buffer', 'dim_data']
>>> distbuffer['__version__']
'0.10.0'
>>> len(distbuffer['dim_data']) == 1 # one dimension only
True
In process 0:
>>> distbuffer['buffer']
array([0.7, 0.5, 0.9, 0.2, 0.7, 0.0, 0.5])
>>> distbuffer['dim_data']
({'size': 30,
'dist_type': 'u',
'proc_grid_rank': 0,
'proc_grid_size': 3,
'indices': array([19, 1, 0, 12, 2, 15, 4])},)
In process 1:
>>> distbuffer['buffer']
array([0.1, 0.5, 0.9])
>>> distbuffer['dim_data']
({'size': 30,
'dist_type': 'u',
'proc_grid_rank': 1,
'proc_grid_size': 3,
'indices': array([6, 13, 3])},)
In process 2:
>>> distbuffer['buffer']
array([ 0.1, 0.8, 0.4, 0.8, 0.2, 0.4, 0.4, 0.3, 0.5, 0.7,
0.4, 0.7, 0.6, 0.2, 0.8, 0.5, 0.3, 0.8, 0.4, 0.2])
>>> distbuffer['dim_data']
({'size': 30,
'dist_type': 'u',
'proc_grid_rank': 2,
'proc_grid_size': 3,
'indices': array([10, 25, 5, 21, 7, 18, 11, 26, 29, 24, 23, 28, 14,
20, 9, 16, 27, 8, 17, 22])},)
2.4. Block, Block¶
A (5 X 9) array, with a Block, Block (‘b’ X ‘b’) distribution over a (3 X 1) process grid.
The full (undistributed) array:
>>> full_array
array([[ 0., 1., 2., 3., 4., 5., 6., 7., 8.],
[ 9., 10., 11., 12., 13., 14., 15., 16., 17.],
[ 18., 19., 20., 21., 22., 23., 24., 25., 26.],
[ 27., 28., 29., 30., 31., 32., 33., 34., 35.],
[ 36., 37., 38., 39., 40., 41., 42., 43., 44.]])
In all processes, we have:
>>> distbuffer = local_array.__distarray__()
>>> distbuffer.keys()
['buffer', 'dim_data', '__version__']
>>> distbuffer['__version__']
'0.10.0'
The local arrays, on each separate engine:
In process (0, 0): >>> distbuffer['buffer']
array([[ 0., 1., 2., 3., 4., 5., 6., 7., 8.],
[ 9., 10., 11., 12., 13., 14., 15., 16., 17.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 0,
'proc_grid_size': 3,
'size': 5,
'start': 0,
'stop': 2},
{'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 0,
'proc_grid_size': 1,
'size': 9,
'start': 0,
'stop': 9})
|
In process (1, 0): >>> distbuffer['buffer']
array([[ 18., 19., 20., 21., 22., 23., 24., 25., 26.],
[ 27., 28., 29., 30., 31., 32., 33., 34., 35.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 1,
'proc_grid_size': 3,
'size': 5,
'start': 2,
'stop': 4},
{'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 0,
'proc_grid_size': 1,
'size': 9,
'start': 0,
'stop': 9})
|
In process (2, 0): >>> distbuffer['buffer']
array([[ 36., 37., 38., 39., 40., 41., 42., 43., 44.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 2,
'proc_grid_size': 3,
'size': 5,
'start': 4,
'stop': 5},
{'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 0,
'proc_grid_size': 1,
'size': 9,
'start': 0,
'stop': 9})
|
2.5. Block, Block¶
A (5 X 9) array, with a Block, Block (‘b’ X ‘b’) distribution over a (1 X 3) process grid.
The full (undistributed) array:
>>> full_array
array([[ 0., 1., 2., 3., 4., 5., 6., 7., 8.],
[ 9., 10., 11., 12., 13., 14., 15., 16., 17.],
[ 18., 19., 20., 21., 22., 23., 24., 25., 26.],
[ 27., 28., 29., 30., 31., 32., 33., 34., 35.],
[ 36., 37., 38., 39., 40., 41., 42., 43., 44.]])
In all processes, we have:
>>> distbuffer = local_array.__distarray__()
>>> distbuffer.keys()
['buffer', 'dim_data', '__version__']
>>> distbuffer['__version__']
'0.10.0'
The local arrays, on each separate engine:
In process (0, 0): >>> distbuffer['buffer']
array([[ 0., 1., 2.],
[ 9., 10., 11.],
[ 18., 19., 20.],
[ 27., 28., 29.],
[ 36., 37., 38.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 0,
'proc_grid_size': 1,
'size': 5,
'start': 0,
'stop': 5},
{'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 0,
'proc_grid_size': 3,
'size': 9,
'start': 0,
'stop': 3})
|
In process (0, 1): >>> distbuffer['buffer']
array([[ 3., 4., 5.],
[ 12., 13., 14.],
[ 21., 22., 23.],
[ 30., 31., 32.],
[ 39., 40., 41.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 0,
'proc_grid_size': 1,
'size': 5,
'start': 0,
'stop': 5},
{'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 1,
'proc_grid_size': 3,
'size': 9,
'start': 3,
'stop': 6})
|
In process (0, 2): >>> distbuffer['buffer']
array([[ 6., 7., 8.],
[ 15., 16., 17.],
[ 24., 25., 26.],
[ 33., 34., 35.],
[ 42., 43., 44.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 0,
'proc_grid_size': 1,
'size': 5,
'start': 0,
'stop': 5},
{'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 2,
'proc_grid_size': 3,
'size': 9,
'start': 6,
'stop': 9})
|
2.6. Block, Block¶
A (5 X 9) array, with a Block, Block (‘b’ X ‘b’) distribution over a (2 X 2) process grid.
The full (undistributed) array:
>>> full_array
array([[ 0., 1., 2., 3., 4., 5., 6., 7., 8.],
[ 9., 10., 11., 12., 13., 14., 15., 16., 17.],
[ 18., 19., 20., 21., 22., 23., 24., 25., 26.],
[ 27., 28., 29., 30., 31., 32., 33., 34., 35.],
[ 36., 37., 38., 39., 40., 41., 42., 43., 44.]])
In all processes, we have:
>>> distbuffer = local_array.__distarray__()
>>> distbuffer.keys()
['buffer', 'dim_data', '__version__']
>>> distbuffer['__version__']
'0.10.0'
The local arrays, on each separate engine:
In process (0, 0): >>> distbuffer['buffer']
array([[ 0., 1., 2., 3., 4.],
[ 9., 10., 11., 12., 13.],
[ 18., 19., 20., 21., 22.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 5,
'start': 0,
'stop': 3},
{'dist_type': 'b',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 9,
'start': 0,
'stop': 5})
|
In process (0, 1): >>> distbuffer['buffer']
array([[ 5., 6., 7., 8.],
[ 14., 15., 16., 17.],
[ 23., 24., 25., 26.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 5,
'start': 0,
'stop': 3},
{'dist_type': 'b',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 9,
'start': 5,
'stop': 9})
|
In process (1, 0): >>> distbuffer['buffer']
array([[ 27., 28., 29., 30., 31.],
[ 36., 37., 38., 39., 40.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 5,
'start': 3,
'stop': 5},
{'dist_type': 'b',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 9,
'start': 0,
'stop': 5})
|
In process (1, 1): >>> distbuffer['buffer']
array([[ 32., 33., 34., 35.],
[ 41., 42., 43., 44.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 5,
'start': 3,
'stop': 5},
{'dist_type': 'b',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 9,
'start': 5,
'stop': 9})
|
2.7. Block, Cyclic¶
A (5 X 9) array, with a Block, Cyclic (‘b’ X ‘c’) distribution over a (2 X 2) process grid.
The full (undistributed) array:
>>> full_array
array([[ 0., 1., 2., 3., 4., 5., 6., 7., 8.],
[ 9., 10., 11., 12., 13., 14., 15., 16., 17.],
[ 18., 19., 20., 21., 22., 23., 24., 25., 26.],
[ 27., 28., 29., 30., 31., 32., 33., 34., 35.],
[ 36., 37., 38., 39., 40., 41., 42., 43., 44.]])
In all processes, we have:
>>> distbuffer = local_array.__distarray__()
>>> distbuffer.keys()
['buffer', 'dim_data', '__version__']
>>> distbuffer['__version__']
'0.10.0'
The local arrays, on each separate engine:
In process (0, 0): >>> distbuffer['buffer']
array([[ 0., 2., 4., 6., 8.],
[ 9., 11., 13., 15., 17.],
[ 18., 20., 22., 24., 26.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 5,
'start': 0,
'stop': 3},
{'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 9,
'start': 0})
|
In process (0, 1): >>> distbuffer['buffer']
array([[ 1., 3., 5., 7.],
[ 10., 12., 14., 16.],
[ 19., 21., 23., 25.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 5,
'start': 0,
'stop': 3},
{'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 9,
'start': 1})
|
In process (1, 0): >>> distbuffer['buffer']
array([[ 27., 29., 31., 33., 35.],
[ 36., 38., 40., 42., 44.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 5,
'start': 3,
'stop': 5},
{'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 9,
'start': 0})
|
In process (1, 1): >>> distbuffer['buffer']
array([[ 28., 30., 32., 34.],
[ 37., 39., 41., 43.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 5,
'start': 3,
'stop': 5},
{'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 9,
'start': 1})
|
2.8. Cyclic, Cyclic¶
A (5 X 9) array, with a Cyclic, Cyclic (‘c’ X ‘c’) distribution over a (2 X 2) process grid.
The full (undistributed) array:
>>> full_array
array([[ 0., 1., 2., 3., 4., 5., 6., 7., 8.],
[ 9., 10., 11., 12., 13., 14., 15., 16., 17.],
[ 18., 19., 20., 21., 22., 23., 24., 25., 26.],
[ 27., 28., 29., 30., 31., 32., 33., 34., 35.],
[ 36., 37., 38., 39., 40., 41., 42., 43., 44.]])
In all processes, we have:
>>> distbuffer = local_array.__distarray__()
>>> distbuffer.keys()
['buffer', 'dim_data', '__version__']
>>> distbuffer['__version__']
'0.10.0'
The local arrays, on each separate engine:
In process (0, 0): >>> distbuffer['buffer']
array([[ 0., 2., 4., 6., 8.],
[ 18., 20., 22., 24., 26.],
[ 36., 38., 40., 42., 44.]])
>>> distbuffer['dim_data']
({'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 5,
'start': 0},
{'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 9,
'start': 0})
|
In process (0, 1): >>> distbuffer['buffer']
array([[ 1., 3., 5., 7.],
[ 19., 21., 23., 25.],
[ 37., 39., 41., 43.]])
>>> distbuffer['dim_data']
({'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 5,
'start': 0},
{'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 9,
'start': 1})
|
In process (1, 0): >>> distbuffer['buffer']
array([[ 9., 11., 13., 15., 17.],
[ 27., 29., 31., 33., 35.]])
>>> distbuffer['dim_data']
({'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 5,
'start': 1},
{'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 9,
'start': 0})
|
In process (1, 1): >>> distbuffer['buffer']
array([[ 10., 12., 14., 16.],
[ 28., 30., 32., 34.]])
>>> distbuffer['dim_data']
({'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 5,
'start': 1},
{'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 9,
'start': 1})
|
2.9. Irregular-Block, Irregular-Block¶
A (5 X 9) array, with an Irregular-Block, Irregular-Block (‘b’ X ‘b’) distribution over a (2 X 2) process grid.
The full (undistributed) array:
>>> full_array
array([[ 0., 1., 2., 3., 4., 5., 6., 7., 8.],
[ 9., 10., 11., 12., 13., 14., 15., 16., 17.],
[ 18., 19., 20., 21., 22., 23., 24., 25., 26.],
[ 27., 28., 29., 30., 31., 32., 33., 34., 35.],
[ 36., 37., 38., 39., 40., 41., 42., 43., 44.]])
In all processes, we have:
>>> distbuffer = local_array.__distarray__()
>>> distbuffer.keys()
['buffer', 'dim_data', '__version__']
>>> distbuffer['__version__']
'0.10.0'
The local arrays, on each separate engine:
In process (0, 0): >>> distbuffer['buffer']
array([[ 0., 1.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 5,
'start': 0,
'stop': 1},
{'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 9,
'start': 0,
'stop': 2})
|
In process (0, 1): >>> distbuffer['buffer']
array([[ 2., 3., 4., 5., 6., 7., 8.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 5,
'start': 0,
'stop': 1},
{'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 9,
'start': 2,
'stop': 9})
|
In process (1, 0): >>> distbuffer['buffer']
array([[ 9., 10.],
[ 18., 19.],
[ 27., 28.],
[ 36., 37.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 5,
'start': 1,
'stop': 5},
{'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 9,
'start': 0,
'stop': 2})
|
In process (1, 1): >>> distbuffer['buffer']
array([[ 11., 12., 13., 14., 15., 16., 17.],
[ 20., 21., 22., 23., 24., 25., 26.],
[ 29., 30., 31., 32., 33., 34., 35.],
[ 38., 39., 40., 41., 42., 43., 44.]])
>>> distbuffer['dim_data']
({'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 5,
'start': 1,
'stop': 5},
{'dist_type': 'b',
'padding': [0, 0],
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 9,
'start': 2,
'stop': 9})
|
2.10. Block-Cyclic, Block-Cyclic¶
A (5 X 9) array, with a Block-Cyclic, Block-Cyclic (‘c’ X ‘c’) distribution over a (2 X 2) process grid.
The full (undistributed) array:
>>> full_array
array([[ 0., 1., 2., 3., 4., 5., 6., 7., 8.],
[ 9., 10., 11., 12., 13., 14., 15., 16., 17.],
[ 18., 19., 20., 21., 22., 23., 24., 25., 26.],
[ 27., 28., 29., 30., 31., 32., 33., 34., 35.],
[ 36., 37., 38., 39., 40., 41., 42., 43., 44.]])
In all processes, we have:
>>> distbuffer = local_array.__distarray__()
>>> distbuffer.keys()
['buffer', 'dim_data', '__version__']
>>> distbuffer['__version__']
'0.10.0'
The local arrays, on each separate engine:
In process (0, 0): >>> distbuffer['buffer']
array([[ 0., 1., 4., 5., 8.],
[ 9., 10., 13., 14., 17.],
[ 36., 37., 40., 41., 44.]])
>>> distbuffer['dim_data']
({'block_size': 2,
'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 5,
'start': 0},
{'block_size': 2,
'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 9,
'start': 0})
|
In process (0, 1): >>> distbuffer['buffer']
array([[ 2., 3., 6., 7.],
[ 11., 12., 15., 16.],
[ 38., 39., 42., 43.]])
>>> distbuffer['dim_data']
({'block_size': 2,
'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 5,
'start': 0},
{'block_size': 2,
'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 9,
'start': 2})
|
In process (1, 0): >>> distbuffer['buffer']
array([[ 18., 19., 22., 23., 26.],
[ 27., 28., 31., 32., 35.]])
>>> distbuffer['dim_data']
({'block_size': 2,
'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 5,
'start': 2},
{'block_size': 2,
'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 9,
'start': 0})
|
In process (1, 1): >>> distbuffer['buffer']
array([[ 20., 21., 24., 25.],
[ 29., 30., 33., 34.]])
>>> distbuffer['dim_data']
({'block_size': 2,
'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 5,
'start': 2},
{'block_size': 2,
'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 9,
'start': 2})
|
2.11. Unstructured, Unstructured¶
A (5 X 9) array, with an Unstructured, Unstructured (‘u’ X ‘u’) distribution over a (2 X 2) process grid.
The full (undistributed) array:
>>> full_array
array([[ 0., 1., 2., 3., 4., 5., 6., 7., 8.],
[ 9., 10., 11., 12., 13., 14., 15., 16., 17.],
[ 18., 19., 20., 21., 22., 23., 24., 25., 26.],
[ 27., 28., 29., 30., 31., 32., 33., 34., 35.],
[ 36., 37., 38., 39., 40., 41., 42., 43., 44.]])
In all processes, we have:
>>> distbuffer = local_array.__distarray__()
>>> distbuffer.keys()
['buffer', 'dim_data', '__version__']
>>> distbuffer['__version__']
'0.10.0'
The local arrays, on each separate engine:
In process (0, 0): >>> distbuffer['buffer']
array([[ 29., 30., 34., 28.],
[ 2., 3., 7., 1.]])
>>> distbuffer['dim_data']
({'dist_type': 'u',
'indices': array([3, 0]),
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 5},
{'dist_type': 'u',
'indices': array([2, 3, 7, 1]),
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 9})
|
In process (0, 1): >>> distbuffer['buffer']
array([[ 33., 32., 35., 27., 31.],
[ 6., 5., 8., 0., 4.]])
>>> distbuffer['dim_data']
({'dist_type': 'u',
'indices': array([3, 0]),
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 5},
{'dist_type': 'u',
'indices': array([6, 5, 8, 0, 4]),
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 9})
|
In process (1, 0): >>> distbuffer['buffer']
array([[ 38., 39., 43., 37.],
[ 20., 21., 25., 19.],
[ 11., 12., 16., 10.]])
>>> distbuffer['dim_data']
({'dist_type': 'u',
'indices': array([4, 2, 1]),
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 5},
{'dist_type': 'u',
'indices': array([2, 3, 7, 1]),
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 9})
|
In process (1, 1): >>> distbuffer['buffer']
array([[ 42., 41., 44., 36., 40.],
[ 24., 23., 26., 18., 22.],
[ 15., 14., 17., 9., 13.]])
>>> distbuffer['dim_data']
({'dist_type': 'u',
'indices': array([4, 2, 1]),
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 5},
{'dist_type': 'u',
'indices': array([6, 5, 8, 0, 4]),
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 9})
|
2.12. Cyclic, Block, Cyclic¶
A (5 X 9 X 3) array, with a Cyclic, Block, Cyclic (‘c’ X ‘b’ X ‘c’) distribution over a (2 X 2 X 2) process grid.
The full (undistributed) array:
>>> full_array
array([[[ 0., 1., 2.],
[ 3., 4., 5.],
[ 6., 7., 8.],
[ 9., 10., 11.],
[ 12., 13., 14.],
[ 15., 16., 17.],
[ 18., 19., 20.],
[ 21., 22., 23.],
[ 24., 25., 26.]],
[[ 27., 28., 29.],
[ 30., 31., 32.],
[ 33., 34., 35.],
[ 36., 37., 38.],
[ 39., 40., 41.],
[ 42., 43., 44.],
[ 45., 46., 47.],
[ 48., 49., 50.],
[ 51., 52., 53.]],
[[ 54., 55., 56.],
[ 57., 58., 59.],
[ 60., 61., 62.],
[ 63., 64., 65.],
[ 66., 67., 68.],
[ 69., 70., 71.],
[ 72., 73., 74.],
[ 75., 76., 77.],
[ 78., 79., 80.]],
[[ 81., 82., 83.],
[ 84., 85., 86.],
[ 87., 88., 89.],
[ 90., 91., 92.],
[ 93., 94., 95.],
[ 96., 97., 98.],
[ 99., 100., 101.],
[ 102., 103., 104.],
[ 105., 106., 107.]],
[[ 108., 109., 110.],
[ 111., 112., 113.],
[ 114., 115., 116.],
[ 117., 118., 119.],
[ 120., 121., 122.],
[ 123., 124., 125.],
[ 126., 127., 128.],
[ 129., 130., 131.],
[ 132., 133., 134.]]])
In all processes, we have:
>>> distbuffer = local_array.__distarray__()
>>> distbuffer.keys()
['buffer', 'dim_data', '__version__']
>>> distbuffer['__version__']
'0.10.0'
The local arrays, on each separate engine:
In process (0, 0, 0): >>> distbuffer['buffer']
array([[[ 0., 2.],
[ 3., 5.],
[ 6., 8.],
[ 9., 11.],
[ 12., 14.]],
[[ 54., 56.],
[ 57., 59.],
[ 60., 62.],
[ 63., 65.],
[ 66., 68.]],
[[ 108., 110.],
[ 111., 113.],
[ 114., 116.],
[ 117., 119.],
[ 120., 122.]]])
>>> distbuffer['dim_data']
({'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 5,
'start': 0},
{'dist_type': 'b',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 9,
'start': 0,
'stop': 5},
{'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 3,
'start': 0})
|
In process (0, 0, 1): >>> distbuffer['buffer']
array([[[ 1.],
[ 4.],
[ 7.],
[ 10.],
[ 13.]],
[[ 55.],
[ 58.],
[ 61.],
[ 64.],
[ 67.]],
[[ 109.],
[ 112.],
[ 115.],
[ 118.],
[ 121.]]])
>>> distbuffer['dim_data']
({'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 5,
'start': 0},
{'dist_type': 'b',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 9,
'start': 0,
'stop': 5},
{'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 3,
'start': 1})
|
In process (0, 1, 0): >>> distbuffer['buffer']
array([[[ 15., 17.],
[ 18., 20.],
[ 21., 23.],
[ 24., 26.]],
[[ 69., 71.],
[ 72., 74.],
[ 75., 77.],
[ 78., 80.]],
[[ 123., 125.],
[ 126., 128.],
[ 129., 131.],
[ 132., 134.]]])
>>> distbuffer['dim_data']
({'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 5,
'start': 0},
{'dist_type': 'b',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 9,
'start': 5,
'stop': 9},
{'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 3,
'start': 0})
|
In process (0, 1, 1): >>> distbuffer['buffer']
array([[[ 16.],
[ 19.],
[ 22.],
[ 25.]],
[[ 70.],
[ 73.],
[ 76.],
[ 79.]],
[[ 124.],
[ 127.],
[ 130.],
[ 133.]]])
>>> distbuffer['dim_data']
({'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 5,
'start': 0},
{'dist_type': 'b',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 9,
'start': 5,
'stop': 9},
{'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 3,
'start': 1})
|
In process (1, 0, 0): >>> distbuffer['buffer']
array([[[ 27., 29.],
[ 30., 32.],
[ 33., 35.],
[ 36., 38.],
[ 39., 41.]],
[[ 81., 83.],
[ 84., 86.],
[ 87., 89.],
[ 90., 92.],
[ 93., 95.]]])
>>> distbuffer['dim_data']
({'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 5,
'start': 1},
{'dist_type': 'b',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 9,
'start': 0,
'stop': 5},
{'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 3,
'start': 0})
|
In process (1, 0, 1): >>> distbuffer['buffer']
array([[[ 28.],
[ 31.],
[ 34.],
[ 37.],
[ 40.]],
[[ 82.],
[ 85.],
[ 88.],
[ 91.],
[ 94.]]])
>>> distbuffer['dim_data']
({'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 5,
'start': 1},
{'dist_type': 'b',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 9,
'start': 0,
'stop': 5},
{'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 3,
'start': 1})
|
In process (1, 1, 0): >>> distbuffer['buffer']
array([[[ 42., 44.],
[ 45., 47.],
[ 48., 50.],
[ 51., 53.]],
[[ 96., 98.],
[ 99., 101.],
[ 102., 104.],
[ 105., 107.]]])
>>> distbuffer['dim_data']
({'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 5,
'start': 1},
{'dist_type': 'b',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 9,
'start': 5,
'stop': 9},
{'dist_type': 'c',
'proc_grid_rank': 0,
'proc_grid_size': 2,
'size': 3,
'start': 0})
|
In process (1, 1, 1): >>> distbuffer['buffer']
array([[[ 43.],
[ 46.],
[ 49.],
[ 52.]],
[[ 97.],
[ 100.],
[ 103.],
[ 106.]]])
>>> distbuffer['dim_data']
({'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 5,
'start': 1},
{'dist_type': 'b',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 9,
'start': 5,
'stop': 9},
{'dist_type': 'c',
'proc_grid_rank': 1,
'proc_grid_size': 2,
'size': 3,
'start': 1})
|