Pull Demo¶

This notebook performs a very simple dot product of the same two random (square) matrices on all ipengines. It then gathers the results from all ipengines and confirms that all results are the same. This mostly demos how the push, execute, and pull pipeline works in the IPython Parallel client.

1. Allocate random matrices locally¶

In [113]:

import numpy as np

In [114]:

N = 2000

In [115]:

a = np.random.randn(N, N)

In [116]:

b = np.random.randn(N, N)

2. Compute their dot product locally¶

In [117]:

c = np.dot(a,b)

3. Initialize the IPython Parallel Client¶

In [118]:

from IPython.parallel import Client

In [119]:

cl = Client()

In [120]:

cl.ids

Out[120]:

[0, 1, 2]

4. Fetch a 'view' from the IPython parallel client¶

In [121]:

v = cl[:]

5. Import NumPy on all IPEngines¶

In [ ]:

v.execute('import numpy as np', block=True)

6. Push both random matrices to all IPEngines¶

In [122]:

v.push(dict(a=a, b=b), block=True)

Out[122]:

[None, None, None]

7. Compute dot product on all IPEngines¶

NOTE: We add 'print c' to the execute statement below in order to demo the display_outputs() function which shows the stdout, if any, of all remote engines as a result of executing the command. In general you do not need to print variables in order to interact with them.

In [123]:

r = v.execute('c = np.dot(a,b); print c', block=True)

In [124]:

r.display_outputs()

[stdout:0] [[ -4.67202217 104.39252912 -78.37783997 ..., -29.30709274 6.66403977 29.92526411] [ 39.22517891 73.37959579 -140.66938625 ..., -30.40772693 9.21601775 29.80644042] [ 31.60094895 -26.90207753 42.07903369 ..., 6.16564342 -40.9135798 -57.82278587] ..., [ -25.57059809 -63.64479316 22.76541772 ..., 1.68819694 -12.9543937 -61.07450388] [ 15.3646617 57.48457183 -34.15856279 ..., 7.34530517 -16.26106107 -17.58811896] [ -13.26202943 75.42065094 74.86911918 ..., -0.55862744 6.72326022 10.38620045]] [stdout:1] [[ -4.67202217 104.39252912 -78.37783997 ..., -29.30709274 6.66403977 29.92526411] [ 39.22517891 73.37959579 -140.66938625 ..., -30.40772693 9.21601775 29.80644042] [ 31.60094895 -26.90207753 42.07903369 ..., 6.16564342 -40.9135798 -57.82278587] ..., [ -25.57059809 -63.64479316 22.76541772 ..., 1.68819694 -12.9543937 -61.07450388] [ 15.3646617 57.48457183 -34.15856279 ..., 7.34530517 -16.26106107 -17.58811896] [ -13.26202943 75.42065094 74.86911918 ..., -0.55862744 6.72326022 10.38620045]] [stdout:2] [[ -4.67202217 104.39252912 -78.37783997 ..., -29.30709274 6.66403977 29.92526411] [ 39.22517891 73.37959579 -140.66938625 ..., -30.40772693 9.21601775 29.80644042] [ 31.60094895 -26.90207753 42.07903369 ..., 6.16564342 -40.9135798 -57.82278587] ..., [ -25.57059809 -63.64479316 22.76541772 ..., 1.68819694 -12.9543937 -61.07450388] [ 15.3646617 57.48457183 -34.15856279 ..., 7.34530517 -16.26106107 -17.58811896] [ -13.26202943 75.42065094 74.86911918 ..., -0.55862744 6.72326022 10.38620045]]

8. Pull the results from all IPEngines¶

In [125]:

dot = v.pull('c', block=True)

In [126]:

print dot

[array([[ -4.67202217, 104.39252912, -78.37783997, ..., -29.30709274, 6.66403977, 29.92526411], [ 39.22517891, 73.37959579, -140.66938625, ..., -30.40772693, 9.21601775, 29.80644042], [ 31.60094895, -26.90207753, 42.07903369, ..., 6.16564342, -40.9135798 , -57.82278587], ..., [ -25.57059809, -63.64479316, 22.76541772, ..., 1.68819694, -12.9543937 , -61.07450388], [ 15.3646617 , 57.48457183, -34.15856279, ..., 7.34530517, -16.26106107, -17.58811896], [ -13.26202943, 75.42065094, 74.86911918, ..., -0.55862744, 6.72326022, 10.38620045]]), array([[ -4.67202217, 104.39252912, -78.37783997, ..., -29.30709274, 6.66403977, 29.92526411], [ 39.22517891, 73.37959579, -140.66938625, ..., -30.40772693, 9.21601775, 29.80644042], [ 31.60094895, -26.90207753, 42.07903369, ..., 6.16564342, -40.9135798 , -57.82278587], ..., [ -25.57059809, -63.64479316, 22.76541772, ..., 1.68819694, -12.9543937 , -61.07450388], [ 15.3646617 , 57.48457183, -34.15856279, ..., 7.34530517, -16.26106107, -17.58811896], [ -13.26202943, 75.42065094, 74.86911918, ..., -0.55862744, 6.72326022, 10.38620045]]), array([[ -4.67202217, 104.39252912, -78.37783997, ..., -29.30709274, 6.66403977, 29.92526411], [ 39.22517891, 73.37959579, -140.66938625, ..., -30.40772693, 9.21601775, 29.80644042], [ 31.60094895, -26.90207753, 42.07903369, ..., 6.16564342, -40.9135798 , -57.82278587], ..., [ -25.57059809, -63.64479316, 22.76541772, ..., 1.68819694, -12.9543937 , -61.07450388], [ 15.3646617 , 57.48457183, -34.15856279, ..., 7.34530517, -16.26106107, -17.58811896], [ -13.26202943, 75.42065094, 74.86911918, ..., -0.55862744, 6.72326022, 10.38620045]])]

9. Check remote results against local results¶

In [127]:

assert np.allclose(c, dot[0])

In [128]:

assert np.allclose(dot[0], dot[1])

In [129]:

assert np.allclose(dot[1], dot[2])