Brtdku

I have the following function that calculate the propagation of a laser beam in a cavity. It depends on many parameters that are stored in a dict called core_data, which is a basic parameter set.

def propagate(N, core_data, **ddata):

    cd = copy.deepcopy(core_data)  # use initial configuration

    cd.update(ddata)  # update with data I want to change

    cavity = get_new_cavity(cd)  # get new cavity object

    P = 

    for i in range(N):

        cavity.evolve(1)

        P.append(cavity.get_power())

    return P

If I want to change a parameter and see its effect on the laser, I can just call the function like, for instance

P0 = propagate(1000, core_data, L1=1.2, M5=17)

This works very well.

Now, I would write a function that passes this function to a ProcessPoolExecutor, with the values of **ddata being iterated over using the same key. It should work, for instance, like this (simpler example):

propagate_parallel(1000, core_data,

                   L1=np.linspace(1, 2, 2),

                   M5=np.linspace(16, 17, 2))

And should then do this in parallel:

propagate(1000, core_data, L1=1, M5=16)

propagate(1000, core_data, L1=1, M5=17)

propagate(1000, core_data, L1=2, M5=16)

propagate(1000, core_data, L1=2, M5=17)

Something like this works for my case:

xrng = np.linspace(110e-30, 150e-30, Nx)

yrng = np.linspace(6.6e-9, 6.7e-9, Ny)



futures = 

with confu.ProcessPoolExecutor(max_workers=Ncores) as pool:

    for y, x in it.product(yrng, xrng):

        futures.append(pool.submit(propagate, RTs=1000,

                                   core_data=core_data,

                                   gdd_dm=x, dwl_filt=y))

The problem is that this is not flexible and I cannot get this into a nice function, as written above. It should be a function that can be called like this to reproduce the code from above:

propagate_parallel(1000, core_data, gdd_dm=xrng, dwl_filt=yrng)

How would I pass the keys from **ddata with the iterated values of that corresponding key?

FYI, I used:

import numpy as np

import concurrent.futures as confu

import itertools as it

You are looking for iterating over the cartesian product.

Here is a way to iterate over a cartesian.

from itertools import product

import numpy as np



L1=np.linspace(1, 2, 2)

M5=np.linspace(16, 17, 2)

dconf = dict(data=5)

size = L1.size

loop_size = size**2



def propagate(N, data, modifiers):

    data.update(modifiers)

    out = 

    for i in range(N):

        out.append('%s : %s : %s : %s'%(i, *data.values()))

    return out



mod = (dict(L1=i, M5=j) for i, j in product(L1, M5))

m = map(propagate, np.arange(2, 2+loop_size), (dconf,)*loop_size, mod)



for outer in m:

    for inner in outer:

        print(inner)

This you can adapt to your code, and if you really need to go parallell (with all that this means in terms of info split between cores) maybe take a look into Dask.

Hope this is enough to get you going.

edit:
your question is quite hard to actually pinpoint.
Is your question really how to just achieve the simple "function call"?
I suppose one answer is just to make a wrap function, something like...

def propagate(N, data, modifiers):

    ...



def call_propagate(N, data, L1_, M5_):

    mod = ...

    m = map(...

    return m



for outer in call_propagate(1000, dconf, L1, M5)

    for inner in outer:

        print(inner)

I think I was somehow blocked... I kept thinking how to keep a variable name (for instannce L1) and pass this as a variable to another function.

@ahead87: Already your first sentence unblocked me and I realized that **data can be handled simply via a dictionary. So, in the end, I simply needed to transform the input dict into a list of dicts for the next function, like so (with some irrelevant parts being snipped):

def propagate_parallel(RTs, cav_data, **ddata):

    keys = list(ddata.keys())

    values = list(ddata.values())

    futures = 

    res = 

    with confu.ProcessPoolExecutor(max_workers=32) as pool:

        for i in it.product(*values):

            futures.append(pool.submit(propagate, RTs=RTs,

                                       cav_data=cav_data,

                                       **dict(zip(keys, list(i)))))

    for fut in futures:

        res.append(fut)

    return res

In the end, I think I finally understand **kwargs, and that it can be handles as a dict. Thank you!