html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/issues/1914#issuecomment-435717326,https://api.github.com/repos/pydata/xarray/issues/1914,435717326,MDEyOklzc3VlQ29tbWVudDQzNTcxNzMyNg==,10928117,2018-11-04T23:07:56Z,2018-11-04T23:07:56Z,NONE,"@jcmgray I had to miss your reply to this issue, I saw it just now. I love your code! I will definitely include xyzpy in my tools from now on ;-).","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,297560256 https://github.com/pydata/xarray/issues/1914#issuecomment-367677038,https://api.github.com/repos/pydata/xarray/issues/1914,367677038,MDEyOklzc3VlQ29tbWVudDM2NzY3NzAzOA==,10928117,2018-02-22T13:15:11Z,2018-02-22T13:15:11Z,NONE,"@shoyer Thanks for your suggestions and linking the other issue. I think this one can also be labelled as the ""usage question"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,297560256 https://github.com/pydata/xarray/issues/1914#issuecomment-366833780,https://api.github.com/repos/pydata/xarray/issues/1914,366833780,MDEyOklzc3VlQ29tbWVudDM2NjgzMzc4MA==,10928117,2018-02-20T00:27:36Z,2018-02-20T00:27:36Z,NONE,"After preparing list similar to ``[{'x': 0, 'y': 'a'}, {'x': 1, 'y': 'a'}, ...]`` interaction with cluster is quite efficient. One can easily pass such a thing to ``async_map`` of ``ipyparallel``. Thanks for your suggestion, I need to try few things. I also want to try to extend it to function that computes few different things that could be multi-valued, e.g. ```python def dummy(x, y): ds = xr.Dataset( {'out1': ('n', [1*x, 2*x, 3*x]), 'out2': ('m', [x, y])}, coords = {'x': x, 'y': y, 'n': range(3), 'm': range(2)} ) return ds ``` and then group together such outputs... Ok, I know. I go from simple problem to much more complicated one, but isn't it the case usually? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,297560256 https://github.com/pydata/xarray/issues/1914#issuecomment-366819497,https://api.github.com/repos/pydata/xarray/issues/1914,366819497,MDEyOklzc3VlQ29tbWVudDM2NjgxOTQ5Nw==,10928117,2018-02-19T22:40:17Z,2018-02-19T22:58:02Z,NONE,"For ""get done"" I had for example the following (similar to what I linked as my initial attempt) ```python coordinates = { 'x': np.linspace(-1, 1), 'y': np.linspace(0, 10), } constants = { 'a': 1, 'b': 5 } inps = [{**constants, **{k: v for k, v in zip(coordinates.keys(), x)}} for x in list(it.product(*coordinates.values()))] def f(x, y, a, b): """"""Some dummy function."""""" v = a * x**2 + b * y**2 return xr.DataArray(v, {'x': x, 'y': y, 'a': a, 'b': b}) # simulate computation on cluster values = list(map(lambda s: f(**s), inps)) # gather and unstack the inputs ds = xr.concat(values, dim='new', coords='all') ds = ds.set_index(new=list(set(ds.coords) - set(ds.dims))) ds = ds.unstack('new') ``` It is very close to what you suggest. My main question is if this can be done better. Mainly I am wondering if 1. Is there any built-in iterator over the Cartesian product of coordinates. If no, are there people that also think it would be useful? 2. Gathering together / unstacking of the data. My 3 line combo of ``concat``, ``set_index`` and ``unstack`` seems to do the trick but it seems a bit like over complication. Ideally I'd expect to have some mechanism that works similar to: ```python inputs = cartesian_product(coordinates) # list similar to ``inps`` above values = [function(inp) for inp in inputs] # or using ipypparallel map xarray_data = ... # some empty xarray object for inp, val in zip(inputs, values): xarray_data[inp] = val ``` I asked how to generate product of coordinates from xarray object because I was expecting that I can create ``xarray_data`` as an empty object with all coordinates set and then fill it. --- ### Added comment Having an empty, as filled with ``nan``s, object to start with would have this benefit that one could save partial results and have clean information what was already computed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,297560256 https://github.com/pydata/xarray/issues/1914#issuecomment-366740505,https://api.github.com/repos/pydata/xarray/issues/1914,366740505,MDEyOklzc3VlQ29tbWVudDM2Njc0MDUwNQ==,10928117,2018-02-19T16:20:15Z,2018-02-19T16:23:31Z,NONE,"Let me give a bit of a background what I would like to do: 1. Create an empty ``Dataset`` of coordinates I want to explore, i.e. two np.arrays ``x`` and ``y``, and two scalars ``a`` and ``b``. 2. Generate an list of the Cartesian product of all the coordinates, i.e. ``[ {'x': -1, 'y': 0, 'a': 1, 'b': 5}, ...]`` (data format doesn't really matter). 3. For each item of the iterator compute some function: ``f = f(x, y, a, b)``. In principle this function can be expensive to compute, therefore I'd compute it for each item of list from 2. separately on the cluster. 4. ""merge"" it all together into a single xarray object In principle ``f`` should be allowed to return e.g. ``np.array``. An related issue in [holoviews](https://github.com/ioam/holoviews/issues/2341#issuecomment-365925725) and the notebook with my initial [attempt](https://gitlab.kwant-project.org/qt/cookbook/blob/2f03d563342be6f5b85190faac24656457f1647f/xarray_holoviews_gridded.ipynb). In the linked notebook I managed to achieve the goal however without starting with an ``xarray`` object containing coordinates. Also combining the data seems a bit inefficient as it takes more time than generating it for a larger datasets.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,297560256