home / github / issues

Menu
  • Search all tables
  • GraphQL API

issues: 1307523148

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
1307523148 I_kwDOAMm_X85N7zhM 6803 Passing a distributed.Future to the kwargs of apply_ufunc should resolve the future 33886395 closed 0     10 2022-07-18T07:31:28Z 2024-01-09T18:21:15Z 2023-12-19T05:40:20Z NONE      

What is your issue?

I am trying to scatter an large array and pass it as keyword argument to a function applied using apply_ufunc but that is currently not working. The same function works if providing the actual array, but if providing the Future linked to the scatter data the task fails.

Here is a minimal example to reproduce this issue

```python import dask.array as da import xarray as xr import numpy as np

data = xr.DataArray(data=da.random.random((15, 15, 20)), coords={'x': range(15), 'y': range(15), 'z': range(20)}, dims=('x', 'y', 'z'))

test = np.full((20,), 30) test_future = client.scatter(test, broadcast=True)

def _copy_test(d, test=None): return test

new_data_actual = xr.apply_ufunc( _copy_test, data, input_core_dims=[['z']], output_core_dims=[['new_z']], vectorize=True, dask='parallelized', output_dtypes="float64", kwargs={'test':test}, dask_gufunc_kwargs = {'output_sizes':{'new_z':20}} )

new_data_future = xr.apply_ufunc( _copy_test, data, input_core_dims=[['z']], output_core_dims=[['new_z']], vectorize=True, dask='parallelized', output_dtypes="float64", kwargs={'test':test_future}, dask_gufunc_kwargs = {'output_sizes':{'new_z':20}} )

data[0, 0].compute()

[0.3034994 , 0.08172002, 0.34731092, ...]

new_data_actual[0, 0].compute()

[30.0, 30.0, 30.0, ...]

new_data_future[0,0].compute()

KilledWorker

```

I tried different versions of this, going from explicitly calling test.result() to change the way the Future was passed, but nothing worked. I also tried to raise exceptions within the function and various way to print information, but that also did not work. This last issue makes me think that if passing a Future I actually don't get to the scope of that function

Am I trying to do something completely silly? or is this an unexpected behavior?

{
    "url": "https://api.github.com/repos/pydata/xarray/issues/6803/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed 13221727 issue

Links from other tables

  • 2 rows from issues_id in issues_labels
  • 8 rows from issue in issue_comments
Powered by Datasette · Queries took 3.92ms · About: xarray-datasette