id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,active_lock_reason,draft,pull_request,body,reactions,performed_via_github_app,state_reason,repo,type 374025325,MDU6SXNzdWUzNzQwMjUzMjU=,2511,Array indexing with dask arrays,13190237,closed,0,,,20,2018-10-25T16:13:11Z,2023-03-15T02:48:00Z,2023-03-15T02:48:00Z,CONTRIBUTOR,,,,"#### Code example ```python da = xr.DataArray(np.ones((10, 10))).chunk(2) indc = xr.DataArray(np.random.randint(0, 9, 10)).chunk(2) # This fails: da[{'dim_1' : indc}].values ``` #### Problem description Indexing with chunked arrays fails, whereas it's fine with ""normal"" arrays. In case the indices are the result of a lazy calculation, I would like to continue lazily. #### Expected Output I would expect an output just like in the ""un-chunked"" case: ``` da[{'dim_1' : indc.compute()}].values # Returns: array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]) ``` #### Output of ``xr.show_versions()``
INSTALLED VERSIONS ------------------ commit: None python: 3.7.0.final.0 python-bits: 64 OS: Linux OS-release: 4.18.14-arch1-1-ARCH machine: x86_64 processor: byteorder: little LC_ALL: None LANG: de_DE.utf8 LOCALE: de_DE.UTF-8 xarray: 0.10.9 pandas: 0.23.4 numpy: 1.15.2 scipy: 1.1.0 netCDF4: None h5netcdf: 0.6.2 h5py: 2.8.0 Nio: None zarr: None cftime: None PseudonetCDF: None rasterio: None iris: None bottleneck: 1.2.1 cyordereddict: None dask: 0.19.4 distributed: None matplotlib: 2.2.3 cartopy: 0.16.0 seaborn: None setuptools: 40.4.3 pip: 18.0 conda: None pytest: 3.8.2 IPython: 6.5.0 sphinx: 1.8.0
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2511/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 608974755,MDU6SXNzdWU2MDg5NzQ3NTU=,4015,apply_ufunc gives wrong dtype with dask=parallelized and vectorized=True,13190237,closed,0,,,2,2020-04-29T11:17:48Z,2020-08-19T06:57:56Z,2020-08-19T06:57:56Z,CONTRIBUTOR,,,,"Applying a function to a data array with `dtype = complex` returns one with `dtype = float`. It seems to work before commit 17b70caa6eafa062fd31e7f39334b3de922ff422. #### MCVE Code Sample ```python import numpy as np import xarray as xr def func(x): return np.sum(x ** 2) da = xr.DataArray(np.arange(2*3*4).reshape(2,3,4)) da = da + 1j * da da = da.chunk(dict(dim_1=1)) da2 = xr.apply_ufunc( func, da, vectorize=True, dask=""parallelized"", output_dtypes=[da.dtype], ) assert da2.dtype == da.dtype, ""wrong dtype"" ``` #### Expected Output `da` and `da2` should both have the same `dtype=complex`. #### Problem Description To me it seems to me that the kwarg `meta=None` somehow causes dask to allocate a float array and ignore the dtype kwargs (which seems to be carried through correctly down to `dask.array.blockwise.blockwise()`) . I'm not familiar with the apply_ufing and the dask code, so I can't tell on which end the bug sits. #### Versions
Output of xr.show_versions() INSTALLED VERSIONS ------------------ commit: None python: 3.8.2 (default, Apr 8 2020, 14:31:25) [GCC 9.3.0] python-bits: 64 OS: Linux OS-release: 5.6.5-arch3-1 machine: x86_64 processor: byteorder: little LC_ALL: None LANG: de_DE.utf8 LOCALE: de_DE.UTF-8 libhdf5: 1.10.5 libnetcdf: 4.7.3 xarray: 0.15.2.dev47+g33a66d63 pandas: 1.0.3 numpy: 1.18.3 scipy: 1.4.1 netCDF4: 1.5.3 pydap: None h5netcdf: 0.7.4 h5py: 2.10.0 Nio: None zarr: None cftime: 1.1.1.2 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: 1.3.2 dask: 2.12.0 distributed: 2.14.0 matplotlib: 3.2.1 cartopy: 0.17.0 seaborn: 0.10.0 numbagg: None pint: None setuptools: 46.1.3 pip: 20.0.2 conda: None pytest: 5.4.1 IPython: 7.13.0 sphinx: 3.0.2
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/4015/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 484212164,MDExOlB1bGxSZXF1ZXN0MzEwMTQwMTI2,3244,Make argmin/max work lazy with dask,13190237,closed,0,,,7,2019-08-22T20:55:49Z,2019-10-25T11:53:42Z,2019-09-06T23:15:19Z,CONTRIBUTOR,,0,pydata/xarray/pulls/3244,"As @shoyer pointed out in https://github.com/pydata/xarray/issues/3237, nanargmax/min from numpy or dask should be used when not working on object arrays. Also, nanargmin/max were added to the nputils module so they should be using bottleneck if available. - [x] Closes #3237 - [x] Tests added - [x] Passes `black . && mypy . && flake8` - [x] Fully documented, including `whats-new.rst` for all changes and `api.rst` for new API ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/3244/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 483280810,MDU6SXNzdWU0ODMyODA4MTA=,3237,``argmax()`` causes dask to compute,13190237,closed,0,,,4,2019-08-21T08:41:20Z,2019-09-06T23:15:19Z,2019-09-06T23:15:19Z,CONTRIBUTOR,,,,"#### Problem Description While digging for #2511 I found that ``da.argmax()`` causes compute on a dask array in ```nanargmax(a, axis=None)```: https://github.com/pydata/xarray/blob/131f6022763d35edf461b11857d7c4ec6630b19d/xarray/core/nanops.py#L120 I feel like this shouldn't be the case as ``da.max()`` and ``da.data.argmax()`` don't compute and it renders the laziness useless. #### MCVE Code Sample ```python In [1]: import numpy as np ...: import dask ...: import xarray as xr In [2]: class Scheduler: ...: """""" From: https://stackoverflow.com/questions/53289286/ """""" ...: ...: def __init__(self, max_computes=0): ...: self.max_computes = max_computes ...: self.total_computes = 0 ...: ...: def __call__(self, dsk, keys, **kwargs): ...: self.total_computes += 1 ...: if self.total_computes > self.max_computes: ...: raise RuntimeError( ...: ""Too many dask computations were scheduled: {}"".format( ...: self.total_computes ...: ) ...: ) ...: return dask.get(dsk, keys, **kwargs) ...: In [3]: scheduler = Scheduler() In [4]: with dask.config.set(scheduler=scheduler): ...: da = xr.DataArray( ...: np.random.rand(2*3*4).reshape((2, 3, 4)), ...: ).chunk(dict(dim_0=1)) ...: ...: dim = da.dims[-1] ...: ...: dask_idcs = da.data.argmax(axis=-1) # Computes 0 times ...: print(""Total number of computes: %d"" % scheduler.total_computes) ...: ...: da.max(dim) # Computes 0 times ...: print(""Total number of computes: %d"" % scheduler.total_computes) ...: ...: da.argmax(dim) # Does compute ...: print(""Total number of computes: %d"" % scheduler.total_computes) ...: Total number of computes: 0 Total number of computes: 0 --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) in 12 print(""Total number of computes: %d"" % scheduler.total_computes) 13 ---> 14 da.argmax(dim) # Does compute 15 print(""Total number of computes: %d"" % scheduler.total_computes) 16 ~/src/xarray/xarray/core/common.py in wrapped_func(self, dim, axis, skipna, **kwargs) 42 def wrapped_func(self, dim=None, axis=None, skipna=None, **kwargs): 43 return self.reduce( ---> 44 func, dim, axis, skipna=skipna, allow_lazy=True, **kwargs 45 ) 46 ~/src/xarray/xarray/core/dataarray.py in reduce(self, func, dim, axis, keep_attrs, keepdims, **kwargs) 2120 """""" 2121 -> 2122 var = self.variable.reduce(func, dim, axis, keep_attrs, keepdims, **kwargs) 2123 return self._replace_maybe_drop_dims(var) 2124 ~/src/xarray/xarray/core/variable.py in reduce(self, func, dim, axis, keep_attrs, keepdims, allow_lazy, **kwargs) 1456 input_data = self.data if allow_lazy else self.values 1457 if axis is not None: -> 1458 data = func(input_data, axis=axis, **kwargs) 1459 else: 1460 data = func(input_data, **kwargs) ~/src/xarray/xarray/core/duck_array_ops.py in f(values, axis, skipna, **kwargs) 279 280 try: --> 281 return func(values, axis=axis, **kwargs) 282 except AttributeError: 283 if isinstance(values, dask_array_type): ~/src/xarray/xarray/core/nanops.py in nanargmax(a, axis) 118 if mask is not None: 119 mask = mask.all(axis=axis) --> 120 if mask.any(): 121 raise ValueError(""All-NaN slice encountered"") 122 return res /usr/lib/python3.7/site-packages/dask/array/core.py in __bool__(self) 1370 ) 1371 else: -> 1372 return bool(self.compute()) 1373 1374 __nonzero__ = __bool__ # python 2 /usr/lib/python3.7/site-packages/dask/base.py in compute(self, **kwargs) 173 dask.base.compute 174 """""" --> 175 (result,) = compute(self, traverse=False, **kwargs) 176 return result 177 /usr/lib/python3.7/site-packages/dask/base.py in compute(*args, **kwargs) 444 keys = [x.__dask_keys__() for x in collections] 445 postcomputes = [x.__dask_postcompute__() for x in collections] --> 446 results = schedule(dsk, keys, **kwargs) 447 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)]) 448 ``` #### Expected Output None of the methods should actually compute: ``` python Total number of computes: 0 Total number of computes: 0 Total number of computes: 0 ```` #### Output of ``xr.show_versions()``
INSTALLED VERSIONS ------------------ commit: None python: 3.7.4 (default, Jul 16 2019, 07:12:58) [GCC 9.1.0] python-bits: 64 OS: Linux OS-release: 5.2.9-arch1-1-ARCH machine: x86_64 processor: byteorder: little LC_ALL: None LANG: de_DE.utf8 LOCALE: de_DE.UTF-8 libhdf5: 1.10.5 libnetcdf: 4.7.0 xarray: 0.12.3+63.g131f6022 pandas: 0.25.0 numpy: 1.17.0 scipy: 1.3.1 netCDF4: 1.5.1.2 pydap: None h5netcdf: 0.7.4 h5py: 2.9.0 Nio: None zarr: None cftime: 1.0.3.4 nc_time_axis: None PseudoNetCDF: None rasterio: 1.0.25 cfgrib: None iris: None bottleneck: 1.2.1 dask: 2.1.0 distributed: 1.27.1 matplotlib: 3.1.1 cartopy: 0.17.0 seaborn: 0.9.0 numbagg: None setuptools: 41.0.1 pip: 19.0.3 conda: None pytest: 5.0.1 IPython: 7.6.1 sphinx: 2.2.0
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/3237/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 481110823,MDExOlB1bGxSZXF1ZXN0MzA3NjczNzgx,3221,Allow invalid_netcdf=True in to_netcdf(),13190237,closed,0,,,6,2019-08-15T11:32:56Z,2019-08-22T20:20:42Z,2019-08-22T20:12:11Z,CONTRIBUTOR,,0,pydata/xarray/pulls/3221,"Hi all, I prepared a little PR which could close #2243 and would allow for a IMO clean way of writing data with complex dtypes (and others). What do you think? TODOs: ---------- - [X] Closes #2243 - [X] Tests added - [x] Passes `black . && mypy . && flake8` - [x] Fully documented, including `whats-new.rst` for all changes and `api.rst` for new API ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/3221/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 260279615,MDU6SXNzdWUyNjAyNzk2MTU=,1591,indexing/groupby fails on array opened with chunks from netcdf,13190237,closed,0,,,2,2017-09-25T13:37:43Z,2017-09-26T08:15:45Z,2017-09-26T05:36:26Z,CONTRIBUTOR,,,,"Hi, since the last update of dask (to version 0.15.3), iterating over a groupby object and indexing using np.int64 fails, when the DataArray was opend with chunks from netcdf. I'm using xarray version 0.9.6 and the 'h5netcdf'-engin for reading/writing. To reproduce: ``` import xarray as xr arr = xr.DataArray(np.random.rand(2, 3, 4), dims=['one', 'two', 'three']) arr.to_netcdf('test.nc', engine='h5netcdf') arr_disk = xr.open_dataarray('test.nc', engine='h5netcdf', chunks=dict(one=1)) # This produces the error: [g for g in arr_disk.groupby('one')] /usr/lib/python3.6/site-packages/xarray/core/groupby.py in _iter_grouped(self) 296 """"""Iterate over each element in this group"""""" 297 for indices in self._group_indices: --> 298 yield self._obj.isel(**{self._group_dim: indices}) 299 300 def _infer_concat_args(self, applied_example): /usr/lib/python3.6/site-packages/xarray/core/dataarray.py in isel(self, drop, **indexers) 677 DataArray.sel 678 """""" --> 679 ds = self._to_temp_dataset().isel(drop=drop, **indexers) 680 return self._from_temp_dataset(ds) 681 /usr/lib/python3.6/site-packages/xarray/core/dataset.py in isel(self, drop, **indexers) 1141 for name, var in iteritems(self._variables): 1142 var_indexers = dict((k, v) for k, v in indexers if k in var.dims) -> 1143 new_var = var.isel(**var_indexers) 1144 if not (drop and name in var_indexers): 1145 variables[name] = new_var /usr/lib/python3.6/site-packages/xarray/core/variable.py in isel(self, **indexers) 568 if dim in indexers: 569 key[i] = indexers[dim] --> 570 return self[tuple(key)] 571 572 def squeeze(self, dim=None): /usr/lib/python3.6/site-packages/xarray/core/variable.py in __getitem__(self, key) 398 dims = tuple(dim for k, dim in zip(key, self.dims) 399 if not isinstance(k, integer_types)) --> 400 values = self._indexable_data[key] 401 # orthogonal indexing should ensure the dimensionality is consistent 402 if hasattr(values, 'ndim'): /usr/lib/python3.6/site-packages/xarray/core/indexing.py in __getitem__(self, key) 496 value = value[(slice(None),) * axis + (subkey,)] 497 else: --> 498 value = self.array[key] 499 return value 500 /home/herter/.local/lib/python3.6/site-packages/dask/array/core.py in __getitem__(self, index) 1220 1221 from .slicing import normalize_index, slice_with_dask_array -> 1222 index2 = normalize_index(index, self.shape) 1223 1224 if any(isinstance(i, Array) for i in index2): /home/herter/.local/lib/python3.6/site-packages/dask/array/slicing.py in normalize_index(idx, shape) 760 idx = idx + (slice(None),) * (len(shape) - n_sliced_dims) 761 if len([i for i in idx if i is not None]) > len(shape): --> 762 raise IndexError(""Too many indices for array"") 763 764 none_shape = [] IndexError: Too many indices for array > /home/herter/.local/lib/python3.6/site-packages/dask/array/slicing.py(762)normalize_index() 760 idx = idx + (slice(None),) * (len(shape) - n_sliced_dims) 761 if len([i for i in idx if i is not None]) > len(shape): --> 762 raise IndexError(""Too many indices for array"") 763 764 none_shape = [] ``` I'm getting the same error when doing `arr_disk[np.int64(0)]` Thanks Uli ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/1591/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue